We’ve all committed the cardinal sin of hardcoding credentials into code at some point. It’s tempting to take the shortcut and get on with debugging. The changing threat landscape has taught just how critical this failure to comply with security best practices can be. There have been several high-profile cybersecurity incidents that have targeted individuals of an organization to gain credentials to their systems. The LastPass breach post-mortem details the painful experience of the organization losing their DevOps credentials and the trust of the community along with it.

This attack was highly targeted and sophisticated and hindsight is 20/20. Even so, as a developer, these incidents should highlight just how critical the system credentials are for hackers. The same credentials we are so used to working with on a daily basis to login to Dev and Production systems. It’s almost getting to the point where devs should be running as far away as possible from handling system credentials locally.

Securing credentials with Azure Keyvault

Azure Keyvault is a Saas based key management solution, one of many available in Azure. Azure Key Vault offers a FIPS 140-2 Level 1 software based or a FIPS 140-2 Level 2 HSM based solution. The standout feature of Azure Key Vault is the ease with which access to secrets can be managed, as well as the ease with which secrets can be stored, rotated or deleted via API, GUI or CLI. Details about Azure Key Vault can be found linked here and there are many quick-start guides available. Here we will focus specifically on a scenario as described below.

Scenario 1: DevOps Team Lead

You are the lead for a DevOps team and as such also responsible for onboarding new team members, managing access and credentials and making development as simple as possible.

Step 1: Create an Azure KeyVault for your team

Each Azure Key Vault is a resource in your Azure subscription which carries it’s own IAM structure. Although there are two options available, it is better to go out-of-box and use the AzureRBAC for access control. This way it is possible to create a Key Vault and have fine-grained access control, and avoid any misconfiguration issues with manually managed Vault access control. The Key Vault specific IAM roles are shown below:

You decide to create one Key Vault named “azDevOpsKV01”. Once provisioned, you will get a URL for your key vault in the form “hxxps://azDevOpsKV01.vault.azure.net/”

Step 2: Create an access governance structure

The next step is to use the role definitions to identify the groups needed to govern access. You decide on the following access governance structure:

• Key Vault Admin - azDevOpsKV01_admin: Yourself and a backup. Will have total control over this KeyVault.
• Key Vault Reader - azDevOpsKV01_reader: All team members as they need to be able to see list of secrets (but not access it)
• Key Vault Secrets User - azDevOpsKV01_secAccess: Since this is a small project with nly API access requirements, you are only storing secrets. In the future, equivalent groups may be need to govern Certificates.

Step 3: Create a request access and access review process

Using the Identity Governance feature, you created access packages for your team. This enables them to request access to the correct access groups as listed above. For example, a new member has joined the team. They were automatically added to the reader group and were able to get knowledge transition including the ability to see the names of the secrets stored in the Key Vault. Now they have submitted a request to access the Secrets User group to be able to read the secrets and consume it. You approve this request. Because you also setup an access review process, you know that after 60 days a review process will be kicked off where you can review all developers who have access to the credentials and remove access as needed. Your own access as administrator is reviewed by your manager, thus making the GRC team very happy.

Scenario 2: New DevOps Team Member

You have recently joined this team and have completed the onboarding steps as outlined by the lead. You now have access to the correct Key Vault and have the document referencing the names of the various API secrets and credentials used in the project. You need to utilize these in your python code to connect to your application API.

Step 1: Validate your account has access to the secret in the Key Vault

After receiving notification that your request for access to the Secrets User group was approved, you login to the Azure Portal and navigate to the Azure Key Vault. You validate that you can indeed see the Secret under the Object menu. Clicking one of the secrets opens up the blade showing the current version, and clicking this brings up the masked secret value. As a conscientious developer, you decide to not copy the secret into Notepad++.

Step 2: Call and retrieve the secret via key

As part of knowledge transition you received a python snippet in a file (azKVCaller.py) that the team uses to retrieve secrets at run time. The python code looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Example code that uses "az login" to authenticate and authorize access to your Keyvault from a py script
# Remember: DO NOT LOG YOUR SECRETS

from azure.core.exceptions import ClientAuthenticationError
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

def getAzKvSecret(vaultURL,secretName):

    # additionally_allowed_tenants parameter is needed when working on trial Azure subs as users may use their live.com id to access the tenant
    # exclude_shared_token_cache_credential is needed because there maybe expired tokens in cache. IDKW.
    credential = DefaultAzureCredential(additionally_allowed_tenants=['*'], exclude_shared_token_cache_credential=True)
    try:
        secret_client=SecretClient(vault_url=vaultURL, credential=credential)
        secret = secret_client.get_secret(secretName)
    except ClientAuthenticationError as ex:
        print(ex.message)
        print ("*** Did you do az login? ***")

    return secret

You write a short code to retrieve the secret named “APICaller” to verify this code works. You run az login and are prompted to login via the browser. You complete the required MFA prompt and your credentials are now cached. You run your code and validate that you get the first 8 characters of the secret. This confirms your access has been granted.

1
2
3
4

import azKVCaller as azkv

secret=(azkv.getAzKvSecret("https://azDevOpsKV01.vault.azure.net/", "APICaller"))
print(secret.value[:8])

You can now use this code to call the API and do not have to store the secret anywhere locally. The GRC team is very very happy.

Scenario 3: OMG! The API secret got leaked. Time to rotate.

As the Team Lead, you get notified by the SecOps team that the API secrets for the group needed to be rotated because of a suspicion that they had been leaked. The AzureAD team wants to issue you a new API secret.

Option 1: Give the application team temporary access to the Key Vault

As the Key Vault admin, you can temporarily assign Secrets User access to the AzureAD team so they can login directly to the Key Vault GUI and update the secret with a new value. The Key Vault allows the creation of a new version of the secret. This way the secret need not travel over the wire to you via email or some other method that could potentially be compromised. Once validated that the new secret works, you can remove their access from your Key Vault.

Option 2: Update version of the key yourself

Assuming you are able to receive the secret over a secure method, you can update the key yourself.

A big advantage of either option is that since the secret is always retrieved programmatically, updating/rotating the secret does not break code for your developers. They can validate everything works after the update and move on. A side-effect may be that you find the developers who did decide to hardcode the secret into their process :-)

In conclusion…

Azure Key Vault and similar solutions are a powerful way to secure credentials, certificates and keys and truly enable secure DevOps methodology. As can be seen in the use cases above, not only are the secrets safe and accessed on-demand, but the process of rotating and changing secrets in a centralized manner means less disruptions for the developers. There are further controls available to ensure secrets cannot be stolen such as ensuring network level controls on the Key Vault itself. This is explained by the excellent video below.

Recommended viewing - Locking down the Azure Key Vault