Unity Catalog to Azure Key Vault: No more dbutils.secrets()

In this article, I’ll walk you through granting Azure Databricks secure access to Azure Key Vault with an Azure Databricks Access Connector, and then centrally governing that access with Unity Catalog Service Credentials. Unlike the approach of calling dbutils.secrets() to create Azure Key Vault–backed secret scopes (which are restricted to a single workspace, invisible primarily to platform administrators, and hard to manage with fine-grained permissions) Unity Catalog Service Credentials are workspace-agnostic, included in Unity Catalog governance layer so you have visibility and possibility to easy manage granular access permissions.

1. Create Access Connector

First, head to "Access Connector to Databricks" in Azure and create a new access connector. Although Databricks has writing that you can use the same connector to connect to storage accounts and other Azure resources, in the case of Key Vault, I prefer a separate connector as it allows granular access control.

Once the access connector is created, we can copy the "Resource ID" (we’ll need it later).

Step 1: Create a new Access Connector for Azure Databricks in the Azure portal and copy the Resource ID for later use

2. Give Access To the Key Vault

In Azure Key Vault, we need to go to IAM and grant the "Key Vault Secret User" role to our access connector (which we’ll find among managed identities). Now, the connector can read our secrets.

Step 2: Grant the 'Key Vault Secrets User' role to your Access Connector in Azure Key Vault's IAM settings

3. Service Credentials

Next, we need to go into databricks to Unity Catalog’s credentials and create a "service credential", which will use our new access connector. We need to choose the name and paste the "Resource ID", which we copied earlier. Then, the most crucial part is that we need to specify which users, service principals, or groups will be able to read secrets in databricks.

Step 3: Create Unity Catalog Service Credentials using the Access Connector Resource ID and configure user permissions

4. Retrieve secrets

Now, we can go to the notebook, install the azure-keyvault-secrets library (in that case, we add it to a serverless environment), and start reading secrets.

Step 4: Install the azure-keyvault-secrets library in your Databricks serverless environment dependencies

We can copy the Key Vault URL from the Key Vault properties.

# import Azure SDK library
from azure.keyvault.secrets import SecretClient 

# service credentials from Unity Catalog
credential = dbutils.credentials.getServiceCredentialsProvider('key-vault-access-connector')
# Azure Key Vault url - copy it from vault properties
vault_url = "https://myvault.vault.azure.net/"
# init client
client = SecretClient(vault_url=vault_url, credential=credential)

Although the code to read the secrets is longer than in dbutils.secrets() (at least for now), the fact that we can manage the connector’s access to the key vault through Unity Catalog and decouple that from workspace settings makes it a best practice.

# Take the secret and print the first four chars for test; otherwise, it will be redacted
secret_value = client.get_secret("my-secret").value
print("Got secret:", secret_value[:4], "...[redacted]")

Step 5: Successfully retrieve secrets from Azure Key Vault using Unity Catalog Service Credentials - note how the secret value is automatically redacted in the output

Remember that many resources can be connected without secrets by using managed identities, which is always a better choice when possible.

Finally, make sure to set a private link from serverless to your key vault, but this is a topic for another article.

Looking Ahead

As organizations increasingly adopt Unity Catalog for comprehensive data governance, integrating secret management into this unified framework becomes not just a best practice –it's essential for maintaining security and compliance at scale.

The slightly longer code syntax is a small price to pay for the significant governance and security benefits you gain. Your future self (and/or security team) will thank you for implementing this approach from the start.

---

Hubert is an advisor and Databricks SME to SunnyData. For more of his content, follow him on ln/HubertDudek and stay tuned to ln/SunnyData for more articles like this.