Grant individual permission to secretsread in Unity Catalog

Recently, I published an article on registering credentials in Unity Catalog, which demonstrated how to create a Service Principal connection for retrieving secrets from Azure Key Vault. This approach allows you to govern Key Vault connections at the Unity Catalog level, giving you precise control over who can access your secrets.

The solution enables you to execute code like this to retrieve secrets:

from azure.keyvault.secrets import SecretClient 
credential = dbutils.credentials.getServiceCredentialsProvider('key-vault-access-connector')
vault_url = "https://myvault.vault.azure.net/"
client = SecretClient(vault_url=vault_url, credential=credential)
Although the code to read the secrets is longer than in dbutils.secrets() (at least for now), the fact that we can manage the connector’s access to the key vault through Unity Catalog and decouple that from workspace settings makes it a best practice.
secret_value = client.get_secret("my-secret").value
print("Got secret:", secret_value[:4], "...[redacted]")

The Challenge: All-or-Nothing Access

The current approach governs the connection to the Key Vault effectively. However, when you grant someone access to the credentials, that user gains access to all secrets within that specific Key Vault.

This led me to an important question: "Can we implement more granular access control and govern permissions based on individual secret names within Unity Catalog?"

In other words, why can't we have individual secrets in Unity Catalog and grant team members access to specific secrets only?

The Solution: Unity Catalog Functions with Service Credentials

Recently, I discovered that Unity Catalog batch UDFs support SERVICE CREDENTIALS, the same credentials used for Key Vault connections.

CREDENTIALS (  key-vault-access-connector AS kv DEFAULT )

This opened up an interesting possibility. So I decided to change those credentials so only the Service Principal can read and write the function:

CREATE OR REPLACE FUNCTION secrets.get_my_secret(environment STRING)
RETURNS STRING
LANGUAGE PYTHON
HANDLER 'batchhandler'
PARAMETER STYLE PANDAS
ENVIRONMENT (
  dependencies = '[
    "azure-keyvault-secrets==4.8.0"
  ]',
  environment_version = 'None'
)
CREDENTIALS (
  `key-vault-access-connector` AS kv DEFAULT
)
AS $$
def batchhandler(batch_iter):
  for _ in batch_iter:
    import pandas as pd
    from azure.keyvault.secrets import SecretClient
    from databricks.service_credentials import getServiceCredentialsProvider
    vault_url = "https://dudekkeyvault.vault.azure.net/"
    client = SecretClient(vault_url=vault_url, credential=getServiceCredentialsProvider('kv'))
    secret_value = client.get_secret("my-secret").value
    yield pd.Series( [secret_value] )
$$;

How It Works: The Permission Model

The key insight is how Unity Catalog handles permissions for UDFs with service credentials:

Source: SunnyData / Hubert Dudek

And that function is registered in Unity Catalog and can be used to read secrets.

According to the Unity Catalog documentation: "The UDF creator must have ACCESS permission on the Unity Catalog service credential. However, for UDF callers, it is sufficient to grant them EXECUTE permission on the UDF. In particular, UDF callers do not need access to the underlying service credential, because the UDF executes using the credential permissions of the UDF creator."

This means that when you grant a user EXECUTE permission on the function, they can only access the specific secret returned by that function but not the underlying service credentials or other secrets in the Key Vault.

So I granted the test user just execute, and the test user can now only read that specific secret:

Recommended Architecture: Secret-Specific Functions

Based on this approach, I recommend the following architecture: I would create a separate function for each secret or group of secrets, storing them in the schema ‘secrets’. Only authorized users would be able to execute these functions. For example, you can have one function which, thanks to params, can return multiple secrets.

This approach provides true granular access control, where each team member can access only the secrets they need for their specific role.

A nice addition:

Thanks to access to SparkContext, you can create some additional logic by leveraging Spark session context:

session_user = tc.getLocalProperty("user")

This allows you to implement user-based logic within your secret retrieval functions, adding another layer of security and auditability.

Important Security Considerations

Normal dbutils returns the word REDACTED when a secret is retrieved to the notebook/console (although there are a few ways to bypass this). Since it prints clear text, use it responsibly and grant secret access only to trusted, responsible individuals.

Conclusion

This approach transforms Unity Catalog from a Key Vault-level access control system into a granular, secret-specific permission model. By leveraging UDFs with service credentials, you can implement fine-grained access control that aligns with the principle of least privilege, ensuring users only access the secrets they absolutely need.

The result is a more secure, manageable, and scalable approach to secret governance in your Databricks environment.