Automated Tags Clean The Mess

The number of assets in Databricks can grow exponentially, leading to a flood of undocumented and unused tables. The best way to manage this clutter is through proactive communication. By tagging delinquent assets, you can notify owners directly that their tables are scheduled for deletion while publicly flagging the missing documentation.

Automation tagging simplifies our automation rules and enables us to incorporate direct prompt notifications.

First: Tag Creation

Let’s head to “Governed Tags” in Unity Catalog and create a missing-documentation tag.

In SQL, the command is:

CREATE GOVERNED TAG `missing-documentation`;

We can also leverage system tags, such as system.certification_status, to mark a table as “deprecated” or “certified”. This setting, which we will be implementing, is also accessible through the table's UI.

Let’s Automate

First, enable the preview feature by clicking your profile in the top-right corner of the screen and selecting “Previews” (though this step may no longer be required depending on when you read this). 

Marking Inactive Tables As Deprecated

To automate this process, we first need to create a new automation rule under governed tags.

Our first priority is to identify unused tables. The tag condition menu offers a dozen useful options beyond simple prompts to help target those assets.

We can target tables that haven’t been queried for 90 days and automatically apply the  system.certification_status tag. At the same time, we can send notifications to table owners or specific users. This highlights an important best practice: assigning individual users as table owners rather than service principles ensures that these automated alerts reach an actual person. 

Once the automation is created, we can either perform a dry run or enable it directly. While automations can be scheduled, we will run this manually for now. 

Once enabled, we can execute the automation and observe tables being marked as deprecated within several minutes. 

Use Genie to Find Undocumented Tables

Another use case involves setting up a scheduled automation and using a Genie prompt to identify tables that lack documentation or comments. 

Once enabled and executed, a great way to verify the results is to open the tag details and inspect where it has been applied.

In addition, all asset owners will start to get notifications.

Reward for Good Behavior: Removing the Undocumented Tag

What if someone corrects the issue? We can set up a reverse process that automatically removes the missing-documentation tag once comments are added. 

Managing All Automation Rules in One Place

Now we can review all our automation rules in a central dashboard.

Keep in mind that automated tagging relies primarily on Unity Catalog metadata (schema, usage statistic) rather than deep data scanning. If you need to analyze and classify table contents directly, you should use data classification instead.

What’s Next

Currently, automated tagging is limited to tables and columns, with support for additional asset types coming soon. Key requested features include auto-tagging individual columns and specific files stored in volumes. Keep in mind that for unstructured data in volumes, you can also leverage Unity Catalog data classification features.

TL; DR

Databricks assets can quickly proliferate into an unmanaged collection of unused and undocumented tables. By setting up automated tagging rules in Unity Catalog, organizations can automatically flag inactive tables as deprecated and highlight assets that lack proper documentation. Automations rely on metadata like usage stats and schemas, enabling automated notifications to table owners and scheduled cleanups. As users fix documentation errors, counter-automations can automatically clear warning tags, creating a self-governing data environment. Continued updates to Unity Catalog promise expanded tagging capabilities across more asset types like volumes and files.