The Translator’s Gone: Inside Databricks' Direct Engine
Among all the extraordinary announcements from Databricks, there is one that I’m particularly excited about, and that may have passed unnoticed. Bundles, already one of Databricks' most important features and a personal favorite, have just been handled a level up: direct engine is here. The idea behind this post is to learn: what it is, why you should care, and why it can be quite bothersome.
What is Databricks direct engine?
Up to this point, the engine behind DABs had been Databricks Terraform Provider, which is itself built on top of Golang. Each time a databricks bundle command was run, Terraform was called to do the "dirty work" behind the scenes. That’s why, if at any point you reviewed the state of your bundle deployment, you would probably find something like this:
See the terraform.tfstate file there? That’s the trail left by Terraform processes running. Terraform is a language designed to create and manage resources, so it makes absolute sense to build DABs on top of it. It is a great translator from a CLI language into the resource world, and probably one of the key reasons why it was possible to develop DABs. However, every great translator comes at a cost.
Suppose you don’t understand Dutch, and you want to interact with a new friend from the Netherlands. A translator makes conversation possible, but we can agree that the situation isn’t as ideal as actually knowing Dutch. There is lag between the moment you speak, the translator understands your words, and reproduces them in the new language. Part of the eloquence and flair that comes so easily to you in your first language are literally lost in translation, and any newly invented word requires you to teach it to the translator first before it can be used.
This metaphor can help us understand the advantages that direct engine brings:
By getting Terraform out of the conversation, “translation lag” is removed and deployment times are shortened.
Introducing new words, which would represent new parameters for resources or even a whole new resource, is easier. There is no need for an official Terraform Provider Release to unblock the new resources. This can already be seen with the addition of Genie Spaces on Automation Bundles with the new engine.
Eloquence is restored, so new, more advanced steps of planning and validation are suddenly available.
The direct engine does exactly that: it takes the translator out of the equation, going directly from the CLI language to Go.
What are the costs of the direct engine?
If this is so brilliant, why wasn’t it done before? For the same reason translators exist. Knowing every language is not possible. Databricks maturity (and probably the help of AI) makes the effort possible and worthwhile.
The main problem for the happy Databricks user is how to change from Terraform to the direct engine.
For new bundles, the solution is very simple. Upgrading the CLI version to one greater than 1.3.0 gets you the direct engine by default, with all of the advantages and without moving a finger. Just to be absolutely sure, it is also possible to use the following notation on databricks.yaml.
bundle: engine: direct
For existing bundles, it's trickier. All the information of the deployment is stored in the Terraform state files we saw before. Direct engine has its own type of state, in JSON files that are called resources.json and have a different schema. A migration is needed to get from the old state into the new version.
Migration is not a single command, but three steps. First, a deploy of the resources to ensure that the state is up to date.
databricks bundle deploy -t my_target
And lastly, verifying. The new plan command should run without issues.
databricks bundle plan -t my_target
What happens if I don’t migrate to direct engine?
At first glance, not much. Deployments will continue working with the old engine. However, new resources will only be available for the new engine, and it’s possible the Terraform engine is completely deprecated at some point in the future.
Just running the deployment with the engine: direct annotation in the yaml without the migration commands does not cut it either. A simple warning is what appears.
Conclusion
Learning the language beats hiring a translator. That's the whole story of the direct engine: less lag, fewer middlemen, and the freedom to say new things the moment they're invented instead of waiting for someone else to learn them.
For new bundles, it costs you nothing. For existing ones, it costs you one migration per target. That's the bothersome part I promised at the top, and it's real, especially in CI/CD, where a bundle can quietly keep speaking Terraform while your databricks.yml insists it's fluent in direct. Do the migration deliberately, verify each target with plan, and then forget Terraform was ever there.
There are also a couple of fun details we skipped here, around firewalls and local vs. remote state, so go dig into the official docs: https://docs.databricks.com/aws/en/dev-tools/bundles/direct.
TL;DR
DABs (Declarative Automation Bundles) used to run on Terraform behind the scenes as the "translator" between your CLI commands and actual resources. The new direct engine cuts Terraform out entirely, talking straight to Go.
Why it matters:
Faster deployments (no more "translation lag")
New resources/parameters ship immediately, no waiting on official Terraform Provider releases (Genie Spaces already shipped this way)
More advanced planning/validation becomes possible
The catch:
New bundles: free upgrade, just use CLI ≥1.3.0, or set bundle: engine: direct in databricks.yaml
Existing bundles: need a 3-step migration since old Terraform state (terraform.tfstate) isn't compatible with the new state format (resources.json):
databricks bundle deploy -t my_target
databricks bundle deployment migrate -t my_target
databricks bundle plan -t my_target (verify)
Skipping migration doesn't break anything immediately, but you'll silently stay on Terraform (with just a warning) and miss new-engine-only resources. This is especially risky in CI/CD.
Bottom line: Do the migration deliberately per target, verify with plan, then Terraform is out of the picture for good.
