Resources and insights
Our Blog
Explore insights and practical tips on mastering Databricks Data Intelligence Platform and the full spectrum of today's modern data ecosystem.
Genie Code Analysis: Two Weeks Later
Databricks Genie Code hit a 77.1% task success rate in production data science workflows — more than double what general-purpose coding agents achieve. But that performance is entirely conditional on the quality of your Unity Catalog metadata. SunnyData's two-week evaluation breaks down what works, what doesn't, and the governance layer you need before you go live.
5 Databricks Patterns That Look Fine Until They Aren't
Five common Databricks coding patterns — including undocumented API calls, manual SparkSession instantiation, and hardcoded Spark configs — that pass code review but fail silently in serverless environments or during platform migrations. For each anti-pattern, this post explains why it breaks and shows the correct native Databricks approach using DABS, the Databricks SDK, and dynamic job parameters.
Databricks Lakewatch: The Future of Agentic SIEM
Databricks Lakewatch replaces the traditional SIEM model with an Open Security Lakehouse — storing 100% of telemetry in open formats at up to 80% lower TCO. AI agents reason across years of unified data to detect and respond at machine speed, closing the visibility gap that legacy SIEMs were structurally forced to create. Early customers include Adobe and Dropbox, with broader availability following Private Preview.
Lakeflow Connect Free Tier: $35/Day Back in Your Budget
Databricks' permanent Lakeflow Connect free tier delivers 100 DBUs per workspace per day — covering up to 100 million records of ingestion at no additional compute cost. For enterprise teams running multiple workspaces, that's over $255,000 in avoided annual costs. This post breaks down the economics, architecture, and what it means for teams still paying a third-party ETL tax.
The Lakehouse Finally Has Real Transactions
Learn how Databricks multi-statement transactions use Unity Catalog catalog-managed commits to guarantee atomic updates across multiple Delta tables — with a step-by-step walkthrough.
Deduplicating Data on the Databricks Lakehouse: Making joins, BI, and AI queries “safe by default.”
Learn 5 proven deduplication strategies for Databricks Lakehouse. Prevent duplicate data from breaking AI queries, BI dashboards, and analytics. Includes code examples.
Deploy Your Databricks Dashboards to Production
Stop deploying Databricks dashboards manually. Learn how to use Git, Asset Bundles, and CI/CD for reliable, reproducible dashboard deployments across environments.
The Nightmare of Initial Load (And How to Tame It)
Initial data loads don't have to be nightmares. Discover the split Bronze table pattern that separates historical backfills from incremental streaming.
Temp Tables Are Here, and They're Going to Change How You Use SQL
Learn how temporary tables in Databricks SQL warehouses enable materialized data, DML operations, and session-scoped ETL workflows. Complete with practical examples.
Hidden Magic Commands in Databricks Notebooks
Discover 12 powerful Databricks notebook magic commands beyond %sql and %python. Learn shortcuts for file operations, performance testing, and debugging.
5 Reasons You Should Be Using LakeFlow Jobs as Your Default Orchestrator
External orchestrators can account for nearly 30% of Databricks’ job costs. Discover five compelling reasons why LakeFlow Jobs should be your default orchestration layer: from Infrastructure as Code to SQL-driven workflows.
Databricks Workflow Backfill
Use Databricks Workflow backfill jobs to reprocess historical data, recover from outages, and handle late-arriving data efficiently.
Snowflake and Databricks: How to balance compute
Compare Snowflake and Databricks compute models. Learn scaling strategies, cost optimization tips, and when to use auto-suspend, multi-cluster, and autoscaling.
Lakebase: The best of both worlds
Lakebase brings transactional speed to Databricks analytics. Get single-record retrieval in milliseconds plus unlimited data processing in one platform.
Managing Databricks CLI Versions in Your DAB Projects
Prevent Databricks deployment failures caused by CLI version conflicts. Step-by-step guide to version management in DAB projects with CI/CD automation.