Data Intelligence for All: 9 Ways Databricks Just Redefined the Future of Data

The Databricks Data+AI Summit 2025 brought numerous announcements that reinforce the vision of a unified Lakehouse that integrates data and artificial intelligence. Throughout the event, key innovations were presented that mark the direction of the platform.

Below, we summarize the main announcements:

Lakebase: Integrated Transactions in the AI Era

Databricks Lakebase is a new fully managed transactional database (OLTP), built on PostgreSQL and deeply integrated with the Lakehouse. It emerges to overcome the limitations of traditional operational databases, which are often isolated from the analytical platform and require inflexible manual integrations.

Lakebase adopts a modern architecture with compute and storage separation, allowing independent scaling and supporting advanced features like branching (environment cloning). By being integrated with the Databricks ecosystem, it enables direct connection of transactional data with analytics and AI, accelerating the delivery of intelligent applications.


💼 Why it matters for business:
Lakebase facilitates the creation of AI-native operational applications without leaving the unified platform, reducing architectural complexity and ensuring that critical data is easily integrated for analysis.

Learn more: Announcing Lakebase Public Preview | What Is a Lakebase?

Agent Bricks: Self-Optimized AI Agents with Your Data

Agent Bricks is a new tool that radically simplifies the development of generative AI agents (chatbots, assistants, etc.) specialized in your company's data. Instead of having to manually adjust hundreds of parameters and test endlessly, with Agent Bricks you simply describe the task or domain in natural language, and the platform automatically handles the evaluation and fine-tuning of the agent.

This solution, backed by Databricks' Mosaic AI research, optimizes agents to achieve higher quality at lower cost – for example, choosing the best prompt, model, or fine-tuning technique for your use case. It has already been tested in production by companies like Flo Health or AstraZeneca to deploy accurate and reliable AI agents in days instead of weeks.

💼 Why it matters for business? Agent Bricks reduces the barrier to incorporating conversational AI or intelligent assistants into business products and processes, by automating the most complex parts of construction and optimization. This allows organizations to put AI agents into production faster, with controlled quality metrics and aligned to business objectives, without needing an army of LLM tuning experts.

Learn more: Introducing Agent Bricks | Mosaic AI Agent Bricks

Databricks Apps: Data and AI Application Platform within the Lakehouse

It's now possible to create and deploy interactive data and AI applications directly in Databricks thanks to the new features of Databricks Apps. This integrated app hosting platform allows data teams and developers to quickly build custom applications – interactive dashboards, generative AI applications like corporate chatbots (RAG apps), self-service analysis tools, etc. – all within Databricks' governed environment.

By abstracting infrastructure (it's serverless and secure by design), Databricks Apps eliminates the need to move data to external applications, maintaining governance and reducing risks of duplication or data leakage. Since its preview last year, more than 20,000 apps have been created with this platform, demonstrating interest in simplifying data application delivery.

💼 Why it matters for business: With Databricks Apps, companies can deliver internal and customer-facing solutions faster, leveraging centralized data. An analytics team could now develop, for example, an indicators portal or an AI application for customers without deploying separate servers, achieving shorter time-to-market and ensuring that the data used is always under control and updated.

Learn more: Announcing General Availability of Databricks Apps | Databricks Apps Documentation

Databricks One: Data and AI at Executives' Fingertips

Aimed at non-technical business users, Databricks One is a new unified and simplified user experience so that C-levels and analysts can access the value of the Lakehouse with the least possible amount of friction. This interface, with renewed design, allows corporate leaders to consume BI dashboards and AI results, ask questions in natural language through the AI/BI Genie assistant, and use data applications created on the platform – all from the same place.

AI/BI Dashboards and Genie: Intelligence Made Accessible

Central to the Databricks One experience are the AI/BI capabilities that make data intelligence truly accessible to business users:

AI/BI Dashboards provide an AI-powered, low-code solution for conventional business intelligence with:

  • No-code, drag-and-drop interface for creating interactive visualizations

  • Built-in scheduling and exporting capabilities for operational reporting

  • Cross-filtering and high interactivity enabling dynamic data exploration

  • Deep Unity Catalog integration ensures governed access to trusted data

AI/BI Genie offers a conversational interface that transforms how business users interact with data:

  • Natural language queries: Ask business questions in plain English and get instant insights

  • Multi-modal responses: Receives answers via text summaries, tables, and visualizations

  • Continuous learning: Remembers clarifications and improves responses over time

  • Ensemble AI approach: Leverages multiple specialized AI agents and LLMs for accuracy

In essence, Databricks One transforms the platform into an enterprise portal for data and artificial intelligence, where an executive can consult indicators, explore insights, or interact with an analysis chatbot, without requiring deep technical knowledge. It will be available in public beta in summer 2025 and promises to bring data science even closer to the front office.

💼 Why it matters for business: This democratizes access to data and AI for decision-making. Corporate users gain autonomy to obtain answers and visualize information in real-time, while the data department maintains security and governance. In practice, Databricks One can create a more data-driven culture, where business decisions are made with quick and reliable insights, without bottlenecks or ambiguous interpretation of results. The AI/BI capabilities specifically address the long-standing challenge of making advanced analytics accessible to non-technical users, potentially eliminating the need for separate BI tools and reducing the complexity of the analytics stack.

Learn more: Introducing Databricks One | Data intelligence for business teams

Databricks Free Edition: Lakehouse for All, at No Cost

In a bid to democratize learning and adoption of Data & AI, the new Databricks Free Edition was launched, a free version of the Databricks workspace with almost all the functionalities of the Enterprise edition. Anyone (students, developers, startups) can learn and experiment on the same platform used by millions of professionals, without paying a thing.

This free edition, based entirely on serverless architecture, includes a wide range of features that were previously only available to paying customers, enabling practice from SQL analytics to machine learning in an enterprise-level environment. Additionally, Databricks Academy has made all its self-directed courses free to meet the growing demand for talent in data and AI.

💼 Why it matters for business: This initiative will expand the ecosystem of users familiar with Databricks while allowing organizations to explore the platform without cost barriers, fostering innovation and rapid prototyping in small teams before scaling. In sum, it increases accessibility to cutting-edge data/AI technologies, something critical in a context where competitive advantage depends on talent and constant experimentation.

Learn more: Introducing Databricks Free Edition | Get Started with Free Edition

MLflow 3.0: Unified MLOps for Generative and Traditional AI

Databricks announced MLflow 3.0, a significant evolution of its open MLOps platform, now oriented to fully integrate generative AI (GenAI) use cases alongside traditional machine learning. MLflow 3.0 unifies the development of classic ML models, deep learning, and GenAI applications in a single platform, eliminating the need for independent specialized tools.

It incorporates new capabilities designed for generative AI challenges, such as production-scale traceability of executions (logging prompts, outputs, costs, and latency), a renewed quality evaluation system with LLM judges, mechanisms for collecting human feedback, and complete versioning of prompts and AI applications. This enables a complete lifecycle: debug with traces, measure quality automatically, improve with expert feedback, track changes, and monitor everything in production.

💼 Why it matters for business: With MLflow 3.0, organizations get a de facto standard for governing both their traditional predictive models and their new generative agents. The platform offers comprehensive observability and control over previously opaque AI systems (e.g., chatbots), helping detect biases, align behaviors, and meet quality and compliance requirements. In practice, a data team can iterate faster on natural language applications with the peace of mind of having metrics, versions, and auditing just as they would with a conventional ML model.

Learn more: MLflow 3.0: Unified AI Experimentation, Observability, and Governance | MLflow Documentation

Lakeflow and Lakeflow Designer: Unified Data Engineering (Ingestion, ETL, Orchestration)

To address the complexity of enterprise data pipelines, Databricks launched Lakeflow, its unified data engineering solution that encompasses ingestion, transformation, and orchestration on the Lakehouse platform. Lakeflow replaces the need to combine disparate ETL, streaming, scheduler tools, etc., offering a Databricks-native end-to-end solution.

With Lakeflow, teams can handle all pipeline phases in a single framework, with integrated governance and monitoring – for example, integration with Unity Catalog for traceability and unified access controls.

The platform includes dedicated components:

  • Lakeflow Connect: Managed ingestion connectors for databases, enterprise applications, files, and real-time streams.

  • Lakeflow Declarative Pipelines: Scalable pipelines based on the new open standard Spark Declarative Pipelines, integrated with a modern IDE for ETL development.

  • Lakeflow Jobs: Native orchestration with advanced control flows, real-time triggers, and complete monitoring.

Additionally, Lakeflow Designer was announced, a visual no-code interface with drag-and-drop support and natural language to build production-quality data pipelines without writing code. What's designed in this tool automatically translates into declarative pipelines governed by Lakeflow, allowing business analysts and data engineers to collaborate without needing to redo work.

Spark Declarative Pipelines: Open Source Innovation

As part of the Lakeflow announcement, Databricks also revealed its commitment to open source by donating its core declarative ETL framework to Apache Spark as Spark Declarative Pipelines. This battle-tested technology, which has powered thousands of customer workloads for over three years, will now be available to the entire Apache Spark community.

Key capabilities:

  • Dramatic productivity gains: Companies like Block reduced development time by over 90%, while Navy Federal Credit Union cut pipeline maintenance time by 99%

  • Unified batch and streaming: Single framework handles both daily batch reporting and sub-second streaming applications

  • Declarative approach: Engineers describe what their pipeline should do using SQL or Python, and Spark handles the execution automatically

  • Enterprise-proven: Built on the Spark Structured Streaming engine with automatic dependency tracking, checkpointing, and error handling

💼 Why it matters for business: Lakeflow eliminates the costly fragmentation of data tools, reducing the overhead of integrating and maintaining multiple platforms. This translates into cost and time savings, as a single solution handles data ingestion (with a growing library of connectors), transformations (declarative SQL or Python), and reliable pipeline orchestration. With Lakeflow Designer, more people in the organization can participate in creating data flows (for example, an analyst building an initial pipeline) while maintaining best practices, as the system guides with built-in intelligence and everything remains under the same governance policies. The open-sourcing of Spark Declarative Pipelines also ensures that organizations aren't locked into proprietary solutions - they can benefit from this innovation regardless of their platform choice. Lakeflow will accelerate the delivery of data ready for analysis or models, allowing teams to focus on adding business value instead of dealing with complex infrastructures.

Learn more: Lakeflow Product Page | Spark Declarative Pipelines Blog

Native Apache Iceberg Support: Open Formats Under Unified Governance

In the data management realm, Databricks announced complete support for Apache Iceberg on its platform, marking a milestone toward eliminating table format lock-ins. Unity Catalog (Databricks' unified catalog) now allows creating and managing Iceberg Managed Tables within Databricks, as well as reading and writing those tables from external engines using the Iceberg catalog REST API.

These managed Iceberg tables benefit from automatic optimizations (for example, auto-clustering of data for performance) and work similarly to Delta tables, integrating with advanced platform features like SQL queries, MosaicML, Delta Sharing, or materialized views.

Additionally, Unity Catalog can now federate Iceberg metadata from external catalogs (AWS Glue, Hive Metastore, Snowflake, etc.), allowing governance and querying from Databricks of Iceberg tables that reside outside, without needing to migrate them. All this drives the vision of a truly open Lakehouse, where data can be in Delta Lake or Iceberg interchangeably, under the same umbrella of security and governance.

Unity Catalog Metrics: Unified Business Semantics

Beyond table format support, Unity Catalog now addresses one of the most persistent challenges in enterprise data management with Unity Catalog Metrics (Public Preview, GA this summer). This capability makes business metrics and KPIs first-class assets in the lakehouse, eliminating the confusion and misalignment caused by inconsistent metric definitions across teams and tools.

Revolutionary approach:

  • Define once, use everywhere: Create metrics once in Unity Catalog and use them across AI/BI Dashboards, Genie, Notebooks, SQL, and Lakeflow jobs

  • SQL-addressable: Unlike metrics trapped in BI semantic layers, Unity Catalog Metrics are fully accessible via SQL from any tool

  • Governed by default: Certified metrics come with auditing, lineage, and fine-grained access controls out of the box

  • Upcoming integrations: Support planned for BI tools like Tableau, Hex, Sigma, ThoughtSpot, and observability tools like Anomalo and Monte Carlo

💼 Why it matters for business: This Iceberg compatibility offers greater architectural flexibility and avoids "lock-in" to a single format. Companies can adopt Databricks without fear of being tied to a proprietary format, as they can choose the optimal table format for different use cases (for example, Iceberg for integration with certain open source tools, Delta for others) without losing centralized control.

It also facilitates multi-cloud or hybrid strategies, by being able to unify access and governance of data dispersed across different systems. Additionally, Unity Catalog Metrics solves the persistent problem of "metric sprawl" where different teams use different definitions for the same business KPIs, leading to confusion and misaligned reporting. By standardizing metric definitions at the data layer, organizations ensure everyone works from the same trusted definitions, promoting data reliability and business alignment. Ultimately, it's a guarantee that the Lakehouse platform embraces open standards while providing unified business semantics, something valuable for IT leaders when planning their long-term data infrastructure.

Learn more: Unity Catalog Documentation | Unity Catalog Metrics Session | Apache Iceberg Support

Lakebridge: Accelerated Migration from Legacy Data Warehouses

Finally, for organizations still operating traditional data warehouses and wanting to modernize, Lakebridge (formerly BladeBridge) was introduced. This free component automates up to 80% of the tasks in a migration process from a legacy data warehouse to the Databricks Lakehouse.

Lakebridge offers end-to-end support: profiling and evaluation of the source system, automated conversion of SQL and objects (for example, converting proprietary SQL codes to standard SQL over Databricks), data validation and reconciliation to ensure results match, all with minimal manual intervention. In practice, the tool can double migration speed compared to traditional methods, simplifying what is typically a complex and error-prone project.

💼 Why it matters for business: Lakebridge helps accelerate digital transformation by facilitating the migration of old analytical platforms to the modern Lakehouse architecture. For a CIO or CTO, this means less cost and risk when migrating decades of data and business logic, and a faster transition to a unified data and artificial intelligence environment. By automating the heavy lifting (e.g., converting hundreds of SQL procedures), the team can focus on re-designing solutions with best practices and not on repetitive tasks, getting the organization to benefit sooner from the scalability and flexibility advantages of the Lakehouse.

Learn more: Introducing Lakebridge | Fast, predictable migrations to Databricks

Conclusion

Together, these announcements underscore Databricks' strategy of offering a comprehensive data and AI platform that is open, scalable, and focused on business use cases. For business leaders, the news means: simpler infrastructures (fewer data silos and fewer isolated tools), greater agility to develop AI and analytics solutions, and the ability to govern the entire data and model lifecycle with confidence.

Technologies like Lakebase and Lakeflow indicate that even the most complex operational and data engineering workloads can now coexist natively with advanced analytics; tools like Agent Bricks and MLflow 3.0 show how cutting-edge AI can be implemented with stringency and speed; and features like Databricks One or Free Edition drive democratization of access to both the platform and knowledge in data/AI.

In a competitive environment where "every company is a data company," these innovations will allow organizations to accelerate their path toward being truly driven by data and artificial intelligence, maintaining control of their assets, and aligning technology with strategic business objectives.

Next
Next

End The Data Engineering Nightmare with Metrics.