Unity Catalog Implementation Enables AI-Powered Insights at dichter & neira
dichter & neira, a leading market research firm in LATAM serving major brands like AB InBev, Coca-Cola, and Nestlé, partnered with SunnyData to implement Unity Catalog and AI/BI Genie on their existing Databricks platform. This transformation enabled natural language interaction with massive retail datasets while ensuring strict data segregation for competitive intelligence across their diverse client portfolio.
Key Metrics
INDUSTRY: Market Research
SOLUTION: Natural Language Processing, Business Intelligence, Data Governance, Unity Catalog Migration, AI/BI Genie
PLATFORM USE CASE: Unity Catalog, AI/BI Genie, Delta Lake
Fast-paced retail and CPG (Consumer products and goods) growth across the globe is driving the market research sector to deliver real-time insights while managing an overflow of consumer and competitive data. Companies must provide intuitive self-service analytics to business users with minimal technical expertise, while ensuring strict data privacy and segregation of competitive intelligence.
dichter & neira (d&n), a leading market research firm in LATAM, serves companies like AB Inbev, Coca-Cola, Nestlé, P&G, and Unilever. They offer customized dashboards through web portals, providing clients with product information, stock data, consumer trends, and competitor intelligence.
The Problem: Breaking Down Technical Barriers
Despite their sophisticated data infrastructure, d&n business users were limited to static dashboard visualizations and struggled to interact with large datasets using natural language queries. As a result, they relied on technical teams for dynamic insights, creating bottlenecks.
d&n's reliance on Hive Metastore also prevented them from using advanced Databricks features, such as Mosaic AI and AI/BI Genie, essential for staying competitive in the current space. Additionally, with the sensitive nature of the competitive intelligence data they handle, d&n needed to ensure strict data segregation to prevent cross-client data exposure, while still offering meaningful insights to each client.
The Solution: Conversational Analytics
The SunnyData team developed a comprehensive transformation strategy addressing immediate AI implementation needs while maintaining business continuity.
SunnyData helped d&n build a solid foundation for adopting AI by first preparing its infrastructure for scalable AI workflows. The first step was upgrading d&n's data repository from Hive Metastore to Unity Catalog, a move that opened the door to advanced Databricks features. This process not only unlocked access to Mosaic AI and AI/BI Genie but also enabled more refined data governance through granular access controls. They streamlined orchestration by reducing 20 Azure Data Factory pipelines to just 5, and migrated them to Databricks through Workflows. This streamlined process improved efficiency and enhanced visibility and control over the client’s data processing.
To address data segregation requirements, SunnyData implemented row and column-level filtering via Unity Catalog, allowing d&n to control data access at a more granular level. These governance functions ensured that competitors' data remained properly anonymized while still being accessible to each client.
AI/BI Genie was deployed to empower business users to interact with data through natural language queries.
The solution was enhanced with a custom LLM Agent capable of understanding client-specific industry jargon, acronyms, and abbreviations. This added a layer of reasoning on top of the data, enabling the agent to offer more informed, context-rich insights and even suggest new metrics or analysis areas that users might overlook. It also meant a significant boost to user autonomy, as business users could now access and interpret the data independently.
Note: This is an illustrative representation of the interface.
Ensuring data integrity during the migration of approximately 400 tables was a major focus. By deploying SunnyData’s proprietary Data Validator Accelerator, the team accelerated the time required to perform validations by 65% compared to the original estimates of manual conversion and manual QA processes. This provided d&n with confidence that their data remained intact and reliable as they transitioned to the new platform.
The Results: Empowered Data Teams & Users
Through improved core synchronization, d&n experienced a 2x reduction in processing time, going from 2 hours to just 10 minutes for their data workflows, leading to faster insights delivery. With the migration to Unity Catalog and the introduction of AI/BI Genie, business users gained direct access to data through natural language queries, eliminating technical and process barriers that translated into fulfilling requests from the IT and business teams in hours or even days.
Governance and security of the data assets were also significantly enhanced, with granular access controls ensuring that each client's data remained segregated and secure. d&n could now confidently manage access levels across clients while maintaining the anonymity of competitors' information.
One of the most notable outcomes was the empowerment of d&n’s internal data team. Thanks to comprehensive knowledge transfer during the implementation, the team is now equipped to implement similar AI-driven solutions across other business units, reducing their dependency on external consultants. This internalization of expertise positions d&n for future growth, as they can now scale these capabilities independently across the business.
