The "chat to data" promise vs. reality: evaluating native text-to-SQL models
By Etienne Delate - AI engineer at Euranova
Disclaimer: This article summarizes findings representing a snapshot of the ESAD project’s ongoing evaluation as of June 16, 2026. We are actively scaling these results across broader datasets.
Executive Summary: Key Takeaways for Data Leaders
- The Production Gap: While native Text-to-SQL tools achieve high accuracy on pristine, synthetic data (Genie: 83.1%, Cortex: 81.8% on Spider), their execution accuracy drops significantly on real-world messy data (Genie: 57.4%, Cortex: 55.4% on BIRD) and completely fails on massive enterprise dimensional models (0% on TPC-DS).
- Methodology & Unbiased Evaluation: To bypass dialect translation biases, we evaluated models using a strict Custom Execution Accuracy (CUSTOM-EX) metric directly within their native engines, ensuring a fair baseline comparison.
- AI is "End-of-Chain": LLM interfaces cannot compensate for uncurated data warehouses. Production viability depends heavily on the quality of the underlying data pipeline and explicitly governed semantic layers (YAML models vs. Unity Catalog metadata).
- The Accuracy vs. Latency Trade Off: Full-scale dataset evaluations reveal distinct operational profiles. Databricks Genie AI/BI achieves higher overall accuracy across both clean and dirty datasets and consistently outperforms Cortex on the most complex queries, but it operates at a higher median latency (~12 seconds). Conversely, Snowflake Cortex Analyst is significantly faster (median latency ~3 seconds) but exhibits lower resilience, completely failing to generate SQL for 8.9% of prompts on messy data.
Baseline Performance Summary
Tab. 1 Execution accuracy and latency of Snowflake Cortex Analyst and Databricks Genie on BIRD dev, Spider test and TPC-DS (Scale Factor 10)
Introduction: The "Chat to Data" Promise vs. Reality
Every data-driven organization today shares a common goal: democratizing data access. The promise of Large Language Models (LLMs) is that business users can ask questions in natural language and immediately receive the right data. This paradigm aims to alleviate the bottleneck of waiting for specialized data teams to write complex SQL queries.
Organizations are rapidly adopting the native AI tools offered by their primary cloud data platforms due to ecosystem lock-in. However, the core business problem behind impressive commercial demos is consistency. LLMs are fundamentally stochastic; the same prompt may yield different SQL logic on different attempts. Furthermore, modern platforms are shifting from pure "Text-to-SQL" generators to Agentic Systems. Unlike standard generators, agentic models execute the generated query, analyze the output, and use it as an intermediary step to deliver a final answer.
Our objective is to provide objective insights into the true capabilities of two major native agentic solutions—Snowflake Cortex Analyst and Databricks Genie AI/BI—by testing their core generation engines under strict conditions.
The Evaluation Protocol: Moving Beyond Commercial Demos
To test how these platforms perform as complete systems, we structured our evaluation pipeline around three core pillars: the model, the data, and the metrics.
1. The Model: Testing Agentic Systems via Single-Shot Generation
While true agentic workflows allow models to iterate and self-correct based on database errors, evaluating baseline intelligence requires isolating the core generation capability. We enforce a strict single-shot approach: the AI is given exactly one attempt to generate the correct SQL query. We bypass multi-turn clarification loops to test the platforms as automated, end-to-end systems. This differs from complex ETL generation tasks by focusing strictly on immediate, conversational data retrieval.
2. The Metrics: Execution Accuracy and its Limitations
Evaluating Text-to-SQL models strictly on the generated code (Exact Match) is flawed because two structurally different SQL queries can yield the same correct answer. Instead, we rely on Execution Accuracy (EX): executing both the "Gold" (human-verified) query and the AI-generated query and comparing the resulting tables.
However, EX has limitations. Traditional implementations like BIRD-EX are vulnerable to "empty set" false positives, where a broken query coincidentally returns no rows, matching an empty gold result. Conversely, Spider-EX relies on perfect data cleanliness, lacking defensive data casting.
To conduct an unbiased evaluation, we developed CUSTOM-EX, a four-step pipeline that checks row counts, validates column volume, performs order-agnostic column mapping, and finally ensures tuple-level equivalence. Additionally, we evaluate purely through native execution rather than transpiling queries between SQL dialects, as automated translation introduces semantic biases and silent execution failures.
Other metrics monitored include the Valid Efficiency Score (VES), which calculates the relative execution efficiency of the AI's query against the gold standard to penalize slow, unoptimized code.
3. The Data: Integrating Business Context & The Testing Grounds
For an AI to generate accurate SQL, it must bridge the gap between user intent and physical database schemas. To ensure parity, we programmatically generated semantic layers for both platforms:
Snowflake Cortex Analyst utilizes a centralized YAML semantic model outlining table logic, synonyms, and dimensions.
Databricks Genie derives its context directly from Unity Catalog metadata, specifically leveraging COMMENT ON TABLE/COLUMN statements and primary/foreign key constraints injected programmatically.
The Pipeline Architecture: Our technical setup harmonizes dataset knowledge, injects metadata into the respective semantic layers (YAML or Unity Catalog), queries the native Virtual Warehouses/SQL Warehouses via Python SDKs, and pipes the output into our CUSTOM-EX scoring engine.
We subjected the models to three distinct datasets:
Spider (Syntax Test): Evaluates fundamental structural intelligence (multi-table joins, nested subqueries) on pristine, clearly labeled data.
BIRD (Real-World Trial): Measures resilience against uncurated, "dirty" enterprise data featuring missing values, inconsistent formatting, and internal business jargon.
TPC-DS (Scale Test): Evaluates models against highly complex dimensional snowflake schemas and advanced OLAP workloads at massive scale.
Key Results and Operational Realities
Scaling the evaluation across the full datasets reveals a stark contrast between controlled environments and real-world complexity, while highlighting distinct architectural trade-offs between the two platforms.
- Controlled Environments (Spider): On clean data, syntactic parsing is highly capable. Databricks Genie achieved an 83.1% overall execution accuracy, slightly edging out Snowflake Cortex Analyst at 81.8%.
- Real-World Imperfections (BIRD): When exposed to dirty data and jargon, performance plummeted. Genie reached 57.4% accuracy, while Cortex followed closely at 55.4%. However, Genie demonstrated superior stability; Cortex completely failed to generate SQL for 8.9% of the messy BIRD prompts, and generated syntactically failing SQL in another 2.5% of cases.
- Enterprise Scale (TPC-DS): Both platforms registered a 0% success rate on massive dimensional models, unable to navigate advanced OLAP requirements in a single shot
Tab. 2 Custom execution accuracy (%) by difficulty level. BIRD levels are the official annotations, Spider levels are computed from the gold SQL
Throughput and Cost Considerations:
While accuracy is paramount, enterprise adoption is constrained by execution speed and platform mechanics:
- The Latency vs. Stability Trade-off: Cortex Analyst is heavily optimized for speed, boasting a median latency of roughly 3 seconds (3.6s on BIRD, 2.9s on Spider). In contrast, Genie prioritizes complex reasoning over speed, operating at a median latency of roughly 12.3 seconds.
- Cost: Cortex Analyst operates on metered consumption, costing roughly $17 USD per 100 requests generated, accumulating meaningful operational costs at scale.
- Throughput Limits: Genie generation can be free but constrained by a strict API rate limit of 5 questions per minute per workspace under standard tiers, requiring substantial architectural workarounds for high-volume enterprise pipelines.
Takeaways for Engineering and Management
The sharp drop in accuracy from the pristine schemas of Spider (peaking at 83.1%) to the messy reality of BIRD (falling to ~57%) and TPC-DS (0%) definitively proves that Text-to-SQL AI must be treated as an "end-of-chain" system.
The models fail when confronted with undocumented jargon, complex snowflake schemas, or dirty data, with Cortex Analyst even failing to generate SQL entirely for nearly 9% of messy real-world prompts. You cannot use Cortex or Genie to bypass a poorly architected data warehouse. To unlock their value, data engineering teams must first clean the underlying data, simplify analytical schemas, and build explicit, highly curated semantic layers. The AI is only as smart as the structured metadata pipeline preceding it.
What’s Next: Deepening the Evaluation
As we continue the ESAD project, our next steps are focused on three core areas:
Detailed Error Taxonomy: Deepening our analysis of structural failures (Attribute Mismatches, Missing Conditions, Clause Errors) to understand exactly how the LLMs misinterpret relational algebra.
Validating Advanced Metrics: We will test platform-specific optimizations (like Gold Query injection) and measure the Valid Efficiency Score (VES) to ensure generated queries are not just accurate, but computationally performant.
Annex
Evaluating at Enterprise Scale with TPC-DS
1. Why We Chose TPC-DS TPC-DS is essential for evaluating Text-to-SQL models against massive, enterprise-grade data warehouse environments. It explicitly stresses models across three dimensions:
- Dimensional Modeling: Requires navigating an interconnected snowflake schema with 7 fact tables and 17 shared dimensions without generating Cartesian products.
- Advanced OLAP: Simulates complex workloads requiring deep subqueries, rolling windows, and advanced aggregations (e.g., ROLLUP, RANK()).
- Massive Scale: Verifies if generated queries are syntactically optimized for execution at up to 10 TB scale.
2. Transformation Challenges Adapting an RDBMS query optimizer benchmark for LLM evaluation required significant engineering:
- Semantic Synchronization: Business logic had to be manually codified and programmatically mapped with strict parity into Snowflake's YAML and Databricks's Unity Catalog metadata.
- Dialect Transpilation: Original benchmark queries (e.g., Netezza, Oracle) required translation and rigorous statistical auditing to prevent "silent semantic failures" arising from implicit casting or NULL handling differences.
- Reverse-Engineering Prompts: Natural language questions had to be generated retroactively from the raw SQL templates.
3. Limitations Explaining the 0% Success Rate Both Snowflake Cortex Analyst and Databricks Genie AI/BI achieved a 0% Execution Accuracy on TPC-DS. This failure stems from both model capability limits and methodological constraints:
- Navigational Failure: Models consistently failed to correctly traverse the complex logic required across 24 interconnected tables in a single-shot setting.
- Prompt Bias: The reverse-engineered questions inadvertently mirrored relational algebra (e.g., explicit grouping instructions) rather than ambiguous human phrasing, failing to accurately test the true "semantic gap".
- Insufficient Statistical Volume: The benchmark's 99 core query templates provide too few distinct logic paths to establish a statistically significant performance delta between high-tier LLMs.
Lexicon
- Agentic System: An AI architecture where the model does not just generate text, but executes actions (like running a SQL query), analyzes the output, and uses that environment feedback to inform a final result.
- CUSTOM-EX (Custom Execution Accuracy): A proprietary evaluation metric that validates the correctness of AI-generated SQL by executing the query and verifying row counts, column mappings, and tuple equivalence, bypassing rigid syntax penalties.
- Semantic Layer: The codified business logic (descriptions, synonyms, relationship maps) provided to an AI to bridge the semantic gap between a user's natural language question and the raw database schema.
- BIRD: A benchmark dataset representing messy, real-world enterprise data, heavily featuring ambiguous schemas, dirty data formats, and external knowledge dependencies.
- Spider: A baseline benchmark dataset containing clean, unambiguous schemas, used to test an AI's fundamental ability to formulate structurally complex SQL (e.g., joins, nested queries).
- TPC-DS: A massive, enterprise-grade decision-support benchmark featuring highly complex dimensional models (snowflake schemas) and advanced OLAP workloads.
Bibliography
- [1] D. Pihulski, K. Charchut, V. Novogrodskaia, and J. Kocoń, "LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL," Dec. 09, 2025, arXiv: 2510.02350.
- [2] K. Akillioglu, A. Chakraborty, S. Voruganti, and M. T. Özsu, "Research Challenges in Relational Database Management Systems for LLM Queries," Aug. 28, 2025, arXiv: 2508.20912.
- [3] F. Lei et al., "Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows," Mar. 17, 2025, arXiv: 2411.07763.
- [4] J. Li et al., "Can LLM Already Serve as A Database Interface? A Blg Bench for Large-Scale Database Grounded Text-to-SQLs," Nov. 15, 2023, arXiv: 2305.03111.
- [5] T. Yu et al., "Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task," Feb. 02, 2019, arXiv: 1809.08887.