Skip to main content

MotherDuck Acquires Tower to power Data AgentsRead

START FREE
BACK TO VIDEOS
StreamAI, ML and LLMsBI & VisualizationSQL

Do AI Agents Need a Semantic Layer?

2026/08/26

TL;DR: Instead of joining the semantic layer argument, Jacob Matson and Alex Monahan just ran the eval. Same warehouse, same questions, one agent grounded in plain-text context with SQL snippets, one working through a real semantic layer built with Malloy. SQL-based context hit 100% on their benchmark. Malloy peaked at 95% and used more tokens. But the semantic layer still earns its keep as a contract for change.

The experiment

They pulled 26 golden question-answer pairs and the data manual from the DABStep benchmark, then had Fable generate both a Malloy semantic model and a MotherDuck Guides context layer — no Malloy written by hand. A cheaper model answered roughly 400 held-out questions through each layer. The setup and results are in how AI writes the semantic layer, and the harness is open source in the agentic-malloy labs repo if you want to run it on your own data.

What the evals showed

With SQL snippets in context, agents scored 100%. Through Malloy, the best run reached 95% and burned more tokens. The models needed extra turns to work in a language they'd barely seen in training — fifty years of SQL on the internet is a lot of pretraining data to compete with. The agents also cheated: in early runs, about 40% of the time they compiled the Malloy, read the generated SQL, and just edited the SQL directly. The harness had to forbid raw SQL to get a clean comparison.

Why keep a semantic layer at all

Change a metric definition once and every consumer picks it up on their next refresh. That contract between data producers and consumers is where the real value sits, and it grows with org size. A ten-line Malloy query can also replace hundreds of lines of generated SQL, which makes review far easier for whoever has to sign off.

Falling model prices change the answer

Mid-experiment, a price drop made the smallest frontier model the most cost-effective choice. Eval runs went from about $500 on a large model to $4–5. The optimal model changed while the experiment was still running, which is the argument for owning your evals. MotherDuck's AI analytics eval field guide covers how to build your own.

Practical takeaways

Use context (like MotherDuck Guides) to route agents to the right semantic models with progressive disclosure. Bootstrap a context layer from your query history with something like SQLGlot. Consider a two-tier design: governed metrics first, transparent fallback to SQL. And don't ask an agent how confident it is. Measure it.

FAQS

Not for accuracy, at least not in the tests MotherDuck ran. An agent grounded in plain-text context with SQL snippets scored 100% on a DABStep-derived benchmark. The same setup through a Malloy semantic layer peaked at 95% and burned more tokens. Where a semantic layer pays for itself is elsewhere: governed metric definitions that update every consumer at once, a stable contract between the people producing data and the people consuming it, and query logic concise enough that a human can actually review it.

A semantic layer translates. It defines tables, joins, and metrics in a structured language that compiles down to SQL. A context layer navigates. It holds documents, definitions, and the relationships between them so an agent knows where to look and which model or metric to reach for. Ontology is the relationship subset of context. In practice the two work together — context routes the agent to the right part of the semantic layer.

Yes. In this experiment, Fable generated the entire Malloy semantic model from 26 question-answer pairs and a data manual. Zero lines written by hand, including advanced constructs. Using Malloy to answer questions was harder. Models have seen orders of magnitude less semantic-layer code than SQL in training, so they needed extra turns and occasionally tried to fall back to raw SQL anyway.

In this experiment, no. The Malloy path used more tokens and more time than SQL-based context because models needed extra turns to work in the less familiar language. The efficiency gain is on the human side: a ten-line Malloy query can express what takes hundreds or thousands of lines of generated SQL, making results far easier to interpret and review.

Not necessarily. Open source tools can convert between semantic layer languages if you're locked into one. And a semantic layer is useful context even if the agent never executes it directly. Including the definitions in your context layer alongside example SQL from your query history gets the knowledge out of your BI tool without a full rebuild.

Related Videos

View all