We set out to prove that AI could build and operate a semantic layer. It could. But for us, the most important insight had nothing to do with the semantic layer.
Many companies have found using a semantic layer to be helpful for managing metrics, but does that still hold in the AI era where agents write great SQL?
We posed two hypotheses when we set out to test the semantic layer, using Malloy as our runtime of choice.
The results? We could indeed construct an accurate semantic layer in Malloy in a “hands-off”, agentic manner. However, it was much more expensive to operate than simply using Markdown + SQL. In rough numbers it: consumed 2.5x more tokens, was less accurate (95% vs. 100%), and slower.
You can see the Malloy models that were built autonomously and the full experiment details in our repo. There are examples of the validation scaffolding and instructions we added to our harness to provide semantic model design feedback to the agent and teach even the frontier models more about Malloy.
We chose Malloy for a variety of reasons, including its symmetric aggregation capability, which correctly calculates summary metrics like averages even across joins that multiply your row count. We also valued its completeness of vision (as Gartner might say!). Malloy has an elegant design that handles the edge cases that you encounter when battle testing with real users. Another key factor was that Malloy is open source. Malloy also is more composable than SQL, making Malloy views more reusable than SQL ones. Part of our hypothesis was around reusability increasing efficiency.
We learned a few things directly from this:
But there are also some implicit learnings from this that we think are much more interesting than “just the facts.” Once intent and evaluation are explicit, implementation artifacts (i.e. the semantic layer itself) become disposable. This is not a critique of Malloy! It is an observation about all of software in the AI era.
As such, the cost of regeneration becomes more important than the cost of maintenance. The result of this line of thinking is that the semantic layer shifts from being “the ultimate source of truth” to a mere projection of business behavior, with the actual durable representation of expected business behavior remaining as question-and-answer pairs. If code becomes cheap, nearly free to write, so shifts the representation of what really matters to our business.
In short, your data tests become more valuable than your semantic layer.
While this experiment does not prove that all software can or should be regenerated, it does show that when we can target specific behavior and evaluate the results quickly, regeneration becomes practical (and maybe even preferred).
Given how cheap and fast it is to (re)generate code, it is time to reconsider how we approach the problem entirely. In the old world, we would implement decisions in code, then preserve, maintain, and patch time after time, accumulating historical behavior inside the artifact. In the new world, we are not subject to the same constraints. This means we can start with capturing expected behavior and then use AI to generate a sufficient implementation. When the new models come out, we can simply regenerate it.
The linchpin that makes this possible is keeping evals alongside every expected behavior. Our semantic layer no longer needs to be a precious, slowly accumulated artifact, but instead a replaceable output of a more durable system. That system can be better aligned to the actual questions the business needs answers to.
In our testing, we had two inputs that are provided as part of the eval set: (1) a manual describing the data and the system, and (2) a collection of questions and expected answers. Given these two pieces of information as a foundation, we could regenerate a framework for answering questions into an arbitrary language (in our case, Malloy).
Said another way: we build a system that could recreate a semantic model given a solid foundation. The output artifact itself was transitory: we could create a reliable semantic layer from any number of frontier models given the same inputs.
This doesn’t mean that code is no longer important. It often contains years of valuable decisions and edge cases. But we would argue that code may no longer be the ideal place to preserve intent. If the same behavior can be reproduced in another implementation (see the Bun rewrite in Rust), the underlying intellectual property did not disappear with the original code.
The durable thing to preserve is the behavior of the organization. The implementation is merely one expression of that behavior. Of course, question-and-answer pairs are not necessarily the entire durable representation - they are just a useful intermediate that makes expected behavior concrete and measurable.
What we are really getting at here is that question-and-answer sets become a starting point for organizational memory. When organized into business capabilities, intents, and dependencies, this creates a map of organizational behavior that helps AI (and humans) immediately understand the business. This parallels software development as well: effective testing is critical, but so are design docs and documented coding standards.
Given this map, our agents can cleanly retrieve the right context for a given problem. In MotherDuck for example, we provide context right next to your data with Guides so that our MCP can answer your questions more accurately over time. This beats software as built today on nearly all counts - which is optimization for execution, not understanding. By treating the software as transitory, we gain understanding of a record of decisions and a way to retrieve them in an understandable way.
If SQL based context and metrics are more accurate than the semantic layer, do we need it any more? Our experience leads us towards yes - but what you need it for can be very tightly confined to three core reasons.
First, you need a way for the data platform team to modify the underlying data models without breaking all downstream consumers. A semantic layer is a valuable abstraction above tables (far more flexible than database views) where a change can be made in the data layer without disrupting consumers.
In contrast, the second need is a way to make a single update to a metric effectively push down to all consumers. Sometimes you need to avoid change, other times you need to make that change universal. Accounting for the new realities of the business in the semantic layer should cause even existing reports to reflect an accurate state of the world.
Lastly, the guardrails a semantic model provides allow a central set of data or domain experts to have more control over customer answers. If an agent is constrained to writing queries against pre-existing semantic model definitions, it provides more determinism in how metrics are calculated. It allows one part of the organization to take on more of the complexity burden from consumers. The experts, whether engineer, analyst, or executive, can make analytics more consistent for the rest of the company.
Taken together, a semantic layer’s value comes from what it enables operationally, not from being the final repository (single source of truth) of organizational intent.
Given these results, we should design our data processes very differently.
If the semantic model can be generated, organizational effort can shift to codifying the business expectations of that model. That could mean manual question and answer development, but going all the way down that road looks suspect. Just look at Meta putting their engineers to work doing data labeling - it likely had many negative downstream consequences. Manual labeling may be suitable to start, but in a “do things that don’t scale” way to bootstrap the process.
Perhaps authoritative answers can be passively ingested. This has some challenges in existing systems - often a database only sees the SQL query, not the true question. But any text to SQL interface could log this information easily. Except, how do you know the answer fulfilled what the asker requested? Are we documenting slop?
As in many things in AI, a balanced approach that combines what AI is best at with human expertise looks most promising. If your business users are chatting with a Slack bot (perhaps even a QuackBot Slack Bot) to get answers, maybe the best path is a feature to certify a correct answer. The secret sauce could be an emoji! If your organization lives in Claude, maybe that looks like an MCP that looks for data questions and suggests saving the answer centrally. The devil is in the details though - every question should be time scoped so it stays an effective, static test.
Organizational knowledge beyond SQL queries matters as well. When a system is being designed, be that an inventory management process or a new mobile app, the design decisions of that system affect how to analyse its outputs. There are many facets to those decisions that can be documented:
Are you the domain expert? Documenting what you know can help you scale your impact across the company. Know an expert? Interview them, transcribe it, and give them the credit when you teach their wisdom to the whole org.
Documenting system design assumptions has substantial precedent in software engineering as well. As one example of many, the Behavior-Driven Development approach of tools like Cucumber.io directly relates user stories to test cases. Do user stories have a bad reputation? Yes. Done poorly, they take time and don’t add value. But maybe AI changes the equation. Our experiment shows that these approaches are now viable in data modeling. You can “compile” your data tests into the semantic model itself, with the fuzzy compiler of an agentic loop.
A large complex enterprise cannot simply replace its systems with a folder of specifications and ask AI to regenerate everything. After all, they have thousands of employees, tightly coupled workflows, regulatory requirements, legacy IT systems, and operations battle tested over years of work. Telling an enterprise to “extract the intent and regenerate the software” is simply naive. The pathway is not “how do we rewrite the last twenty years of software?”. Instead:
“Where should we accrue knowledge?”
Starting with the next decision, instead of a rework, means systems can continue to operate while the center of gravity shifts towards the organizational memory provided by context. Over time, more implementations become projections of explicit behavior instead of standalone code, naked without context.
For years, data teams have accumulated code because code was the most precise way to preserve how a business worked. AI does not make that history irrelevant, but it may give us a better set of primitives to store intents. This changes the long term asset from being the implementation of the intent into the explicit, testable description of the behavior we want to preserve.
The question that remains: If you were founding your company today with AI as a given, would you choose code as the primary long-term memory of how the business works?
2026/08/13 - Simon Späti
The August 2026 DuckDB Ecosystem Newsletter: giving every agent its own embedded DuckDB, the ADBC extension reaching 30+ external databases, async I/O landing in v2.0, ISO GQL graph queries, Perspective 5.0 pushdown, plus a community spotlight on Kyle Cheung.
2026/08/18 - Alex Monahan
AI agents make building data apps easy, but what about deploying them or making sure they scale? Replit provides a way for just about anyone to build real applications, and MotherDuck is the easiest way to help them scale to larger data sizes. Load data with MotherDuck Flights and you can query millions to billions of rows on the fly!
