0:02Jess: All right, welcome everyone to our webinar. We're excited for you all to join us today. Let's quickly introduce ourselves. I'm Jess, I'm a DevRel engineer at Braintrust, and we also have —
0:18Jacob: Hello, hey everybody. I'm Jacob, I'm a developer advocate at MotherDuck.
0:25Jess: And we both love pickleball and tennis, as we found out talking before this webinar.
0:33Jess: Okay, so let's get into what we're going to cover today. The idea is that as we were figuring out what to work on and collaborate on, we found this issue: back in the day, before AI existed, you might ask your database a question and write it in SQL or code, which is precise — as long as it compiles, you know exactly what you're getting back. But nowadays most people query their database using natural language. It's faster and easier than writing plain code, but it's also a game of telephone, because your agent could quietly answer a question slightly wrong, or differently than what you asked. So what we wanted to do is use MotherDuck and Braintrust together — use MotherDuck to query a dataset, and then score how well it actually performs by using Braintrust to score it, read the traces, and understand how that whole agentic system is performing. We're going to talk through all of that, and we'll leave time at the end for questions. I'm going to pass it to Jacob to give some context on what MotherDuck is, I'll talk about what Braintrust is, and then we'll go into the specific eval we built.
2:07Jacob: Amazing, thank you Jess. I'm going to talk a little about our architecture, which is built on top of DuckDB — I'll get to that in a bit. As it turns out, the architecture of your database matters for what it looks like to use agents with it. The core thing that happened when we were dealing with MapReduce back in 2006 is that our servers had about 2 gigs of RAM and one core. If you look at today, we have many terabytes — I think you can get 32 terabytes of RAM on the biggest AWS instances — and over a thousand cores. That's a totally different world from when a lot of the first wave of big cloud data warehouses were built, like Redshift and Snowflake. The second, related thing is that when we're querying our data, we're often using very small, targeted queries. We don't need giant massive machines to do that, nor do we need hundreds of nodes running at once.
3:20Jacob: Where this gets interesting with agents is that what we've observed with customers is they often need a bunch of small nodes, and they give those to their agents, and their agents just spike queries through. Because they can write way more queries than humans can, way faster. They don't have a lot of context for the data — much less than a human — even if you're doing really great work around providing context. So they're writing a bunch of queries to get grounded all the time. Even if you ask a very specific question that it should know, it's going to ground itself first. That leads to a different kind of workload.
4:03Jacob: This fits in really nicely with DuckDB. DuckDB is a super fast single-node query engine. What does that mean? It runs on one node. There's no distributed systems tax, no MapReduce — it's just `GROUP BY` and aggregates, the SQL words for the same thing. It's super lightweight, and we can scale it up to really big nodes if we need to, but queries only stay on a single node. As a result, our queries run really fast because there's no network penalty, and it's easy to vectorize across one machine with hundreds of cores compared to splitting that out across a big distributed system.
4:44Jacob: How MotherDuck fits in — we're building a platform that runs all this for you. The first thing is serverless compute, so you only get billed for when it runs. This is important for agents: they're going to spike, you need a lot of compute available for a short period of time, and then it's done. The next thing is hyper-tenancy. What this really means is that each agent or each user gets their own isolated compute sandbox. Because of this, you can very safely run unsupervised AI and data workflows, and you don't end up in a scenario where you spend hundreds of thousands — or maybe not, but tens of thousands — on a single query. We see people do that on other platforms where their agents are being a little too aggressive. And if you're running on shared infrastructure, maybe something like Postgres replicas, you start seeing "my query affects your query." That problem is eliminated in the MotherDuck sandboxing approach. The last thing is we've got a bunch of really cool AI-native integrations. The MCP server is the entry point, although we have a great CLI as well. MCP is what we're going to be talking about mostly today in terms of the integration point with Braintrust — well, sort of, but as the method for running these questions. We've got all these affordances to make it easy to plug in really small instances, or get really big ones if you need to, and then scale them out as needed. So that's what we've got. Jess, why don't you keep going?
6:30Jess: Cool. So Jacob has talked about how MotherDuck makes it easy to point an agent at your database and get an answer back. But the catch is that the answer you get back might not be correct — or it might not be fully correct. That's where Braintrust comes in. Braintrust, if you've never heard of it, gives you a systematic way to measure your agentic system so you can catch regressions, hallucinations, and quality drift, and make better product decisions.
7:10Jess: There are two big pillars to our product. There's observability, which I'm sure most of you have heard of, and there are evals. Evals are the core primitive we're going to be talking about in today's workshop, because we ran an independent eval which we'll get to in a sec. The way I like to explain evals is that it's just a way to answer a question about your system. For example, it might answer something like: should I be using Kimi K3 to run my agentic system, or GPT-5? Or if you're Netflix, you can measure something like: is my Netflix AI movie recommendation system actually recommending movies that people want to see?
7:59Jess: The way it works — if we look at the graph on the right — is it takes real inputs, runs them through your AI system, and then scores that output based on whatever verticals you want to score on. It could be toxicity, accuracy, hallucination, humor, tone, things like that, and it gives back some sort of percentage. Based on that, you now have a quantitative way to measure your AI system, and you can also look at the traces and see exactly what's going on.
8:41Jess: To visualize how MotherDuck and Braintrust work together in this eval, this is the flywheel that's going to be your conceptual model. You start at the top: develop. That means we're developing an agent that queries over our data. In our case, we created an agent that answers questions about a dataset using the MotherDuck MCP to write its own SQL query. Then we go to the right: evaluate. Using that agent, we score the agent's answers against some sort of golden correct answer — we'll talk about how we get that in a second — and that gives us a quantitative score. The next step is observe. Once you get those scores measuring how the agent performed, you're going to not just look at the final percentages but also look through the entire trace: read through the model calls, read through every query, to understand why you're getting the score you're getting. If the score is lower than you expected, you're going to read through the traces to understand why. And the final step is improve. Once you understand why your agentic system is performing a certain way, you can tweak your system prompt or change your code to actually improve it. That's the conceptual loop of how MotherDuck and Braintrust work together in this eval.
10:28Jess: Okay, so we've teased the eval a couple of times — let's actually get into what it is. I think there are some pretty interesting numbers and findings here. My mental model for running an eval is that you always start with a question. As I said, an eval is really just a way to answer a question about your system. Today, the question we're asking is: how often does an agent using the MotherDuck MCP server get the right answer when querying a dataset, when does it get the wrong answer, and why?
11:08Jess: There are three main components of an eval. The first is your dataset. For ours — and Jacob will cover on the next slide how we created it — it was 40 analytic questions across a synthetic company warehouse, and each question came with a known correct gold answer. The second portion is running the task. For our task, we had Claude Sonnet 5 answer each of these questions from the dataset using the MotherDuck MCP, which hands the agent a schema and metadata and then has it write its own SQL queries. The final part is the scoring system. We match the answer the agent gives when completing the task with the expected correct answer, and give it a numeric score based on how well it matches. Then we aggregate that score across the entire dataset to get an overall holistic score percentage.
12:26Jess: Let me give you a better picture of our dataset. This is a dataset that Jacob built, and he'll explain how he built it in a second. Essentially it has 12 different schemas and 82 tables across about 2 million rows. It covers direct-to-consumer, wholesale, finance, ops, marketing — basically any data you would have across a synthetic company. There were 40 questions about the synthetic company: 10 easier and 30 harder. An example of an easier question would be something like: as of September 31st, 2026, how many unique products had launched and were active? The reason this is easier is that all you have to do is look at one specific table and filter down by a certain date to get the number.
13:29Jess: A hard question would look something like: which product produced the most net realized contribution from commerce initiated during H1 of 2026? This is harder because the agent needs to look across multiple different sales channels to understand net contribution. It's also not necessarily a column that exists in a table, so the agent has to calculate that itself. It needs to understand business terms — "realized" means real prices, so it has to look at actual and contract prices, not just list pricing. And there's a time constraint: it has to figure out what H1 is. So that's an example of what an easy versus a hard query looks like.
14:31Jess: Okay, and then I'll hand it to Jacob to explain how he built this. It's a really good synthetic dataset — I think it's interesting to go through.
14:41Jacob: Thank you, Jess, I really appreciate it. I've been working on evals on top of MotherDuck — specifically the ability of agentic harnesses to answer questions against our data — since at least the beginning of 2026 now. You learn a lot through that process. The first thing is that there are a bunch of academic datasets that exist. Spider is one, BIRD-Bench is another, DABstep is one I'm particularly fond of. But a lot of the LLMs know them pretty well, so they don't serve as a good basis for building an eval.
15:16Jacob: I come from a background in accounting — I mentioned this in the chat, but for those who didn't follow it, I was VP of Finance before I worked in tech. Actually that's not exactly true; I worked at a tech company, but not on the technology side. Anyway, I said: I wonder if I can build an agentic ERP, which is basically a way for me to build a cohesive dataset. That's where it started. I leveraged agents as much as I could to build that thing out, and really focused on realistic procure-to-pay and order-to-cash company data. Those were the initial starting points.
16:05Jacob: So it was something novel that the model couldn't have memorized, and that I could then build out questions against. Once we had a dataset we liked — which I'm still tuning, and it may be something I publish in the near term as open source, we'll see how far I can get there — I gave my agent access to commonly known benchmarks and said: okay, given the data that I have, how do I start shaping this into a good eval set, along with my own knowledge of what I built in the codebase? That generates a set of questions for us, and then we start doing our own eval on the eval set — which is down in the bottom right, where the mouse is. We need to refine this set. The initial questions might be pretty good; they might not be hard enough. Actually, some of the tuning here — since Opus 4.5 or 4.6, around November of 2025 — the ability for agents to look at your database and answer questions has massively improved. So a lot of these things you have to intentionally make hard, especially when you're building something synthetic.
17:14Jacob: I ran it anyway. I ran it on some fast, cheap models, looked at the traces, improved the clarity of our questions and supporting docs, and then kept running it in a loop until I got to something I felt converged on what a good set was. That's how we ended up with the two parts that make this possible: the underlying company dataset, and the question-answer pairs we can then use to build the eval.
17:43Jess: It's really cool. We call it a synthetic dataset because we created it synthetically, but it's important to mention that Jacob, with his extensive business knowledge, looked through the dataset and made sure it was good and audited the questions and answers.
18:00Jacob: I will just say that secretly I love that we're doing evals now, because it really scratches the accounting itch for me.
18:08Jess: And I'm glad that you audited the questions and answers, because I do not understand any of these terms — I had to look up a lot of what everything meant just to read the traces myself. So that's so funny.
18:24Jess: But it was a really interesting dataset, and it's cool how we built it. Okay, the last portion is how we scored the answers. Thanks to Jacob, each question in the dataset also had its own golden answer. But he also provided two more pieces of metadata. The first is the answer type — whether we're expecting back a number, a string, a list, things like that, so the agent knows what to look for. The second is the tolerance: how close counts as right.
19:09Jess: For example, if the agent gives back $2,922.73 but the correct answer is $2,922.74, we allow for the tiny rounding gap. Or fuzzy matching: if the answer is `paid_search` but the agent gives back "paid search," that counts as correct. So we also provide the tolerance rule for what we allow as the correct answer. Besides that, it's pretty much deterministic. If it matches, it gets a score of one, or 100. If it doesn't match, it gets a zero. That's how we scored the agent.
19:55Jess: Okay, so let's get into the actual numbers and what we learned. The agent scored 100% on the easy questions and 64% on the hard questions. So it passed all the easy questions with flying colors and struggled a bit on the hard ones.
20:21Jess: One interesting thing we found is when we separated out the scores by the bucket of questions in the dataset, the finance and ERP questions scored really well, and the agent really struggled with marketing and wholesale questions.
20:51Jess: When I looked at the traces, it was mostly because in general the finance and ERP answers already existed in a specific table in the database — something like an accounting ledger or a list of unpaid bills. Whereas for the marketing and wholesale questions, there was generally no existing table, so the agent had to reconstruct the answer somehow, which is why it scored a lot lower on those.
21:25Jess: The reason we were able to build that story and understand why marketing and wholesale scored lower is that in Braintrust you can look through traces. I've been using the word "trace" a lot, but you might not have a conceptual understanding of what it is, so this is a screenshot of what a trace looks like in Braintrust. For my job I go through traces a lot, but this might be your first time looking at something like this. On the left-hand side is what a trace looks like. Each of these rows is a span, and a span is synonymous with any sort of action. A span could be a call to the MotherDuck MCP to list the tables, read a table's columns, or query something. It can also be a call to the model — a call to Claude Sonnet 5 to go through reasoning steps and things like that.
22:28Jess: In Braintrust you can click on any of these spans to open it up and see exactly what SQL it wrote and what it got back, plus any metadata like latency and cost. So with traces you get more insight into how the agent thought about a problem, and you can follow the story of how it progressed through its logic to get the final output. These are the things I look at when I'm reading through the entire story of how the agent performed.
23:17Jess: Another interesting thing I found looking through the traces is a breakdown of what endpoints are called in the MotherDuck MCP. On this graph on the left, these are the endpoints that exist within the MCP, and you can see a large majority of them are just querying and listing columns. To me that means the agent spends a lot of time reading table schemas and catalog metadata to try to find its way around the data.
23:50Jess: Something else I found interesting is that for the harder questions, the agent tends to what we call thrash more. When an agent is working on an easy question, it made an average of about seven tool calls and three column lookups. On a harder question, it actually doubled that — around 11 tool calls and five column lookups. So the harder the question, the more the agent has to dig and the more queries it ends up making. And it doesn't always arrive at the correct answer — it just means it's trying harder to find it because it's struggling.
24:40Jess: I mentioned the agent scored 64% on the hard questions. We wanted to look at specific traces where it failed and why. What we found — for two of the traces we looked at, and this spans more of the traces — is that it's not really MotherDuck's fault, it's not really the MCP server's fault, that it's scoring low.
25:13Jess: For example, in the first trace we looked at, the question was: as of this date, which product produced the most net realized contribution from commerce initiated during H1 of 2026? It counted the contribution wrong because it counted direct-to-consumer only and dropped wholesale, which was about 60% of what the correct number would be. So this is more the agent not really understanding what net realized contribution is — I don't even understand what it is — but the idea is the agent doesn't really understand the phrasing or wording of the question itself, so it wasn't able to look up the right data.
26:05Jess: Another one I saw is a question that asks: on this date, which active launched SKU had the highest total committed inventory capital per unit shipped across both channels during the preceding 90 days? The answer was off by just one number — the correct answer was 215, the agent gave back 214. It was off by one unit because the phrasing was "preceding 90 days," and the correct answer was inclusive of the 90th day while the agent's answer was not. So this is a semantic thing, right?
26:50Jacob: Yep, totally.
26:52Jess: So as we can see from the traces, it's not that the MotherDuck MCP is failing, but rather that the agent isn't understanding the question, or maybe isn't understanding what the business terms are.
27:06Jess: This is the biggest learning we had: the biggest lever we found in improving the agent's score is actually the context it has. When Jacob handed me the dataset, he also gave me a 450-line manual with context — basically what I'd define as a field guide, with terms and cutoff rules and things like that. I ran this eval one time with the agent not having the manual, and one time where it did. In the first run, without the manual, it got a 50% accuracy score. The second time, with the manual, it got 73%. So that's a very big upswing, and it proves that giving the agent context on things like business terms really helps it perform better.
28:06Jess: Examples of things in the manual: defining that realized revenue uses contract price on a wholesale order, not list price. Or that an active launched product has to be launched, flagged active, and not yet past its discontinuation date. So those are the kinds of things you'd see in the manual — it explains business terms and what they mean in the specific terminology of what you'd find in the database itself.
28:44Jess: So yeah, this was a big learning: just providing the agent more context really helps it perform a lot better on these evals. And I think that's it. To summarize: we built an agent that uses the MotherDuck MCP server to query your data using natural language and get answers back, because it wrote SQL for you. Then we used Braintrust to measure how good those answers were. By looking through the traces and the scores, we found a lot of really interesting things — for example, understanding the impact context has, as well as the breakdown of the types of questions the agent struggles on, like marketing questions versus finance questions. Hopefully that helps you understand what Braintrust is, what MotherDuck is, and how to run an eval and look at your traces. I think we can open it up for questions now — I believe Liv and Jacob have been in the Q&A section probably answering a bunch of things already.
29:55Jacob: Yeah, we have some really good questions actually.
29:58Jess: You want to just read them out?
29:58Jacob: Yeah, sure. I'm just going to go through the chat first and then we can jump into Q&A.
30:12Jacob: "Did the model solely rely on schema for context, or did you provide additional metadata?" Yes — and Jess had that in the slide showing the difference between no context and with context.
30:27Jacob: There's a related question. From Kendra: "I'm not asking this very well, but can it learn as you run multiple evals?" Jess, do you want to start with that — what people have built like that — or do you want me to take it?
30:50Jess: Yeah, I think what Kendra might be talking about is what we like to call self-improving agents. For people who haven't heard the term: if you have a system that can measure how well an agent performs, you can extrapolate that to automation, such that an agent can say "hey, I'm scoring 50%," look at its own traces, understand why, improve itself, run again, and improve itself again.
31:22Jess: Conceptually, I think we're hopefully converging on that as the technology gets better. I'd say Braintrust — we're still trying to figure out product-wise if that's where we want to go, it's not certain right now. We still think there needs to be a human in the loop to look at the traces and understand things. I myself have tried to automate that process, and the technology just isn't good enough for it. When you have an agent try to understand why its traces are not good, it doesn't always get to the root cause. That's just what I found — I don't think AI is good enough to diagnose its own traces yet.
32:09Jacob: Yeah, I've spent a lot of time specifically in the text-to-SQL domain, or whatever we want to call this natural language querying. The most important thing is looking at the traces — that's sort of unavoidable. That said, I've found that human-in-the-loop, using whatever agent you're using — whether it's Claude Code or Codex or whatever — and just having it consume the traces and turn that into improved context, really works. We didn't do any of that loop here in this experiment, but I suspect that with four or five passes on that loop we could probably get this to a very high percentage of accuracy. I think it's still human-in-the-loop for now.
33:00Jacob: Methods I've personally used internally, and that we use for our own data at MotherDuck, that work really well: literally just what we talked about. Look at the traces, have an agent help you identify common traps. Sometimes, at least for us in the data warehousing space, I'd realize we have an ambiguity that's triggered upstream because we use the same word in multiple different objects to mean different things — so how do we make that more specific? Very challenging to identify, but that's what we ran into.
33:34Jacob: Okay, another question: "So the implication is that for your data zone of a warehouse you'd need to do the synthetic benchmark process, coverage depth gaps up to the human. By extension, do you explicitly include parameterized tests, such as different filters and time ranges?" I'll start with this one and then hand it to you, Jess. Where this points me is: how do you give the right context to your agent? Where the Braintrust–MotherDuck fit is really nice is, well, if you're going to define context in MotherDuck — which you can do — how do you know that it's improving the efficacy of your answers? You need to run evals to do that. What we're trying to figure out — and again, this is a product question for us too — is how far do we guide down that path of: how do you know if your context is good? How do you know if it's bad? How do you know if it's drifting? These things are extremely important to measure over time as we become more and more agentic in the way we interact with our data. Anything you want to add, Jess?
34:46Jess: Yeah, one thing I'd add is that it's not always a synthetic dataset that you'd need to use. Jacob, I believe at MotherDuck you run internal evals —
35:00Jacob: That's right.
35:02Jess: — I'm assuming it's not all synthetic datasets.
35:02Jacob: Oh no, we run a lot. The most important eval is the one on our real warehouse for our real business analytics.
35:09Jess: Yeah. So a lot of the time, if you're trying to do this yourself, you might actually use some subset of production data to run evals. So it's not always synthetic data.
35:22Jacob: Yeah, absolutely.
35:22Jess: For this one, we didn't want to share actual production data with the public, so it was a synthetic one — but luckily Jacob was able to human-vet it for us.
35:36Jacob: Yeah, indeed. Okay, there are two related questions here that I'm going to bundle together. The first: "I know we're talking about evals, but to get a good text-to-SQL result, do we first design some semantic layer?" And: "Is there semantic layer functionality in MotherDuck?"
36:01Jacob: This is a really good question. There's been a lot of conversation around semantic layers and AI, and what we found is that context is more effective than semantic layers for enabling your agent to answer questions. We have some research we can publish on that too. Really it comes down to two things. Agents, or LLMs, have like 50 years of training for SQL — they're so good at writing SQL, they understand it really well. And second, there's no unified spec across semantic layers. They're all over the place. Furthermore, most of the ways they're implemented are behind paywalls and not in the public domain, and that has made it really hard to use them effectively. We found that even when they do get the answer right, they use about three times more tokens than just using SQL, because you have to give them so much context about how to use the semantic layer.
37:04Jacob: That being said, I think semantic layers serve two very important purposes. The first is that it's a contract between your consumers and your data creators — super important that you have that. The second is that when you're building business analytics at scale, you need some place to make one update and cascade it across every place. So don't think about it as being for agents — it's for the kind of annoying things that do matter at scale, which is really governance and update-once. But it's not really helpful for agents, in our experience.
37:47Jacob: All right, this one's for you, Jess: "Can we build this kind of visualization of the results in the Braintrust platform?" If you go back a few slides to your summary tables, like this one — you had a couple of other analysis ones. Just talk about what's possible in Braintrust.
38:11Jess: Yeah, so the answer is yes. However, in my time at Braintrust I've never actually used the native Braintrust UI to build my dashboards. I've used the Braintrust CLI to get the numbers, and I have a skill to build these dashboards or graphs you're seeing, mostly because I write a lot of blogs and articles, so these are our branded ones. It's the same — you can do the same. I just get a little more freedom to tweak the UI and the colors the way I like. But yeah, it's a plug also for the Braintrust CLI and MCP server, because you can grab those numbers from the traces and put them into some sort of graph generator. It's all possible in Braintrust as well — I just like having a little more freedom to tweak some of the parts of the graph.
39:19Jacob: Yeah, plus one to that. The Braintrust CLI is amazing to interact with, especially when you're using an agent.
39:27Jess: I agree.
39:45Jacob: All right, I think this one's interesting, about how you actually run the eval mechanically: "How do you reconcile the different answers — gold truth versus LLM answer? Can you share some techniques? LLM-as-judge, word overlap, semantic similarity, etc. Does Braintrust provide this natively?"
40:07Jess: Yeah, so if I understand the question correctly — for this specific eval we used a very simple judge, a deterministic judge. Because Jacob had an answer: if the correct answer is 42 and the agent also produces 42, we say that matches, it gets 100%. If the agent produces 67, it doesn't match, it gets 0%. So behind the scenes it's just simple Python — if x equals y, it gets this score.
40:46Jess: If we were to run a different eval — if I asked the agent to tell me a story and I were judging it based on how good the story is — we might use something more like LLM-as-a-judge, where we're maybe using GPT-5 to measure how good the story is. We might define certain pillars: the story is good if it's funny, and it has a punchline at the end, and it's less than 300 characters, or something like that. For this specific eval it's just a deterministic score, but we support all sorts of scoring — deterministic, LLM-as-a-judge, and human review scoring in Braintrust. So it supports everything. It just so happens that this specific scoring system was very simple.
41:50Jacob: Yeah, I designed it that way on purpose, to be honest, so that we could build something as verifiable as we can make it. There's already a lot of non-determinism in using LLMs to build all this, so it can be a little hairy.
42:11Jess: Agreed.
42:11Jacob: Okay, good answer. I see some comments from Ken: "Can you define what these two terms mean and the differences?" I'm assuming you mean context versus semantics. I'll let you reply in the chat if that's what you mean, and I can take that up separately — I don't want to get too much into the weeds. Okay, great, I see that.
42:48Jacob: Really quickly: if we think about the difference between semantics and context, especially for text-to-SQL — generally a semantic layer is defined in a programming language, or maybe YAML or JSON, and defines the shape of specific metrics. Usually what we want to use that for is what I said earlier: keeping it as a contract between consumers and producers, and being able to update one place to update all of our dashboards or whatever's consuming that thing.
43:20Jacob: Context is a broader set of that. Context can include your semantic layer — and in fact that's how we treat it at MotherDuck internally. The way we do that is we use a feature in MotherDuck called guides, which lets you put markdown into MotherDuck, into your database, and then query it with the MCP. We didn't use it for running this eval, because when we started that feature was not live yet. It is now live. You can think about that as the superset of all the things you might need to give to your agent. It can include things about your dataset, things about third-party datasets, instructions on who to send a message to if you see an error. We're seeing customers use it for all sorts of things. Really it's just a way to put information with the agent when you're asking questions. And then obviously the way you improve it is with evals.
44:12Jacob: Speaking of which, this is an interesting question that I don't know the answer to, so I'm going to give it to you, Jess: "What does Braintrust think about in terms of zero-knowledge proofs and self-verifiability algorithms for privacy and sensitive-data evals?"
44:31Jess: What does Braintrust think about zero-knowledge proofs, self-verifiability algorithms for — I don't know the answer to that question.
44:42Jacob: Okay, sorry, I didn't mean to give you a trick.
44:42Jess: No, no, no, this is good. We should have an answer to this question. Okay, we'll follow up in an email on that one, I think.
44:53Jacob: Okay. "Blog post idea" is the comment in the chat. Let me see if there's anything else in the chat we need to talk about. Okay: "Do you use both online and offline evals? Any monitoring beyond just having someone look at the traces?"
45:09Jacob: This is such a good question. My initial thinking as it relates specifically to the MotherDuck domain is definitely: we start with offline and then we figure out a way to build online into it. Unfortunately, right now it's extremely agentic and soft from my perspective. I don't have an answer where it's like "oh yeah, I do this a specific way." It's more like we sample the queries, we get feedback — human feedback is always the most helpful. Whenever the CEO puts a comment in the data team channel and says "hey, this number looks wrong," that is extremely important, right? That's one version of an online eval. But yeah, I don't have a great answer. I don't know, Jess, if you have a better answer, especially around evals for systems that are not just text-to-SQL.
46:04Jess: Yeah. What we ran in this specific example is an offline eval, because none of the data is connected to production at all. An example of how it'd be connected — maybe not for this eval, but let's go back to my previous example of asking an agent to tell me a story. I might run an offline eval where I generate a bunch of stories and perfect my scoring system for judging how good the story is.
46:37Jess: So let's say I run my scoring system on a first pass and it gives all these stories a certain score, and I look through them and don't agree with how my agent is scoring things — I think this story is better than it says, or worse. So offline evals are a way to calibrate your scoring system to what you as a human would actually score something. This might take a couple of iterations, where you go through and say "I need to tweak this scoring system to the way I want it to be." Usually at a company this is the PM doing it.
47:14Jess: Then when you're happy with that scoring system, that's when you bring it online. When you start to get your production logs in, you set your scoring system on that. What we mean by online is: as soon as the production log comes in, it immediately gets scored, live in real time. So if you're a PM looking at all your production traces and logs coming in, all of them will have some associated score. And because you ran your offline evals, you trust that score — you agree with it, because you've tuned it to be what you think it should be.
47:50Jess: So as Jacob says, you run your offline evals first and then tie it back to your online evals, because you've tweaked your scoring system to a point that you trust it. Then at that point you'd probably look at whatever aggregate score you're getting and say, if it's too low, alert, or you need to make some sort of code change. So that's how the flow would go between offline and online.
48:20Jacob: Indeed, makes a ton of sense. Let's see, is there anything else in terms of questions? Like one or two more and then wrap up.
48:37Jacob: I don't see anything we've missed from a theme perspective. I do see a question here from an attendee about model or data-model drift detection using different divergence metrics. My experience running evals on this — and Jess, tell me if I'm wrong — is that I don't really think about traditional divergence metrics when I'm thinking about evals. I don't know what your perspective is and how you measure this.
49:01Jess: I think this would be really interesting. We have run evals before on multi-turn conversations — for example, very long multi-turn Claude Code conversations — and tried to look for when drift happens. I have never heard of the specific divergence metrics being mentioned, but I think it's totally doable in Braintrust. It's something you'd probably pass in as a spec to the LLM-as-a-judge to look for as you go through the multi-turn conversation. So that's something you'd pass in as context to your scoring system, so it can judge for drift based on whatever framework you're trying to use.
49:51Jacob: Yeah, I think that's right. Yes, there is risk of drift in these models, of course, especially when we talk about text-to-SQL and natural language querying. The thing I really think about that's slightly different is: yes, you need some way to tune the loop, which we didn't really talk about today. What does it look like once you run the eval — what do you do about it? The next thing is you figure out a way to improve it.
50:24Jess: Yeah, exactly. This is the right slide. And what that impro— sorry, go ahead.
50:32Jacob: Oh, sorry, go for it.
50:37Jess: No, I was going to say, obviously we're leaving this "improve" step as an exercise for the reader at this moment. But that's the important thing: why run an eval in the first place? Well, it's so you can do that — so you can get more accurate. It's really dependent on whatever the root cause is. Sometimes when I run evals I run into issues where there's no retry or backoff logic implemented, so you go into the code and implement that.
51:07Jacob: Right, sure.
51:07Jess: It really just depends on whatever the issue is. In our case it's: okay, we find that we get a low score because the agent doesn't have context, so you build some sort of manifesto and give it some context. It really depends what the issue is, and that's very dependent on your system.
51:22Jacob: Right, and on the MotherDuck side, we've recently built an affordance called guides, which is a place for you to put context so you can start running these types of loops effectively. Obviously, inside of a database it's fairly trivial to repeat these types of things, so the scope is constrained enough that we have a very tractable problem set. It's great fodder for using an AI agent to improve the surface area of the effectiveness of your database, which is really cool.
51:52Jacob: Do we have any other questions? I don't see anything else, Jess — I think we're okay.
51:57Jess: Yeah. Thank you everyone for joining us in this webinar. Thank you, Jacob, for collaborating with us on this — this was awesome. I had a lot of fun running this eval, a lot of cool learnings. We should be reaching out to you afterwards with a recording, and feel free to reach out to us with questions. Thank you for joining us and giving us your time today.
52:17Jacob: Thanks everybody, we really appreciate it. All right, see you. Bye.
