0:00Welcome, everybody. Thank you so much for joining us. Good morning, good evening, wherever you are in the world. We're going to get started here in just a minute or two as people filter in. In the meantime, let us know in the chat where you are joining us from. And if you feel so inclined, let us know your favorite DuckDB function or piece of DuckDB SQL syntax. We always love to nerd out on what people find most useful.
0:31My name is Garrett. I'm on the product marketing team here at MotherDuck, and I'm joined by my colleague Bev, who I will let introduce herself as well. I'm a customer engineer here at MotherDuck, so I'm the one who makes sure all your technical questions get answered and your experience with MotherDuck is as smooth as possible.
0:55Very cool. Okay, we've already got California, Arizona, London. I saw some German in the chat as well. Hopefully the Berlin or German summer is as nice as it is here in Seattle. Just in Berlin for We Are Developers, I guess the biggest conference in Europe. Super cool. Folks from all over. I think there were like over 10,000 people. It's pretty crazy.
1:19Oh, shout out Seattle. I'm down the road here in East Lake. Okay, anyways, I think it would be a good time to get started. Again, thanks for joining us. What we're going to do today is we're going to talk about guides, which is our new context layer feature here at MotherDuck.
1:36Something we're very excited about, just knowing how much our community and customers are doing with analytics agents and how important it is to feed them the right context about a business and a database and how to write correct SQL and all of that and how important it is.
1:56So we're going to talk about that today. We're going to spend like 10 minutes or so just kind of setting the stage on what guides is, why it matters, a little bit of slide where, and then we're going to spend most of our time on a demo that Bev has prepared.
2:11If you have questions, please pop them in the chat. We'll take them as we go. If you haven't joined us for one of these live streams before, we like to keep things pretty organic. So just pop your questions in there and we will get to them throughout.
2:30Cool. So why don't I go ahead and share my screen here and we will get going.
2:38All right. I think you're seeing that probably right, Bev. Yeah, I see it. Okay, sweet. Then I'll hand it over to you. And please, you know, I'll just be your slide monkey. Let me know when to click forward.
2:58So guys, why did we, why did we develop guides? This is the hot topic nowadays, this context layer. I've been around in the industry for quite some time and I have watched this intractable problem of defining your data and explaining your data and making your data understandable to people who are not familiar with it.
3:28Who are not particularly data engineers and watching it fail miserably for like the last 30, 35 years. Now that agents are involved, it's become so much more important because now you have non-humans looking at your data and trying to make sense of it.
3:47So the mother ducks approach to this very intractable problem is in fact guides and what they kind of are like is like a readme and an onboarding doc to a new hire, except it's stored in the warehouse. It's versioned and it's queryable in SQL.
4:07Storing it in the warehouse allows you to do it and forget it. You don't have to remember to point an agent at a YAML file or a markdown file or a skill. It's just right there in the database as a database object. It is referenced for every natural query that comes through a remote MCP server.
4:30The first thing it's told to do is consult the guide before you start writing any SQL. So now you have a reliable place to put your business knowledge.
4:43So I'll let you flip. Yeah, what is context? That is a very good question. We all know what semantics are. If you've been around as long as I have, you might remember ERDs, entity relationship diagrams. Yeah, those were kind of kludgy.
4:59Semantics have become a better answer to this, but a semantic layer can't define a term, a business term. That's what context is. And if you know how an LLM actually works behind the scenes, LLMs are using something called contextual embedding in every single process they do.
5:23So they take all the words in your natural language query, they turn them into mathematical numbers, and then they go look at a probability that this is what you mean.
5:35Well, if you can be much more specific in your language and much more specific in telling the agent what you mean, that process is much, much more reliable and much, much faster and very many, many less iterations.
5:55So that's what context is. Context for through our MCP is when you put a prompt in and the agent decides it needs help, which is we have to admit most of the time.
6:12The very first thing it's going to do is get query guide and get guide pulls from personal and organizational guides, and then it's going to allow it to respond and act appropriately.
6:27So a guide is a guides can be nested. So you can have that initial guide, an organization guide, and that organization guide can then call other organization guides and can call personal guides.
6:44Hmm. What's a personal guide? We give you a section in that in the guide context that allows you to put your own, your own interpretations, your own requirements, your own needs in that when you come through.
7:04For instance, suppose you always like dark mode. I mean, this is a pretty simplistic example, but if you always like dark mode, your dives will get produced in dark mode. If your organization has a very defined revenue, that'll go in the organization guide section. This is all access primarily through the MCP.
7:26The other thing I really, really like about this model that our approach is that your agent can write back into a guide. So as an agent is inferring, guessing, assuming and exploring, it can write that path back into a guide, which can then bubble up to admins and to people who are paying attention.
7:53And they can take that in those those times when an agent was stumbling a little bit and redefine so it doesn't stumble again. So that's a very powerful part of our guides.
8:10There's already some great questions in the chat that we're going to. I'm not only not putting them up on the screen because we're going to answer them very shortly. Some questions about guide hierarchy and editing guides in a single session that you're that you're already talking about. So I think it's cool to see what everyone's intuition is for for the architecture here. So appreciate that, folks. Let's go next. Yes, absolutely.
8:35OK, so, yeah, what's the result of all this?
8:38Better agent queries, very consistent flights and standardized dives. And if you don't know what a flight or a dive is, check out our Web site. And there's some great concept documents in there that will tell you what a flight and a dive is. But basically, a flight is pipeline creations and dives is your dashboarding visualizations.
9:02But what I'm really going to focus on is better agent queries, because that's really the heart of the matter. Guides fix bad queries. They just, you know, we we really want people to use natural language. But as we know, people who are non-technical at Mother Duck, we call them non-technical ducks and TDs.
9:26And they like our salespeople, you know, that they don't want to know schemas. They don't want to know SQL. They just want to ask, you know, what's going on with my customers?
9:38If a non-technical person comes in with a very broad question, that could end up with a lot of back and forth with the agent, a lot of clarifying questions, a lot of token usage, a lot of time.
9:52And what a guide will do is it will eliminate a lot of that because there will be direction and context and common error patterns and compute resource requirements and everything a non-technical person doesn't want to think about that can all be put into your guide. And it just makes, you know, more context equals better agents.
10:15And here's the proof in the pudding. So this is a wonderful DABSTEP benchmark. And if you don't know what the DAB benchmark is, let's see. I'm trying to. Do you have a good explanation, Garrett, for the DAB benchmark?
10:39Yeah, yeah, yeah. So DABSTEP, you know, the researchers love doing their like benchmark acronyms. It stands for data agent benchmark for multi-step reasoning. And essentially it's like 400 or so pretty hard questions about analytics from a benchmarking data set.
11:02It includes like common human shaped errors like misspelling column names or misdescribing a schema as well as just like kind of standard ambiguity the same way that a business user might ask questions.
11:19So it's a pretty nice test for agents to actually try to parse, like, OK, what does a human mean? How do I write the sequel? And then can I receive the correct result? And it's stratified a little bit. And I would definitely, you know, if you're into agent analytics, I would definitely recommend reading the research paper behind this.
11:38But it's just a great stepping stone for understanding agent performance on pretty like real world shaped analytics questions. And, you know, when you pass one of these questions to an agent without any context, you know, predictably the performance is pretty much scales with model intelligence.
12:00Generally, as you push the frontier of the most recent LLMs, you get better performance. So, you know, Opus 5, Table 5 will outperform previous generations. And that's fairly predictable.
12:14But the big difference in performance when it comes to actually getting correct answers, especially from less expensive, maybe less intelligent models, is providing context. And so, you know, our benchmarking here, like we're able to achieve about almost 100 percent correctness on this benchmark just by providing context about the data to the agent.
12:39And so what we're not doing is like training on tests or kind of like stuffing benchmark answers into the context guides themselves. It's more of just like describing the shape of the data, describing quirks in queries that are uncovered through multi-step sessions in order to achieve that result.
13:00And so this actually was done, this benchmarking was done with Gemini 3. 5 Flash, I believe, which is an older generation of model. It's much, much less expensive and fast, much faster than, you know, the recent Claude series, for example.
13:20And we're still able to do this performance pretty drastically just by giving the agent context about what the data actually means and how to query it. Sorry, did I go off the rails here too much?
13:31No, that's fine. Actually, I gave this diagram to an agent. I'll have it remain nameless. And it basically said that this hard section here, by having the guides in there, this was like a two cent query on a two cent prompt.
13:52It's that efficient. And Garrett's point that you can downgrade your model and still win with the context layer is absolutely correct. And hard queries are defined as those multi-step joins and the business logic questions. And that's where agents stumble the most.
14:13Agent doesn't know your business. If you don't give it the correct context around your business, it is going to stumble. So this is, you're spot on, Garrett. Oh, man, there's so many good questions here. I am going to stop and take one because I'm being nerd sniped a little bit.
14:30Hold on, I'm going to present it here. Ryan asks, wants to learn more about query consistency, says semantic layers were traditionally done with the goal to get consistent results on import, especially externally facing metrics. Does well-modeled data and a guide solve that problem? Would you mind if I take a quick stab at this, Bev? Please do.
14:54Okay. This is such an interesting problem. I won't do like the five or ten minute rabbit hole on semantic layers, but I will say that we found in our benchmarking and working with customers that well-modeled data and guides does solve a lot of these problems.
15:13I want to be clear about one thing, which is that when we are using analytics agents to write SQL for us, LLMs are non-deterministic. And so there is no such thing as the query engine equivalent of an LLM that always returns the same response.
15:34And if you are using different classes of LLMs, that's like doubly true. And so there's always a little bit of non-determinism in our results.
15:44But what we found pretty consistently is that the combination of clearly and well-modeled data, the same way that a colleague would expect using best practices such as clear column names, a medallion architecture, so forth, plus guides to actually provide agents with business context.
16:04And then a solid suite of evals or evaluations for actually understanding, okay, if I upgrade to, you know, Cloud Opus 5 or if I use Gemini Flash 3. 5, how does that change the correctness of the queries being written? Those three practices are pretty dang good for returning accurate results.
16:27So we're never going to be able to control for a business user misinterpreting some data or asking a wrong question or using the term revenue when they meant profit. Like, those are all human problems that I think are going to be continually, you know, something we need to work on and something that organizations need to build on top of.
16:50But just to get to the point, like, those three practices have been really transformational for getting good answers out of analytics agents. And that's not really anything new. Like, these are classic data engineering practices that are doubly important for using agents. Sorry, I did rabbit hole a little bit, but this is such a cool question. Thanks, Ryan.
17:12Yeah, the consistency factor is a huge one. And in fact, I've done some a little bit of testing with guides to find out if, in fact, I switch from agent to agent with a guide. Am I getting the same answers? And I have found that my consistency is in the 90 percent tiles.
17:34I know we have some some slides on some more details about guides and the hierarchy. So I want to get to that really quickly. One more. Hannah asks, how are guides different from skills and Claude? This is so important. If you look at a guide and we'll show an example in a moment, it actually looks quite similar. Agents are very good at parsing markdown files.
17:56That is how Anthropic and OpenAI and the other labs have built agents to access things like memories and skills and session logs. But the key difference is that guides live in the data warehouse next to the data. Yes.
18:14So if you move on from Anthropic and need to migrate all your skills or I don't know how many of us have tried to take a skill and push it down to the rest of the organization while keeping it updated and version and source controlled. Those are all messy, annoying things to do.
18:33And we think that context related to data should just live in the warehouse alongside the rest of your enterprise data. And my other big point on this is you don't have to point your agent at external files. You don't have to remember to reference a skill. It's built into the MCP.
18:55So as we all know, humans are notoriously bad at remembering things. It's just not one of our strong suits. And if you have this accessible to your agent every single time, your consistency, reliability and accuracy are just going to go much higher. Very true. Thank you, Hannah. OK, I think our next slide is the hierarchy.
19:19Yes. Excellent. So for those of you who still remember card catalogs in libraries, you can sort of think of a the hierarchy of a guide as, you know, the Dewey decimal system. Right. You can have it inherit all the way down.
19:39You can have guides that work as topic functions, as folders. You can have ownership at the user, the org level, and you can have query and tool specific guides. So you can fine grain this as much as you want. This is Markdown. So as when I when guides first came out.
20:00Matt, I went up this you could do anything with this. I mean, you can get as specific on a term as you want. It's entirely up to you.
20:14The agent is going to call the first org. I don't know if there's a Yeah, the first org guide the that will kick everything off. And then as it is parsing that you're the
20:28prompt, it's going to look for other guides that reference anything that's in that prompt. So this is let's see. Yeah. What's the we did our revenue
20:42query snippets, that's user a nested topic of forecasting. For flights and dives, which are the other mother duck products that go along with this, they ride along
20:56automatically, without the agent having to search for them, you will see that the context that you put into your guide will also pertain to your flights and your dives. So it's all in one neat, organized place.
21:13And if you this is a little bit of a just a visualization here, but for folks familiar, which hopefully is everyone looks like a file system, right? We have a top level topic, which is our
21:25revenue billing topic. And then below that, we have an organization level guide. So MRR and ARR definitions, those are shared across the organization and reference the
21:39subscription table. We also have an example guides can be user level as well, say for testing or experimentation, or something unique to how you work with data. So we have, you know, this
21:52revenue query snippets example, and then we have a nested topic within this revenue billing top level topic. And so that will allow an agent to discover guides related to
22:04forecasting for a forecasting model. I think there was one other question in the chat earlier, I didn't promote it, but it was basically asking about using the column and table level
22:18comments for adding context. We found that this file system model actually works a lot better for agent search and retrieval, there's just less for the agent to stuff into its own context window. And having that
22:34hierarchy enables it to traverse the top level topic, find the right guide, and then load that guide into its own context to write a query.
22:46That's actually a very good point. There is a context limit for an agent. So being able to slice and dice this up into smaller chunks makes it much more efficient. And you really can't stuff
23:03everything you need to say about your, your business in column annotations. Are column annotations still useful? Absolutely. I mean, you know, they're highly encouraged, but they can't do the whole job. So having those in conjunction with your guide just gives you that much more power.
23:23Cool. So the one thing to this is an example of a guide, which is to give you an idea of what it looks like. It's Markdown, it's plain Markdown, it's human and agent readable. It is easy to edit, it can be ported from system to system. And if you put, if you make that that effort to really get your business logic and your business decision
23:52terminology, and every single company has its own little acronyms, and its own little internal buzzwords. You know, when I started at Mother Duck, they kept talking about ducklings. And I tried to pretend I knew what that meant. You know, so having those kinds of definitions in your guide as plain English, just makes everybody's life so much easier. And all of those, all of that, you know, as you can see, there's a lot of
24:21input, you know, as you data engineers and data scientists are doing your work, you are coming up with definitions for things. And being able to put that in a central place that everybody can read and plain English, that's gold.
24:35Cool, I'm just scanning the questions real quick. There's so many. I wish we could do we should do a whole follow up session on like the practice of enterprise knowledge management. Because there's so much history here and so many things that folks are experimenting with.
24:55That's actually a really good point. Because like I said, I've been in the industry for a long time. And I've watched all the attempts at this in the last 35 years. And it hasn't been pretty. You know, it's been a very complicated topic, topic, and it's a bit intractable. And what I'm always interested in is how everybody else is approaching this problem. I would love to see everybody get together and say, here's what we're doing. Here's what we're doing, kind of coalesce that information.
25:25Hey, if all of that was easy as plug and play, we wouldn't have jobs. Do you mind if I take this one? Actually, this is something I love about, about guides and also flights and dives are other kind of agent first products at Mother Duck, which is that you can manage them as code.
25:45I'm going to skim through this one a little bit, just because I know we have we also have some demo content to get to. But I would encourage folks to check out our blueprints repo down at the link here at the bottom. Gerald, if you're still on, maybe you can chuck that into the chat. Appreciate that.
26:02Where some folks on the team have put together a really nicely structured way of managing these assets as code. And since guides are just marked down, we can do that same thing. And I just want to walk through this pipeline for doing that.
26:20So in a GitHub repo, we can manage our canonical source of truth guide, as well as the blueprints repo has an example YAML file just for configuring CI and GitHub actions. Say I want to make a change to a guide, change the definition, add some context for the agent. I can open a PR on that guide, which will automatically deploy on a branch.
26:49And you'll notice that we have these SQL functions such as MD create guide and the update guide. This is another interface for working with guides as well as our other agent native tools.
27:03Of course, the MCP is the best way to do that. However, you can also connect to my other doc with any compatible client, such as the duck db CLI and use a SQL function to make that update manually or encode directly from your local machine.
27:22So if I open a PR on that change, it will deploy a branch scoped change on my user. So essentially, I get a private guide in my mother duck account that reflects the change that I'm that I want to make.
27:43And so I can test and query and run some evals on that local or sorry on that user scoped guide before I push to my organization so that I can be sure that the agent performance is adequate, that the change I've made actually helps with the queries being written, for example.
28:03And then once I merge the PR, assuming everything looks good, the pipeline calls that update guide function and pushes it into an organization level guide in the catalog. And so it's just that, you know, it relies on a GitHub action to make those changes, but it's just a nice flow for managing guides as code.
28:24And again, this is like something that one of the engineers on our team built as part of creating dives, which is our data visualization tool. And we've carried that forward through our other features as well. So if you're familiar with if you want to manage all of your assets as code, this is a nice way to do it.
28:46Okay, thank you for letting me do that. I love this topic. Anything else you want to get to that before we demo? I'm just going to take a quick look at the chat. Oh, man, there's so many good.
29:10This is a pretty hot topic. Yeah. Okay, there's one that says, who manages topic definition collisions? I think this is like, just like BI tools say, right, like this, you know, having organization and user level guides doesn't like abstract away the problem of an organization agreeing on definitions.
29:33These are human problems that the agents will not proactively solve for us. You can certainly like YOLO into the void and have your agent, you know, argue about whether finance or product has the right definition of MRR. But, you know, this is on us as data engineers or as owners of this business context area to actually maintain.
29:57Nothing has ever presented prevented like a business user from firing up Excel and doing wrong analysis and presenting it to leadership. And the same way like we don't have, we can't abstract away those guardrails to the machines to prevent colliding on definitions.
30:18I will say that the guides as code workflow is intended to maintain centralized ownership of that context.
30:27So I think that the patterns that we've already established in access controls and, you know, other sort of like organization level controls for semantics and for tooling and for code review, like all of those things are super relevant, if not even more relevant. I'm just going to pop this one up because this one is funny.
30:51Heard you like agents, so you got an agent for your agent. Yes, exactly. Some people will go down this rabbit hole. I think that they will be disappointed that at the bottom of it is human behavior. Good old fashioned human behavior. OK, thank you, everyone, for the questions. I'm going to try to get to a few more. But I know that has prepared a cool demo. So I'll turn that over to her.
31:13OK, so what I did is we have a database that we use that has all sorts of interesting statistics, whatever. Would you would you just zoom in once or twice so we get a slightly larger font? Can I get in? Is that better? Maybe one more. You don't mind.
31:38I'm trying to find the zoom. Do you know where it is? You just hit the command plus one or two more levels. Is that helping? I don't think so. Maybe top top right where it says finish update. You have a three dot menu. And you're from what we got.
32:05Let's see over here. No, I don't have anything that that makes it zoom. I'm sorry. I can do a full screen, but that's about it. OK, that's all right. Let's just rock with it. Just give us to help us understand what's happening. Yes.
32:23So I asked a question of this agent that without a guide to go and look at this data and come up with this. Let's see if I go over here. I can probably read it a bit better. I asked it to answer a question and it went off and came back with a lot of follow up questions. You know, that back and forth that you go through.
32:46And at the end of it, it came up with an answer. Can you scroll down on that, Garrett? Or do I scroll down on that? Go ahead. Controlling the screen. Yes. It's all me. OK. And what I loved at the very end, this was all done without a guide, is this phrase right here.
33:08Since this metric doesn't exist as a defined term anywhere in the account, want me to save this definition as a mother duck guide? So it's consistent. I'm like, what a great idea. Also, he was wrong. This is not correct. This is not the correct answer.
33:26So we did create a mother duck guide for this, and I'm going to feed this back to Claude. I'm just going to pick up the whole thing and feed it to him and say, this is what I want you to use. And I want you to answer the question after I've given it to you.
33:50So I'm going to say, let's see if I can just, I don't know if I can pick up the whole thing. Let's download it. And I'll just say while you're doing this that I noticed you're using Google Docs to review all the content of the guide yourself. That's a great intuition because you're actually a human getting eyes on this thing.
34:11I'll say that as of I believe this week's release, which should be happening in the next few hours, you'll be able to review all that content inside mother duck as well. So just more things are coming. Just a little bit of a stopgap while we release the ability to edit and view guides inside the product, too.
34:34So I'm going to tell it to use this markdown file. I can't type file and create a guide in mother duck. And off he goes. I'm happy to happen to use Claude, but of course, we're agent agnostic.
34:56So basically, your workflow here was how did how did you get that that draft of the guide that you just paste it in to generate that for you? Did you write it? No, actually, Claude did generate it. What we did internally is we were running this as a demo and we realized, OK, yeah, those numbers aren't right.
35:17So I went back to Claude and said, can you produce a markup of a guide? And then went through and double checked it, you know, as anybody should and modified it and put it in a in a doc X file that I could access. So your workflow was you were doing some exploratory analysis. You realize, you know, I'm not going to be able to do that.
35:41Because you're an expert on the data. And so you said, hey, Claude, like, that's not right. You should query in this other way. Can you draft a guide that specifies that context is basically, basically. So, OK, it's very it's very it's always funny to me how similar this is to like how you would work with a new colleague. Yes.
36:00You know, you'd be adding to the documentation, for example, of how to use the query pattern as that person's learning. Well, in fact, that is something that I have heard from customer feedback. I have heard people say, oh, it's like having another member of your team. The guide is having just somebody who's been in the company for a really long time. And they're like, oh, I'm not going to be able to do this. I'm not going to be able to do this.
36:23I'm not going to be able to do this. Yeah. They're all interns. I love when people call the agents interns. That's that's great. It's true. And every every person who is new to a company, which an agent would be, has to be, you know, engaged in the process of creating a guide. And so, you know, I think that's a really important thing to keep in mind. It's true.
36:46And every every person who is new to a company, which an agent would be, has to be incorporated into the company culture, the company definitions, the company slang, you know, all of that has to happen. And a guide is a great place to do it. For sure. So this is going to take it a couple minutes.
37:08And what's fascinating is when it's done and I'm on Sonic five, folks, I am not. I am not using Fable. I'm not using something that's super expensive. If I upped it, it'd probably be a bit faster. But I can do this on a less expensive model, which is becoming a factor.
37:28You know, you're starting to hear quite a bit about people talking about token usage and how to monitor token usage and how to streamline token usage because it's going to happen. Well, hopefully Claude isn't having an incident. I know their reliability is a little shaky. Well, while that is working, yes. Well, that's chewing. There was one question earlier I wanted to get to.
37:51So Hale asks, Do you eval or measure performance on live queries on a mother dog? Is there an out of the box way for me to do so for my organization's queries? Such a good question. And evals are becoming so important in understanding LLM performance or agent performance on tasks like Dev stuff that we talked about earlier. Today, there's not an out of the box way to eval and mother dog.
38:15It's a really interesting area of product exploration for us. I will say that just yesterday we had a live stream with our partners at Brain Trust, who are building a whole platform for agent evals, where we showed how to build your own eval for analytics agents.
38:36And so definitely check that out on the videos page of our site. I hope that we'll have more tangible recommendations there very soon in our documentation as well. But it's something that's super important and rapidly evolving. Such a good question. So I'm going to show what Claude came back with.
38:58And what he actually told me is this is being saved as a personal guide because I don't have the ability. I don't have the permissions to make it an org wide guide. And what it says right here, if you want it visible to your home team, someone with the right permissions can call site guide access on that you ID to flip it to organization. But I it's relevant for me.
39:22So now when I ask this question, it's still going to use that guide. So if I go back and ask Claude again to it is now going to read that guide for me. So you want to take another question, Garrett, while this is chewing? Yeah, there was one.
39:41Gerald, if you wouldn't mind finding I don't know if the brain trust evals video is published yet, but if we can't if it's not published yet, Pat, we will get it to you somehow or another, I promise.
40:00Oh, hey, Alexi, thanks for joining. Yeah. Alexi asks, so a guide is basically a package with Markdown content and a SQL query. Can there be multiple definitions in one guide or do you want one per guide?
40:14Just on semantics there, really quick, the SQL query that we showed is just written in plain text in Markdown. So it's not a SQL file that's referenced in the package. It's literally just text in the Markdown that the agent is parsing.
40:30Can there be multiple definitions in one guide or do you want one per guide? Such a good question. There are so many definitions or dimensions to how an agent performs retrieval and sort of negotiates how to write a SQL query.
40:49I think the way that we have been describing best practices for guides is very similar to best practices for data, data modeling. Don't repeat yourself. Right. Everyone. Nothing that I mean, these are all old concepts, very old concepts. Like, don't repeat yourself. Have separation of concerns as much as possible.
41:12Avoid like stacking definitions onto each other or just piling context on every time that there is a new question that isn't quite answered.
41:22So I hate to come back to this idea that like there's nothing new under the sun, but really it's the practice of good data modeling and managing context and pruning things that are not relevant any longer. You do definitely want to separate those things as best as possible.
41:45And I think that's going to be, you know, as we enter this, like this world of agents as primary analytics consumers, that's going to be the thing for us to solve as an industry is how not to bloat all this context up every time we want to answer a new question. Just one more call on this one more table. Right. Like that'll fix it.
42:09I'm also going to show this is our mother duck UI. So I was going to show you that your guides sit actually right here in the UI. So there's the one I just created. Oh, if you click into that, then there it is in plain English. So you can see exactly what it's referencing.
42:29Let's see if he's back. He's back. And now the number is correct. That was the number I was looking for. So what the guy changed versus my first pass gave me an understanding. He actually it shows that he read that. I'm sorry. I was referred to Claude as he.
42:48It read the requirements in that Markdown file and knew exactly what to do to get the correct answer. And so you what you change. And that was kind of a long guide to try to read. But what tell us one more time what you what the guide actually changed.
43:09So on shipped orders, I had told it you need to exclude this piece. This piece canceled orders don't make sense when you're looking at, you know, total revenue cost of goods sold reversal for restocked returns. Yes, this is exactly what I told him.
43:28I told Claude to do exactly this unit cost times accepted quantity for returns and using the receive date as the realization point. So Claude picked up all of these things that Claude just didn't have that knowledge before we had.
43:43There was no definition of cost of goods sold given to that agent. So the agent has to go to its reference, its training and guess. Well, they use the word infer, but infer is in fact guessing. So by giving it an actual definition for what I want, then I got the correct answers.
44:08There's one other question here. I'm sorry, I'm just perusing chat. So this is an easy one. I would give myself a softball. One ask, what's the best practice to keep query and flight guide separate? Would that simply be separate folders?
44:31We've actually taken care of this for you with reserved topics. So we reserve topics or folders basically for flights slash and dives slash. So for example, flights being our Python runtime for data pipelines and data enrichment, you can keep all of your guides related to creating pipelines in that reserved topic.
44:58So if you have a poorly documented API that you are often writing pipelines against or if you want to specify back off and retry logic, things like that, you can do all of that under the reserved topic for flights and the agent will not conflict that with any other standard query behavior.
45:22Because, you know, we definitely want to keep that Python agent world separate from SQL agent world. Yeah, I think we have time for a few more and then we will let folks go. Are you are you much of a DBT expert of. I have the rough edges for it, but let's give it a whirl.
45:44Are you are you much of a DBT expert of. I have the rough edges for it, but let's give it a whirl. OK, we will tell you if we don't know, Joji, thank you for this question. Joji asks if we use DBT, how is this guy different from defining each metric at the column level or documenting the metric definitions in the description field in a YAML file?
46:09DBT is extremely good at this at the semantic layer. And yes, my suggestion would always be to take that DBT information and include it in your guide because it is the definition of what the metrics are supposed to be. But it doesn't give the full story.
46:33It doesn't DBT doesn't allow you to put in the anomalies and the ambiguities. It's much it's much stricter than that, whereas guides allow you to do that. But they should definitely be working hand in hand. No question about it. Yeah. Huge fans of DBT here at Mother Dog.
46:55I would say that you alluded to something important, which is like the expressiveness of Markdown versus YAML when it comes to an agent. For defining like those specific deterministic definition metric definitions in a YAML file like that is the right tool for the job.
47:15But if you have a table, for instance, that is messy, that reflects like a mid-year change in your payment processor or, you know, something else that like is hard to define in in YAML and you still need it to be queryable by an agent. Guides is a great fit for that.
47:35Yeah. Same definition, multiple places like we'll be doing this for 100 years. Right. Like I said, it's been an intractable problem for many decades, but I think that YAML files and Markdown files can work hand in hand. Yeah. And I think they're both necessary. We're mostly talking about the context layer, but DBT really does handle the semantic layer quite well.
47:57Yeah, exactly. And all the stuff of downstream systems. Right. Like that is a DBT. That is a DBT shape problem. Absolutely. So I appreciate that question, Georgie. Thank you. All right. I'm going to search for one more, maybe, and then we will let folks go. Also, I really appreciate the engagement from everyone in the chat.
48:20Sorry if we can't get to everything. Some of these are so deep. I would definitely ask if this is something you're interested in checking and talking about in more detail, pop on over to the MotherDoc Slack community. It's totally free to join, of course, and you will find some other data and context enthusiasts.
48:43Hannah asked a question earlier about RLHF, and I'm trying to find the reference that they are referring to, which I want to say, sorry, Hannah, is actually this one. Does this mean you can edit guides in the session? Excuse me. And apologies if I'm butchering the intent of your question.
49:03I'm certainly not a RLHF expert, but what I think you might be alluding to is kind of like the loop of learning and editing.
49:16And you can imagine a system where an agent is exploring data, writing queries, running a suite of evals, and then editing its own guides in order to improve those evals. That is not something that we have built out of the box, which is certainly possible.
49:36There is so much happening kind of at the forefront of, like, continual learning and, you know, task-based self-evaluation. And we would love to see what you build there.
49:48I think I could totally imagine a world in which a frontier model is grading and evaluating a suite of sub-agents on some of these tasks and can also could potentially improve its own guides and context just by running many, many queries and asking many questions across an eval set. So such an interesting topic.
50:11Yeah, I would definitely, like, welcome folks to share what they're thinking about there in the Slack channel. I know some people internally have begun playing with loop engineering, if you will. I think that's the hot term for things like this. But just to be very specific, you absolutely can edit your guides in a session as an agent.
50:34One thing that I do now as a standard practice when I'm working with any of my agents is I tell them I have a standard phrase that now says, any inference, assumption, guess, please document and put in your personal section in the guide so I can look at them, bubble them up to the org. And as the next ones come through, do the same thing.
50:58And I'm finding it's becoming more and more less and less things they have to infer, guess or assume. But is that an automatic iteration? It is not. So it still requires a human to bubble it up. I'm glad these things still require humans, personally. Okay.
51:22I think we'll end it there. Thank you again, everyone, for the engagement and for joining. Jump on over into the Slack and let's continue the discussion. Thank you, Bev, for coming on as a guest and sharing all of your wisdom and your demo. Really appreciate it. Thank you for having me. All right. Until next time. See you all.
