1:07Jacob Matson, the duck father. Welcome to the super data brothers. Thank you ryan I'm so pumped to be here and to chat with you. Uh, looks like you're saying hello from beautiful Orlando. That's right. Yeah. So, I just wrapped up the Gartner Data Analytics Summit down here in Orlando. We have a lot to talk about what came out of that. Uh but um yeah, wrapped up yesterday and now I'm chilling here for a couple days before I head back to Detroit enjoying the warm weather. Yeah, so it's great to have you on the show, man. I've wanted to have you on for a while to talk about lot of stuff in particular what's is going on with DuckDB and MotherDuck and what you guys are doing with LLMs both as far as your product is concerned
1:50and also your personal research into LLMs and how they work in the analytic space. So we're going to dive deep into all that stuff as I survive the windstorm here. But before we do that, I want you to give everybody a little bit about your background. Oh, sure. Yeah. You know, I definitely come to data in a very earnest way, I would say.
2:15So I actually started my career working in accounting. I have a business degree and a computer science degree. And, you know, I had like this delusion that like, oh, if I like work on the business side, I won't be sitting in front of a computer all day. Incorrect. Extremely
2:37incorrect. But yeah, so, you know, I worked in accounting and I kind of, you know, worked on a lot of the things that are like classic accounting problems, all the things that, you know, people know and climb the ladder. And, you know, I got to the point where I was like, you know, my career
2:55goal was like, I want to be a CFO by the time that I was 30. And I finally got into those rooms and I was like, this sucks. Like, whoops. Like, this is like not, I don't really care about this. I really just cared about like, you know, climbing the ladder. And so, you know, what was true though
3:18about my career was that I got really good at using larger and larger calculators. Okay. So that started off with like Excel, you know, you know, 65K rows, right? And then expanded, you know, Excel got bigger and you can now do a million rows in Excel. And then, and then it was like, okay, Excel's not fast enough. I need to learn SQL. We were, we were Microsoft shop. So
3:43it's like SQL server and Power BI and Tableau at the time, Tableau desktop. And so I learned those things. And then eventually, you know, I got my data got too big for SQL server. And I said, okay, I need to do something else. And that's when I kind of discovered DuckDB probably in 2021, maybe
4:022022, kind of in early days, maybe 0. 4 DuckDB, something like that. And it was like, it was like a light bulb went off. I was like, this is the thing I want to work on. Like, this is the piece of software that I have always wanted to always wanted in my career. It took things that analysis
4:23and data that I was doing that was taking me hours in Excel was practically immediate. And this is even before LLMs in DuckDB. And, you know, so I was super, super pumped when I saw what Jordan and Tino were up to with MotherDuck. You know, I was always staying close to them. And eventually the opportunity came up for me to kind of join MotherDuck. I was actually the VP of finance and
4:47operations for a Series A stage startup before I joined MotherDuck on the DevRel side. So they took a little bit of a risk on me. And hopefully it's been paying off so far. But it was super, super fun to be working now, kind of on the pure tech side, as someone who brings operator experience, right? So like, I'm always like grounding us and like, you know, what does this actually mean for
5:10people to use? Yeah, like, how does it help people like me who like really care about, you know, solving a problem? Like, you know, it's really cool that it's a vectorized database, and it runs locally on my laptop, but like, why should I care about those features? Right. And so I kind of bring that perspective, you know, to the team. And I think so far, it's been it's been awesome to be a part of it.
5:32Yeah, well, listen, I'm a huge fan of you guys. And the product, I use it personally to run Super Data Brothers. So like the Super Data Brothers business back end is MotherDuck and Claude. Like that's, that's basically it. Yeah, really, it's, it's ridiculous. And we'll get into that. Now, listen, folks, if this is your first time attending a Super Data Brothers show, this is not a
5:53prerecorded interview between Jacob and I, this is live. I am sitting, I'm sitting on the balcony at at the Ritz Carlton in Orlando right now. And that means that if you ask a question, there's something you're dying to know. I will read it live. I will ask Jacob your questions. The more you talk in the chat, the more fun it is for us and more fun it is for you. Our regulars, like a lot of
6:18edgy know that he says, Hi, Ryan. Nice hat, Jacob. Welcome back to the show. This is one of our every week viewers. We love you guys for showing up every week and always bringing the heat to the comments section. So, so keep that rolling here. And, and before we move on, of course, I do have to thank today's sponsor, GoodData. Now two things I want you to check out from GoodData. These are
6:41both papers by Donald Farmer. Donald Farmer is one of the OGs in our industry. He did some of the very first research on in database, data mining and analytics at Microsoft. Then he was head of product at Qlik. Now he's an independent consultant, super sharp guy. I love talking to him. I talk to him every chance I get. These two papers I want you to read. One covers analytics as code. This is about
7:04how in the modern era, especially era of LLMs, your analytics tool needs to have good APIs, SDKs, all that stuff, or else it's just not going to get the job done. The second paper is about the analytics lake. This is an emerging architecture that GoodData is building, which kind of combines your context, your compute and your analytics in a single layer that makes it very easy for AI
7:29applications and agents to work with. So go ahead and check both of those out. And thank you to GoodData for sponsoring the show. So that kind of gets us to, let me change the layout here. There we go. That kind of gets us to one of the things I really want to talk to you about, which is this.
7:48Coming out of Gartner, every presentation was about context and semantics. It didn't matter what the title of the presentation was. You could walk into a presentation that was called, I don't know, How to Build Better Charts, and all they would talk about was context and semantics. The big takeaway from Gartner and from a lot of the people there was if you don't have a semantic
8:13layer and a context layer and all this complicated knowledge graph ontology, all this stuff, your agents are going to fail. It's all going to fall on its face. It's not going to work. Now, at MotherDuck, and you in particular with the research you've done, you kind of have a very different take. And it's not just a vibe, like you've done some actual research on this.
8:34So why don't you tell us about your findings when it comes to whether or not you need all this context and semantics stuff to get value out of LLMs and agents? This is such a good question. This is where I'm so happy that I bring a little bit of a different context than everyone else, because I did start my career in accounting. We've been
8:55measuring our businesses since the 1400s, since the Medici's introduced double entry accounting, with two tables, a ledger and a sub ledger. We don't need a lot of tables to understand the complexity of our business. Obviously, we've added a bunch of other stuff. And there's a reason
9:19why databases and various systems are shaped the way that they are. I always like to think about this notion of durable primitives. What things are going to survive a long time? We have
9:39spreadsheets from 5,000 BC. From antiquity, there's some guy logging a complaint about a copper meter or something. Like a cuneiform tablet. Certain things just survive. They've
10:02been battle tested. My perspective as someone who's taking a very simple approach to this stuff, which is like, hey, I can model an entire business that's doing billion dollars of revenue in a spreadsheet. Why do I need 10,000 tables to do that? I've been taking a very kind of bottoms
10:23up simple approach to it and seeing awesome results. One of the things that I found out when I was looking at this bird bench data set. Yeah, talk about that. It's a really awesome data set. It's one of the first Text-to-SQL benchmarks that came out. I was working on just a small set,
10:47500 questions across, I think, 11 data sets called mini dev. The full data set is 14,000 questions across like 50 data sets or something. That's huge. You can run a huge one. I was just running on a small one just to kind of test to see if this thesis was right. My thesis was like, hey, if you have simple data models, you can get really good answers from Text-to-SQL. What I found
11:11out was that basically, if you can define your question in a way where the words clearly map to the data model, LLMs are extremely good at answering the questions, like 95%. Then the things that they get wrong are just like, hey, you need to define maybe when your fiscal year starts. Actually, they're pretty bad at dates. There's a whole bunch of date handling stuff you
11:35can add to make it work better. When I say a whole bunch, I just mean a few prompts or like, hey, maybe when someone asks you what date it is, run the SQL command, select current date, instead of asking your memory or whatever because they'd never know what date it is.
11:51No. I saw someone say they told their OpenClaw agent to cancel their flight today because they needed to go somewhere. It canceled their return flight instead of their flight there. Anyways,
12:10just didn't know where they were. To tie that out, I found that simple models are durable primitives and LLMs behave a lot like humans. If you build a simple model, you can get really,
12:29really good results. That being said, the state that most people are in, which I think is why you're probably hearing that on the Gartner side, is they have an SAP database and they copy that into data mix. They have 100,000 tables. Which one do you go to? What are the joins? All this
12:49stuff. These are really hard questions to answer. It's hard for a human too. You need the SAP manual, which I'm sure is like 5,000 pages next to you as you're writing your queries. Yeah. It's a German company. The manual is at least 5,000 pages.
13:06Yeah, exactly. It's so funny. Very valid reasons the system is built that way. And so my perspective is basically like, hey, maybe we . . . Door one is you take the really complex data model and you add all this context and you add a semantic layer and then maybe you
13:30will get good answers out of it with AI. Option two is you just simplify everything and you build the simplest model that you can and you make it easy for humans to use. And by doing that, you also make it really easy for LLMs to use and you can skip all that other stuff. And that's kind of what we're working on. That being said, the feedback we're hearing
13:49repeatedly from customers is like, hey, Jacob, great theory. We have 10,000 tables. Right. And so we're, of course, working on some things to make that easier too. One of the things that you uncovered in your research that I thought was fascinating was when you look at these bird benchmarks,
14:09they typically show LLMs passing what? Is it like 55%, 60% or something like that? But what you uncovered was that a lot of these areas where it's graded as failing, that it's actually like, yes, the SQL is not perfect, but the way in which it's failing is not a way in which a human being would ever care about. In other words, it's giving the right
14:33answer. And correct me if I'm interpreting this wrong, but it's giving the right answer. If you ask how many widgets were there, it's saying 10,000 and 10,000 was correct. But the SQL it generated to get the 10,000 had some squirrely stuff that you wouldn't have done as a human being. Okay. That's sort of correct. Okay. Yes, true.
14:56I think that a lot of the data and annotations around it were generated by previous generation LLMs and they generated so much, right? That I think the quality of it wasn't super great. So what I found in my research was that if I gave really good descriptions of what was in the data, like enriching and using comments in an LLM to generate them and clear questions that were
15:21already . . . I didn't modify the questions at all and pass those to a state of the art LLM. In my case, I was mostly using Gemini 3 Flash. It's just so cheap and good that it's a great model. That we were getting like 60% accurate, let's say. And so that part is true. But what I was finding when I was looking deeper was that a bunch of them were actually
15:45wrong in subtle ways. So A, the SQL was wrong and also the answer was wrong. And it just got to the point where I was like, am I losing my mind? How is this possible? So just for more context, the state of the art I think on that benchmark is like 82% for an LLM and this really awesome paper. What's really great about that benchmark
16:07is a bunch of awesome people have contributed open research on it. And it's been really, really helpful in informing my own point of view by reading their papers, which is awesome. And so AT&T actually submitted, I think, the top one as of a month ago when I last checked. And they had a bunch of really great techniques in there. But when I was looking at it,
16:28and the human benchmark is like 90, 92, 93% I think on this. It's like a human, if you give them the same data, they get that percentage. And I was just looking at the gap between them and trying to figure it out why. And what I figured out is there's just a ton of annotation errors. And actually, there's a bunch of papers that have came out that were published on this topic. They're like, hey, there's a bunch of annotation errors. The answers are straight up wrong. And so what happened with the folks who were kind of maxing this benchmark
16:53is they basically added the errors, the annotation errors, just got kind of into the context of their models so that they were able to answer the question in a way that matched the SQL and the answer, but it was not correct from a technical perspective, which is very funny. I guess from a real world perspective, it was incorrect. It was technically
17:17correct. It matched the gold SQL, and the answer was correct because they were training. They basically integrated the errors of annotation into their models. Yeah. So that was an interesting discovery along the way. And I published all my results. It's on GitHub. You can find it on my personal GitHub, mattsonj slash birdbench, bird dash bench,
17:41which has also what I call, kind of in a funny way, like the platinum answers, which are my human plus AI reviewed fixed answers for that data set. And just to give you a sense of the scope, out of 500 questions in the mini dev set, I have 150 platinum answers.
18:00So that was roughly the error rate that I found, so 20% to 30%. So basically the gap between what I was getting, that 60%, and the human was purely annotation errors, which is crazy. So yeah. Okay. So now in your research, I mean, you guys are using, and your contention is really, no LLMs,
18:23if you give it a simple enough model, really are good enough to just write the SQL. So I mean, how are you, like, are you guys applying that at MotherDuck today? Like, is that, you know, internally, are you able to do this? Like, hey, our models are simple enough that we don't have this problem? It's a really good question. So the answer
18:47is yes and no. So our internal data warehouse, I think, is fairly well designed. And so we do use it to ask questions. You can get really good answers back from it. We use Claude internally, for the most part, when interacting with it, although there's people who use Codex and Cursor and a few others via the MotherDuck MCP primarily. You know, the challenge, of course,
19:10is, like, where there are complex joins that you need to really know some domain knowledge about, or time series is still tough. But, like, in general, we use it all the time. One thing that I think, excuse me, is an emerging pattern that we're noticing internally is, so we've got an awesome set of dashboards that we have internally, you know, using a
19:33traditional BI tool. And those are maintained and kind of known as the source of truth for lots of metrics, right? And so what that lets us do is I use that as my baseline so that I can also use SQL to kind of riff on it and say, okay, well, I know that, like, my page sessions yesterday was, you know, this number. So that means that I can actually slice and dice it, you know, and as long as my numbers are kind of footing from a top-level perspective,
19:57I think about it like an accountant, right, which is. . . Yeah. these numbers need to put to each other, right? And so I'm kind of doing that math kind of on a consistent basis to say, okay, I know this is right. I know. And so then I know that I can look at kind of the slice by these different dimensions and say, okay, this, this looks right to me too. And of course I can look at the SQL and I do, right. But, you know, the LLM
20:19generates the first, the first leg for me. And then I, you know, for me, it's, it's a purely as a reviewer stage, right. Which I think is the loop that we're seeing internally and broadly, we're seeing adoption of across the entire org for people who are like, Hey, I have a data need. I wasn't able to get it met before kind of through traditional methods. I can now use the MotherDuck MCP to self-serve. Yeah. And we'll, we'll, we could kind of do a deep dive on the
20:41MotherDuck MCP. I want to come back to it. I use it like every day, to be honest with you. So, I mean, this is not a sponsored episode. Okay. I just, I'm a big fan of MotherDuck and, and the MCP. But I want to zero in a little bit on, you know, coming back to the message at Gartner and we've kind of talked about a couple of facets of it, but, you know, what would your advice be
21:05to someone who's getting all this message about, Oh, you need a semantic layer. You need to go out and you need to acquire this middle layer between the complexity of your analytic sources and any downstream LLM applications. And without it, it's just going to be a disaster. That is one road. It seems like you're proposing the other road might be, well, can you actually just get a
21:27simpler data model that you can give to the LLMs? How do you make that decision? What path would you take? I mean, there's a Steve Jobs quote that I think about a lot, which is basically like, simple is harder than complex. Right. Because you have to, you have to basically simplify your thinking and it takes, it takes a lot of time to do that. Right. And that's a very human thing to
21:50do. And so I kind of lean more towards that, which is like, how do I find, how do I zoom out and think about the through line in the problems I'm trying to solve so that I can build a data model that is going to let me do that. Right. And that means it has to be simple. It has to be simple because I want humans to use it, but I also want my LLMs to use it. Right. But like I'm
22:14designing it for humans first. And so by, by its nature, it's going to be simple. You know, I think that, you know, if you're in the state where you have these very, very complex models, you know, one of the things that's amazing for LLMs is you can just say like, Hey, like here's a deep research type task and you can spin them up and you can run a bunch in parallel.
22:38And I almost would say, I would almost like challenge people to say like, can you take that deep research paradigm and say, Hey, like, here's our business model. Here's the problem we're trying to solve. Or like, here's our data model. Here's the problems we're trying to solve. Like find a through line here, find the simplest possible data model that we can potentially implement and then help me implement it. Right. That would be one way that I would start about,
22:58that I would think about it. The other thing is we've found like adding comments to like every, every, every database has this like comment or description field for columns and tables. I think it must be part of the spec. I've never used it. Okay. I've worked on databases for
23:1715 years now, basically. And I have only ever seen descriptions on columns in like the Microsoft example, like databases, like the, the, the Northwind type stuff, the whatever. And so actually in the MotherDuck MCP, I think you've done the same thing, but like we've actually added it, made it so that comments are surfaced to the LLM when you ask questions about a table.
23:42And so comments are a great way to inject context. Right. And so what can you put in a comment? Well, you can describe what the, what the column is, right. But you can also say, Oh, this column joins on this table over here. Right. You can give it a way to introspect the relationships between the tables. And that's super powerful. Right. Of course, there's other ways to do that,
24:03but because you can describe it just in text, it's very, very powerful. What doesn't work on the other side of those columns is you just, just describe the, like a longer description, like do a text, expand the column name. Like that's bad. Don't, don't do that. Like the main thing you want to do for LLMs is you want to add context. They don't possibly know already. Right. So if they read the column name and say, Oh, well, that means this. And then you just put that,
24:28you know, 10 word description in the column. Useless. Don't do that. But if you say, Hey, this, like in our company, we use this word to mean that great. They're not a column. I think context is still super important. Right. Context is going to improve the accuracy of your ability to answer questions. And so I think the other thing that partners with that is evals. Right.
24:49And so when I talk about evals, what I really mean is like some sort of way to run tests on a nondeterministic system and figure out how often it's right. And so like with text to SQL, I think about this as the notion of like question and answer pairs. So natural language question and an answer that is, that is, you know, either text or numeric that you can train against. Right. And
25:11so, so I've been doing a ton of that and that's really like kind of the core learning loop I've been building is like question answer pairs and underlying data. And then I don't care what the SQL is. I care about the answer. Right. The LLM writes a SQL and it returns an answer. And I care if the answer is correct. Just like an end user. Yeah. Yeah. They don't care what the SQL is. Yeah. I mean, I think there's like a whole, I have like a overarching theory there that I can
25:34get into that is crazy, but we'll, we can talk about it later. Yeah. And so, and so I think we'll, we can talk about it later. Yeah. Now Ramona Truda, who will be on the show later this season has yeah, she's great. Yeah. She's she's incredible. So she says, well, here's the exploit to that. The comments are toxic payload. The LLM will just treat it as code. And Ramona,
25:59I think has done a lot of research on how to poison LLMs, you know, this way. You know, so I don't know if you have any base response to that. I mean I think like obviously like you need good security controls on what goes in there.
26:20You definitely can corrupt. You could probably corrupt in very subtle ways that'd be very hard for users to detect by putting bad comments in. Right. But that's actually why you want to run evals. Right. Because if you run evals, you can detect drift in your answers. Right. So it's actually really important that you you have some sort of baseline. Right.
26:43And you know, what I love about the MotherDuck model is that you can have a bunch of really small databases, as many as you want. They're isolated from each other. You can run an insane amount of evals at a time for a very cheap price. And I'm actually doing that all the time now. We are kind of working on the research stage, but it is progressing into the point where we're
27:03going to start using it internally, which I think is the next step. And, you know, we'll share we'll share parts of that, you know, broadly to an open source, which I'm really excited about. Now one, you know, before the show, we had a couple of comments on the episode when I posted it on LinkedIn. One of them. So a couple of them I want to throw your way. One was, well, how good
27:27are your like how good do your column names and table names and descriptions have to be for this to work? I think a lot of people are operating in environments where someone eight years ago gave the column a meaningless name and and they feel stuck with that. Yep. Okay. I will answer that question very precisely, which is the most important thing
27:51for an LLM to be able to answer questions about your data model is the column names and the table names. Right. They have to make sense. If it wouldn't work for a person, it's not going to work for an AI. Certainly. And, you know, I kind of I'm not sure how to think about this concept. Right. So like people are very familiar with this notion of like medallion architectures.
28:15Yep. Right. Like, you know, bronze, bronze, silver, gold or like, you know, raw, you know, intermediate and conformed or whatever. I've kind of been like riffing on this idea that like for if you're building something for AI, you needed something else. You need something like a you need like a titanium layer. You need the layer for robots. Right. And so AI is actually an
28:37opportunity for us to revisit a lot of those assumptions. Right. Like one of the one of the biggest hacks, if you want to get good results out of out of AI asking questions to a table is just use one big table. Right. Like that actually works really well because then you reduce the
28:55surface area for a joint to be a problem. Right. That hurts my soul as like a OLAP enjoyer who like loves star schema. Absolutely. But like those are great. Right. And like so I think there's there's lots of ways to do it. But I will say that like the most important thing,
29:19like elements are reading the table names and the column names 100 percent of the time. If they're bad, you're going to inject junk. Right. Just like Ramona earlier said, like, hey, you can inject a toxic payload in the comments. Your table names and your columns have the same problem. Those things, you know, if you're using, you know, select star throughout
29:42your pipeline. Well, that means that it's hard for LLM to inspect what's happening in there and figure out how things are related. Right. So like all these little best practices like don't really matter that much for humans. It's like, all right, we're using select star because we're lazy, whatever. Well, actually, it's really helpful to like give, I don't know, your dbt pipeline or whatever, whatever you're using to build kind of the underlying data to the model so we can
30:03understand where things are coming from. Right. And do things like help you write documentation. If you're being lazy and writing stuff like select star now, you now, you know, it's like, oh, well, I now need to buy a lineage tool and the lineage tool will help me understand what's happening in the data. And then we can give that to the LLM and the LLM can help us write it. It's like, what are we doing here? One more tool, like we need to simplify, we need to simplify,
30:26not add complexity. Yeah. Another comment that we had before the show was basically, oh, Jacob's research says that the LLM was 95% correct. Well, 95% is way too low. Anything other than a hundred is unacceptable to end users. And I definitely understand where that's coming from.
30:51And in a very tightly walled kind of BI garden where you can't do anything other than what's already been modeled in BI by your IT team, maybe you can expect superficially that level of accuracy. But I guess I would argue, and I'd like your take on this, that like actually most,
31:10the human authored accuracy rate is not 95%. Even in BI tools, the gap between what people maybe intended to pull or what they believe they're pulling, but they don't know that you have to do this one, you have to set the current record flag to Y on this particular dimension.
31:33So I guess I would argue like actually the human error rate, even in the best, most robustly modeled systems is probably, brings you below 95% accuracy. I don't know what your take on that is, but that's definitely, that's one of the big pieces of pushback you'll get to LLM authored anything is people will say, well, if the robot can't hit a hundred,
31:54then I don't want it. Yeah. I mean, I think here's what I would say, right? I began my career in accounting. My job was to find errors in human produced financials, right? And even at companies with really smart, incredibly diligent accounting teams have errors. Okay. That doesn't
32:18mean there's fraud. That's different. Fraud is different. Okay. But they're going to make mistakes. Okay. 95% is probably, I would actually kind of agree. 95% is probably too low for an unattended system. Okay. But we're building attended systems. Human are in the loop. I'm not talking about replacing humans. I'm actually saying like, this is actually amazing.
32:40What a great time to be an analyst. We can do better analysis. Like why would we not use this as an opportunity to do better? And 95% of it works great. Let's spend time on the one that doesn't. Right. Let's build awesome stuff that was never possible before. You know, I think like that, that's how I kind of think about this stuff. So you think like, you know, the first thing is,
33:02yeah, 95% probably not high enough for an unattended system. I agree. Right. We need a hundred percent if it's going to, we're going to rely on it. But also systems that we're relying on are definitely less than a hundred percent today. I mean, I found errors in public company financials, right? Like, okay. You know and what do you do when you find that, right? You're definitely not opening up a prior period and making a change. Okay. Like, you know what they
33:24do when they find that at those accounting firms? Like, well, I guess we're going to bury that in the current year. Like, what are we talking about here? Like these things, the most important thing is that we have a consistent basis of understanding. Right. One of the things I think about for my career is, is I worked in, you know, analytics and data kind of in the early days,
33:47you know, we were running, I built like web servers, we were running on like Rackspace and things like that. Right. And so we had our own analytics around how things worked and like how many hits we were getting. And then Google Analytics came out and was like, here's your real number. Like, why are these different? Okay. I have no idea why they're different. Okay. I mean, like I do, but like, it's so nuanced and like, what I care about is the direction of the line.
34:09And that's almost always the same between those two numbers. Right. Between like what my web server says, and this is like in a pre, pre-Cloudflare, pre-caching days. Right. So like, now it gets crazy. We got Cloudflare in there. Who the hell knows what's missing what. But, you know, what we care about is directionality on most of our analytics. Okay. And so that means
34:31that like repeatable is actually really important. And on the repeatable ones, we should actually have a human in the loop and just say, okay, yeah, this looks good. It's repeatable. But, you know, yeah, I think there's definitely room to get more precise with how I want to think about it. But also like, you know, 95% is a gift. It's a gift we didn't have a year ago. Like,
34:52yeah, treat it as a gift. That's how I think about it. Yeah. I mean, I can tell you, I'm fine with 95% accuracy. Let's imagine the accuracy of what I'm doing on, you know, through MotherDuck with the MCP and Claude is 95%. I'm so thrilled with that because 100% of what I'm doing, I would not have been doing otherwise. Right? Yes, exactly. Yes.
35:15So give me the 95, right? And I'm not building these autonomy, like I'm in the loop, right? I don't have some autonomous agent that's doing Superdata Brothers ad spend decisions with my credit card number, right? I'm doing classic analytics stuff, making decisions about, you know, ad spend and guests and all that sort of thing. And I'm in the loop and when it's wrong,
35:37you know, hopefully I can detect when it's wrong. But that's, you know, it comes down to that, right? I would not be doing this stuff. All right. So we've had a couple questions come in here. Let's do it. Give me the hard ones. All right. Yeah. So let's see. 95% inaccuracy on what size database? And are we saying validated this across pre-aggregated data?
36:01Are we talking about every permutation of analytical calculations you could possibly do? I guess the question is like, what does 95% mean when we say 95% how you could possibly handle every permutation of what you could calculate or the variability of how well your database is even modeled? Actually, the next comment is actually kind of answering it, I think, which is like, we're getting some great stream of consciousness here.
36:28So, yeah, I mean, I'm talking about 95% as relates to the BirdBench dataset, right? Does that generalize to your dataset? Like, I have no idea. Like, you need your own evals, right? The BirdBench data that I worked on and the related eval harness is free. It's open. You
36:47can, please, please steal my harness. Please use it, right? You will know what it looks like for your data. I think like that's, but like, it's a great question, right? Which is like, I'm not saying generally that 95% of it's right. What I'm saying is for the ones I reviewed, that's what I saw. Okay. You know, that being said, my practice, I use it all the time. Okay. I do human in the
37:11loop stuff all the time. For me in practice, it's at 95%. It might be even higher, right? And now, you know, I've worked in SQL. I've written SQL for 10 years. It's easy for me to do that.
37:23So like, that's probably not true for someone who is not data, you know, it doesn't have a fingertip feel for the data, right? But I think like, this is like, it's such a good question, which is like, what I will say is like, for me as an analyst, everything I'm writing about rings
37:46true. And I have data to back it up. That's why I wrote about it. It was not like, I was surprised by the outcome, actually. I did not expect that we would be this tight this soon. But the last 5% is going to take forever on those benchmarks. And in the real world, you know, what I'm pumped about in the future is, you know, giving people the power to run their own eval so they can know
38:10what that percentage is for their own data, right? I just don't know what it like, I don't know that yet. That's the next stage actually for the research. So thank you for leading me into that. That's what I'm working on right now. Awesome. Thanks for the great question. Now, we have a couple of good questions. But before I get to them, you need to do me a favor. If you're watching this on YouTube, I need you to click on the like, subscribe button and drop a
38:33comment on the video to help us grow the channel. It would help us out enormously. And likewise on LinkedIn, you know, go ahead and follow me on LinkedIn. You'll get a notification and an invite when we go live. We go live every Thursday at 12 o'clock, interviews with great guests like Jacob. So follow us on both channels and tune in for the live show next week.
38:55We have a question from Jenna Jordan. Jenna asks, to get to that 95% accuracy, what do you not leave to the LLMs in the input process? So this is actually, I think, a really good question, right? This is such a good question. Jenna, by the way, hello, Jenna.
39:14We know each other in person, so hello virtually from my basement. So the best thing to add for context to LLMs is things it doesn't already know. Okay, so I talked about this briefly earlier, which is like adding column comments that simply are text expansion of the column.
39:33It's actually fairly, like all it does is actually increase the expense for you to run your eval or to ask questions. It's very poor. So what I like, so the things I add are things that could possibly are not, that it wouldn't know without looking at the data first. Okay, so I actually
39:54have a data profiler that I've been working on and building that adds column comments. for instance, things that I found effective in my evals. So, things that are effective are like telling it what type of data it is. Is it typically used as a fact or a dimension? Is it, what other columns are known to join on this, right? And so, we can do that by
40:15doing, and how do we get that data? We get that data by two kind of things I think about. The first one is we run like, DuckDB has this summarize function, which tells us a bunch of information about the data, like the types of the columns, which we actually already know that because the LLM can know that already, so we don't need to add that. But like, it tells us like, you know, is it like, if it's an integer column, what are the min and max
40:38values, right? Is this an enum or is it a numeric range? So, like there's statistical information we can populate it with. The second thing is we can use query history to say, hey, these tables are commonly used together on these keys, right? And we can do, there's a whole area of research I'm sure people are exploring around like, how do we use query
41:01history to like, bootstrap context? I'm doing a little bit of that here too, which is like, okay, I can know what my gold joins are by like, looking at the queries of my CEO, right? Now I know what my, like, what's the definition for these metrics? Great, let me rank, I'm going to rank the queries from certain people higher than lower. And so, there's a couple
41:25of ways, but I would say, again, if it's something the LLM can read and understand and like, get a good description of without any human input, never add that to your context. You want to only add things that expand or clarify, right? If it's like, only you, that only your context, like human context adds. So, that's what I found to be super effective.
41:49And I think there's actually a lot of research in that space too, which is like, clarifying things that it already knows is actually not important, but clarifying things it could not possibly know is super important, right? Where it really comes down to is like, let me give you a very tangible example. If you ask for what the revenue was last year, it can probably actually reason about revenue fairly reasonably, because there's so much
42:10data in the training set. However, last year may mean fiscal year. And you need to define that somewhere, that like, hey, when I say last year, I actually mean, you know, the calendar year starts in February, or sorry, fiscal year starts in February. If it doesn't know that, it might just give you calendar year instead and just assume that year means year, right? And so, figuring out a way to do that, I think it's super important. And
42:32then like, reinforcing that, you know, you would need to reinforce that across multiple metrics, right? And so, if you do that, you have a, how I kind of think about this stuff broadly when I'm doing evals and like, self-learning on this stuff, is I think about it like a Sudoku puzzle, okay? So, if I have a set of question-answer pairs that I know are good, right, that are my training set, and as I define one of them and I enrich the context
42:54on one of them, it actually reduces the possibility space for other answers to also be correct. And so, once I lock in a set of 20 or 50 or 100 kind of golden metrics or golden questions, now I have the, now I basically have a plausible factor of things that like, I can't, I have a bunch of rules kind of baked into that set of 100 questions and answers that like,
43:18exclude a bunch of wrong answers from ever being, from ever being presented to the human. Is it perfect? No, but I'm working on figuring out how to make it, you know, as good as possible. Where are you storing that, like those question-answer pairs? Is that in MotherDuck? Are you making that available to the LLM in a markdown file? Like, what are you doing?
43:37That's such a good question. You know, I have a very versatile database at my fingertips that I use, you know, it doesn't cost me anything. Yeah. So, I'm definitely using a database for everything. Good. Yeah, like JSON columns, let's just throw, or like a VARCHAR, you know, let's just throw, let's throw a big markdown, markdown, sure, let's throw that into a column.
44:01Yep, that makes sense. Ramona makes the point, you know, humans do the same thing. Tell me more means give me context I didn't have before. And I think that's a very astute way to think about it, right? The things you need to give the LLM are the things that it can't possibly know. The stuff that's unique to your business, or the stuff that's unlikely to be well represented in the
44:23training set. That's right, out of distribution. Give it the out of distribution stuff, right? Yeah. I do want to shift gears a little bit. We have about 15 minutes left. I want to thank you all for being an awesome audience, and all the great questions and comments we got flowing in today. As usual, the Super Data Brothers show attendees bring the heat with the questions.
44:44They do, I love it. I want to talk a little bit about dives, actually. So, I will lead by saying, I love MotherDuck Dives. I use them personally, again, to help run Super Data Brothers. Can you tell us what is a dive? What does it do? And why would you use one? Oh, man, I love dives. They're
45:07like one of my favorite things that we've worked on this year. But I'm going to tell a little bit of story about how we got there, because I think it's actually really informative of why we did it the way we did. So, we released our remote MCP in December. We've had a local MCP you could use for quite a while. Ryan, I know you were a local MCP user. Are you fully converted to remote now?
45:29Yeah, only the remote. Okay. Yeah, only remote. We've now superseded the capabilities of the local, which is great. Great sign there. So, we released it, and what was happening immediately was, I sit in our Seattle office, and I sit next to some guys on our sales team, and they were like, all right, I need to figure out what this thing is. What the heck is this? I need to tell customers
45:51about it. And I'm like, oh, you can just ask questions. They're like, okay, let's ask some questions. And so, they were like, all right, help me prepare for my next call. It's with this customer. Go look at our internal data warehouse and just give me a lay of the land so I have the right context when I'm talking with them. And so, they started building those out, and they started
46:12building out these really complex artifacts, which is like the Claude version of just an HTML or JavaScript file that they could then share. They're building like dashboards and stuff, and then they're like, oh, we can use skill. Claude's like, oh, do you want to put this in a skill so you can use it again? And so, they're like, yeah, sure, put it in a skill. So, they're like slacking around and emailing skills to each other to build these reports so they're better
46:35prepared for their calls. And so, I'm sitting there next to them, and I'm just like, this should be programmatic. There should be some sort of way to do this. And so, that was like the kernel of the idea for Dives. And it was great because as we're building it, I'm sitting next to the sales team like, okay, here's the next thing. How does this matter? Is this worse? We really built it for
46:56ourselves first, which has been amazing. But anyways, what Dives are is they're artifacts in Claude that you can then save to MotherDuck and share and update automatically, right? So,
47:17they're basically used in a very similar way to like Excel. I think about them as like a perfect metaphor for Excel. All my Excel usage has basically been replaced by Dives at this point, which is insane, insane to say, but is true. Everything that I would just throw into Excel and the chart goes into MotherDuck instead. It's so easy to get the data in using our MCP or just
47:42loading it with like the command line that I just say, okay, here's the data set that I'm looking at. Help me build some charts. Help me understand this information. And Claude is just so curious that it makes it great. And so, basically, it's in our MCP tool. You can say build a Dive after you've done some analysis, and it will help you build a visual story around what you're doing, and then you can save it and share it inside of MotherDuck. So, that's kind of the core of what
48:05it is. You can kind of think about it like dashboarding, but it's more like Excel in my opinion. And then it kind of takes all of that stuff. We're talking about like, hey, like 95%, that's probably not good enough. Well, when you're in the loop and you get something that's right and good, you then want to save it so that other users are not subject to the 95% constraint. You don't want them like running a skill that has this air mess, that has this chance of going
48:29totally off the rails. So, that was the main genesis was stop running the skill, let's just save the end thing as something you could refresh and share. And so, that's kind of what we built out. And I built some really fun NBA ones that I've been having a lot of fun with. I refreshed my NBA data pipeline, so that now works automatically. You get like really rich insights
48:50and analytics out of it. It's really fun. It's really fun to build. And everything, what we like to say now internally is like, if you look at a dashboard, you're like, hey, how do I add this other thing to it? I'm just like, that's one prompt away. Just ask Claude, it'll add it. And it's been insane to like just, now everyone's just like, oh, that's one prompt away. Like, yeah, I want it to interact. But when I click on that, it drills down. And I'm like, great,
49:14that's a prompt. Which is a awesome, awesome interaction point and our users are loving it. As one of your users, I love it. So, how are you seeing people use these? Because what it really is, I mean, it's like you're building kind of an app, essentially, right? Like it's a JavaScript
49:38app that, and it can do whatever it is you need it to do. So, if you need spreadsheet, let's say you're in accounting, you need spreadsheet-like functionality. Okay, you can do that. I'm a BI guy. I want to make interactive reports and dashboards. Okay, you can do that. You can really, I'm almost like anything you can imagine. Really, it can build. And then it saves them into DuckDB
50:01or in MotherDuck. So, there's essentially, they're available. You've got your database and then these almost micro applications that exist, that are built on that data. They exist in the same place. How's it going? Like what sort of adaption are you seeing with your customers and what are they doing with it? So, I think there's like two. So, I'll talk about internal use cases
50:25first and then we'll talk about customers. So, internally, it's kind of like there's two kind of core ways that I'm seeing it being used. The first one is by people who we call NTDs. That's non-technical ducks. That's like our sales team. Okay. So, they're basically using it to build, do self-serve to better inform them on what their day-to-day looks like. Building a data,
50:49basically, every person on our sales team has a slightly different one they're using. In the past, it would be like they would either all get the same thing or get nothing. They'd be like, you know what? Sorry, that's too deep on the BIQ. No. And so, we're seeing them do that and we're seeing them also do some just awesome stuff that I was never thought was possible, where they're using actually skills as like data pipelines, which is bonkers. So, if they have
51:13like something they need repeatedly, like that's not in our data warehouse, they're using like skills to like, they're embedding SQL queries in their skills and then be like Claude run this and it'll give them the data they need. These are like very small specific slices of what they're trying to do every day. So, that's one use case. The other thing is our engineers who are like,
51:33especially the ones that are like more AI, less ready to embrace AI, are kind of like seeing, oh, well, something that I would just kind of wonder about in the past, like maybe some observability question or like looking at some logs, et cetera. They're like, oh, okay, now I can just use Claude to like solve this thing that I just couldn't do before because I
51:55don't know the primitives of like how to drag and drop the chart on the thing, right? They're doing like, they're solving really hard complex problems and they want to be able to visualize improvement over time, like gradient descent type stuff or whatever. Now, they just do it without having to like reach out to the data scientists to do it. And they're doing really
52:17awesome, really awesome stuff internally. And then on the customer side, we're seeing very much the same, right? Non-technical folks who are like being able to self-serve kind of for like as an extension to like an extension of their internal data team, right? So like for internal needs. And then we're starting to see people, like I think the number one ask for dives is embedding. So they want to be able to build dives and then like embed it in their application so
52:42their downstream customers can use it. And so that is what actually we're working on next is making that easy to do. So that's kind of how it's working. We're seeing lots of adoption. But folks are really finding it a powerful abstraction. And it works, you know, because it's part of the MCP. You can connect it to Cursor, ChatGPT, Claude, Claude Code,
53:06all of these different places and you can get awesome results. So yeah, it's been pretty fun to work on. So what would you say if someone, oh, I guess I don't need a BI tool now. Yeah. I mean, I think like one thing that I would say is there's still things that, you know,
53:25need that higher level of accuracy, right? And one thing that has been really powerful about us for using dives is that because we have a really strong like source of truth already in our internal dashboards. Can I say who we use or is that okay? Yeah, no, go for it. We use Omni
53:45internally. It's great. It's amazing. And so basically all the long tail is so like, you can think about like BI as the core things, right? Hey, it's important that like everyone knows these same metrics. And then we use dives to basically riff on all of those metrics we already know about, right? And one thing that's really cool is Omni's semantic layer is extensible.
54:10So we've just basically loaded it into our data warehouse and you can inject this context wherever you want. You can basically be like, hey, go to this table over here, read the context file on the Omni, you know, on the Omni metadata and then use it in our dives. And so we do that all the time basically as a step to bootstrap the context so that you have a higher probability of getting
54:30the right answer the first time. Awesome. So we've got about five minutes left. So there's some questions that we're probably not gonna be able to get to. I apologize. I do my best to read as many as I can. But there were just too many good ones today. Ramona asks, what are some edge cases that you're still figuring out? Or any data that just refuses to comply? Yeah, I mean, I think
54:55I haven't found data that's refusing to comply yet. But I think like, we're, I'm in the stage of transitioning from research to application on this stuff. And so right now what I'm working on is like self-healing context. And then that will move into, I see there's a couple of related questions like expanding the benchmark to continuous user-based question answer set.
55:18Yes, that's next. That's the next thing that I'll be working on or working with the team on. I think that will not just be me. But yeah, I don't, I don't have, I wish I had a better answer to this question. I would say that like everything in my research at this moment is pointing towards a better interaction between the data modeling and the question asking.
55:41Okay. And what I mean by that is you need a feedback loop that says, hey, this question is being asked frequently. And it's a hard question because there's complex calculations and complex joins. Why is that not already a view in your database? Okay. How do you push that in and have like some human in the loop review to say, oh, maybe we should add, we should add a view
56:04for that or add a macro for it or whatever. Right. Add a function that returns that thing. Right. I actually, I actually was talking to a buddy who said the best data warehouse he ever worked on was actually with SQL server and everything you worked on was a function. So everything's defined as a function where you'd say select star from a function and each function
56:22would let you answer a set of questions. So I think that is another take on potentially what I would call a semantic layer or context layer, but we'll see what happens. Yeah. I mean, this is a space that's moving so fast too. That's the other thing, right? Like what was seemed possible, what seemed impossible two months ago or three months ago became very
56:45possible. And I know in talking with you and Ryan too, like, you know, the latest iteration of the frontier models really did like, just did so much to enable a lot of the stuff we're talking about here. So, you know, where it's going to be in five months, who knows? And I guess that's my last question for you is like, so what's next? You guys are doing this research,
57:09you know, both for you personally and MotherDuck, where are things headed? Man, I wish I knew. That's an honest answer right there. I'm working. So I think like there's a few things that I would just be honest about, about AI. The first one is like, I don't know how anyone manages like sprints anymore. I have no idea. First off, like everything is either like possible in 30 minutes or, you know, multiple weeks. It's
57:34impossible to know in advance, just like from an engineering perspective. So good luck on that. The second thing is like, how the, how the heck am I supposed to submit talks that are like six months from now? Yeah. Like, what? I'm like working on a couple right now. I'm just like, working on a couple right now. I'm just like, I'm like, what, what the heck? So like for me
57:54personally, you know, it's super exciting because like, I can't tell if it's like continuing, will it continue to be exponential from here? Will it be linear? I have no idea. You know, it's very hard to, for me to read the tea leaves. I expect it will be more linear than exponential. I think like we're probably through the steep part of the curve at the moment. I mean, I could be wrong, but that's my suspicion. I think for MotherDuck, you know, it's continuing
58:17to build out, you know, more enterprise-y type of features. What that means is things like role-based access control that is more fine grained, row-level security, you know, higher scale partition handling, and a few other things like that. You know, but you know, I feel really good about where the platform is at. We're making great progress there. And then, you know, in terms
58:41of what the AI interactivity pieces look like, man, I think we're just going to, we want to, part of why we built Dives as part of the MCP so that people could use it in Claude Code is like, if you squint, maybe like Claude is the interaction point for a lot of your apps.
59:01And you know, we think that might be a possibility. So that's just kind of on deck. We'll see. And then I'll just say, Ramona, yeah, I would love to connect, but anyone, you know, feel free to ping me if I didn't get to your questions. Ping me on LinkedIn, or you can find me on Twitter, or MotherDuck Community Slack. I'm always happy to talk with the community. So,
59:25you know, just wanted to shout that out quickly. Yeah. Yeah, absolutely. Yeah. So, you know, Jacob told you how to get ahold of him. Reach out to him if you want to keep talking. I want to thank you all for attending. Once again, great audience. Like, subscribe on YouTube, follow me on LinkedIn, or connect with me on LinkedIn as well. Both are great. And yeah, stay tuned. We'll be back next week with another show. Jacob, thanks for joining
59:49me. I knew you'd be awesome. You delivered, my friend. So thank you and have a good day, everyone. I'm going to go join my wife at the pool. See y'all. all right.
