0:00Welcome to the Analytics Engineering Podcast, featuring conversations with practitioners inventing the future of analytics engineering. Jordan, it's good to talk to you. You were actually on season two, I think that was about a million years ago at this point. Why don't you take a minute for people who have not listened to the entire back catalog
0:22of the podcast, just to reintroduce yourself and share a little bit about what you do, what you're up to. Yeah, sure. I think actually the last time I was on was just starting MotherDuck and was still kind of trying to figure out what the hell we were doing. And, you know, thankfully you gave us a platform to expostulate about some of it.
0:43Oh my God, we've already used the word expostulate, I love it. I didn't say hypothecate yet, I was trying to work that one in. It was a, you know, it was the prediction market where people like throw in random words and then you can place bets on it. Anyway, you know, so, you know, we're building MotherDuck.
1:05We built a data warehouse based on DuckDB, sort of like Snowflake, but faster, less expensive, easier to use, all sorts of more goodness, more AI, you know, better for AI, better for applications. And we've been going for about four years since the last time I was on.
1:29And yeah, I think we're doing some really, really cool stuff and the, you know, the probably we'll get into this, but the, you know, the AI era is really making things a lot more, a lot more interesting and heating up the data space like it hasn't been for a while. That's shocking. I kind of believe it.
1:51So, okay, I want to hit some background stuff and then we can get to the, you know, AI and everything else. Sure. But, you know, you are not directly affiliated with DuckDB, the organization that produces DuckDB. Yes. You, I think there is a relationship there.
2:14Can you remind everyone, what is that relationship between MotherDuck and DuckDB? Yeah, sure. So when we started, we, you know, basically it was like the recognition that, hey, DuckDB is amazing. It's really well done. These people really know what they're doing. It's going places. And I actually, you know, reached out to Hannes and Mark, the co-founders of DuckDB Labs,
2:37to see if they'd hire me to build a SaaS service, because I'm like, hey, somebody should build a SaaS service out of this. And they said, no, we really want to just focus on the database, but we'd partner with you. And so that was sort of the, you know, the initial kind of founding of MotherDuck. And so what we did is rather than saying like, hey, well, you know, thanks for building an
3:01awesome open source project, we're going to go take it and try to make a bunch of money on it. We were like, hey, we want to make sure that we're good partners and good stewards of open source. And so we gave them a co-founder share of the company when we started. So they were kind of economically incentivized for us to be successful.
3:20We also did a lot of, you know, funding of development in DuckDB, you know, because with the belief that the faster DuckDB moves, the better it's going to be for us. Like, you know, and they also build a lot of custom features just for MotherDuck, you know, things like, you know, because at the time, most people were just running DuckDB on their laptop.
3:42If you're running DuckDB on a laptop, you don't really care about, you know, write concurrency, for example, like being able to, you know, do multiple write queries at the same time. And but for us and for, you know, when you're building a cloud data warehouse, that's obviously super important. And so we worked with them on a lot of adding a lot of these types of features.
4:06But we do, it is still an arm's length relationship. You know, they come to our board meetings, they, you know, we talk to them a lot. We have, you know, we, we have, you know, a great, a great relationship. I think one of the other interesting things was, you know, when we started, it was like, we had this development agreement and we're like, okay, well, we could, we could basically
4:28hash out like, here's like this very detailed defined description of what happens if this happens in this and like, you know, highly litigated and, you know, legalese. Or we just say like, you know, look, none of this is going to work unless we trust each other. And so, you know, we, we just chose to trust each other.
4:49And I like hanging out with Mark and Hannes and I think they're good guys. You know, I have no reason to believe they feel that much different. And and so it's, it's just sort of, it's just sort of works based on this sort of this just sort of high, high trust relationship between the two companies. That's very cool. You and I have talked a bunch of times over the years.
5:10I actually did not realize that they were on your cap table. That makes a ton of sense. And you know, alignment of interests in these long-term relationships really, really matters. Back in, back in the early days of dbt, I, I never even considered that maybe I could, you know, find somebody out of Google or Microsoft or somebody like that and to, to commercialize
5:35dbt and I could like stay working on the open source stuff. I mean, that, that seems like a great setup. I think going back to 2018, maybe I would have taken that deal too. That's pretty cool. You might have given up on that if you, if you'd asked.
5:52Can you, it is fascinating because there, you know, you mentioned the arm's length relationship. There is obviously a set of stuff that if you want to build that stuff, it's going to go into dbt proper versus another set of stuff that is not, is like outside the charter of dbt proper.
6:17And, and I assume you folks would want to build that and, and exactly where that line gets drawn means a great deal to, uh, you know, what you folks can commercialize. Do you have some kind of like clear delineation of like where things go? You know, I think, um, there isn't one that's like, that's clearly, this clearly written down.
6:41I mean, obvious, the obvious one is like, we're building a hosted service and they're not. And so, um, you know, and I, and I think from a, from a, you know, open source business model, uh, you know, SAS is, I think is the, to me is the, is the most obvious and cleanest of the, of the business models. Like, Hey, you pay us to run this in the cloud.
7:03Uh, you could do it yourself, but it'd be a lot more work and you'd have to do all this other stuff. And, um, you know, I mean, there's like, you know, Postgres hosting is a multi billion dollar, uh, you know, revenue, revenue business. And, and so, you know, there is the, Hey, we build the SAS service.
7:22Um, DuckDB right now is, is a, you know, uh, doesn't have any concept of users. So if you want any sort of authorization, um, uh, you know, or, uh, uh, authentication, like there's, there's just like, like a bunch of stuff that, that, you know, for a data warehouse. And if you're going to run it in an, or in a, you know, meaningfully sized organization,
7:47you, you know, it's not really quite suitable for, for that. So there's like really no such thing as a grant statement, right? Correct. Wow. Did, is there a grant statement in MotherDuck? Yes. Huh. So you've had to build an entire user model on top of DuckDB. Yes. I mean, we, I mean, which is like, you know, that's the stuff that we, that's the stuff
8:08that we're, that we're adding and, you know, we have SSO and we have, you know, all of those kind of things that you would need to, um, you know, to sort of build a real data warehouse. Um, and it's possible that DuckDB will add some of those things. You know, I think it's, um, over, over time. Um, but you know, I think the way I think about it is like, Hey, they gave us a big
8:29headstart and if we can't win with a, with a headstart that big, then like, you know, we've, we've done something, we've done something meaningfully wrong. And I think in general, they don't really have interest in building all of that, all of those sort of enterprise mechanisms. And I think that there is sort of a clear line of like, Hey, enterprise, you know, businesses need these, this kind of things.
8:50And like, and if you're just trying to run on a laptop or, or, or kind of a, um, you know, very small organization, just trying to make your data, you know, build a great, um, uh, you know, query engine against data that's an iceberg, whatever. Like they, um, you know, you don't need those kinds of things.
9:11And, um, and I think, you know, I think we are the, uh, we're the obvious answer for people who do, which I think is actually good for us. I mean, they just launched something they called Quack, which was for a long time, DuckTV was just embedded. So, um, it was, it was clear where the line is. It's like, Oh, well it's an embedded database versus a, versus a client server database.
9:32Um, and now they have a service, you know, that, that you basically can, you can stand up on EC2 and you can connect to it from, you know, from, from anywhere, uh, using, you know, it's an HTTP, you know, um, mechanism, but I think it's actually great for us because like a bunch of people are going to try that and then they're going to be like, Hey, I need, I need users, I need auth, I need like these, these things.
9:56And then we can, we can, we can step in and, and provide that we need, you know, backups and we need, you know, these types of, of things. Um, I, I think this whole thing is fascinating. I really, I, I did not understand, uh, just how stripped down, uh, DuckTV was at, at the core. That's a lot of scope for you to innovate on top of.
10:17Do you have any, via this, you know, arm's length partnership, do you have any intel on like just how many, how, how widely used DuckTV as a whole is? I, we have some intel on, on it. Um, and I think we see, we see reflections of it.
10:36I do believe that like even the, virtually everybody, every enterprise has DuckTV somewhere, whether or not, whether or not they know it. Um, but I, we very often will talk to the JP Morgan or some, like some, um, some big banks, some big enterprise. And, you know, I think at first they're like, Oh, and we don't, you know, we don't use any
10:59DuckTV, but we, you know, and then they're like, Oh, actually we use DuckTV all over the place. And like, and once they start trying to, you know, actually think, think about it and, you know, cause it is, it's just so useful and convenient. And I think one interesting thing is like, you know, often in, in, in open source projects, there's a big pipeline from the open source, the open source to the commercial is like,
11:21Hey, you use the open source until it gets to be sort of mission critical. And then you go and you use the, uh, the, the, the hosted version, uh, or the fully supported version. And we see less of that in DuckTV because a lot of use cases are just are like, Hey, I'm, I'm like, like using it to query my, this parquet file is sitting on my laptop.
11:43There is not a, there's not really a, you know, It's like a useful for a cell in a Jupyter notebook. Yeah, exactly. And, um, but I think the quack is actually going to change that. It's like, cause they clack offers a, you know, the, the DuckTV RPC, it offers sort of a clear, like, Hey, I can see why you would need then, you know, to go, to go with, um, with MotherDuck and the, and the hosted version of this.
12:08DuckTV, it incidentally, they do offer support for, for DuckTV, DuckTV labs does. Um, and you know, there's a lot of enterprises that, that are using DuckTV heavily and hedge funds and, you know, various, various folks that, um, they pay, they pay DuckTV labs, um, for, you know, enterprise level support. I'm curious, and you started to get into this with the funnel from open source to the hosted
12:30version, but, um, who are your customers? Uh, are, are they, are they people who are moving from the open source solution or are they, they adopting Greenfield? Are they, are they moving from another technology? Are they big companies, little companies? Like who uses MotherDuck today? Um, so most of them are small-ish companies, um, which is, you know, what we've, what we've
12:55been, been targeting, um, tech companies, startups. Um, but it's basically, you know, anybody who wants a data warehouse. And so we, we are, do we do a lot of people migrating from Snowflake, from Redshift, from BigQuery, um, couple from Databricks, uh, some of it's just gold tier stuff. So like the presentation, presentation layer, um, a lot of it, um, but a lot of it is starting
13:19to be starting to be silver tier. Um, we're starting to get, you know, larger, larger, um, organizations, um, a decent handful of sort of AI, you know, AI, AI companies. And, um, the other kind of, so there's a data warehouse, um, uh, you know, cause we can be, you know, we're a lot less expensive, we're faster and easier to use than Snowflake.
13:41Like it's, um, and so people who feel like Snowflake is overkill, people who feel like, Hey, I don't actually, you know, have that much data, but you know, my bills are going up, like, you know, going up like crazy. Um, there are also people who just like DuckDB and they want to use DuckDB. Um, the other, the other big use case that we get is people who are building applications where they're the, um, they want to show analytics to their end users.
14:05And like the tenancy model that we have, like is, uh, is really ideal for that. And so the tenancy model is every end user of MotherDuck gets a separate instance of DuckDB in the cloud. So if you have 10 users, you, we will spin up 10 different instances, uh, for those, for those 10 users. So none of them step on each other's toes.
14:27Uh, they all, they all start in, you know, within a hundred milliseconds. So there's virtually instantaneous startup. Um, we can scale those up to virtually the, you know, the largest size of EC2 instance, um, which is roughly the equivalent of a Snowflake 3XL. So like we can scale up to if you, you know, if each user needs a 3XL or less, like, you know, we can scale to, uh, to that size.
14:49Um, and you know, and then we, you know, and as soon as you stop using it, we, you know, it, uh, they, they shut down. Okay. But you were just talking about applications, uh, companies who build products that they want to expose analytics to their end users, but you don't mean that like, let's say
15:08Wealthfront is powering their front end via, uh, MotherDuck that like every user of Wealthfront, of which there are, I don't know, hundreds of thousands or millions, um, that each, each user who has a mobile app in front of them is like getting a DuckDB instance, right? So that's, I mean, that's one way that we can run it. Um, probably if you have hundreds of thousands of users that are concurrent, like then no,
15:32you wouldn't want to do that. The other way we run it is something we call read scaling, which is just, if you have like, you set a limit of like, you know, eight, 12, 64, uh, concurrent instances. And so we'll basically fund or funnel each connection through, um, you know, through a different, a different DuckDB backend.
15:53So it'll be basically like we scale out these, these DuckDB replicas on the backend. Uh, and that, you know, that lets us, those people can strain how many of those they're using and, um, but also make, make it so that, you know, if they, as, as they scale, uh, it's easy to just to turn that number up and turn it, turn it down. So there will be some like, um, you know, concurrent, you know, multiple users using
16:18the same, using the same backend, but, um, you know, it, uh, you know, a DuckDB is very fast. I think our median, our median query time is, I think it's three milliseconds. Uh, Oh, sorry. I just, okay. All right. We can move on. I just needed a minute to absorb that. So, yeah, I mean, so we can, I mean, you can run a lot.
16:43I mean, if, if you're meeting query time is three seconds, you know, it three milliseconds, you need, but it was three seconds. Oh, yeah. You need a lot of hardware to handle like a lot of concurrent users. And then, but if it's three milliseconds, you know, you can do, um, you know, 300, 300 queries per second. Uh, uh, and, you know, all from all from different users.
17:05And so, so, uh, it, it, you know, can scale pretty nicely. You know, I, I, I don't have a dog in this race. Uh, and so I'm, I'm not trying to prove anyone's faster, cheaper than anyone, but, uh, I, I do want, you know, you've made some, some big claims here. I want to at least give you the opportunity to put some substance behind them.
17:26If, if DeckDB is faster than or cheaper than, um, other, other alternatives, um, what, why is that at a high level? Is that because of business model choices that you've made? Is it because of the technology itself? Like where, from whence do these savings come? Great question.
17:47So, um, you know, in, in big query, every, every query that you, that you run has to go through a lot of hops in order to get to the thing that's actually running the query. And then if your query does anything non-trivial, it's got to go through more hops. Data's got to be shuffled around the network. Uh, you know, there's like, there are all these, like so many different moving, moving
18:12pieces and, um, and that those all add latency. So like, so we can do, you know, we can do client measured. I mean, so like the three milliseconds I mentioned, that's that server side mentioned, uh, measured, but client side measured, like we can do, you know, 10, 20 millisecond queries measured from the client and that includes the round trip to, to the server.
18:33So yes, you do have to be in the same data center, uh, in order to, to get those, those kinds of latencies. So a lot of it is just the sort of the distributed, the mechanism to build distributed databases adds, adds latency. Uh, and it, it makes, you know, they're designed for throughput. Um, and you know, I remember, I remember working on BigQuery and my manager said, I don't care if you add, we're making a change.
18:57I don't care if you add a second to every query because that doesn't matter. We're handling giant queries. Like, but if you're running a dashboard and your queries take a second, you know, you add a second to every, every, every query, that means every user has to sit and wait a second. That's really annoying. And, um, and so just what we have designed for is latency rather than, rather than throughput.
19:20And, you know, I think the single, single node, uh, the single node helps. I think also the, just the database, the core engine of DuckDB, um, you know, because the energy and DuckDB has been focused on making a great single node engine, um, versus making a distributed system where all these things can go wrong and all these things can happen and have to shuffle all this data.
19:41Like they've been able to just build a super fast engine and, um, and that, you know, that shows, I mean, that shows, you know, like our customers notice it, you know, but if you look at like ClickBench, for example, like, you know, our, our smallest, our, you know, our standard instance, which is.
20:00$2. 40 an hour is something like five times faster than the Snowflake 2XL, which is $64 an hour. And I mean, that's Clickbench, that's one benchmark, it is a benchmark that was, you know, it's not our benchmark, it's Clickhouse's benchmark. And it's not huge amounts of data,
20:23but it's, it's, you know, largely representative of the sizes of data that a lot of people have in like, and our customers find kind of similar, similar kind of price performance, you know, trade-offs. Okay, so, so let's, let's talk about data size, because this is one of the things that
20:40I think you made a big splash about, you know, years ago, the, the small data movement. You know, you had a couple of blog posts that talked about how most data is actually not that big. And yet we architect most of our data systems around this small percentage of the data that actually is
21:02quite large. Has, is there anything you would update about that, that story years in? Is that ended up ending up being true? And how does it impact what customers you work with? And, and, you know, what use cases you can do for them? Yeah, I think so. One of the key things is,
21:26I think that there's kind of, there's these two independent axes of scale. There is the size of data that you have. And, you know, clearly there are people that have petabytes of data, you know, large data does exist. So to, to, to say that large data does not exist is kind of, it's just telling people the opposite of what they, what they actually know.
21:48But then the other axis is actually compute size. And, and so one of the interesting things is that just because you have large data doesn't necessarily mean you need large compute. And a good example is if you're looking at, you know, the last hour worth of logs, which is what often people look at what's, you know, what's happening now, or the last day worth of transactions
22:10or the last week, it doesn't actually require that much, that much compute. You don't have to scan, you might have a petabyte of logs over 10 years. If you're just looking at the most recent stuff, like you're not going to have to, you don't have to scan all of that. So you don't really need the mechanisms of big data, you know, the big, you know, the big compute mechanisms to,
22:29to, to handle that. So like, and I think the, the kind of the other, the other way that we can handle, handle the, you know, large, large data is just the, is the, the tendency model. Because very often what you have is, you know, data, you have lots of users, you have lots of end users,
22:51and their data is largely partitioned. And so we can actually, you know, with MotherDuck, you can have each user is running the separate MotherDuck instance that is looking at a different slice of the data. And so you can scale out to like actually using a large working set of data. But we are scaling to multiple, multiple instances that each one is looking at a smaller,
23:12smaller subset. The other side is big, is big compute. And I think there's, there are big compute, small data cases, the big compute, small data cases is, you know, likely your BI tool, you might have like, you know, 500 users in your company that are all using, you know, looking at the same dashboards and slicing, dicing, the same things, the amount of data that you're looking at
23:33for your, you know, for your gold tier of your BI is probably pretty small. But you need a lot of compute to be able to handle all those users. And so that's where the, you know, the read scaling that I was talking about, where we scale out to multiple, multiple instances on the on the back end. So the those three quadrants, like the small data, small compute, obviously kind of Duck TV,
23:55MotherDuck can handle that well, the big data, small compute, the small data, big compute, like those are those are things that we can handle quite well. And those are probably like 97%. Then there is the like, they're legitimate cases for I have a lot of data, and I have a lot of computation that I need to get done. And so we have two mechanisms to do this one is,
24:18well, we scale up to like these giant instances I mentioned, like, you know, these, we call them giga 192 cores, terabyte and a half of RAM, like there aren't a lot of workloads that won't fit in that. And, and so we, you know, we have, we have customers that have, that have knocked those over by the what the workload that they're doing. But generally, that's because there's a bug. And those
24:40are those kinds of things that we're working through, because Duck TV wasn't really designed with that scale in mind. And it's sort of getting better and better at at handling that scale over time. The last thing is there are some times where like, that's not enough. And so I think DuckLake is is kind of our big bet on on on being able to handle this or iceberg for that matter.
25:05But like, you know, if, if your your MotherDuck data is a is a managed DuckLake table, then we can actually give you access to the same underlying files that are sitting on S3. And so you could run that on Spark. So it's like, you know, yes, like, it's not that satisfactory from our perspective to like, hand off, you know, say, hey, we're not going to handle this workload.
25:27But it is one is like, hey, you know, because very often, it's just I need to rebuild this table, I need to do this, like, every once in a while, I need to just like, there's some there's something like, that just I have to I have to clear up all the PII and my data set or something and like, and so like, you know, Spark could be a reasonable a reasonable way to do that using using DuckLake or using using iceberg.
25:50Okay, I you're doing great at anticipating the the next question that I have for you. I was going to go to to iceberg and lakes next. And, you know, I think a lot about lakes these days, because the, you know, obviously the dbt five trend merger, we've been very,
26:13you know, out in front of what we're calling open data infrastructure, which, you know, the lake sits at the heart of that. I see data lakes as a great way to get DuckDB in the conversation.
26:33Because if your data, if your data is fundamentally accessible to any compute engine, then you can make choices really easily about how this stuff works. Do you currently see DuckDB and MotherDuck used as a, as like one of many compute options on top of data lakes?
26:56Or is that still kind of a, an unusual architecture that people haven't moved to yet? I think we're seeing worse, you know, we're seeing more and more of it, we're seeing, you know, customers that are using that are using iceberg, or they're ingesting. So I think often people do want to for their for their like gold tier analysis, you do want to have something that's a little bit more kind of compact and using using a,
27:20you know, even if you have a lot your data to typically is stored on iceberg, to keep the gold tier in something that's that's managed is, is not so is not so bad. And so we'll see people ingesting from iceberg, we do have a bunch of customers that are using that
27:37are using iceberg. And I think, you know, we see a lot of like, people have not quite wrapped their heads around, you know, how, if you're going to use iceberg, you do give up some things, you know, and like, like, there's a customer that we were working with that, you know, they're doing,
27:56you know, millions of single row updates, you know, a day. And that, like, generates, like all of this mess, and, and, and makes it super hard, and super slow. Who the hell is doing updates in their analytical database in the first place? Well, you know, Okay, I yeah, that makes sense. I think that even the iceberg creators would tell you that that's
28:21not a good pattern for for iceberg. Yeah, I mean, but like, you know, if you're moving from ice, if you're moving from snowflake, you're moving from BigQuery, like, it's like, you know, you just it's what you're, it's sort of the way you're, you know, you're streaming data in, you know, and, and you're the naive way to write it is just to do single, single row, single row updates.
28:42So we are seeing, we are seeing more and more of that. And, you know, and I think the other the other key use case, I mean, for that is like, okay, somebody wants somebody wants to try out MotherDuck, their data is a snowflake. And so what what can they do, they can, they can export the data to iceberg, reimport it back into snowflake, you know, as an iceberg managed
29:07a snowflake iceberg managed table, I don't know what the, you know, the nomenclature for that is the snowflake world. But from their snowflake workload perspective, nothing has changed, because, you know, they get on their same snowflake instance, their workloads on it. But now that data is visible, you know, it can be visible to us. And then we can we can actually run, you know, run queries on it. And if we can be, you know, dramatically less expensive and
29:29dramatically faster than than snowflake, then it's sort of a big, you know, I see this as a big opportunity for us. But, you know, I think also just, you know, most of our customers are on the smaller side. And, and so they are slower to adopt iceberg than I think some of the some of the bigger enterprise customers who are, who are worried about lock in, etc. I think a lot of the
29:53smaller customers, the reason they use iceberg is just, I mean, I would say a lot of it is just there's a lot of excitement and hype around iceberg. And they're like, Hey, I want to give this a try. Or this seems like the the cool thing to do in the way everybody everything is going. And I think one of the things they find is like kind of the, the tooling is, is behind the hype,
30:15in terms of maturity. And, you know, I'm sure that's going to change over time. But you know, it does it does slow down some of that that adoption. We've done a lot of like core Duck TV, MotherDuck, I want to move into a post you wrote, you wrote this post, that is is just like a podcaster's dream, because you make about 100
30:37predictions in it. And each one of them has has good things to fight about. So I want to, I want to read back to you, one of your paragraphs, and then I have a question for you about it. So here's your paragraph that you wrote. ETL is highly vibe codable, extract,
30:59transform, load or extract load transform pipelines are generally generally contained fairly straightforward code that will be easy for an LLM to generate. There are good open source connectors. And even if they were not, LLMs can consume documentation for an API and build a connector fairly easily. The transformations themselves are typically relatively straightforward and can be usually specified in SQL. The ingestion and transformation side of the modern data stack
31:23would seem to be ripe for disruption. So here's my question for you. Screw you, Jordan. I think that there's stuff in there to agree with. I want to give you an opportunity to say a little bit more because before I start telling you why I disagree with parts of it.
31:46Well, I thought you might, you know, be interested in that, the things, you know, I mean, like, so just as a, from a journey perspective, like, we, you know, we launched our MCP server in, in, in December. And this is a little bit around the way, roundabout way of getting,
32:08getting to your question. But, and we found that like, all of a sudden, like, you could just, you know, in Claude, ask questions and get answers. And then the other thing we noticed was like, like, holy shit, Claude is really good at building data visualizations. Like, yeah, you did dives. You did answer. Was it called answering machine?
32:31Yeah. So we could, we launched something called dive. So it was like, well, after we realized that we're like, Hey, like, well, what does it take to become a BI tool? What does it take to sort of add BI, you know, BI support or at least add visualization support. And we're like, well, you know, the things that Claude comes up with are actually, you know, are the data's contained. It's not, it's not, it's not updated. It's not hosted. It's not shareable. And so we said,
32:56okay, what if we replace the data that Claude, you know, dumped into this TypeScript file with a SQL query hosted on a MotherDuck. And now you have, you know, basically a dashboard. And that was sort of the root of our dive product. Like we started out saying, Hey, this is not BI. This is sort of a narrow, narrow use case. And then as we started to realize,
33:18like, wow, this is like, this is like, it was harder to draw the line between what we were doing in BI and realizing that we had really stopped using our internal BI tool. We were just using this for everything. And like our head of customer success was using it. Our sales team was using it. Our, like our support team was using it. Everybody kind of at the company was,
33:39was, was using it like crazy. And then we said, well, what are the other, what are the other sort of adjacencies that, that are, that are kind of the Claude is good at. And I actually, I had talked to somebody who was, you know, thinking about starting a, thinking about starting a company and he'd built like this data,
33:58this basically vibe coded data ingestion solution. And, and the thing that shocked me was it was running in Claude. They had no front end, they had no UI. They had like their whole company was an MCP server and they could do, they could pull data from a, from a source. And then it would,
34:20it would show these mermaid diagrams that were the, the, your schema and you could approve the schema that it was going to translate things to. And then it would, you know, it would load the, it would load the data. And, and it was like, wow, this is, this is amazing. Like, and, and you can, you know, and this was in September, October, this is before the models, you know,
34:43got, got a lot better. And so it's like, it, it, it seemed to me that like there, there was a lot of opportunity for, you know, for LLMs to be building, building data pipelines. And, and, you know, via, you know, from, from natural language. And then you also look at like,
35:05you know, yeah, the ability to say, if you wanted to say, Hey, pull my data from HubSpot, you know, like the LLM could figure out how to do that, you know, either from, you know, an open source connector or from the, the, you know, HubSpot API, API documentation. And so it did seem like there is the, that side of that side of the stack is sort of
35:29at least open to a lot of, you know, a lot of, a lot of change and a lot of people doing things, doing things differently. I realized that like to productionize it and, you know, make things actually, actually work and work over time that there's a lot, there's a lot more, I mean, just like there's more to software engineering than writing code. I think there's a lot more to
35:48data engineering than sort of building your data pipeline. And so like, I think that's, that's where it starts to get, to get interesting and figuring out what's going to actually happen. The world that like everything is in right now in, in the data ecosystem, because the,
36:08the sources of competitive advantage or the sources of like your, your moat there are changing super quickly. And I, I totally agree. It actually reminded me of it when you were talking before about, you know, you said, I feel like we have, you know, we got a headstart versus, you know,
36:30DuckDB, the open source project. And if we can't manage to stay ahead and to use that, that headstart, then like we deserve whatever we're going to get. And in the same way, I think that, you know, whether you're, you're open source or whether, you know, whatever, whatever the kind
36:51of original software era advantage you might've had was like, it buys you time, but it doesn't buy you like infinite time. And so, so I think that it is, it is totally fair to say that data
37:11pipelines of 2030 are not going to look identical to data pipelines of, of 2020. And we're going to need to, you know, figure out what agents want when, when they build data pipelines and monitor
37:29them and update them with changes and everything. What is, what is, what do agents want out of an analytical database? What's your, what's your best hypothesis? You know, I, you know, this is actually what my board asked me at the last, the last meeting,
37:50actually, that was what they asked is like, is that what they asked was like, how do you make it so that, you know, agents are going to use your, your database versus others. And that's something that I've been sort of, we've been trying to figure out and I, and I wish I had a great exam, a great answer. Because I think you, people have seen the success of, of Neon and
38:14Superbase. I just spun up a Neon database the other day. On the, yeah, on the Postgres side, you know, to, to interact with agents and like agents are, you know, using those like crazy. But if you think about like, like agents, they need to store data somewhere. And Postgres is a great way to, great way to do that. And a lot of it can be ephemeral, a lot of, you know, and a lot
38:38of it can, you know, you're building an application and the application, you know, needs a database, and you're going to spin up one using, using one of these tools. It is a really interesting question, why would you need an analytical database? And that I think is a bit more, is a bit more hand wavy to me, like the, yeah, I think there will be cases where, you know, the agent needs to
39:02interact with larger amounts of data, the agent needs to do kind of aggregations, the agent needs to kind of answer, answer some harder questions. And, and so I think that that's, I think that that's going to happen. And we do see it, we do see it happen. We do have, we do have agent-based use case. We have actually have a lot of users building agent platforms on top of MotherDuck.
39:27Oh, fascinating. And so that's, you know, one of the, Airbyte just announced their like agent platform and, and it uses us, us under the covers. And, and, you know, we have, there's a handful of other companies that are doing, are doing, doing similar, similar things to that. But there isn't the same sort of like, oh yeah, agents are going to just start hammering snowflake instances.
39:50I wish there were, because like our architecture is amazing for agents, because if you have like a hundred agents and your agents are branching and like, like you know, our tenancy model works really nicely with that. And we were like, you know, like, I don't, if you just think about it, like, if you're, if your agents are hammering snowflake, then like, that, you know, sounds like an incredibly expensive thing to have them do.
40:16And, and so I want there to be these agent use cases, and I think we would be the right database for it. I don't know what those would be. I'm interested in your, actually, your thoughts on it. Have you seen analytical use cases for, for agents, other than agents that are involved in the analytics process, which I think is actually also a really interesting way of,
40:40of thinking about it? Okay, so I'm so glad that you differentiate those two, because a lot of times people in the data world think about agents as being synonymous with, like, conversational analytics agents, where, like, a human asks a question, an agent, whatever, writes a bunch of SQL, runs queries, whatever, and then gives them an answer back. But, but that is,
41:04like, kind of only the most basic, obvious agentic flow. And in fact, I really believe that over time, many more analytic queries will get written by agents that own a business process,
41:24like insurance claim review, or whatever, than agents that are, like, specifically acting as a data analyst and, like, responding to a human-initiated query. And, and I, I totally agree
41:44that, that DuckDB's architecture has a, is, like, very compelling from an agentic standpoint. I think one of the, you know, you talked about some of the maybe more sophisticated advantages, but I think a lot of times the biggest advantage is the dumbest one. It's just that agents all of
42:04a sudden really make everything about the local environment. And, you know, your, your agent wants to own a computer to do stuff with, and it then wants to interact with software that it can install itself. Like, you know, how many things has Claude Code asked you to brew install? And, and, and so
42:29the fact that you can have your own environment with your own analytic database that runs locally, and, and then, you know, you go through the productionization process and you, you, you know, move to a hosted scalable service, great. But, you know, like brew install BigQuery is not a thing.
42:49Yeah. Yeah. I mean, it's a great point. I think, and I, and like, not to like toot our own horns again, but I think like the, the, the way we, we architect working with DuckDB is our client is DuckDB. So if, if an agent decides to install DuckDB and it's doing a bunch of stuff locally one of the things you kind of find out quickly, if you're running, you're running stuff locally is
43:10like it's very easy to use all the compute on your machine, however much you have and use all the memory on your machine, no matter how much you have. And our architecture means that the, the steps needed from going to the, from this local DuckDB operation to Claude MotherDuck operation is it's just a change the name of your database. Because if the name of the database starts with
43:34MD colon, then it runs in the Claude. If it doesn't, it starts, it runs, it runs locally. So you don't have to install anything differently. As part of the, I mean, that's one of the things, the advantages that we have of working with DuckDB labs is that we sort of get this sort of built in, built in support. And so, yes, if the agent, if there's agents that are building,
43:53that are doing stuff locally with DuckDB, there is actually a great then kind of graduation to, to MotherDuck case where it's like, Hey, this is just too slow. It's pulling all this stuff down locally, or the container that I'm using is, is small and only has one CPU. Cause it was supposed to just be an orchestrator. Like it's really easy to just sort of push that off into
44:12the, into MotherDuck. Gosh, everybody fricking hates it when CEOs say crap like this, but I've, I've been doing a lot of building with Claude Code recently. And, you know, you, you go through the transition very quickly where like, you know, everything starts local. And then, you know, I was doing all my inference in Olama until recently. And then you realize that like,
44:37I mean, like my local Mac mini just doesn't have that much memory and it's just not that fast. And so you like end up relying on API services as you, as you scale it out. And so the, the I think the continuity, like the, the, the ability to like make decisions about when you go Claude and start, you know, swiping a credit card is a real advantage. Okay. There's, there's another
44:58thing you said in this post and this is a little bit less of a gotcha. But I wanted to see if you had more to say about this, because I don't totally get it. You said when it all shakes out, my bet is that there ends up being one form factor that people settle on. It will consist of an agent swarm for data management backed by a query engine for doing the actual analytics.
45:22Agents can handle change and adapt the system in real time. They can prepare insights directly for users. What is it specifically the term agent swarm? Like what, what, what, when, when you wrote this, like what's, what's the picture that lives in your head? Um, so I did, I actually wrote a, a followup to this one. Um, I called it Watertown. Um,
45:46Oh yeah. I didn't get to read that whole post, but it's, it looks fascinating. A takeoff of Gastown, you know, which, you know, the, with sort of the agent swarm for, you know, like, how do you, how do you run like lots of Claude Codes? Um, you know, you kind of need some, somebody to keep track of them. And, um, and, and so one of the ideas
46:05is really that, um, like as your data, as your data is coming in, there's going to be agents that are going to like do quality control. They're going to run, there's agents that are going to run evals, uh, that, that detects like something is something is goofy. Um, I was talking to somebody
46:26from open AI, uh, a couple of, a couple of days ago, and I was kind of asking them how they dealt with context. Cause context is a big sort of hot button, hot button issue. And one of the things that is for core, for core things, for core concepts, they basically turned their con context meeting. Like when I say revenue, what I mean is here's
46:45the calculation. Um, when I say, you know, this, here's the, here's the calculation, like, or these two tables should, should be joinable one-to-one. Um, and, uh, and what they do is they have evals for all of those. I mean, and so like, so making sure that when I say, you know, tell me
47:05the revenue on February 1st, that it's exactly this number and that, you know, cause it's agents are going to, you know, LLMs are going to be interpreting it and then turning it into something you want to make sure that like, do you always get the same number? And so, um, and that's operationalizable by an agent that can validate that that never changes. And so I think there's,
47:26so there's agents that sort of do data governance and then they can data observability and they can sort of flag, flag what's happening. Um, I think then there's agents that also will can look at actually what's happening and then can add their own context and their own context and saying like, Hey, look, the, you know, this, this field is always a U S state name that is capitalized.
47:49Like that's like, that is information that, that it can, that it can, that it can glean, that can be useful for, you know, anytime you need to join, anytime you need to look something up. Um, and, uh, you know, I think there's on the other side, then there's, there are agents that like, they can, you know, look at the chat transcripts. Like when I talk to Claude, I tell it,
48:12you know, I often say like, you know, I want to know what's happening, you know, with our paying users are paying users. And what that means is they're in either the, you know, the capacity plan, the business plan or the light plant. Um, I just gave that the agent, and I just gave Claude information that could be captured and, uh, and used so that the next time somebody else
48:35might not even be me ask that same asks, like, tell me about paying users. It should be able to automatically know that. So I think there's another agents that are, that's going to basically be taking that curating context, taking, you know, things from different users, collapsing, collapsing those. Um, uh, and, uh, uh, I, I think, you know, I think Anthropic is
48:57sort of calling this dreaming where it's taking like memory and just sort of distilling it in terms of what is actually active, active memory, which I think is cool, which is a cool name. Um, and then, so I think like, there's just going to be, and then there's going to be agents that are going to be sort of figuring out what it thinks you might like, you know, it's like, Hey, what's going on? Like there's, you know, revenue shot up today, you know, like I look at our revenue
49:21dashboards all the time and like, and like every once in a while it'll go like this. And like, sometimes that's a good thing. And sometimes it's actually not because it's like somebody, somebody sort of doing something that they hadn't intended. And you kind of also want to keep an eye on, keep an eye on those things. And so like, um, that's the kind of thing that an agent could figure out, be able to like, to show me when those kinds of things happens.
49:44So that's sort of the agent swarm idea that I was thinking is like, is that I think that, um, there's going to be a mechanism that, and I think it's, it's like, it's important that that's a single to single entity or a single, like, because I think there's like, there's communication and shared and shared knowledge and shared context across all those cross organizations. So I think
50:08most likely, or at least to start out with is probably going to be somebody who's going to provider is going to come up with that. I think a lot of companies are going to come up, you know, are going to build pieces of that on their own because they're just going to realize, Hey, like, I can have an agent that, you know, continually runs queries against my analytical databases to
50:26make sure that certain invariants are never, are never violated. Um, and, and that these things that come from our, our context, which we have written as like, these are the things that like the metrics that we have for our company, that those are, that those, that those stay, that those stay valid because, you know, people hate, people hate, you know, curation of things
50:49as much as they hate writing in the first place. They really are very bad at keeping it up to date, but I think this is, this is that also agents, agents can help out with. So that was a little bit of a long-winded answer to, uh, to, to that, to that question, but, uh, hopefully it helps. I did not get that picture from just reading that, that one paragraph, but, but, uh, hearing
51:08you talk more about it is fascinating. I, I really agree. Um, I think that, you know, my mental model for it is like, you know, I think Benedict Evans, I don't know, a couple of years ago said something like, you know, generative AI is like having a thousand interns. Um, and,
51:27and, you know, here it's, it's essential, like imagine that you had a thousand data analysts and that they could like literally look at every table every day, every column in every table, and just like imagine all the ways that it could break or do weird things. And they could also look at every single metric and your entire, uh, semantic layer. They like, you can just have
51:52enough eyeballs to like have attention on everything. And the, um, the thing, this kind of loops back around to a thing that we've been talking about the entire time, but like that drives a tremendous amount of usage, uh, both of like inference, you're going to be spending a lot
52:15on tokens in that world. And hopefully, uh, you know, Quinn's getting real cheap and, uh, you, you know, we can, we can run a ton of inference like that. Um, but it's also going to, like in, in many ways, you can imagine tokens getting a lot more, a lot cheaper, a lot faster
52:36than the physics of like running table scans. Like we've, we've kind of figured out how to do table scans at least reasonably well. And DuckDB has got, you know, a lot of efficiencies there, but you've, you've just got to be able to expect to pay less money than we currently are. If you want to have that type of capability inside of your data infrastructure.
53:01Yeah, I, uh, a hundred percent. And I think, you know, there's, you know, the Jeevans paradox where like, when stuff gets expensive, you find more stuff to do with it. And that may be a case where it's like, Hey, we can, we can dramatically make this, you know, make analytics less expensive and move more locally. And, and, uh, um, and then, but I think people will, they'll find, they'll find more ways to sort of keep their, keep their bills similarly
53:26similar and, or even, or even have it go up. But I think the, the good part is, you know, you're adding value. Like there's you're adding value. You're like, you're not going to have like as much of, you know, a user trying to, you know, a human trying to debug what the hell is going on on with like, why is this, why is this dashboard showing this weird number? Um, it's,
53:48you know, it's going to be, the agents will have flagged something, you know, well in advance and, or you can even ask the agent, he was like, why, where did this weird number come from? And it's going to be able to sort of look at your pipelines and look at things. And it's like, Oh, well, here's what's actually, here's what's actually going on. I think that's a good place to leave it, Jordan. Thank you so much for joining me. This has been a super fun, taking a journey through MotherDuck history and talking about where
54:13this crazy world is headed. Thanks, Tristan. It's really great to chat with you.
