0:01So I just arrived in Brussels for the second edition of the DuckCon conference and I'm going to take the opportunity to ask a few questions to the creator of DuckDB. DuckDB
0:25It's very great to see you in person in this beautiful weather in Brussels. What was going on in your mind before the first commit of DuckDB? So DuckDB is of course coming out of our research at the CWI, the Dutch Research Institute for Computer Science and Mathematics, and where I have been working for the last 10 years. And I've been working in a group called Database Architectures.
0:49And in that group, you know, we do research on how data systems should be built, you know, maybe think a bit about the trade-offs that we face, the architecture, the general sort of experience. And there was this interesting thing that happened when we started talking to some data practitioners, specifically the R community, by the way, who are very friendly and I'm still really appreciative that they were so nice to me.
1:11It turns out that they weren't using databases at all, right? Like they weren't, they were storing their data in CSV files. They were like, some of them had like hand-rolled data frame engines on top of like these things. And they had these data frames in memory, they were running stuff on that. And it was all pretty slow and limited because of, yeah, the way these engines were structured.
1:34And they really didn't like to use databases. And I was really confused by that because I really love databases. And I thought, okay, the R people are working a lot with data. They're facing real data issues like stuff being too big. Why are they not using our stuff, right? And then we started thinking and said, okay, the first big problem is that pulling data between a database and your application program, in this case, an R shell
1:58where you maybe undo analysis, way too slow. Like the client protocols of databases, you know, built in the 80s have never been touched. Row-based, hardcore serialization stuff, not great. So we fixed that and then we showed this to the R people and they said, yeah, this is better, but we still hate managing database servers because the way, you know, if you think about it, client-server setup, like you're running your R thing
2:22and then you're running your like database next to it. And that's a setup issue. You know, you can make this work on your laptop maybe, but you want to send a script to somebody else and it won't run because, you know, they don't have the same setup and then they have to set it up and it's a nightmare to configure these things. Maybe we need to do something like SQLite where there is actually no database server, but the database is in process.
2:46It's a library. It's just linked to the thing. But for analytics, because SQLite is an amazing system, it doesn't do analytical workloads. It does transactional workloads. And that didn't exist. It turns out we invented a whole new class of database systems. It was the in-process analytics thing, which of course is good if you're doing research in this sort of thing. So what we then did was we took another database system and we hacked around in it to make a prototype.
3:10It was called MonoDB Lite. And the MonoDB Lite prototype was an in-process analytical database system, SpeakSQL, and kind of worked. It worked okay enough to get people excited about it. And so we put it out. We promoted it a bit in the R community. People started using it. People got excited. But we ran into some architectural issues there.
3:34For example, if you are running in-process, it's a much more harsh environment. You cannot just exit the process if things go wrong because you're running in somebody else's process. If you exit that, they're going to be very unhappy with you. You have a lot of other requirements like this where you cannot use certain APIs because they change some state in the program that the other thing also looks at, like the locale
3:58or the working directory. It's really trivial stuff, but it adds up. Taking all this out of an existing system proved to be quite hard. reliability issues there. And I felt encouraged to do whatever I thought was best. I decided that I'm gonna, that I'm gonna start with Mark together to build a new database system, which is a monumental undertaking.
4:22It doesn't happen very often and it doesn't, it's not successful 99% of the time because it's a monumental undertaking. And it's actually not great for your career as a researcher because you disappear for five years, hacking around and not writing papers, which is what you're supposed to do. But we kind of knew that we wanted, like what we wanted, right? Because we had this prototype, we knew what was wrong with the prototype. So we knew how to do differently.
4:45There was kind of already a product market fit. We had already seen that this is something that people get excited about. That was really encouraging. But of course it was also really frustrating because getting back to where the prototype was took a long time because you start from scratch. And also, so you have, you know, like SQL is a complex language. Supporting like all these detailed sort of operators,
5:09like it was quite, it was quite a lot of work. And yeah, Mark and I basically did nothing else, evenings, weekends, just hacking on DuckDB for a couple of years. And then we, I think we announced it in, maybe we open sourced it in 2019 and we got a lot of positive feedback, of course. I think also immediately when we announced it, there was also just a lot of positive feedback and people got excited.
5:34The design and sort of the inspiration of DuckDB really came from talking to data practitioners. Yeah. So to recap, it came from your, the AI community struggling with, you know, just local file and anything to work with database. Yeah. And so you end up making a database for them. We, in a way, yes. And I think it's kind of interesting
5:57because we made, it's true, it's fair to say we made it for them. Indeed, we did kind of design it around these interactions with the R people. And yeah, I think that's generally a good idea because you, you know, you have a real problem to solve. You're not just designing into thin air. They will tell you whether they like it or not. And I'm still very thankful for that
6:20because like there were lots of people in that community that were willing to try the crazy stuff that we hacked together that was barely working and gave us like feedback and said, okay, this is really exciting or this doesn't work or. The thing is this kind of process is sometimes missing in the academic world. Yeah, that's true. It's rare. Yeah. It's a, it's, that's a good point. How is, why is that? Why, why did we actually start going talking to people about this?
6:44That's, that's, it's a bit of an accident, baby. It's not supposed to happen. It's maybe we were just weirdos. And, but for me, I always, you know, for me as a, as a researcher, my definition of success is not to write papers. It's to have impact. And how do you have impact? Well, especially in data systems research, you have impact by making systems. I mean, we have a Turing Award in,
7:07in computer science for Mike Stonebraker, not because he writes amazing theoretical paper, but because he made systems like Ingress and Postgres. So I think in this particular space, it is actually, how should I say, it's actually required to, to, to see, you know, to make something that we'll see widespread use in order to achieve impact. And that's exactly what we're doing. Quack.
