Big Data: Funeral or Renaissance?

0:01Welcome, everyone. My name is Aaron Elmore. I am an associate professor of computer science at the University of Chicago and moderating today's debate on if big data is dead. It's a funeral or renaissance. So we're joined with Jordan Tagani, who is the co-founder and CEO of MotherDuck, and Ditsia Parameshwan, who is a co-founder of Ponder and associate professor at the University of California, Berkeley. So we're gonna start off with some brief introductions from the folks.

0:25They're gonna tell a little bit about their background, kind of what big data means to them, and they're going to summarize the respective blog posts that led to this debate. So since Jordan had the post that started this, we'll have Jordan go ahead and kick us off. Thanks, thanks, Aaron. Very excited to be here with you, Ditsia, as well. So yeah, I'm Jordan Tagani. You know, I've worked for a long time on Google BigQuery

0:49kind of at the, you know, outside of the big data era, and I think we capitalized on sort of a lot of the momentum behind that. I started MotherDuck in large part because we saw that, like, the people that were using BigQuery, many of them didn't actually have big data, and that everybody's sort of focusing on these giant data sizes

1:11that may not be kind of what everybody has. Kind of just to summarize the blog post, you know, really, like, most people don't actually have that much data, you know, since I posted, since I wrote the post, a lot of people have been sort of coming out of the woodwork saying, like, you know, this was incredibly validating because as a data engineer or data scientist

1:35or data analyst, like, I've sort of felt almost like I've had this, like, inferiority complex because, like, I wasn't using billions or trillions of rows of data, and, you know, I would go to these conferences and where they talk about the Netflix architecture, and I'd come home and be like, yeah, I'd be super excited, and then I'm like, but what am I gonna do? Run this all on a one-node Spark cluster. And so, you know, for a bunch of reasons

1:59that people don't actually have big data. Kind of there was an interesting part of that, which is that, you know, like, separation of storage and compute really split big data, so what used to be sort of one term into sort of big storage and big compute. And big storage is certainly a thing that people definitely, like, accumulate a lot of data over time,

2:23but I think big compute is really sort of what we're talking about, that, like, actually, for the most part, you don't need to have giant amounts of compute, and when you kind of have a modern, you know, modern data system that has, you know, good metadata management, for example, you end up, like, even if the data size is very large, you end up only reading a smaller proportion of that. Like, you know, and one of the things we noticed on BigQuery

2:46was that 90% of queries were sub-100 megabytes, and, you know, really the, like, people always talk about what their data warehouse size is, which is sort of the total data storage, but that's almost irrelevant for, like, for actually what they need to do. It's really the workload sizes, and the workload sizes tend to be much, much smaller, particularly when you're dealing with sort of columnar storage and, you know, properly partitioned data.

3:11You know, one of the reasons that those work so well is that there's a real recency bias in terms of how data is accessed. Sort of the vast majority of the data that's accessed is really from the last day, last seven days, last 14 days, maybe the last 30 days, and then really after that, it really, really gets quiet. And, you know, one of the things

3:33that's really kind of pushing this is Moore's law. So when big data started being talked about in, you know, the early, you know, 2000s, you know, a single machine might only have, like, one core and a couple of gigs of RAM. Now, you know, you can get pretty basic instances

3:56for not that much money that are, you know, two orders of magnitude more than that. And Moore's law has not been slowing down. It's really, since, you know, in those 20 years, the amount of hardware on a single chip has gone up by three orders of magnitude. So, like, the threshold for what has been big data

4:18has been changing and is going to continue to change. So if it's something that's like five, you know, maybe you need 10 servers now in five years, chances are you'll be able to do that on one. And then just the last part is, you know, data being a liability that, you know, often people don't know what data they have. So they're like, well, I need big data because I have all this stuff sitting around

4:43and, you know, better inventorying, better data management processes can avoid, you know, spending, because the thing is, if your data sizes are increasing exponentially, your costs are also increasing exponentially because the cost of storage, at least on like, you know, public cloud object stores is not actually going down. And so the only way to prevent that is to really pare down the amount of data that you're keeping.

5:08So yeah, that's sort of the summary, you know, chances are, you know, yes, there are some people that have big data and big compute needs, but for the most part, that's, you know, that's not what most people need and they would likely be better off optimizing their systems and designing their systems for the usability of the data rather than the size.

5:33Okay, great. Thanks, Jordan. All right, so let's let Aditya kind of again introduce himself, define what big data means to him and go ahead and kind of summarize his rebuttal to Jordan's post. Awesome. So I'm Aditya Parmeshan. I'm a co-founder at Ponder, also a professor at Berkeley. I've been part of the database research community

5:54for a while, kind of seen trends come and go and big data was a trend or is a trend. And I think my humble plea to Jordan and others is let's not kill the big data buzzword because who knows what will emerge in its place,

6:16the reverse ETL analytical data mesh, maybe. And let's not introduce a new buzzword. So that's my humble plea. So thanks to Jordan for writing a thought-provoking blog post and me as an academic, I couldn't resist like putting all my fingers in there and trying to tease it apart and trying to tear down the argument.

6:40So let me try to make my sort of claim. So the broad nature of my argument is that organizations have big data needs, medium data needs and small data needs. And the key is to be able to be flexible to those organizational needs. And so we should embrace and support kind of the duck-sized workloads

7:03as well as the elephant-sized workloads, right? So let's not prefer one over the other. So responding more explicitly to some of the points that Jordan raised in his blog post and what he described just now, the first point, which is that most people don't have that much data. And I don't necessarily agree with that, but if you think about customer data

7:28or if you think about transactional data, fine, maybe your data isn't big, but the present world is highly instrumented, highly interconnected, even like a small, simple website tracks thousands of user actions across like hundreds of attributes, right? So very quickly it blows up. And if you look at most domains, healthcare, finance, what have you, they have this sort of operational interactions data.

7:51So it's no longer about transactions, it's about interactions, right? So why wouldn't you wanna leverage all of this data for high quality, high fidelity insights or models? The next point that Jordan brought up is like the storage versus compute bit. And I think there's a little bit more nuance than what he mentioned,

8:14which is that even if the average workload's compute needs are small, it doesn't mean that all of the workload compute needs are small. So in fact, a lot of the important workloads compute needs are not small and you want the flexibility to be able to scale up and down depending on the workload. Simply saying, hey, I'm just gonna get a beefy machine and be done with it doesn't quite work because you're at that point, you're ending up doing the,

8:39should you run your own servers versus should you use the cloud argument? And for a lot of organizations, the cloud is the right answer. The ability to flexibly scale up and down is the right answer rather than simply like provisioning upfront and what are usually pretty expensive servers. The next point that I think Jordan brought up is most data isn't queried, right?

9:01And I think there is definitely a recency bias to data, like to querying data and recent data is queried more, but as you're learning with ML and data science, you kind of need to look at all of the data and looking at more data and throwing more compute is gonna lead to better accuracies, right? I mean, if there's one thing that you learned from the chat GPTs of the world

9:24is that they are revolutionizing how we do knowledge work and they wouldn't exist if we hadn't thrown more data and compute. So, and at a much smaller scale, of course, this, we're not talking about those doing massive, massive compute and data at the problem like OpenAI might, this is at a much smaller scale holding true in individual enterprises as well.

9:47So it's better to have the flexibility to scale up and down and for at least a high value workload to be able to operate on as much data as possible. And then I think the final point is around kind of the single machine versus a big beefy cluster. Often you'll find that a lot of the beefy machines

10:10that you might need to provision to run your workload currently is actually pretty expensive. Even with Moore's Law, it's currently pretty expensive. And there are often other workloads that require, that won't fit in that single machine. So you need to provision a cluster of machines. So you end up with the same cloud sized problem, right? Like do you provision your own expensive resources

10:32or do you allow for flexible scaling up and down? And so I think to summarize, I mean, all organizations have small data needs, medium data needs, big data needs on a day-to-day kind of on a query-to-query basis. And we should instead be working on tools that allow you to switch between these modes, flexibly scale up and down, allow users to get insights and build models

10:55and so on, independent of data size. Okay, great. So we can get into questions then, and then we can loss them back and forth, depending on the question, but you know, so I kind of came up around the same time as Aditya through as XML is decaying inside of the database community and web services were decaying and this big data thing.

11:20In my opinion, a lot of it was energized by Google, Amazon, these promises of these kind of like large shops that could run at web scale, right? And part of the traditional definition that I was given for big data or people, maybe this was just the marketing hype was the three Vs of big data, right? So we have velocity, meaning the data coming in and coming in at a really high rate. You've got the volume, which we're talking about a lot

11:44already with the scale of the data and then variety. It's heterogeneous data, it doesn't always, it's not always structured, it doesn't always fit into relational data model. So, you know, you both have talked about kind of like storage and compute, but so is it that you think some of the Vs aren't irrelevant, you know, are no longer relevant, they are irrelevant today. Is it just that we have systems that address these needs so it's not as much of a concern?

12:07And then possibly, you know, maybe going in and then, you know, articulating, you know, more clearly, how would you actually define what big data means in 2023? So Aditya, since you're unmuted, let's start with you. Sure, so I think that like at least two of the three Vs, the volume and velocity needs still exist today, right?

12:31And I think you do, you can't get around leveraging quote unquote, traditional big data tools to address them, right? Like, and for example, if you have a firehose of information, you absolutely need big data tools to be able to handle that. And I think that it's, since you mentioned kind of the big data hype of 2010 and the NoSQL movement,

12:57I think one of the lessons of the NoSQL movement was not that, and the fact that it kind of died down and so a lot of the popular NoSQL systems no longer are as popular or as prevalent today. I think the main lesson to be taken from that is the fact that SQL is incredibly popular.

13:20People will still love to write SQL or most other querying languages. It's kind of a staying power as I think Mike Stonebaker called it, it's intergalactic data speak, right? Like that's what it is now. And I think Pandas, the reason why Pandas exists is because Pandas is similar, right? It's like 25, spoken by 25% of developers

13:44and we believe that it's not going anywhere anytime soon. And so I think as kind of these systems died out, what emerged was systems that did support SQL but could operate on larger data sets. You have the sparks of the world supporting larger data sets and also supporting data sets without schema, right?

14:06So being able to support the variety of the other, the heterogeneous JSON shaped data sets and you had systems like Postgres and MySQL being able to support that. So yeah, broadly, I think the three V's are still incredibly relevant and it's very hard to address them without big data systems.

14:31Cool, so yeah, I'm glad you described what the three V's were because when I heard that question, I'm like Vim, Vigor and Verve. But I guess the first one that, the one that I mostly talk about when I'm talking about big data is volume and how much data do you actually have? And that's mostly what I addressed the post out.

14:53I feel like the variety is less important from a big data perspective because nowadays even Postgres supports JSON and kind of nested queries with arrays and that's becoming popular in all sorts of data systems. Certainly it's in DocDB.

15:19The velocity I think is the one that's still quite interesting. It's like if there are a number of places where you have these fire hoses that have massive amounts of data and you have to figure out how are you going to land that data? I think that there are some non big data-ish solutions

15:41to that that I think may be actually more interesting and actually more scalable from certainly a cost perspective, which kind of move the analysis, the filtering and analysis of that data out to where the data is being generated rather than having to sort of pull it all into a single centralized location. But certainly that is kind of another interesting area

16:05of where you might still need to scale out. But that at least also is kind of a smaller area of the overall kind of compute landscape. And the other thing is it also just sort of leads to additional recency bias because a lot of the computation needed on that

16:30is the stuff that is happening right now. So yeah, great. I remember I think it was Postgres 11 or 12 when they did JSON as a data type, there was the joke going around that they implemented Mongo as a feature. But yeah, so do you think that some of the velocity then is just addressed by the tooling and the services that are offered today? You have things like Kafka, you've got the largest fire hose possible at S3

16:55and dump it in there. And so that's just mitigated by these services and therefore it's less of a concern? Yeah, I think so. But I do think it's still an interesting area because if you do need to process the data as fast as it's coming in and react to it as fast as it's coming, which I think is important, you may actually have to scale out

17:19because you just don't have any options. Anything that you'd like to add on that? Yeah, I think to Jordan's point about being able to filter and kind of subsample or summarize or extract the high value nuggets as the data is being generated and then landing those in a data warehouse.

17:43I think that's pretty difficult in my experience because you often don't have a foresight into what you're gonna be doing with your data, right? And this is why you have the so-called whatever lake house paradigm emerging as something that people are doing, right? Because you store whatever data you're generating, you don't know what you're gonna be using it for.

18:06So you can't really kind of get rid of most of it or summarize it or filter it. And therefore you kind of have to store all of it. And then depending on the query, you look at some of it or all of it and so on. So I think like the volume and velocity problem,

18:27what I'm seeing right now is people are basically saying, hey, I can't really deal with either of those at the moment. So I'm just gonna store it. Storage is relatively cheap and then figure out how to extract, transform it over time, right? So that's what I'm seeing emerges as a paradigm. Well, yeah, I mean, the storage being cheap might depend on who you talk to, right?

18:51It seems like it's that first gigabyte's free kid. Come on over here. Let's get it in. So we had a couple of questions that were sent in via video. So we had one from Andy Pablo over at CMU. So let's go ahead and listen to that and we'll have Jordan respond first to it. Hey, this is Andy Pablo from Carnegie Mellon University, Autotune and Off-Prohibition Productions. My question is, do we think big data systems

19:15are unnecessary at this point because they're economically infeasible? You know, the vendors charge too much and people are spending on them. Or do we think we've just got to the point where single node systems are fast enough and you don't need large scale-out massively parallel systems? Thanks, see ya. It's a good question.

19:37I think, you know, the systems that we have that are scale-out certainly are expensive. In theory, they're expensive because they're providing value commensurate with the cost. But I do think that it's that, you know, they're not necessary. And when you don't have, when you scale-up rather than scale-out.

20:00You know, your systems are so much less complex. You know, every, every programmer wants to implement Paxos, or their own version of Paxos. But you know, if you're not, if you don't, if you if you're just scaling up, you know, you really, you really don't, you don't need that you don't need, you don't need to build shuffle, you don't need to build like, there's just a lot of things that you know, your life gets gets easier.

20:28And, and so I think you can move faster and you can focus on, you know, the things that actually matter rather than like, you know, sort of the, the complexities of dealing with a with a with a scale out system. So I think that we can build more, you know, more interesting systems if we if we scale up rather than scale out.

20:52Anything that you want to add or counter?

21:22Yeah, absolutely.

21:53And so wouldn't you rather just hand it off to someone who does it for you flexibly scales up and down for you. So for a lot of organizations like that, that decision would be a duh, yes, let's have someone else flexibly scale up and down rather than me managing these servers and provisioning for it.

22:10The other cost argument, which may change in the in the future, I agree, is that these these beefy servers that you might want to provide might want to buy in house or reserve for your data analysts or business analysts or data scientists are still extremely expensive. So you're like, in fact, in many cases, the ability to flexibly scale up and down might actually be a lot cheaper than saying, Hey, I'm going to get one beefy server for each of my business.

22:38So if I could just address that, you know, quickly, you know, the you know, the amount of hardware that's in a Snowflake XL is, I think, one hundred and twenty eight, one hundred and twenty eight cores.

22:51The there's an eye for eye instance in AWS that I think is eleven dollars an hour and Snowflake charges roughly forty eight dollars an hour. So you consider there's like sort of the SAS markup on top of that, you know, it's sort of it's not incredibly expensive to be able to to to run that yourself.

23:12And that's, you know, huge. It's also a terabyte of terabyte of RAM, like pretty, you know, there's there's not a lot of workloads that you can't run on a Snowflake XL.

23:23I think most people I think it's about a half million dollars a year, almost four hundred K a year for that. So and I think the other thing is, just as Moore's law increase, you know, keeps going up like, you know, OK, another two years, that's going to double another two years, that's going to double like, you know, another two years, you'll be able to get equivalent of a two XL and then two more years, you know, three XL.

23:45And so the number the amount of workloads that require more than that, I think, are are going to be diminishing. And so as a comment on one thing you said, I was curious about your thoughts on like, you know, this push for repatriation back from the cloud and on prem. Is that something that you both believe or you think it's, you know, it's not going to be worth it for most organizations to have to manage that on their own?

24:09You know, is the cost savings going to be worth it? It looks like we got in a question up here. Cloud computing in general is being proven as something valuable just for a small number of companies. Yeah. So 37 signals on this repatriation. They had that blog post, I think, talking about how they saved five or eight million dollars. So, Jordan, what are your thoughts on that?

24:29So I think, you know, if you are repatriating data or, you know, working, you know, off your own servers, you know, having having fewer servers, fewer. I mean, so like the kind of servers that you buy these days off off the rack are probably going to be, you know, at least 64 cores.

24:49And and, you know, you can you can shove a whole bunch of RAM in it. And so those are going to, you know, probably meet most most workloads that you need, because I think you don't have to necessarily buy the big, big iron machines. The tricky part then comes in is making sure you have separation of storage and compute.

25:08But there are, you know, off off the shelf like MinIO and Ceph and Cohesity and kind of ways you can build that build that in yourself.

25:16But I think if you want to take advantage, I think even, you know, big data or or medium data building separation of storage and compute and making sure you have that is going to be critical to sort of having a having a robust system.

25:33Yeah, that that I agree with. I think it is being able to be flexible even if you are managing your own servers is going to be key. The other thing I want I want I want comment on on pricing because I wouldn't be able to check those numbers.

25:51But what I have seen is like the cloud data warehouses are getting increasingly, in my experience, commoditized and cheap. They have an incentive to drive down prices, just like these servers are getting cheaper to buy and provision. So costs are going down on those fronts as well. So the markups are going down and costs are going down.

26:12People are sensitive to the fact that there is a perception of these cloud data warehouses being expensive. And so, yeah, I think different organizations are going to have to make bets differently depending on their needs. And there would be organizations who have the data engineering talent to be able to provision a large number of machines and deal with the complexities that come with that.

26:35There'd be others who are like, I would rather have someone else deal with it for me. And I think that the small data workloads are not going away anytime soon. I also like that. There's still going to be a lot of experimentation that's still happening in your laptop or on medium sized machines with the understanding that that's not going to be everything. And a lot of the important workloads are still going to need to look at a lot of the data.

26:58OK, so while we're on that, there's a question I wanted to get to, which was kind of like, how do you reconcile small and large systems with a unified interface? So a lot of times people might want to just look at something recent, but Diti's point is, no, you shouldn't be looking at the bigger picture. Maybe someone wants to develop on a smaller set and then go for it. And clearly, there's a scale with the laptop's not going to hold.

27:19So I'll start with Diti. How do you feel about kind of being able to quickly and easily migrate between the small and big data? You know, let's use those terms right now, you know, as an interface. So that's kind of like that is as far as softball questions, it's as softball as they come for me, at least personally because of what we are doing at Ponder.

27:41Ponder's kind of like name of the game or the motto is you should be using the same interface to operate on data at all scales, right? Use Pandas the way you are using it on your laptop and your tiny data set.

27:57Be able to use it the exact same way, even if the data is resident on a cloud data warehouse or somewhere else, right?

28:07So that's the goal. And so I think there is a definite possibility that we can get there for all different forms of query languages and libraries where users use the libraries the way they expect to in a small in the way they would do it in a laptop.

28:28And have preserved that user experience as it ends up going to a larger data set on data warehouse, right? Why should the users have to worry about that? This should be something that should be happening transparently to them. Jordan.

28:46Likewise, it's also a softball question for me because, you know, that's also one of the premises of MotherDuck and what we're trying to build is basically saying you start with DuckDB on your laptop and then you extend into the Claude. You don't have to make any changes. You know, the fact that we're a single node system and a scale up system is really an implementation detail.

29:06It could be a scale out system, you know, under the under the covers and, you know, we might end up having to do scale out the kind of the jury is still out on that one. But in which case I will retract my blog post. But, you know, you know, we have an internal bet about whether we're going to have to do that. I'm taking the no other people or other people are taking the yes.

29:28But yeah, I think the that's there. You know, there's a premise there is sort of a there is obviously still a scale beyond which you need to you need to scale out. And but I just and there are kind of many organizations are going to have some of these workloads where you have to do this kind of thing. But, you know, there is still spark. There is still, you know, systems that they can handle that.

29:51And I guess the other the other thing is that, you know, there's always this performance versus, you know, scale scale out trade off. And, you know, if you're willing to take a little longer, you can do you know, you can do anything on on one node. Great. So there's been a few questions coming in on the chat.

30:12That's kind of asking about what is your perspective on the role of real time systems of real time analytics and kind of how do you see the future streaming analytics? Like in, you know, if you had to kind of squint and guess like what percentage of analytics or valuable analytics do you think today are from streaming versus where they'll be potentially in five years from now? So, yeah. So how do you see this question on the screen? So, Jordan, you want to go ahead and start off with that?

30:35Yeah, first of all, like I'm a huge streaming nerd. I, you know, I think streaming systems are amazing. I think that's a more natural way of actually performing computation, even though it requires you to think a little bit differently about about it.

30:53And, you know, in in Google, like, you know, we used to do a bunch of batch computations like and we switched a bunch of them over to sort of streaming computations. And they just worked better. They were more maintainable. They were more scalable. You could when something breaks, you find out in, you know, 10 seconds rather than like, you know, two days later when the map reduces, you know, stop, you know, stop working.

31:17And and so, yeah, I, I, I think streaming streaming analytics is is a great way of solving problems. It is, you know, there are some tricky things. And I think there are some some people that are, you know, trying to sort of tame some of that some of that trickiness. You know, in BigQuery, we sort of we're very into into sort of, you know, streaming and making streaming possible.

31:40And, you know, reconciling reconciling state, I think, was one of the most the most hard problems. Late arriving data is also another hard another hard problems. That said, like, you know, I think it's always been like two years away. And and I don't know that we're any closer.

32:00You know, now that we've been two years away for 10 years. So I would love to see it be more, more, you know, you know, more common. But I don't I I wouldn't at this point, I wouldn't bet on it. I mean, I work with Mike Franklin. He's been saying they've been working on this since the early 2000s. Right.

32:19It's been a couple years away for a long time in his belief. But I did say you want to follow up on that. Yeah, I'm less bullish on the streaming systems. I think I think in particular, I think it's going to end up being more of a niche analytics use case,

32:35specifically targeted towards observability and things like that, rather than anything related to true inside generation, business intelligence or machine learning and data science. So that's where I see the vast majority of analytics going to be at.

32:52And there's going to be a few niche use cases around observability and reacting to events. But I think it's going to be niche. OK, but so we had a question from Astasia Myers. Let's go ahead and go to X. I think it might be relevant to where we're at right now in the discussion. Hi, I'm Astasia Myers from Quiet Capital. Thanks for fielding one of my questions.

33:16I thought Aditya did a great job highlighting that there is a difference between record keeping data and operational data. He argues that operational data grows quickly and can be bigger. I see this in my portfolio companies with machine generated data. So I'm wondering, Jordan, do you believe that your claim that big data is dead is applicable to operational data?

33:40So it's certainly an interesting question. And I think the you know, to me, it's it's really about the operational data. You know, it can get quite large if you're looking at your click stream logs and there can be a lot of a lot of data coming in. But I would say also a lot of that data, like that's the data that decays very, very rapidly in value over time.

34:02And so, you know, you know, looking at what's going on right now, what's happening, are there alerts, are there problems, problems that are happening? Maybe you look at what happened yesterday. But it's much more rare that you would want to go and you scan like, you know, the last 60 days or two years worth of worth of data.

34:20So I do think that, you know, even though operational data can get pretty, pretty big, I think the fact that you don't you don't usually have to to to use massive time ranges is is really kind of what saves you and allows you to process it in a smaller scale system.

34:41So then pushing back for you a little bit to, you know, balance out the softball. So, you know, if operational data is part of this, what you said, you're not bullish on streaming data and reports in this. But that to me seems like that would be the value of operational data is be able to make real time decisions or near real time decisions. Right. So if you're saying the value of operational data is more archival, you think there's good examples of of use cases that have demonstrated that success.

35:05And then a related question that came from Mike Franklin yesterday. I'll just pitch this in there. Is that like, you know, he's had some conversations with folks. They say when this data starts getting older and older, like Jordan said, there's a decay. But part of the decay actually comes from that. The applications evolve or the users evolve. So it actually gets a lot harder to model that data to say what was the state of things three months ago versus the state of things today.

35:26So totally. Yeah. So I think there is a certain type of operational data that is, again, like observability, streaming, just like what's happening right now.

35:42Like, is my server running hot kind of data? That's not the data that I would consider to be. I mean, that data has certain use cases and interest. But imagine let me paint a different type of operational or sort of event sequence data that could be equally valuable.

36:03Let's say maybe even more valuable. Let's say a user. I think I gave the example of users interacting on a Web site, click on a bunch of things, decided whether or not to make a purchase. Right now, that event sequence of what the user did is incredibly valuable for someone trying to optimize whether a customer.

36:23What are the pain points for customers when they are navigating a Web site? Where are they getting stuck? Why is that not leading to an eventual purchase? And being able to not just looking, look, identify those patterns, but also try to draw inferences and trends over time.

36:42And also maybe build models on this type of person does this type of action. All of that is going to be really important and is only possible with this machine generated data, even if you have things like sensors. And so the other forms of machine generated data include things like sensors.

37:05And again, there's a question of how do you make sense of all of this data with machine learning models? And and even like even if you want to not all of it is just like summarized away and and biased towards recency.

37:21Often you want to look at data over time and draw on insights across all of it, as opposed to just on the most recent stuff. And again, I think what we've seen with the likes of the large language models and more data helps. Right. And yes, for certain types of questions, more recent data helps.

37:43But and these may be some of the questions that event systems, sorry, streaming systems are tailored towards. But the more general data science and ML and AI use cases do require as much data data hungry. They can use all the data as as much data as they can do. The second question on data drift. I think a lot of these machine learning and AI systems can actually learn the drift.

38:05They can learn that pattern change over time while still capturing the underlying phenomenon. Right. So I think they understand that. OK, I don't know. People's likes and popularity and friend networks are growing over time.

38:24That could be a part of a feature. But but they take that into account as they're building the models. Right. So as you train them on. Then so for Jordan, there was a question from the chat from Michael Siegel that was saying, you know, part of the promise, you know, maybe it kind of lines with what Aditya is saying, is that with big data, your models and your decisions get simpler because, you know,

38:46you just have more data on your things. But the articulation, if you're trying to do more complicated, you might need to do more complicated analysis or more complicated methods or models with smaller data. Any thoughts on that? Or. Yeah, I think also like some some, you know, you know, the experience of the large language models is that they there's not everybody is going to have to train these things.

39:09And, you know, having, you know, training, training some of these big models, you know, obviously has to be done on massive clusters and and, you know, with specialized hardware. But I think for most people are not going to need to have to have access to to those things. You know, I think, you know, Aditya's answer about, you know, about, you know, being, you know,

39:31there are there are some times when you do want to have to do some sort of deeper computation. It's not just a simple sort of scan filter aggregate and, you know, something that resists summarization over over your data and and use and use more and a longer time time window of data.

39:49Just the one thing that I would caution about from from that is is the longer back you go in your data, the. the more dangerous it gets to draw inferences directly from it based on how the data is being collected today. Because, you know, schemas, schemas migrate over time, you know, the meaning

40:10of something migrates over time. There were like data collection bugs or something, you know, something screwed up in the system that only happened over like December in like 2017. And, and, you know, those kinds of things can sort of be these sort of silently skew the inferences you're making about, about the data. But the point is, the point is still valid. There are,

40:33there are certainly use cases for, you know, broader, broader computation and deeper computation over, over larger sizes of data. So if someone came to you and they were kind of describing, you know, what they want to do, how would you identify the difference between a big data problem and a problem with big data? You know, knowing which is the right case to go? Should I stay small? Should I go big? Do you have thoughts as to how somebody could get that insight or that

40:56knowledge? Or is this just something gained with pure experience and understanding? Well, my, you know, my, my philosophy is just, is start with the simple thing until you, you've demonstrated that you can't do the simple thing. And so, yeah, give it a try. And if you can't scale it up, then scale it out. Aditya, anything to add on that? Or is it? I think, I think that in general, a good rule of thumb.

41:19Okay. So, so Aditya, so Raul Quezada Fernandez asked a question about, you know, outside of operational data, do you find that there's other domains where the rate of data is really outstripping or outpacing the, you know, the growth and compute and memory density? Or do you think it's really limited to kind of sensors, automated devices and operational data?

41:43So I think, okay, so there is two aspects here, right? Like one is the rate of data growth, and the other is the compute needs, right? So one could argue that even if you're, what you're operating on is medium sized data sets, that could be situations where you want to provision a large amount of resources to be able to process that medium data set

42:06in a very, very small amount of time, because it's a really urgent need. So I think simply saying, hey, okay, like the data dictates the compute needs. Again, it's painting in very broad brushstrokes, because I think the timeliness of the need and the size of the workload are both important axes to be considering. And so depending on how timely

42:29the need is, and how much data you're operating on, you should ideally have the flexibility to provision the set of resources that you want. And like I mentioned with the website example, pretty much every industry would have sort of instrumentation of whatever people are interacting with, be it a website. I mean, if you think about healthcare, finance,

42:53retail, all of these scenarios, there's lots of data coming in, right? It's not bespoke IoT sort of situation. The IoT itself is generating zettabytes of, what are yottabytes of data? Sure, but like, even very important, quote, unquote, important verticals with lots of data,

43:14or with lots of, like, that rely on data a lot have this operational data problem, like finance, and retail, and healthcare, automotive, hardware, etc. Great. So, and this is probably for both of you, but maybe more Jordan is, you know, Jordan, you've partly articulated that, look, keeping this old data is a liability, there's potentially like the data quality degrades, there's a cost associated with the storage, there's even

43:37potential liability, you know, security risks or whatever. But you're not saying that there's still no value in it, but it might just need a closer eye and understanding what is that data value to keep or how to keep it, you know, like maybe it is some kind of materialized aggregates or a model based on that data. So, do you think there's good processes in place for people to kind of help

43:57sniff through historical data and see what is the right data to keep, or is it just, you know, kind of a green field right now? Yeah, I don't know. I don't know that there are great ways. I think one is, you know, I think if you have a data lake, that can sometimes lead to sort of sloppy data management practices, that I think using a data warehouse and sort

44:19of proper data, you know, like Schema on Read is sort of lovely for, you know, you know, hacking something and getting it together quickly and being able to make things continue to evolve. But it's really bad from a governance perspective, like, because you don't know, you don't know what you have. And if you don't know what you have, then you don't know what you're not supposed to have, and then don't know who's doing what with, you know, and how and,

44:43and so I think, you know, kind of using kind of more rigorous, you know, either data, either lake house or data warehouse kind of storage management, you know, data cataloging governance, and using that to understand what you have. And like, somebody, somebody was talking the other day about, they have, they have a bunch of data sitting on s3, that they know no one has,

45:07has used in like six years, but they're terrifying, they're terrified to delete it. And like, that's a bad, that's a bad situation to get into. And, you know, I think the only way to, like, sometimes you just have to rip off the bandaid is sometimes it's sort of like, you know, if I, you know, throw out all those, you know, clothes in my closet that I haven't, that I, you know, that I haven't worn in over a year, I, you know, I always think,

45:30well, what if I need that for a costume? Or what if I need that for something else? And it's like, you know, you're probably not actually going to going to need it. And, and, you know, you probably are that you're going to get less value out of it than you, then you think by, by keeping it around, but I don't know that there's a, you know, an easy answer to the problem.

45:51Yeah, when it is in our postdocs together at MIT, we had some interesting conversations with the group with, you know, they had hard storage limits, right? So they had to get rid of data. And they just give this nice analogy of the toothpaste, the fact, you know, the tube, you kind of get, you know, stingier and stingier, but they had to start deleting stuff. And their process for deleting was scary, because it was just emailing like, hey, anyone know what this data set is? Can we recreate it? Do we use a drive? Is it?

46:16You know, it was an interesting exercise. Okay, so I think we're hitting about 45 minutes. I think we wanted to wrap up with you each trying to take the other person's perspective and kind of argue for the other side of it. So with that, Aditya and Jordan, if you want to, you know, switch gears and do that, I guess we should start with Aditya on this one. And say what you think Jordan is right about.

46:42Yeah, so I think definitely there is that you should you should operate on the sizes of data sets that make sense for your, for your workload for your organization. I think the thinking carefully about what makes sense for your organization and picking resources and, and,

47:09and, yeah, picking resources and processes wisely for that organization makes a lot of sense. So for example, if there is a data set that you're never, ever, ever, ever going to look at, probably a good idea to delete it, right? Like, and if it makes sense to

47:29kind of throw out the cruft of your data. So for example, your data stored in an incredibly inefficient format, if you can get the same data set stored in a much more compact format, while not losing any fidelity whatsoever, do that, right? Like, and so all of that is, makes a lot of sense.

47:48And what else? So I think, broadly, thinking about every organization should make the trade off between provisioning resources that are in house versus resources in the cloud,

48:08in a in a careful manner, in a lot of workloads, starting small and starting with your laptop is actually perfectly fine. And it's a good practice, in fact, before you decide to run that $5,000 query that Jordan wrote about in his blog post. So I think start, start with what makes sense,

48:30and then scale up as things go. It sounds like someone in your family was upset as you took that other perspective in the background. It's just great timing to start and you kind of get into that mode of, now I'm upset. They were good until you switched over. Jordan? Yeah, so I think,

48:49you know, you two made a lot of good points. I think particularly the, you know, there are some algorithms that, and some, you know, data complexity, you know, cases where you're going to need to, you know, you're going to need access to a lot of data, you know, large language model training, training data, you know, things that have, where you need all of the fields in a,

49:13in a, in a, in a table, like those, those are cases where scale out may still, may still make sense. Scale up has some, you know, some limitations still, you know, the, you know, the, if it's difficult to get all of the power out of a, out of a large machine. They're, you know,

49:36operating systems have some, some sort of like, some high CPU limitations, you know, getting data on and off the machine is, is, you know, is often a bottleneck. So like, it's not, you know, scaling up to giant machines isn't necessarily a panacea. And, and yeah, there are just, there are, you know, there are, you know, organizations that do have, that do have to deal with lots of data

50:00and are going to have to, to do so for the, you know, foreseeable future. Great. It's probably good to end where people can find out more about you or your organization. So Jordan, you want to tell people where they can find out more about you or more about MotherDuck? Yeah. MotherDuck, you know, motherduck. com. You know, you can, you know, there's a, you can sign

50:19up for our mailing list. You can, you can sign up for early access to, to, you know, our MotherDuck system if you want to see, you know, actually, actually being able to use, to use DuckDB in the cloud. I've got my third hat on, you know, and yeah, that should be pretty easy. Great. And Aditya?

50:45I'll switch my, my hat as well. I don't, I don't have a third hat, unfortunately. So yeah, ponder. io is the place to go for, for Ponder. Ponder is building scalable data science libraries that run wherever your data sits. So if you'd like to try Pandas, which is the world's most

51:09popular data science library on literally any database or data warehouse, come talk to us. And this, we are doing private betas at the moment and likely to have a public beta very soon. And there's also an open source project behind it, which is called Modin, which you can go and check out. So Modin helps you run scalable Pandas on, on Ray and Dusk.

51:34Great. Anything else for me too, or is that good? No, thanks Aaron for, for hosting and thanks to our, you know, people who wrote in with questions and, and our video questioners, Andy and Estasia. Great. Yeah. Thanks so much. This was a great time and an interesting discussion. So we hope everyone enjoyed it and learned something. Totally. Thanks again, Aaron and Jordan. Thanks. Bye.