0:00Hello. Hello, everyone. Good morning. Good afternoon. Good evening. Wherever you're at. Glad that you are here joining us today For our live stream on data engineering is AI engineering. My name is Jerel. I'm on the marketing team here at MotherDuck Joining you from from Seattle And I'm joined with but with Jacob who is one of our illustrious dev rel folks
0:25Here as well also in Seattle. Hi Jerel. Thanks for in the intro Yeah, yeah a couple of housekeeping items before I let Jacob get into it We are recording this and we'll send out the link if you registered And if you didn't register you can find the recording on our website. Hopefully later today in a few hours Depends on how long it takes it to process Yeah, I'd love to hear
0:50Post in chat, you know, where are you? Coming in from are you No, you're up. Are you in the US you who else had you know the day off yesterday? We're coming in hot this morning. It feels like a Monday, but it's actually a Tuesday for us. So We'll have some time for Q&A at the end As well as you know, if there's a maybe some pertinent questions throughout throughout it that I noticed what we can pop it in
1:13I want this to be really, you know, really interactive feel free whenever you want, you know, just post a question in chat Jacob is is a very Smart person and hopefully can answer your question. So well with that I will stop talking and head over to Jacob cool Thank you, Jerel. I really appreciate it I'm I'm really pumped to talk about this today You know
1:36One of the things that I get to do, you know at MotherDuck is really Extend our product and like use it with the latest and greatest AI tech And so I think you know what this what this really is about what this what this kind of webinar is about is really about How ways I've learned to use these things together In ways that are just like super
2:01You know interesting and compelling but also useful right and so, you know We've taken all of this really awesome stuff that the DuckDB has built on their side and we have taken that and Built something really cool with MotherDuck, you know MotherDuck is running DuckDB in the cloud which means you get all of the things like users and backups and
2:23Security and governance and all the things that you need in your database Um that DuckDB doesn't have and so we get to build all that cool stuff and MotherDuck But that's not what you really care about hearing today today. I'm going to talk more about this notion of AI engineering and And Engineering and data engineering and how we got there, so I'm gonna pull my slides up Jerel. I'm gonna kick you off. Bye Jerel
2:49Everyone say bye to Jerel Anyways, this is this is an adaptation of a talk I gave at a conference a month or two ago, but I just wanted to kind of share more broadly with everybody and so I'm gonna just jump kind of into into what I built here Let's see if I can get my presenter mode to work. Perfect
3:11So for those you don't know, I'm a really big NBA sports fan This is a great time of my life because I get to watch these games. It's great at the playoffs. It's a lot of fun And so I built an app this app actually got built I Don't know almost like 18 months ago. Now. You can actually check it. I'll drop the I'll drop the Let me just drop the link in here we go
3:35NBA Box scores that were sell app Great dropped it in you can hit it. You can hit the link If you want just so you can kind of see the thing that I built, but I built it About a year ago and I'll just show you what that looks like because I refactored it recently kind of with AI like how does
3:57It look like to actually build this and do all the things I want. So let's hop in Perfect so here's a demo of it just kind of showing what's available right now. It's making actually real query to the database when you see this We can actually click on these tiles here and immediately load it because it's just a database query So we've actually already loaded everything now
4:18We just can use the power of DuckDB wasm to make this all work. We can do some charts We can transform it really quickly by clicking a button Hell yeah, Jared Spurs and six agree. Let's go boo. Okay. See I'm in Seattle as well. So can't can't stand okay see So anyways, this is actually really cool this game quality thing that is pulling up here that's hidden in the corner
4:40That's actually doing a calculation just browser side, you know comparing millions of rows to each other Which is really cool because it's very fast as well okay, so I've I've coded this right And the original version version of it worked but it was really kind of gross and really hard to understand the
5:05It was really hard to understand The data pipeline because the data sources coming from the NBA are like kind of gross just like these really net like deeply nested JSON for each player For each game and then it's nested by like team and then quarter and then player and then each stat and it's very sparsely populated
5:25Right, because if that player doesn't play they might have like or they they they play like, you know a couple couple minutes maybe they have like two stats populated someone who plays the whole quarter has many more stats and there's a bunch of ones and it's Unclear the scheme is not well-defined So it was really it was really tricky to kind of build it and so I ended up kind of in this weird Python TypeScript world to to look at it
5:49There's nothing incremental every time I was I was running my pipeline. It was it was a full reload When I looked at how I was handling the sequel Kind of in the front end you could put whatever sequel you wanted in there Which was you know, the classic the classic sequel blunder And you know lots of dead code hard-coded numbers all over the place No tests and I never really actually published it
6:12And never made it available to everybody because I was kind of like like this is not quite at the high like at the bar that I wanted the project to be at and so I had my agent look at it and I said, hey, like how do we refactor and build this thing? Let's make it simpler. Let's go all typescript. This is actually my first all typescript project. I'm more of a Python Pythonista
6:36Yeah, I see a comment here if you think the NBA API is gross yes It's it's all bad It's all all API's all API vendors hate you and want you to fail is my my takeaway from working in data engineering You know for the last, you know, I don't know seven or eight years. It's it's horrible. It's it's really really bad
6:58so yeah, a lot of a lot of empathy for that for that and and part of actually why I Move some of this to typescript was just I felt like the handling of the JSON objects was easier for me to reason about anyways So I'm building this refactor, you know have my agent kind of look at it say how do we build this how to make it? better
7:18And I actually started building this initially, you know, the first version of this I built with windsurf and cursor, you know You know over a year ago, and I'm thinking about it. Hey, like I'm doing this sequentially working one at a time You know, this is how I started the project You know when I started it very much in the loop
7:36And then you know in November December of last year, you know, really smart Opus dropped Opus 4. 5 And it was it was too it was too It was undeniable that I needed to upgrade my workflow and get into using using Claude Code and so I did that and
7:58This is interesting for me because I think what what this is like the first time that I really stepped out of the loop on this stuff In cursor, it's very much like hey You're you're seeing the code in the IDE as you're working on it and Claude Code does not necessarily require that as a first Principle to get started and I just kind of have continued to do that Although I often do have VS code kind of open on the side while I'm working on this
8:23My first concern is not like looking at the code my first code. My first concern is like looking at the tests And so I'm a little bit more hands-off than my traditional IDE in that case And then I was like, what if I just like opened up a bunch more windows, right? What if I had like three windows running at a time, right? This is great. I get more throughput and I become the scheduler, right?
8:47And I just kind of use github to coordinate it and said hey like write a plan put it in this issue Okay. Now the executor put pick up this issue, you know, whatever And just kind of handing them off It's actually kind of kind of a tiring workflow to do but it did work and it was very fast It was getting me good results And then, you know I said what if I can use agent teams to do this instead and just put it in one window, right
9:11and so now It's launching itself and it's doing sub agents inside isolated work trees age agents getting a github issue They're working independently reporting back, you know, and and honestly like this this actually like works really helpful for me Because You know, I've I've worked on teams
9:33Both technical and non-technical and and led those teams And so this was very much like oh like now now I'm feeling like I'm kind of meant doing a little bit more managery type Stuff and it seemed like it worked pretty well And then what I realized what I was doing is I was just doing data engineering I'm like wait, hang on. Hang on
9:55What I'm really doing is like figuring out how to plan my work in a way that can be executed Kind of in a graph, right? and so One of the things that I was working on as I was refactoring this project was splitting all of the work into waves of work That could be done together and hit common checkpoints, right? And so I split all of the work into these parts and then I was like, okay now I can start handing this off to different different pieces
10:19Different different sub agents different different agent teams and then you know building and testing after every wave and you know Keeping keeping all my tests green as I'm going and because it's you know The surface area is very visual like I showed earlier You know, it was fairly easy to kind of make this make this all work So One thing I like to kind of go back to is the way I like I always thought about data
10:43Engineering was like how do I like make this as like dumb as possible, right? And so when I was a human running complex pipelines, I always thought about it this way whenever possible, right? I wanted batch instead of streaming. I wanted full refresh instead of incremental I was gonna run cron's instead of kind of linking everything together I was gonna aggregate at the source if possible as previously noted
11:07You know the API's That you're using were designed by someone who hates you and You know on-call rotations are a great way to increase quality I think some of these are true and still remain true, but I think I have a different point of view today I think batch and streaming. I don't really
11:30Care too much around the difference around this You know, that's kind of an AI job full refresh and incremental this again this is about state, right AI can help me build all the code that made this really annoying and make and help me test it, right? So
11:51One and two is solved by agent six, maybe an agent. I don't know actually I kind of like on-call still because it's a great way to, you know, make sure you're on the hook for the things you're building, especially if you're building with LLMs You know if you're building anything important you probably should think about how to make it supportable by other people Anyways, so I think we've evolved a little bit from dumb guy engineering a dumb guy data engineering
12:14Just because we have AI and it can solve a lot of these problems, which is great This leads me to kind of the next thing here, which is man, we're climbing a curve and we're climbing this curve really quick, right? All right, we had tab complete. I don't know if maybe some of you remember when we used to be in Be in like VS code with copilot I don't if you remember that it was like three years ago and we'd hit tab and
12:37It would like autocomplete maybe the words and then we then we got LLM assisted tab and it was amazing, right? I haven't used that interface in a long time now, but you know, that was a that was a point in time We're climbing the curve right now. We add an agent. Okay, great. Now we climb the curve a little bit more now We add a team climb the curve a little bit more right now. We add loops, right? So this can happen autonomously someone mentioned, you know, hey, like what does it look like? Gas town, right?
13:02Yeah, that's agent loops. And of course, you know, the labs are way up here, right? Who knows what the heck they're doing They're doing crazy stuff Well, the point is, you know I'm now like at a hundred X me in 2022 when I wrote that dumb guy did engineering post not not because I got a Hundred times smarter. I wish I could say that that was true. But because our abstractions just allow us to be Be more effective, which is crazy
13:25and of course, this is why You know hate to be the one to break it to you kid. We're all building a Agentic coding frameworks right because if you can figure out this thing you can climb this curve really really really really fast And that's an important curve to climb because it means you can go faster All right, so I talked a lot about you know
13:48Conceptually what I was thinking about and how I did this like let's actually look at what was built By the way, this code is all public. You can take a look at it. You can tell me it's bad You can tell me it's good. You can use it. You can adapt it. You can fork it. I don't care I will share the link in a minute. Actually, I'll just drop it in here Just so you can see it if you really care github. Let me just go Pull it up
14:11NBA box score is perfect. Let me just drop it in the chat if you really care about it All right Let's talk about architecture. So this is kind of how we ended up building it. I tried to make it very simple You know coming from my dumb guy Data engineering roots. I want to make this very easy for me to reason about and therefore easy for me to fix
14:36One left we have our external API's. I'm actually using the play-by-play stats API For Historical data and then I'm using the NBA CDN for live scores and box scores So that's pretty neat we use we use node to run kind of our
14:56Run our process to get the data out and then to load it into MotherDuck and then MotherDuck kind of holds Holds all of this. We're just kind of treating it like a semi lake house. I'll talk a little bit more about that in a minute And then it does all the transforms and then we have you know Set of simple tables that we can serve into the front end to be a sequel with the DuckDB WASM client and then you Know next JS Puts it in nice pretty objects that we can interact with
15:19Okay, so this was kind of like top-level how I wanted to think about it So what is MotherDuck doing in this case I was doing two things We're loading all the data in as JSON we basically so each each response we get from the API We just load straight in to a single row in MotherDuck and then add a bunch of metadata And we do this so that our inserts are really fast
15:42We could Also do transformation and extraction of the JSON which has actually gotten a lot better in the latest version of DuckDB But I decided to just kind of handle that as an asynchronous step So that I could just you know get everything in there and then handle my transformation separately You know
16:07Does upserting on pks its directs equals You know, uh does upserting on pks its directs equal there's no orm, you know I think this is like really really actually helpful too. Is that because You know with the MotherDuck api I can just say hey execute this set the sequel step It becomes really really easy for me to think about this um And then on the front end we've got DuckDB WASM
16:31Um, jake see you have a question in here We'll talk about this at the end around like how this works from an architecture standpoint The the actual answer is is next. js is the front end MotherDuck is just running a kind of it's actually running a like middle tier um, or like one and a half tier kind of thing with with wasm So there's a what database in the browser and then that browser connects to next. js Um, but we can take that more in detail, uh at the end
16:55um And so this is really fun. We can do like temp tables in there We can let people use kind of the same thing, but it's all you know Kind of backed backed by MotherDuck in the cloud, which is really fun Um, or at least fun for me as devrel who gets to work on this, um Uh, this is really fun, too. Um, we'll talk a little bit more about the pipeline engine um, this is a really really fun problem to work on so, uh the
17:20Responses from the nba by play stats api are quite slow. They take about four seconds to return a Uh single game so you make the request you wait four seconds you get the game back Um, they can also be parallelized there is rate limiting on it, but it's not published anywhere and um It depends on the actual load of the system how many transactions per second you get so we End up having to build an adaptive rate limiter
17:45So when we run to get the data out it speeds up and slows down based on the load Which really we only know because sometimes the api gives us a 503 because we're sending too many requests Unfortunately doesn't give us 429s for those of you who are dealing with the rest of the apis. You know what I mean? Um, you also get sometimes the happy path of like empty response 200s Which are great. Um, so you do need to check the responses
18:07Of course as well all the fun things you get to do when you're dealing with rest apis returning json um and so What that really meant is we just you know built a queuing system that said all right Let's let's release these these new calls from a queue as um uh at some cadence and then we would continuously test what the cadence was and if we got a
18:29Failure we would then send it to the back of the queue and try again later, right? Um, very normal stuff. Um, so Uh, but you know, uh, what's really cool about this? I didn't have to vendor any of it, right? And so I just got to build, you know I just built a little little small parts that I needed of you know, fairly complex tooling, um to make it work Um, and it works really well um Yeah, it was nice
18:53um It also you know I think like using an agent to kind of iterate on different specs and do research as you're thinking about this stuff is really Is really helpful, too um now one of the things here is that the uh That what I that I was seeing in the data was that there was some inconsistency from time to time where some some uh, some data would get tied to the wrong objects just like there are errors in the way that the
19:17Data was like stored in the underlying database on the mba side um, and so uh The main thing here is that we built some data quality detectors Um, then we're able to kind of build handling in there when we did find that one thing that was really interesting actually about this part building data quality detectors and I think like for again for those of you who Built kind of data engineering like pipelines, you know the challenges here, um
19:41uh The the stuff I was getting from ai were like, okay well like let's just like set limits on like how many points a player can score in a game or like in a quarter or You know rebounds or whatever and it actually was just all it was doing was detecting outliers that were real real data And so it's like flagging Hey, like, don't, hey, like, hey, I see that there's a, a game here with 30 rebounds. Like, obviously that, that's not possible.
20:06And it was like, no, actually that, that was a real game. There was a 30 rebound game. Um, uh, you know, from this player, I think, I think it was, uh, Jusuf, Jusuf Nurkic on the suns had like a 30 rebound game or something like a couple of years ago. Um, it was crazy. Um, anyways, so, uh, you know, we built, we built that out and then I basically dial a lot of those data quality detectors back to really only test things that were like attributes around like players on teams.
20:30And what that really meant is it built like, um, built like trade detection and then like detection on when players were getting cut and waived. Um, uh, yeah. So, um, anyways, so we're, we're, we're, we're building all the things that, you know, you need to think about what, while we're, we're pulling all this stuff together. Um, and so as I think about this, um, you know, what does this mean?
20:54Like building a parallel, using parallel agents to build a parallel pipeline. Um, and I think I, like when I talked about this, you know, kind of at the end of the previous, um, you know, I said that like wave dispatch sounds like a data pipeline DAG, like here it is, right? Like, here's what I'm thinking about, right? Like the structure of how we built the thing is the same as the structure of the, uh, um,
21:15or sorry, the structure of how we built, you know, um, this data pipeline and use our agents are very, very similar to each other, right? Parallel workers, bounded concurrency, dependency or scheduling, right? Those are similar patterns across different domains. And I think like, that's the key thing that I started realizing as I was working on this
21:34is as I'm thinking about my agent teams, excuse me, I'm realizing, oh, this is just the, these, this, this pattern of workers working in, uh, AI, you know, it's the same patterns that I was using for workers in data engineering, right? Running all these different parallel jobs. Right. Um, and so like, I think for me, this is the real part that I was pulling together.
21:57And yeah, Matt, I apologize for, you know, going into the MBA stuff as it works for me. Uh, you know, I always try to find the, uh, you know, really interesting parts of a domain problem and then, and then drill back out. But, um, this is how I really started thinking about tying these things together. And so what does this really look like? You know, um, we can do a lot of things really fast. Um, this was, this is fast for me.
22:20Uh, this was a couple of days of work, you know, uh, 16 K lines. We had it a bunch of tests. We closed up all our issues. We got 26 seasons. Um, you know, this is, this is great. And now it automatically refreshes, which is even, even better. Um, you know, I personally, uh, uh, enjoy, enjoy that. I learned a lot from building this and then like what worked, what worked.
22:42Um, I think the main thing that I, that worked while building this was, you know, because I built my plan on execution in a way that like the work could easily be parallelized and that we got to move really, really fast, but on the trade off of, you know, spending a little more time thinking about planning. Um, and then I will say the, like, you know, always think about building tests. Right. As you're thinking about one thing that I think applies for AI engineering and
23:05data engineering is like test, test, test, test, test, right. Do it all the time. Do it as much as you can make it easy for it to happen. Right. Um, you don't want to get too far, you know, in a specific PR or set of commits where without testing, because, um, you can make your test, your testing makes something that is potentially a non-deterministic, um, pipelines deterministic, and I think that's really helpful.
23:28And then lastly, like staying in the loop, you know, um, when I was, uh, uh, when I noticed something going the wrong direction, I could redirect, um, you know, really thinking about that from a team lead perspective, I think it's really interesting, like kind of reflecting on how this has been for the last, um, couple of months after I really kind of did this work.
23:47Um, and we can talk about reflections, I think at the end too, like, uh, there's a real debate, I think around this fine point of like, how in the loop do you stay? What are you, how much are you viewing? How much are you not? I don't know if I have a super strong perspective at the moment, but you know, you probably want as little code as possible and to review it as much as possible and, you know, figuring out a way to do that is important.
24:11Um, so one thing I'll talk about here, and I think like, this is, this is another thing that, that I learned through this process was like, um, don't copy my workflow. Um, I spent a lot of time, you know, reading things on Twitter and LinkedIn and like grabbing, grabbing those, you know, best practices that I read about and turning them into, um, ways of working.
24:37And like, it really didn't work for me. I think when you're doing this stuff, potentially you're moving so fast that like you, it needs to perfectly snap fit into your brain. Um, I, I, um, I think about it like driving an F one car, like everything needs to be perfectly aligned to the, your way of thinking and working.
24:59Um, because of the decision, you know, you're, you're moving so much faster than you have in the past. Right. Um, and so thinking of like trying and experimenting with this stuff is important to do so that you find the way that works for you. Um, I think like, and this is really difficult because I think like a lot of teams, um, in the past would be like, okay, here's our way of working. We all do this.
25:22Here's, you know, this is how we do a, B and C and that's just how it worked. And it was fine because everyone was going, you know, pretty slow relative to how fast we can go today, but now we can go so fast that we still need, we still need like some common defined, well-defined interface to work with our teams. I'm not suggesting we don't do that anymore. What I'm suggesting is that like for your specific workflow for generating solutions to the problems that you have, that you should think about how your
25:46brain works for it so that you can move quick and then you can, you know, generalize that back to whatever the common way of working is. But I found that definitely, um, you know, taking other people's, you know, experience or even like applying, you know, I don't know, Gastown or, or G stack or whatever, you know, AI thing that, that can work, um, is really hard.
26:08Um, it's really hard to kind of adapt that for your thinking because it's, it's literally someone else mapped their brain into a set of prompts and skills. And like, um, you know, at least for me, like that doesn't, I, I can't, doesn't even, I tried, I really tried. I wish I could have moved faster because it feels, sometimes it feels like, you know, you're falling behind, everyone else is moving so fast, but the reality
26:31is like, you know, it's, it really comes down to, I think, experience and building it. Um, so I think this takes us into, you know, harness engineering a little bit, which is, is what I built on. Um, you know, there's a question earlier. Did you attempt to let an agent design a task graph? How did that go? If yes, I think like, uh, I would describe that more as like a building a harness, um, uh, which I've done a little bit of kind of in other projects.
26:54Um, not at this one. Um, uh, uh, yeah, Callie, I love this comment. I'm just going to throw it up here. Cause I think this is totally right. Um, uh, yeah, I want to use someone else's framework. Um, I think like part of the brilliant part, uh, um, of Claude Code and what they've done with their, with their harness to their credit is like balanced.
27:20Me being able to put my own thoughts and working on it. Um, along with, you know, the, the way that they're harness engineering it. Right. And I think that's a real balance. So figuring out how to do that is hard. Um, so, you know, I think, I think 2026 is the year of agent harnesses. And of course, um, you know, there's some proof there with, with some of the stuff that OpenAI built with something called symphony, um, which was a hundred
27:44percent at data engineering project. Um, yeah. Uh, yeah, it's definitely, we're checking out kind of how they're thinking about it as a set of skills. Um, anyways, uh, what do I have here? Let's see. Yeah. I mean, I think, you know, one of the challenges as we're, as we're here, right, is, um, bottlenecks shifting, right.
28:08Like every day it feels like one, one thing, right. And so like, what's durable, like what's durable as these bottlenecks shifting. Right. Well, knowing what to build, right. That's the thing that's durable. The things that we always were working on as data engineers and now AI engineers, that's still the stuff that we have to do. Right. Um, all these shifts make the iteration loops faster, which means we can do more experiments and get better answers.
28:30Um, uh, so I think like this is, this is definitely what I'm thinking about is like the loop keeps compressing, but our ability to know what to build that still takes time and experience. And so how do we, how do we continue to connect and build all this stuff together? Um, yeah, so that's, that's kind of what I have here. Um, you know, this is how I've been thinking about how to tie all these pieces together.
28:53Um, yeah, compress the loop, find patterns, build your own way of working. You know, we, we've been doing this for data engineering. Um, now it's time to take that into the AI world too. Um, that's all I had for, uh, this talk. I really appreciate everybody joining, you know, hopefully you learned something and I would love to take some questions on all this stuff. Um, uh, now I'm going to pull the slides off, Jerel, you can join us back.
29:18Uh, throw that. Hi, Jerel. Everyone say hi to Jerel. I'm back. Uh, we'd love to take some questions, uh, you know, about, about any of this. Um, let's see here. Um, yeah, let's do we have any questions in here we want to throw in?
29:39Uh, I think you were, you already answered all the questions that came in earlier. Uh, well, I guess, what do we, I mean, there's maybe more, more than like specific, so I don't want to talk about the, uh, the WASM part, right. Just yet. Um, this is, uh, here we go. Pete's asking, uh, can you talk about how to do testing when your upstream sources are inner, uh, indeterminate or yeah. And outside your control.
30:05Oh, this is such a good question. Um, I always think about this notion as like testable assertions. So like, um, we want to make assertions about the data that we know are always true and then we just test them all the time. Um, I did this before I was at MotherDuck, I actually worked and, and, uh, at a, um, company, like a early stage startup that was doing, uh, telco
30:29analytics and we had like very adversarial APIs upstream of us. And so the biggest thing that we built that was effective was, um, running them as like, almost like invariant checks, like just constantly. And actually we leveraged DuckDB, to do that. Um, that was actually, you know, one of the moments where I was like, I need to go work at the company building stuff with DuckDB, like this is such a
30:53powerful way to think about it. Um, so I think that was, that's how I always thought about it. But you know, my, uh, for those of you who don't know my background, I actually, um, worked in, uh, I'm accounting. Uh, I sat for the CPA exam, for example. Um, never got my, never got my license cause it never went that way. But, um, I did, I did pass. I did pass it. Thank you for asking. Um, uh, but I think about like, how do we test it? How do we audit?
31:17Like, that's always what I'm thinking about, like for better, for worse. And so that's, that's a, that's a good question. Um, yeah. Uh, next one here is from Marcus. How much time did you spend on the plan and waves? Uh, to be clear, each wave, uh, was a gate that needed to be passed before, uh, the agent on team and moved on. Oh, this is such a good question.
31:41Um, so, um, it's probably like 50, 50, like planning versus coding. The thing is, a lot of this has shifted and like, it's really hard for me to answer this really, really concisely or like accurately, because I feel like six months ago it was maybe 80% or sorry.
32:04Uh, like 20% planning, 80% execution because things took longer and the models are just getting smarter. And so I feel like now it's moving more towards like 80% planning, 20% execution. Um, and yes, I did put gates in there. Um, but also I was using work trees, so I could kind of like, one of the, you know, amazing things about using AI for all this stuff is that like merge
32:28conflicts just become an AI problem to solve, not a human problem to solve. And so you can kind of have those work trees assume that the test will pass. And then like, you know, when they, when you do a, um, a PR and, um, a poll, a poll request on that, and you do have to do your merge that it will work. And if it doesn't, um, you know, then you can have, have your, have your agent take a look at it and figure out what's happening.
32:52Um, but again, you're leaving tests constantly the whole time, right? So that if you don't, if you have a, if you have an issue, you know, you catch that as part of, you know, a refactor associated to a merge conflict, for example, um, you know, I, I, it's moving really fast in terms of where this all fits, but like, I would say, you know, it's moving more towards like 80, 80%
33:12planning, 20%, you know, actual execution, but like, um, you know, practically for me, it probably looks more like 50, 50, because I'm doing a lot of experimentation. I just don't know. Sometimes the plan, it'll be like, yeah, we should plan this. And I'll be like, let's just see what works. Like, I don't know. Like, let's plan it. Let's figure it out. Let, let's do, let's try it. And experimentation is so cheap that I tend to do a little bit less planning
33:37than maybe, um, uh, makes sense, but also like your architecture needs to be good. Right. If you can, if you can make your, if your architecture is good and all of the pieces, um, work well together, it's really easy for you to, uh, experiment and like, you know, have an agent go off and like chase some rabbit trail that may work, um, you know, so I think that's, that's a little bit different than,
33:59than in the past where it was really expensive to do that type of experimentation. Cool. Uh, oh, here's an interesting one. Uh, you know, this all seems great for more senior engineers, for those who are more junior may not understand best patterns to approach, uh, how would you recommend someone, uh, you know, more junior work with AI? Such a good question.
34:20Um, so a lot about, I almost think about this as like climbing the abstraction ladder, right? Um, and what I mean by that is like, you, you kind of have to like master each level of that before you go to the next level. And so that means like you should use AI on things that are simple and you understand first. Um, and then you can go to the next level.
34:43Um, and then you just kind of climb, climb the ladder. Right. Um, I don't know if I've got a really good structured way to do it. Um, I will say that like, again, for me personally, when I'm working on like data structures and things, I'm always thinking about like, can I make this into a ledger because I really understand a ledger, right. And for better, for worse, I use ledgers everywhere. Um, so like, because that's an abstraction, I really, really understand.
35:07And so I can just kind of like put that, put that wherever I want and then kind of, you know, get into a domain that is, you know, more complex. Um, you know, I think, uh, the other thing is that, that like, I kind of go back to like the dumb guy data engineering stuff, which is like, keep it really simple. It's like, you know, one thing I love about DuckDB is it's like, okay, I can put it on a VM, I can put CSVs on there. I can look at the CSVs with other tools.
35:32I can then run SQL on them. It's like very, very like, like it's not high level abstractions. I can look at it and understand it what's happening. Um, and again, that's like more related to data specifically, but. Um, you know, I always am thinking about like, how do I translate this? A lot of this stuff in my mind is like translation of language from one domain to another, from one syntax to another. And LLMs are incredibly good at that.
35:56So like, whatever your expertise is, I find that like, you know, saying like, explain this to me as if I was, you know, an expert in whatever. Um, uh, is very, very helpful. Right. Um, I don't write a lot of TypeScript, I'll be honest with you, but I can build something that works in TypeScript, which is incredible. I would not have been able to do that, um, you know, two years ago. So, um, yeah, I, you know, Adam, I think this is a really good question. It's something that like we have to figure out.
36:21Um, but I don't, you know, hopefully that hopefully that's directionally helpful. Hi, Mark is asking, I think it's a fairly long question. Maybe I'll try to skip through to where, you know, uh, questions, uh, you know, on ingestion, uh, I guess I'll just read the whole thing. Uh, you know, one important part on the engineering side is figuring out the data model in there.
36:42People are talking a lot about, uh, importing the data models from the sources to make it end up in dbt, to help drafting the data models for the, you know, dimensions, facts, metrics. Uh, what are your thoughts on agents and retrieving this context from the many sources we can encounter in the wild? Uh, whether it's, you know, APIs, databases, et cetera. Yeah, this is so good.
37:02I think I, I still kind of go back to like, so if you know what your final model is going to look like with like facts and dims and measures and metrics and so on, I think like you can build tests that define what those, what that needs to look like along the way. Right.
37:21Um, you know, uh, from like an MBA data perspective, which was, was working on this pipeline, I built some assertions that said like, you know, Hey, like if a player has a record, they should always have like greater than, you know, uh, zero minutes played. Right. Cause then we can detect, Hey, like there's something weird going on here. Right. And I think actually did detect a few weird, a few weird records where like
37:44someone got substituted in, like drew some stat and then was substituted out. For example, um, uh, which, uh, and no time passed on the clock. Right. Um, which are, were valid and was like, Oh, I guess I got to handle that thing. Right. Um, I did not expect that someone would be able to, you know, have no time passing the clock and have this thing occur. Right.
38:05Um, uh, so I think like my thought about this is like build the testing around it. Um, build the, build a set of assertions that are true about the data or should be true about the data a hundred percent of the time. Right. Um, and then, you know, uh, find out how soft your edges can be. Right. There might be some, there's probably some data where it's like, you know, we only need that 95% of the time.
38:29Um, but there's some data you need a hundred percent of the time. Right. Um, uh, you know, finding out where those edges are, I think just comes down to like actually building the thing and then doing some experimentation on it. Um, but yeah, I think building, building the test is how I would think about all that stuff. Ooh, uh, eval question. Uh, how do you think about model costs versus ability? I found a lot of shops are throwing Opus 4. 7 at everything.
38:53I personally use cheaper models with a bit more structure around tests. Uh, what's your view on that? My view on that is, um, right now we're, we seem to be getting pretty good subsidies by using Claude Code with Opus 4. 7. So, uh, it seems like we should take advantage of it. And that seems also true with Codex and GPT 5. 5, for example, which are great models.
39:15Um, so I think there's a split here on like human loop, which those subsidies seem to be leaning towards mostly, um, and like building agents in prod where you're consuming other, uh, where, where you are consuming like an API. Um, if you're consuming an API to like survey API, um, responses, um, definitely
39:39lesser models can be used. Um, you know, I do a bunch of, I do a bunch of testing kind of on the side with something like a connections, the New York times connections puzzle eval that I run. Um, Gemini three flash in particular is very, very good at, um, solving those types of puzzles. It is really, really good at data analysis as a result. Um,
40:00so I think like, uh, definitely can do it. Um, but like, you do need to think about how you tune them and build evals for them and all that kind of stuff. If you're building, you know, something that's a, uh, you know, hands-off, you know, automated system using these, using these, uh, API calls for sure. Cool. Uh, all right. We'll take a couple more questions and then I think
40:22we'll wrap it up. Um, I'll go out to asking, uh, basically, you know, what is, what is the future of data engineers and data, data, data engineering, you know, in an AI world is regarding, you know, career perspective. What do you think, uh, of data engineer roles compared to kind of AI engineering gen IA roles? Will DE roles get reduced as agents are used more and more?
40:45Such a good question. I mean, I think like I see it as, uh, it depends on if you think it's like the demand for data engineering is finite or infinite. And I kind of believe that it's infinite and that it's a, um, the more data that we use, the more data we need type of problem. Um, I do think that like we are, we're that companies, I think, I think part of the challenge
41:08here is that generally like, uh, data engineering is on the cost side of most companies, not on the profit side. And so, um, that generally means that it's like, okay, well we have this finite mindset about the data we need. Um, but I do think that, um, uh,
41:28using data requires more data. I mean, that's the genius, honestly, of where we've seen with snowflake, right? Is they have, they have insane, um, net revenue retention because the more data you use, the more data you need. Um, and so, uh, I, I guess like my career perspective on this is that like the role is going to certainly is going to change, right? Um, I think a lot of things that,
41:53that, uh, were really hard from a data engineering perspective got a lot easier because we can use AI to solve a lot of these problems in a very reliable way. Um, because a lot of the challenge in, in data engineering is that you have a poorly understood API that needs to fit into a well-understood pattern. And we basically have to build all the scaffolding around it to make it reliable. Well, now AI can just build all the scaffolding
42:17around it to make it reliable, right? Okay. So that means that scaffolding part that you're building is, is not part of the job really too much anymore. Um, so I think, you know, uh, it, it definitely becomes, I think more about how like building an effective data platform and AI and what the architecture looks like, you know, um, uh, I think it's going to be different than
42:41what we're doing today. Um, but you know, uh, yeah, I think, I think, um, certainly there will be, there will be a lot of data engineering happening. I don't know. It's very unclear to me if like data engineering will, well, here's what I'd say. Data engineering is certainly not going
43:03to exist the way that it does today, um, in five years. Right. Um, but the need is still there. And I think the need is going to be a hundred, like a hundred X more. So, you know, what does that look like? I can't tell you, but certainly I think it's, it's interesting. Um, especially kind of like how do you chase all the long tail things that you couldn't chase before? Cause it just was
43:23not cost-effective. Uh, all right. Last question. Uh, where'd it go? What architectural primitives allow your agents to fly when building pipelines? I mean, I think like the biggest thing is just like understanding how things relate to each other. Right. And which work needs to be done
43:43in which order. Um, literally just like looking at it and understanding what the graph of work that needs to be done and what things depend on each other. Um, and AI is actually pretty good at doing that too. Um, you know, I've, I've had the pleasure of working with dbt for the last,
44:01like, I don't know, seven years now. Um, and so, uh, you know, I see, I can see what a DAG looks like by just looking at it. And obviously I came from Excel before that, right. Uh, as an accountant, so it was all DAGs all the time. Um, so really just being able to, to look at the data or look
44:23at the problem in front of me and break it down into very small parts and then figuring out which parts need to happen in which order. Right. And then saying, okay, well now that we know all of these can be, these can be decoupled from these, let's build a good interface for X so that we can decouple this further. Um, you know, figuring out what those parts are and then using, you know, using AI to cross validate it at, um, all helps, you know, um, I think like, you know, the other
44:45thing is just, um, I had the pleasure of, you know, working on the level architecture classes when I was in college. And, um, that means I have a pretty good understanding of the zeros and ones happening at a very low level. And, uh, you know, uh, that means those, those, those, those
45:08primitives, I think, uh, uh, basically a compiler design, uh, very, very much translate to data engineering and AI engineering. So, uh, if you, if you feel like reverse engineering or compiler, uh, or taking a class at that level, I found, I have always found that very, very effective, but, um, that might just be just, that might just be me.
45:30Yes. Well, uh, on that, I think we will end it here. Thank you everyone for tuning in for all your great questions. Uh, if you have more questions, you can join us on our, uh, community Slack, Jacob and the rest of the team. We all hang out there. Uh, I'd love to continue the conversation. Um, with that, I'll send out a link to Slack in the follow-up email, uh, along with the link to the video, uh, with that,
45:55have a great rest of your day. Thank you. Thanks, Jacob.
