1:01All right. Hey, everybody. Welcome to our show today. Very, very excited to have you all. We are going to be talking about MotherDuck and DLT Hub and LightDash and what it looks like to use all these pieces together. Super, super excited that you all are here. I'm going to quickly introduce myself. I'm Jacob.
1:25I work in DevRel at MotherDuck. I do all things MotherDuck-related, which is quite fun. Alina, why don't you tell people who you are and what you do? Hi, everyone. I'm Alina, and I work at DLT Hub as a DevRel already more than three years. So you probably know me, probably not. Hello. Amazing. Yeah, Oliver, go ahead.
1:49Hey, everyone. I'm Oliver. Yeah, I'm one of the co-founders and CTO at LightDash. So I'm going to be walking through how to get set up on top of MotherDuck fully agentically. Yeah, amazing. I think this is our first kind of webinar demo. First off, no slides.
2:07We're doing Claude Code only, and I think that's a really cool new way that things are happening here. And so we're going to start with Alina, and then I will do a little exploration on the MotherDuck side of poking around the data, and then Oliver will light up the dashboard on the end. All right, Alina, are you ready to jump in? Yes. Okay, amazing.
2:29By the way, feel free to put questions in chat. We will pick them up kind of as we go here or at the end. So no dumb questions. Super, super excited. Drop us a note at any time in the chat, and we'll take it from there. So let me first introduce what we're going to build today. I will share my screen.
2:53If you follow the link I shared before, you will find a GitHub repository with a demo description. So what we're going to build today, we will take data from GitHub REST API,
3:08load it to MotherDuck using DLT, and model this data with LightDash and build the dashboard. So we start with my part because every data journey starts with ingestion. So DLT, it's the open source Python library for data ingestion.
3:33And today we're going to build DLT pipeline using only agents. To start, you need UV on your machine. And there are a couple of accounts like Cloud Code accounts or any other agents you have on your machine.
3:54MotherDuck token for your data lending and DLT Hub account and LightDash account. I will talk about it a bit later. So let's start with the UVX DLT Hub init command. So this command will scaffold some DLT Hub workspace, initial workspace.
4:17It gives you everything you need to start working with your agent. So let's do that. I already have a project directory. So I'll skip this and you can do as you like. Let's copy it and go to IDE or terminal, whatever you want.
4:38I prefer PyCharm and run the command UVX DLT Hub init. We will install the latest version. It asks us, do we want to create a virtual environment? Yes, I want. Please do. So what do we have in our project? We have environment.
5:02We have DLT, some scaffolds and agent initial skills. So let's just, OK, we have deployment files. These are, you don't need for this webinar, you don't need to know what these files mean.
5:23Let's just use Claude. You can ask Claude if you don't know what they do. What would be next? Today, we're going to use just a couple of prompts to keep it simple and quick. We will load data from GitHub API to MotherDuck directly without authentication.
5:44And just for three points, commits, issues and contributors. Let's copy it and run. So the other side of building pipelines with agents is that now we have to sit and wait.
6:05So I will try to explain what's going on during this process. For example, you can see that Claude started GitHub router or DLT Hub router. DLT Hub has a lot of skills which helps you to build DLT pipelines.
6:23For example, you can list them.
6:30OK, let's run DLT Hub AI toolkit list.
6:49And see how many actually toolkits DLT Hub AI harness or DLT Hub AI harness contain. So we have skill toolkits for each type of pipelines like SQL database pipeline.
7:07When you take data from SQL database for file system pipeline, when you load files to your database, for example. And of course, the most popular pipeline toolkit is pipeline. So let's see. Start asking questions.
7:25Let's load issues from DLT repository. Submit answers. And instead of installing all these skills during DLT Hub init, DLT Hub router will find the toolkit you need.
7:48For our case, it's figured out that we need REST API pipeline. And for other steps, it will load more toolkits if they are needed. So what's the main advantage of using DLT Hub AI harness?
8:08It's that you get senior DLT user instead of junior. If you try to build DLT pipelines without DLT AI harness, it will take some time to make it clean and efficient.
8:31So the coolest part is that DLT Hub AI harness contains not only ingest stage, but also it will validate your data, your pipeline. It will debug it. It suggests some DLT Hub Pythonic transformations.
8:53It will help you to deploy it. We will discuss later. And yeah, explore your data, build dashboards and reports based on the context agent has.
9:11Since DLT provides a lot of information about your pipeline, including code, schema, data, metadata, traces, logs. Yeah, it has a lot of information.
9:28Yeah, I love I love not having to think about my ingestion pipeline and just saying, OK, use DLT to put this data into MotherDuck. I don't have to think about, you know, pagination on my APIs or back off and retry or schema mapping.
9:48All of those things kind of are included in there and just work, which is for someone like me who spent a bunch of time in the data engineering salt mines. I enjoy not thinking about those and knowing they work. And then, of course, being able to see the code. Right. If there is if there is an issue. Right. I'm not stuck in a tool that prevents me.
10:11That is not that's dragging and dropping. And then if I run into an issue or, you know, the API changed today and it ever doesn't work. You know, it's amazing. I love it. Yeah. Let me quickly copy the credentials.
10:35So, as you can notice, DLT API harness will never ask you for a question or for secrets. It will ask you to set it yourself. If you ever, ever it asks you, please, can you copy paste your credentials? So in this chat, please don't do that. Or if you did, you should take your tokens right away.
10:59Yeah, absolutely. We got a we have a good question, actually. I'm just going to throw into the chat. Does DLT work in Databricks environments?
11:13Yes. If you if you go to DLT Hub dot com documentation, you can find the sources DLT supports. But to be honest, DLT supports any source. So you don't have to look at this section. And we have pre-built destinations here. And you can see that it supports Athena, BigQuery, Klinghouse, Databricks.
11:37Also DuckDB, Duck Lake and of course, MotherDuck. But using SQL database via SQLAlchemy, you can load it into every SQL database, which supports SQLAlchemy. Also file systems. Amazing. Thank you.
11:59Very cool. So we set up credentials. During our conversation, agent built some pipeline. It lives in this Python script. So you should know that DLT is just Python. It's not a framework. It's not a service or something. DLT, open source, it's just Python.
12:21You can define your REST API source in this declarative way. Just set your base URL and define what kind of authentication you use. In our case, we don't want to set any authentication, so we will provide none here.
12:42And set parameters for your endpoints. For example, we set 100 here to reduce number of requests. And of course, resources and points are listed here. So for now, agent built a resource only for issues and points.
13:04Just to keep it simple and quick, it will evolve the pipeline later when it makes sure that it works. So these five lines of code define the DLT pipeline. You set what kind of destination you want. If you want to load it to Databricks, you just type it here.
13:27But today we're talking about MotherDuckk. So let's do that. Usually we load data first to DougDB and test it locally. And then we just switch it to destination to MotherDuckk for production, to load data in cloud. As you can notice here, at limit and dev mode true, agent set it for debugging purposes.
13:50It will do the first run and check on the small amount of data if it works at all. If data is correct, and if we can do the full run later. So let's try. Token is set. It will run it first locally, but I want to deploy it directly.
14:13And schedule, I don't know, daily 3 a. m. So now we have to hit again. What agent will do? It will run the debug skill.
14:32It will run the DLT pipeline with all the possible debugging features, built-in features like dev mode true at limit one. Also, it will enable logging. You can probably see it here.
14:52First, it will show HTTP error body if it's failed. It will switch log level from warning to info to get more information about your pipeline. So after it's done, it will run it first locally and then it will try to debug, deploy it to DLT Hub platform.
15:16A few things about DLT Hub platform. This is app. dlthub. com URL. You can follow it and log in.
15:30I already logged in, so agent will use my DLT Hub account from my local machine, from the terminal directly to deploy pipeline on this platform.
15:51So DLT Hub provides you not only all the AI agent support, but also it's a whole managed infrastructure. You can deploy your pipelines here. Also, not only pipelines, but transformations and reports like marimba notebooks or streamlit applications.
16:14This is our default playground workspace. DLT Hub will mirror this project, this whole directory. It will wrap it into container and run it here.
16:36So you don't have to provide the credentials there. Just set it up in secrets and config tunnels. It will copy it here and run it with all your files. So when it's done, I will do a tour around these sections.
16:57But so far, let's see what's going on. Solve three endpoints, load clean. 100 here means that only one page were loaded. So tables were created. You can see that issues were nested into several sub tables.
17:19Issues, labels, issues, assignees, issues performed via GitHub events. Same for commits. DLT does it by default. If you have nested data and you want to load it to database, relational database, you need to do something with this nested data.
17:40You can just save it as a raw JSON, but it's not very nice for your data analysts. So they have to handle it later. Apply transformations here or any other logic. So DLT does a nesting by default. You can disable it, of course.
18:04You can customize DLT pipelines in very many, many ways because it's just Python. So you can do with DLT everything you can do with Python. So that's easy. So still waiting. Any questions in chat? No.
18:24Nothing in chat at the moment. No, this is this is super cool. I'm curious, what what database did it put the data in? Currently it's MotherDuck. I just asked. No, no. Sorry. Like what database name? What's the alias of the database that it put it in? I'm just curious what decision it made there. OK, I set it up. OK, you have data.
18:48OK, great. It's it should be existed database in your MotherDuck. So I think. Perfect. But somehow it's loaded. I think it might have created a new one. Yeah, that's fine. Just fine. You know, such is life with agents.
19:10Oh, yeah. Actually, yeah. During this short conversation, agent logged me in into my GitHub account.
19:18So we just checked if I already logged in in my browser and automatically connected my local workspace with my Dolti Hub platform workspace. So a lot of things done. So it shows my playground workspace.
19:43I yeah, I can create another workspace and and connect it to my local environments. But I'm fine with playgrounds. Let's continue. So it asked me. which DLT Hub workspace should this deploy into? Create a dedicated one. Use playground. Let's use playground.
20:08Yeah. How far back should the first load reach? Okay, sorry, let's keep it short. Just set to submit. So currently at DLT Hub, we don't build DLT pipelines anymore ourselves.
20:30So all our stack is Claude Codes, DLT Hub AI Harness and DLT Hub Platform. And this is our best practices. Actually, yeah. Also, we collect all our knowledge, all our best practices in this skills, including, for example, performance skill. You can just run this skill.
20:54It's a toolkit. Yeah, that's a set of skills. And ask, can you improve my DLT pipeline performance? I love that. That's a good skill. We got some questions in the chat. I'm gonna pull up a couple. We'll start here with Jay's question. What type of API source is it pulling from? Is it GitHub data being adjusted via REST API?
21:19Is this configured via one of DLT Hub's custom defined integration setups or done via the general REST API connector? Actually, agent will decide it himself. It has another skill. It calls find source. It will check first, do we have verified source already for this source?
21:40Then it will check our DLT Hub. com context list. So this is a base of context for more than 10,000 REST APIs. You probably can find any existing GitHub API here. For example, it will check if, do we have context for GitHub?
22:05Yeah, we have. It contains information about API, how to set credentials, how to set DLT Hub, how to run the EI harness, and so on. And this information, it's mostly for agents. So it's not even for humans anymore.
22:27Context, yeah, it's for agents. And based on all this gathered information, it will decide, should I create it from scratch? Should I reuse any existing source? And based on my experience, it always builds a new REST API pipeline because REST API, it's just a declarative config.
22:50So you don't even need much, I don't know, scaffolds, yeah, or existing sources to build this pipeline. It can build you the GraphQL version as well. Well, that's good that we don't do that as humans anymore. All right, we have another follow-up here.
23:15You know, something that I don't think you touched on yet. What are the options to choose compute for varying workloads? I think this is running on the default size, right? Sorry, again? So like for the, on the DLT side, how do you choose how big of a DLT, there's obviously cloud resources being attached to this. Yeah, sure. How to choose, yeah, yeah. Yeah, by default, it's,
23:41so there's, let's see. So there's, let's check our DLT app documentation. If you go to pipeline operations, job configuration, it says that default is small. It's just two CPUs, four gigabytes memory. You can increase it if you need, yeah.
24:02But use a performance toolkit instead first. Okay, got it. Unfortunately, yeah, we exceeded some rate limits. It will figure it out. For example, it will set limits for pages. Yeah. That happens sometimes, definitely real.
24:27Let's see, another good question here. Can you further customize the DLT agent? Specialized skills, defining transformations, destinations, data shape, not just schema, judgment call stuff, depending on source data. Sorry, it was too fast. No, it's okay. It should be on your screen, too, if you want to look at the screen. I have to, yeah.
24:50Oh, love it, okay, amazing. I'll read it again. This is about customizing the DLT agent. Can we basically predefine ways to define how we do transformations and destination data shape? It's not built-in feature, but it's just a bunch of MD files. You can do whatever you want. You can ask lots.
25:14Please use these kind of transformations for this DLT pipeline, yeah. Yeah, yeah, absolutely. That totally makes sense. I mean, I've typically enforced that type of stuff via tests, so I'll create a set of assertions. Okay, every model should look like this, for example, because may not listen, and then you can run a test and look like, oh, it failed. Whoops, it needs to look like this shape.
25:36But yes, great answer. How's our pipeline going? Yeah, it's already deployed. Amazing. Yeah, you can see it on the DLT Hub platform. It's scheduled at 3 a. m. We can start, run, and see it in real time
26:00because we don't want to wait for 10 hours. But in the meanwhile, let me show you what this weird dashboard job does. It's actually our built-in Marimo notebook for a quick overview what data we loaded. Is your schema correct?
26:21If your pipeline is healthy, I can, like, first stop when your pipeline is, okay, actually it's running, so we have no data in the dashboard yet. We need to read, yeah.
26:43Later, we can find some information about your datasets and pipelines in these sections. Like, it says taps before you go to your destination. Instead of, yeah, jumping back and forth, you can just make sure that your pipeline works correctly
27:06directly on this DLT Hub platform. And also, during the scheduled jobs, you can validate that it's healthy, that there's no schema, drifts, or broken datasets and pipelines. Yeah, okay, our data were loaded to MotherDuck,
27:29to GitHub data dataset. Now we can jump directly into pipelines and see what tables were loaded, how many rows were loaded, what kind of disposition, yeah, loading strategy it has for these two, it's merged. For the photo contributors, it's replaced because it contains only 200 rows,
27:53so it doesn't make sense to make any incremental loading. So these are child tables. They take the parent loading behavior. So also you can see information about your traces here.
28:14So this is good, useful context for your next work, for your transformations, building dashboards job. There's a lot of things you can explore. Yeah, if you go to notebooks now and can look at your data even closer, you can actually build your own notebook
28:37and serve it DLT Hub platform in one command, or you can just ask Claude to do that for you. You can ask it to build the notebook and deploy it for you. Yeah, you can even take a look at your data. So, pipeline info.
28:58Now, data quality, run trace, and so on. Don't want to waste your time on it. So it actually is. What I want to repeat again, repeat again that DLT Hub,
29:22it's actually your agentic layer on top of any destination you have, for example, MotherDuck, and also it helps you to manage all your pipelines, any other kind of jobs. Yeah, thank you. All right. I'm going to hop into MotherDuck next, I think.
29:44Alina, do you have anything else you want to show? I think that's it. You have the data, so feel free to. . . Amazing. All right. Thank you so much. That was really cool. I think at one point in my career, something like that took me like a couple of weeks to build, and we just did it in a prompt. All right, I'm just going to. . . My part is pretty quick here.
30:08You know, MotherDuck is a cloud data warehouse built on top of DuckDB. As Alina alluded to, what's really nice about DLT is you can kind of build everything locally in DuckDB, make sure it all works, and then you just flip a switch, change the destination to MotherDuck, and you have a prod, something that's prod-ready. Very, very cool. I did see there was a handful of questions about like differences around, you know,
30:33where to use DLT Hub versus MotherDuck Flights versus whatever. We can capture some of that stuff at the end of this. All right. So I'm logged into Claude. Hopefully my screen looks good. So I'm connected to the MotherDuck MCP, so I'm just going to do slash MCP. So you see here's what I have connected, MotherDuck. I have a local one that's not authenticated.
30:58Then I have my Claude AI one and Slack and some other things, Burrsell and Paper. The MotherDuck one here with 44 tools is the one that I really care about. So I've logged into the account that we were using, and let's just see if our demo database is in there. So we're just going to say, what is in our demo database in MotherDuck? And so I'm just going to do this, right?
31:21Obviously I can do this in a UI as well. What's really helpful here is because we have a remote, a remote MCP. You can just add it and away you go. You can start asking questions to your data. This is also part of the Claude and OpenAI marketplaces.
31:41So if you want to add it to your enterprise accounts, it's very, very easy to do. It's literally one click and then we have OAuth. This is pretty funny. There are several demo named databases. That's true. One literally named demo is likely what you mean. Yes, that is what I mean.
32:04Very funny. And so you see it's making some calls here. We can see what they are by clicking on them. And we can see it's reading and looking at the data. All right. So it's telling us what's in here. This was probably honestly slower than just logging in and looking at it. But, you know, we got to be agent-pilled.
32:28I'm actually surprised it's this slow. Okay, great. All right. So it looks at, it says, hey, we've got 500 rows in our GitHub data. We've got some contributors issues. You can see we have some schema copies. Yeah, exactly. We're table maxing. Amazing. I know I have lots of demos in there. I did not mean those other demos.
32:55All right. Let's see here. So I'm just going to say, let's see. Any really old issues that haven't been resolved? Let's just see. What can we. . . Well, we'll just see. Hang on. Let's look at issues by age.
33:22All right. So it's going to just, you know, do this in Cloud. Obviously, we're going to have a very lightweight visualization. This is something that, you know, Oliver can sneak into a dashboard and say, okay, maybe we want to have like a health dashboard on top of our GitHub repo, see how things are working, right? And, you know, we can. . .
33:45I think my general perspective, I think, on AI and analytics is generally that as much as, you know, we would love for everybody to fit into a single pattern. We'd be like, all right, I'm just going to apply this. You know, I'm going to read the book about what it looks like to run analytics for companies in my vertical, and I'm going to apply that. The reality is much messier, I think.
34:09And I would say that metrics and data is often more like an archaeology project. We discover them. They do not come down to us from on high, from a book we read. We discover them by looking at the data and spending time in the dirt. And then we get to memorialize the ones that are good in dashboards, right? So that's my perspective.
34:31Maybe someone else has other perspectives, but, you know, I've been doing this for a while. That's kind of my take. Great. So we can kind of see what this looks like. Awesome. So, you know, if we look at this, wow. Pretty good close rates on these. Very, very cool. Clustered in the one-month bucket. Obviously, we didn't load the whole history, right? This is just some of what's in the data set so far. Cool.
34:55So that's kind of how the MotherDuck MCP works. There's a handful of tools. Yes, exactly, Nick. Metrics are archaeology. Very funny. So that's what I wanted to show from the MotherDuck side. Obviously, we have other data in there, but it is very, very easy to kind of get started and start asking questions, right? And so that's kind of what I wanted to show you all.
35:19With that, I think I'm actually going to hand it off to Oliver to take this beautiful data set and start doing some LightDash stuff. Does that sound good, Oliver? You ready? Yeah, that sounds good. Let's do it. All right. I'm going to kill my screen. I kind of wish I could just build off the thread you already had going.
35:38But maybe that's a good thing to start with, which is that we have kind of like three people going through each of these stages. But in reality, you would do all of this in one agent thread, right? And it would have all of the context from like all the work that Elena's done. Yep. Okay, cool. So I'm going to share my screen.
36:02So I'm going to be working in the same repository that Elena was using. And the cool thing about that is that if I have questions or if later on, you know, one of the places often where you end up with some tickets or some bugs or somebody saying that they've noticed something wrong with the data, it's because they looked at a dashboard or something like that.
36:23And one thing that's really awesome is that if you can have the definitions of all of your dashboarding and your BI layer, as well as like the transformations and the ELT scripts from DLT Hub in one place, then usually you have the full lineage of everything. And usually the bug is going to be somewhere in that pipeline.
36:42Because the alternative is that you're jumping into different repos and like asking other teams about where this bug could be. So if you can see all of the loading code from the API all the way through to the dashboard in one repository, then you have control over like the full data stacks with one core thread, which is pretty cool. So, yeah, I'm going to be presenting from LightDash.
37:03So if you're not familiar with LightDash, LightDash is a business intelligence tool. So you have the ability to build dashboards, to build data apps, to do conversational analytics. And all of this is built off a semantic layer. And we're going to build our semantic layer live. And then we're going to deploy it to LightDash. That's the goal.
37:24So if you're not familiar with what a semantic layer is, this will also be a really good intro. But the idea is that we write kind of like a contract on top of our warehouse so that we're free to change the schema in MotherDuck. We're free to change the ELT pipelines and separate all of that like hardcore data engineering work from the consumption layer,
37:46where the end users just say, like, all I care about is seeing like total number of open issues. And if we decide as a company to change the definition of total number of open issues, or we decide to refactor our database or anything like that, we keep the same contract. And all of the agents around the organization can still just keep calling for how many open issues do we have.
38:05And they'll get the new numbers, and they don't need to care too much about the implementation hiding underneath. So I like to think of a semantic layer as basically a contract between the raw data and the ELT pipeline, which might change over time through engineering decisions, and then kind of like the consumption of that data through agents and through humans who have business questions.
38:28And they don't care about how it was implemented. They just want to get their number. They want to get their quarterly results. So the only thing I've done before this demo is I've set up a LightDash project that's completely empty. And I've also connected the LightDash CLI. So the LightDash CLI contains everything I need in order to manage my LightDash instance,
38:50to manage the visualization and the semantic layer. So you might not be surprised. I'm going to open the floor and just leave it on Fable as well. Oliver, I think you need to check out the GitHub demo branch, because in my story I stored a Jaffa API example. What's the branch name?
39:13GitHub demo. Okay. No, sorry, it's demo slash GitHub. Okay. There we go. Thank you. That's awesome. Keen eye. Okay, cool. I should have just asked Paul to do that as well.
39:36I don't know why I madly typed a git command. So the first thing I'm going to do is I'm just going to auto-complete this skill called developing in LightDash. You can actually fetch this. So once you have your LightDash CLI, I did this by just typing LightDash install skills. It does that for you. So I did this right before the demo. I'm going to develop in LightDash. I'm going to do setup, the LightDash YAML.
40:00for this DLT project. Create the semantic there. Tell me which models we have. Build a dashboard to explore it. Amazing. So I'm just going to start with that. Let's see what happens. The Light-CLI has the
40:18ability now and also this call session to see all of the DLT code so it can see how the tables in MotherDuck were materialized. The Light-CLI also has direct access to MotherDuck so it can write raw SQL and it'll pass through to the MotherDuck instance which is really cool because oftentimes you want to do that exploration exactly what they could show which is your
40:42call thread can just go and poke around the schemas and poke around and get little snippets of the data. And so it's looking for that but with the context of having the DLT Hub project it's even more rich. So now our call session can see all of the DLT Hub scripts, all of the business logic that's in there, and then all of the schemas that are in there. So it says it found the
41:03production facing schema is this GitHub data one and it's going and fetching the commits, issues, and contributors. And it's going to go pull those out and then from there we're going to create some metrics. Now just really quickly while this is running I'm just going to jump over to LightDash and I'll show you a project that I already have here. Let me just update. Has
41:28someone redeployed? Let me know Jacob if there's any questions while this is running. Of course. Cool. Go here. Right so what we might want to get to is if my project is filled out I can go here and I can jump in to this is like a dashboard I made previously. And our goal is to
41:50basically build something like this. Now the way someone's put a wacky visualization on here. The way that I would normally build one of these, or you might have traditionally, is that you would essentially go into this UI mode. You're going to just drag some stuff around. You're going to maybe like add a new tile. And it's like a pretty laborious process the truth be told. You know
42:11you're adding these tiles one by one and dragging them around. But Claude can do this for us and it does that by writing a yaml that defines each of these tiles and then a layout. And that's exactly what we're going to do. So let's jump back and see how we're doing over here. I do like the scary revenue tile by the way. That's very you like the revenue is down. There's no doubt about that.
42:36So maybe we're pointing out that in LightDash, there's a couple of ways to build these tiles. So some of them are very simple visualizations that anybody can build by kind of like point and click. But we also have these completely custom tiles that we call apps. And you have the ability here to generate essentially any JavaScript viz that you can imagine.
43:00For example, all of these aggressive falling arrows when the revenue is down. And you can put all of these together on a dashboard. So any visualization that you can imagine you can also put here. For today, I'm going to show the more like on-rails mode of this. But it's worth saying that all of the metrics that you deploy to LightDash, so for example, this definition of revenue, it's available here to build dashboards. But I can also go
43:23and like talk to my agent as well at any time. I was like, how does revenue compare? I was like, how does revenue compare to the last year? And so I can also chat with my data conversation. And here, the access, instead of having raw access to the MotherDuck warehouse
43:45here, I'm giving it access to only the semantic there, so to the metrics. So it's a more on-rails experience. There's a few reasons why you might do that. One is to only expose subsets of data to certain parts of the business. So depending on who's logged into LightDash, I might see different data or I might see different rows or different columns. And I can also provide other
44:06context that depends on who's logged in. Okay, let's go back. And okay, we're going to start writing some files now, which is pretty cool. And I just want to like click through a few of these so that you can see what's happening. Just let me know, Jake, if you think it needs to be bigger. Looks good to me. Okay, cool. So in here, we've got the pipeline that builds all of the data.
44:31We have access to MotherDuck, so that's where our data lives. But now we also have this LightDash directory. And this shows us that we've got three models, one for commits, contributors, and issues. And we're going to go into one of those. And I'm going to be brave and try and use Vim here to look at it. Anything could go wrong. And so this has configured like
44:54a few things. And some of this is data related, but some of it is really just related to visualization. So it's added some descriptions for each of the commits, but it's also adding more of like these consumption metadata. For example, what should the default time dimension be? You know, if I ask to see any of the metrics in this model over time, it has a default dimension.
45:16It also has some like default suggestions for fields that might be interesting as you drill down into your data. So these are all interactions that you can do through LightDash. That's managed here. It's also put it in a group, and it's also described a join that you can do. So that
45:33contributors joins directly to, wait, which one was I on? The contributions. So contributors and contributions join together. So you can see that enriched data together too. So this is all metadata that's really helpful for generating more complex queries down the line. There's also things like a primary key and things like that. So you can start writing tests.
46:00All right. So I'm going to go back to see how Claude is doing. So it's gone and written one of those for each of the models that we have. And now I can see that it's now writing the YAML for each of the charts. So if I go back here to, again, just to look at the structure. And I'm just trying to show a simple LightDash project here. So you can see that in LightDash we've defined a model. And so we've got the commits, the contributors, and the issues.
46:23And then here you can see that we're starting to generate individual charts. So for example, this KPI for commits over time or the issues open per month and things like that. And eventually, we're going to deploy all of this into LightDash. I kind of want to deploy it now, but it's going to try and do it itself. So I don't want to interfere. Here we go. So it's running upload
46:45now to upload all of these into LightDash for us, which is really cool. And just to say that the LightDash CLI has a lot of features hidden underneath that you as a human probably wouldn't use, but the agent loves to use and is in the skills. So for example, it has this command called LightDash lint. And this will go through all of the YAML files because it's writing a lot of YAML.
47:06And you can expect that in a big nested YAML document that even Fable may make some mistakes, or maybe the documentation is slightly wrong. So this lint command basically scans that YAML really fast. It's just a static linting. And it'll provide really detailed, structured information that the agent can read that can go and make those fixes. So what's really nice is
47:30that you can ask for to just go in a loop and just say model 10,000 tables until they're all working and lint is passing. So that's a really nice way to keep these agents on lines. Okay, cool. It looks like we're done. So everything got deployed. So I'm going to jump back
47:47to this ModelDuck project. Cool. And we can see that we have a dashboard here. So I haven't done anything at this point inside of the UI for this project, but I already have, for example, I can see all of the issues over time. It's chosen to split them by open and closed. I can see our
48:06commit rate, so I can see the velocity, some of our biggest contributors over time, and also the breakdown of whether they were a part of the team or it was like an open source pull request. And I can even look at some of the current issues and pull requests that are open against the repo, which is super cool. And what's nice is because this comes through the semantic layer, so LightDash
48:28is kind of aware of all of the lineage up to the point of the visualization as I already have loads of interactivity built in here. So if I can say, oh, well, what are these 52 open issues? I can just ask to view the underlying data, and I can see and drill into all of that issue level data
48:49that's inside of LightDash. So that's everything that we have available to us. And I mentioned there's two parts. So I built this dashboard with all of these tiles on, and that's represented as YAML. But also, all of this comes from a semantic layer. And so I'm just going to quickly show this underneath. So you can see here that it's built out all of these metrics. So for example, the author
49:11count, the total comments, and the total reactions. And what's really cool about this is, for example, here, let's take a look at this chart. And it has the total issue count by state and per created month. And let's just say, I don't know why this would be the case. Let's just say that we decided we wanted to change the definition of issue count to be double what it is today. I don't know why
49:34we would do that. But I'm just trying to think of a reason that we might refactor something. So if everybody in your organization is just generating SQL and storing it and re-executing it, then you have around the business all these different definitions of what it means to have, for example, total number of issues, for example, here. So I'm going to just say to Claude,
49:57I don't know if it's going to do this because it's a bit of an outrageous request, but change the definition of the issue count metric to be double the real number. And what it's going to do is, all it needs to do is, it needs to change the definition of the metric in this
50:16issues. yaml file. And you can do this in the UI, right? We're having fun in Claude, but all of this is also possible. Yeah, exactly. You could just change it here. But the idea is, probably as the data engineer, you would probably be looking after this repo with all of the YAML in. And when you deploy it, that will fan out to everybody in your organization.
50:41So there's a benefit to keeping that definition of total issues locked in here. I was going to try and find this specific metric. Here you go, issue count. And there you can see now it has times two on it, a ridiculous definition for this metric, but we're going to go for it. Now deploy. So now it's warning me, just worth saying, this number is completely wrong.
51:07And I'm like, great, that's fine, and I'm going to go deploy. And so you can imagine that I have this dashboard that has the total number of issues in it. So you can see it's here, like, 180. So hopefully it'll be quite a lot higher. But you can imagine that I might have hundreds of these built around the organization, or maybe thousands. And if I wanted to go and update the
51:27definition of issues, because it changed for whatever reason, that would become extremely expensive. But here I just had to do a one line change in that YAML file. And now you can see, now I have 954, and this is now pushing 350. So if you imagine that in a bigger organization, usually the people that are querying the data that are hitting MotherDuck and getting all of
51:50their, you know, trying to get answers, it's usually quite different from those that are maintaining all of the data inside of MotherDuck. And so this semantic there gives you that contract and gives you kind of a source of truth. And now what you have is a repository that has all of your ELT pipeline, and then all of the business definitions of things that you can query there, and everything all the way through to kind of visualization. And I will upload all of this,
52:14everything that we just generated now, to the repo so that you can see it in one place. I think I'll stop there for questions. Yeah, this is great. I think you hit on this, but I'm just going to pull it up again. Where does the semantic layer live? How can you change it? Yeah, yeah, good question. So the semantic layer lives right in this DLT repo now alongside the
52:39ELT. So actually, maybe I can, can I just push to this branch? Is that okay? Alina, that's a question for you. Can you push to the branch? I'm going for it. Oh, I hate to go, because I need to have one password running. It's a fork, right?
53:01I think this is actually, no, this is actually on the raw one. So I don't know if I can push to it. I'll see. Find out. We're going to find out. No, okay. But anyway, to answer the question, it's in this single repository. And then you keep it in sync with that upload, with the upload CLI. Is that right? I mean, or there's other ways to do that, I'm sure. But like, that's the kind of way that looks. Okay,
53:26great. Amazing. So we have the API pipeline from DLT, but we also have the LightDash models, charts, and dashboards. Yep, yep, yep. Incredible. All right. So obviously, feel free to type questions in the chat here. We're going to take them on. I'm going to jump into a couple here that we saw earlier, and we'll just kind of work through them here. First one is,
53:50flights are using DLT Hub. What is the value of using DLT Hub directly? Higher level flexibility, I guess. That's not exactly right. So MotherDuck flights is just a Python runtime. DLT is a library that you can use inside it, right? So that's different there. Obviously, DLT Hub, and Alina,
54:08feel free to add here, is kind of, you guys are building the whole thing around it, around DLT, to make it serverless compute. It's also observability. It's the data quality stuff that you're showing off. I don't know if there's anything else you want to show off, but they definitely are taking different approaches at different sides of the same problem.
54:33Yes, and answering another question, everything related to DLT Hub, it's a paid product. So DLT open source, it's free, it's open source, and DLT Hub is paid product. And yeah, managed infrastructure, AI harness, and all the features around data quality transformations and observability,
54:55everything is part of the one subscription. Amazing. All right, so I'm going to pull this one up, because I think this is also related. The DLT Hub skills, are they public? Are they on the public skills marketplace, or not? They are on this marketplace, but they are under license. So you should need to have a subscription to DLT Hub, and use it. Yep, okay, got it,
55:20understood. Awesome. Let's see, here's a very specific data engineering kind of shape question, I think is interesting. You mentioned we can turn off nesting. This is, again, when we're talking about building data models. This is for you, Lena. Would that be a good use case when I need to build a raw data layer where the data is preserved as is? It's a very good question. Just recently,
55:42we had a huge migration for our customers. We moved 70 pipelines in five weeks using AI harness and DLT Hub platform. And every pipeline had nesting off, because it was migration from another
55:58tool, which, I can't say, it cannot do nesting, but it's difficult to control it. So they maintain a nesting on the DBT side, on the transformations side. So that's why you need
56:17to turn off nesting, but it's a very good question. Yeah, great question. I personally love to turn it off in the raw layer too, just because that way I can always see exactly what came across, and then we can unpack it subsequently. It's so fast to do it with MotherDuck that I don't worry about it. Okay, let's see here. So what should I choose,
56:40MotherDuck against BigQuery, slash Databricks, snowflake, Postgres, et cetera? So this is two questions. I'll answer the MotherDuck one, and then, Alina, you can take on the DLT Hub question. So I think MotherDuck really fits in today. If you have datasets that are in, let's say, tens of terabytes or less,
57:02MotherDuck is much more cost effective and faster than those other platforms. Postgres is kind of weird to include here. Postgres is an OLTP database. Generally, first off, it's very difficult to run Postgres into the terabyte size unless you really know what you're doing. And generally, people who are talking to us that are in the SMB kind
57:27of mid-market are not running Postgres that big. But they are running into issues where they have to put indexes on everything or use materialized views or all these things in Postgres that don't really – they're failing to scale to serve analytics to their customers. And then you try to put a tool like LightDash on top of it, which assumes certain affordances in terms of analytical querying, and it just doesn't work. Your users open the dashboard, and then they say,
57:50why is it spinning forever? It's no one's fault. It's just that Postgres can't handle those queries. So that's what I would say there is really cost effective and fast under tens of terabytes. Alina, you want to talk a little bit about DLT Hub against the native integration tools? I think this means other things that kind of come out of the box with those platforms.
58:15Okay, I don't know many of these native integration tools, if you mean the SaaS vendors or something else. But the answer will be the same. DLT gives you full control over your pipelines. And compared to vendors like SaaS vendors, it's much, much, much, much cheaper.
58:37And actually, Oliver, I know you use MotherDuckk and DLT Hub for your internal pipelines, right? Can you also answer this question? You probably know the best. So why you would choose MotherDuckk over these ones?
58:58Why do you use? Why do I use MotherDuckk? Okay, there's a bunch of reasons I like MotherDuckk. Firstly, one of the key ones is I always say the product is like the sum of not just the thing, but also the team around it. And the MotherDuckk team
59:19is like so responsive. They're obviously like the absolute experts on DuckDB, which is such a cool technology. And the reason that I like push people to use MotherDuckk is that it works for all workloads, but it particularly shines on like a small workload. Whereas if you take something like BigQuery, there's like this four second floor, you know, to like every query, which is really
59:40cool if you're scanning like terabytes and you get it back in four seconds, that's cool. But if my agent is just spamming like a bunch of tiny queries, like I'm going to make way more progress doing that in MotherDuckk than I am in like a much bigger world. So that's really cool. And also like the amount of cool functions and things that are available. inside a MotherDuck, so they have some like pretty cool,
60:01some wacky joins and things like this to do ranking, and they're always releasing new features there. So that's one of the key reasons I like to use it. I'll just say as a DLT hub and DLT enjoyer, it's just Python. Jamie's point is SAS vendors don't cover all endpoints. I would just say that the REST API affordance is incredibly powerful. I don't think about it at all. AI knows how to use it really well.
60:26And, you know, I wouldn't say I'm one-shotting everything, but basically like 98% of REST APIs, you just say, all right, use DLT and connect it. And I have something working, which lets me get really quickly to, is it valuable for us to make this work for our data engineering team, right? Meaning they're going to move it into production, or is it just like, you know, this data is interesting, but not useful enough, right?
60:51And I think that that part, you know, this notion of like, well, now that all this stuff is so fast, we have disposable pipelines. We have disposable dashboards. We have disposable databases. We can do all this R&D work that we couldn't even do before. And so I think that's really cool. All right, we have one more question before we wrap up. Alina, do you want to add anything? I would love to hear your opinion about DLT hub platforms,
61:14to be honest, and AI harness maybe in the future. We should talk about that offline. I would love to share thoughts. So far, what I've seen so far has been amazing. All right, we're going to have one more question here, and then we'll wrap up. Thanks, everybody, for letting us go a little bit longer. This is for you, Oliver. Can you expand on your talk with data feature about only needing the semantic layer?
61:39Yeah, go ahead. Yeah, good point. So to talk with your data, there's two modes you can flip inside of LightDash. So one is just raw talking to the warehouse, which is kind of what you saw Jacob doing with Core Code and MotherDuckk. And so there you have the ability to kind of like scan the whole schemas. There's also a governed mode inside of LightDash where it only sees the data through the semantic layer. So it sees an API for your data.
62:02And the reason you might do that is a few things. One is kind of like access and controls. So you can control which parts of the warehouse become available to different groups of users. So maybe the finance team in Europe only see the Europe data, or the sales team on the East Coast can only see their own performance data. There's also caching and a lot of performance benefits. So we reuse a bunch of like different queries
62:26and re-aggregate queries in your query history to serve results like super fast in the semantic layer. And the third reason that you would do it is because it also contains, for example, you can put something like a filter on. Let's say you're connected to your warehouse and it has like 20 years of data in, and you would never want to query 20 years of data. You'd only want to query the last six months. You can add rules like that. Like, please only let these groups of users
62:49query like a max six months window or something like that. So it's just a way to put the agent more on Rails versus giving it the more SQL directly. Amazing. That's super cool. All right. Thank you, everybody, for coming and joining us. Talk about, you know, what it looks like to build all this stuff in the age of Cloud Code and make it work together. Alina, thank you so much for your awesome DLT demo.
63:11Oliver, thank you for showing us LightDash. Everyone, we are wrapping up. We will see you on the other side.
