AI codes with outdated libraries. Why is this unsolved?

0:00Mehdi: Hello everybody and welcome to another episode of Explain Analyze Podcast where we go back to the hot news around data and AI and having a discussion with an interesting guest and today I have Dumky with me. Dumky, how's it going?

0:18Dumky: Yes, it's it's going well, yeah. I came across this very interesting analytics game that I think we're gonna talk about first. so maybe let's dive

0:30Mehdi: Okay. Yeah, yeah. So

0:31Dumky: let's dive right into it.

0:33Mehdi: Yeah, let's die let's dive right in. So I have the screen share and for people listening, it's what it can you describe for people that are listening actually?

0:42Dumky: Okay, okay. Yeah, let me let me describe it. So you're looking at the the interface the of a very like early two thousands Windows style w what's that even called? Like a you know an an overview of your company. It's like a dashboard with an interactive dashboard with stuff that you can click. And it's called Analytics Tycoon. it's supposed to like look like one of these early style games that are are mostly text based and you have a lot of levers to pull like you can you can work on your orchestration you can work on your desk dashboards you can hire new people and you see an overview of what you have in the bank how happy your investors are how much tech debt you have have how much trust you have and then as the time goes on right like you go day by day You'll get new requests from stakeholders and you'll see a daily log and there's obviously like unforeseen events happening. and I just thought this was hilarious. I I'm not entirely sure if I agree with everything that's that's happening here, but the general sense of like you hire someone and then they they take a job on from a stakeholder and then over time like they become You don't need them anymore. You can just fire them. and then you progress through the the eras from like typewriter style to basically implementing Excel and SSIS and then you get into like the modern data stack era and you can implement Fivetran and dbt and then after that you get to the Agentic era. And you can have like text to SQL stuff. I th I thought it was very hilarious. I haven't seen what comes after the agentic area. I'd be very curious. but I think in general it is a it's a hilarious take on how the industry works.

2:34Mehdi: Yeah, I'm I'm just clicking around and I say yeah. It's a cron jobs broke overnight. It has already cost you one thousand three hundred twenty nine dollars, not decide what you do, five fights, absorb it free, explain, ap apologize and move on. and then you basically lose trust points. That is interesting. I think I think I think yeah, tr trust is Is is a bit I would say underrated as a as a data team. Like if you, you know, fix things but it doesn't look fix it from the business side because there might be, you know, specific rule that hasn't been take out. so so yeah. yeah, what what do you want to take

3:19Dumky: No, exactly. And and I think there's some very there's some very good points in here. I think trust is definitely a major one. I think the other one is that what we often forget is that there's real money involved in this, right? Not just not just to stuff like what you pay to Fivetran, for example, or what to you pay for your database. But there's people that cost money every day, basically. And and if you're on a data team, then you're costing money to the company. You're you're a cost center. and so what's interesting here is that you do have a fixed budget, but there is a way of like y you have some benefits, you have some some windfalls, you also have stakeholders just paying you for some stuff they do. And so there's actually a quite an interesting lesson on how to manage your data team and think about trust and and data quality and reliability and the fact that you do need to balance all of these with in a sort of fixed budget. Yeah.

4:20Mehdi: Yeah. I think it's a it has been a it always has been like really hard for data team to make money, just in general. Because even if you're doing a good job, it might be serving other departments. So depending on how you structure cost attribution and revenues, right? If marketing say, Hey, we've been able to do successful campaign, this is the result on the sign up. But the successful campaign comes from, you know, data that the data team is, you know, giving. And and it's hard to evaluate from the data team what's the value, like the end value that you provide to stakeholder. so it's funny here because you have monies and so on and you move on, but yeah, in reality it's I think it's pretty hard to make to put the dollar on Specific request or an internal that's paying you.

5:13Dumky: Well well, I I kinda disagree. Like, yes, it's hard to put a dollar amount on like one specific request, but as a team, year over year, there's definitely a cost involved. And I think we are or have been at least too easy in saying, yeah, but this like we provide a semantic layer or we provide data quality, all these like intangible things that allow you to say like, no, but this is this is worth it once you know X happens or Y happens. And I do think

5:45Mehdi: Yeah, but the case the case of making money is always depending on the stakeholder. There is cost involved for the data team, right? But that that's what is hard, is that

5:49Dumky: Yeah, yeah. That's true. That's true. Yep.

5:53Mehdi: as a data team, you don't really control on how much money the data set you provide is gonna do. That's yeah.

6:01Dumky: No, exactly. But but there is value in especially if you think about the way you structure your data teams, right? You can have a central data team, you can have kind of your analysts and data engineer dispatched to to other teams. so I do think there is a way to think about the value of your your people for specific parts of your organization.

6:25Mehdi: Mm-hmm. No, fun I mean fun one. The design is definitely fun. It's from Tycoon, right? Which is another database.

6:32Dumky: I mean there's like a whole series of these tycoon games, right?

6:35Mehdi: Okay, yeah. All right.

6:38Dumky: but yeah, the design like as you go through the ages the design also changes to different style.

6:46Mehdi: Okay. second one I had what I want to tell talk about. Okay, I talk actually something really Pythonist. Did you know that Python 3.15 was there?

7:02Dumky: I I did not actually know. I was like three

7:05Mehdi: And

7:05Dumky: point fourteen is the perfect one, like we're never gonna change that.

7:09Mehdi: No, but the point is that there is new release, but okay, if you would ask me what's the latest Python version, I think I would have to think a couple of minutes to think is it 3.13 or is it 3.14? So and my head kinda, you know, my brain kinda bugged when I saw this headline, okay, Python 3.15 is there. And why is that? It's that I feel like I'm, you know, specifying Python version much less than I used before because AI is basically doing this. But I do often sometimes look at it, and it's often an old version, like 3.11 or 3.12. And so

7:49Dumky: It loves 3.11, yeah.

7:51Mehdi: Yeah, and so if you look actually, so I have two other data s points around the new release. So the r new release.python 3.15 is coming in two months, the official release. Okay. So that's the the first candidate. So f the first data point I want to look is the end of life on Python. And if you can see the end of life of 3.10 goes in two months, basically when the 3.15. 3.10 doesn't sound too far, at least for me. But you know it mi

8:23Dumky: No, I well I I mean I remember switching from 2.7 to three point so it makes me it makes me feel

8:29Mehdi: Yeah. Me

8:31Dumky: old.

8:32Mehdi: exactly. But so the the the reflection there is like do you have also the same like I feel I don't keep up with like the same version. I think it's even dangerous in terms of like libraries compatibility and so on because you know agent like takes what is the most available version right now, right? And as a best practice usually I always do N minus one. So what is the latest minus one? Because then I'm pretty sure all the library the basic library have done had the time to you know migrate because it's been a year basically right so I do n minus one. and so but I as I said like I'm pretty sure if I if we do a test right now I'm pretty sure your AI is gonna recommend 3 dot twelve. you could we we could to try this quickly. but that means that first AI is completely biased by what is available right now. And what they see and what they've been trained, which is a lag, right? And second, that's that's a potential security system, like aside from dependency, you know, breaking, just like explode, that has been already fixed and others. and another last data point I wanted to mention is that if you look at duckdbstats.com, which is the aggregating downloads of DuckDB package on python only and you look at the top python version you see 3.12 and look look at 3 look at 3.13 3.13

9:59Dumky: Very interesting. And three dot 3.10, like as a big yeah. Nine and yeah, yep.

10:07Mehdi: is so 3.10 is second on the most downloads so you have so that's all time download right But it's the second ranking on the most downloads Python version is 3.10, which is end of life in two months. So I'm I'm I'm pretty sure and look at 3.9 is already end of life and it's four place. And so normally to be healthy, you should be around 3.13, and 3.13 rank fit place. Actually I should have the late latest thirty days on that, but yeah.

10:40Dumky: Yeah, and I think so so what I remember like yeah.

10:43Mehdi: What do you think about that?

10:45Dumky: It there there's a couple of things I think are interesting. So I remember 3.11 had some big performance improvements that were quite quite significant. so that's like if there was one that I would have to like safely pad on, at least for an AI also to to take that, it would be 3.11. And it's interesting to me that that something like 3.10 is still so so prevalent. The other thing I think is again a how do you say like a problem with LLM models in general, right? So they're tr they have their training data which has a cutoff point, and there's still no solid way of having some kind of current affairs, current state of the world understanding compared to the cutoff date of the training day. Like you still I I sometimes build my own agents for various purposes. You still need to inject the date today's date into the context. Otherwise the agent doesn't know what like even what day it is. So yeah, no, no it's not very crazy that it would pick a Python version if it doesn't even know what today's date is, basically.

12:00Mehdi: No, that's a that's a good point. But also it it feels like, you know, you need some skill file around this which is completely stupid, which is hey, check the latest NPM because this is valid for Python, but any programming language, of course, will be the same. check all the latest

12:15Dumky: Yeah, yeah. Yeah, but but and so I wonder whether it's it's in your harness or in a skill file if or even in the model itself if it can have an understanding of the like a conceptual understanding of the programming ecosystem and how to check for all the dependencies and the end of life dates and new features

12:42Mehdi: It do check, I think it could be in the training or in the harness, I guess, like because I do see, you know, Claude Code, for example, checking more doc much more documentation than it used to before, like if you compare a year ago. so it could be there. But I think it was nice thing to go out that got my intention this this week. I was like, Wow, 3.15, I would never bet that this

13:00Dumky: Yeah, no, that's super interesting. Yeah.

13:05Mehdi: was already out. And then it just got me thinking, but okay, let's let's be just

13:10Dumky: Yeah. Which is funny. I d I don't know how

13:11Mehdi: Cautious about what

13:12Dumky: this is for you, but I do feel in a way it it almost feels like the world is at a standstill. Like my attention has shifted from the Python ecosystem, the data ecosystem, to the agentic AI stuff. And so in my mind, I'm not keeping up with the the the programming stuff necessarily. So I I have that feeling like we're still at like 3.13 Python 3.13, 3.14.

13:39Mehdi: That's true. And I think like all the the new cool bits of the latest Python version aren't used that much because again s you have your model which is trained among, you know, a big type of data and it's it's hard to move the needle there and to say actually there is better way to do it now in Python. You shouldn't use this unless you know and you keep up with the with the latest feature. so yeah.

14:07Dumky: Yep. Yeah. So I guess that that brings us to the the second problem, right? So if this is we talked about the demise of Stack Overflow before and what would happen, like if no one's asking or like answering questions on Stack Overflow anymore. Like are we just stuck in time with our our models? So this is I would say a similar thing. Like if no one's talking about 3.15, or at least not enough people. Does that influence the training data of the model and will it then have a harder time understanding what the actual features of Python 3.15 are capable of and why it should pick that over an earlier version, basically?

14:49Mehdi: Yes, I I mean I yeah. You're totally right. And I think the TLDR is that it's not because we are handing over all the coding aspect to agent that we should not be careful about that because I think it's easy to, you know, not think about that anymore. Just version dependency security, what's the new paradigm that the programming language has introduced? I think it's important to still keep up with the high level blocks at least. What else do you have? Agent Craft. Wait, did you did you have like so many games over there? What is this?

15:20Dumky: Yeah, so this is I don't why I don't know why this came up in my my feed. I I saw this together with the other one, which is called Agents View. and it basically kind of reminded me so this is a Agent Craft is hilarious. It's like a World of Warcraft kind of RTS strategy game style, Age of Empires style game. And it allows you to sort of orchestrate your agents as if they were characters in this in this game.

15:55Mehdi: yeah yeah. There was there was one that did the Warcraft sound on clothes when Infinite task, you know, Okie dokie? Like the the real sound. I don't know if you saw that yeah, yeah. Yeah. Okay.

16:05Dumky: yeah, yeah, yeah. Yeah, these ones are also like prompt, prompt, prompt. It's it's a hilarious way to to think about your agent. And and so the main point, of course, is that we're in this phase where one it we need to rethink how we interact with agents and the the amount of tokens and time and attention and tasks we're spending on and putting into these agents, right? And so this is just a a funny, hilarious way to get an overview of what's happening in your ecosystem. And I do think like to be honest, I think people are very tuned or attuned to perceiving maps, right? So spatial awareness is a thing that humans are really good at. And so it does make sense to have some sort of map. For your agents or your code base or something like that. So it makes it memorable, I would say. I don't think necessarily this is a a good way to do that. It's definitely a fun way. it

17:04Mehdi: Yeah, I do. Yeah, yeah, yeah.

17:07Dumky: also shows you like what's possible with with again with an LLM and some creativity and determination. Like you can create a full game out of this. The more useful part or aside to this is the other one, which is called Agents View. So that is basically a

17:25Mehdi: Okay. You put it there, yeah? Yeah, yeah.

17:27Dumky: yeah. a kind of you can have it as a desktop app or as a CLI thing that that serves a webpage or a dashboard. But basically it just just checks all your folders for sessions you had with Claude, with ChatGPT, Codex, everything, and it kind of shows you what you're doing over time. How you're using certain skills, which sessions are using the most tokens. so it allows you to kind of keep a view on what's going on on your local machine, basically. of course this is just your local machine, and I think we talked about this before, right? There's a ton of stuff that you need to do around this in an organization to really get an understanding of where is my token spend going? Are we actually doing things in an efficient, effective way? or are people just, you know, building crazy games with their their tokens? so I think this is an attempt at this

18:22Mehdi: Yeah. No, but I I I think I saw this yeah.

18:27Dumky: is an attempt at creating like a new insight into this new world of of tokens. And I think it's funny that they use the sort of git or GitHub style what's it called, like commits chart for this.

18:40Mehdi: Yeah, yeah, the Yeah. No yeah, that's that's interesting. That's Visibility mostly what I've been struggling for example, I see there is a search session but I've implemented like you know a search physic session between various tools like I'm using Claude Code and Codex, right? And so just that for example I feel like it's quite missing like fuzzy search between existing session, cross tooling.

19:07Dumky: Yeah, yeah. So I had to create my own my own skill to search across different tools, right? So things work fine if you just have one tool, like if you just use Cloud or or a Codex. but once Yeah, exactly.

19:18Mehdi: Yeah. But that's not the future, right? I think yeah, I think we need I mean especially when people are cutting off models that you are using. So so yeah that's like you you need to be able to switch to different different agents. And speaking of that, I have a link which is not there. so I'll put it Sorry. She's this one. Just a side note. you you've seen this one? subscribe our okay.

19:51Dumky: I have not seen this one. Wow, Stripe is interested in acquiring OpenRouter. That is big.

19:56Mehdi: Yeah, yeah. So that was a week ago, a bit more than a week ago. So Stripe Intox you acquire OpenRouter for a potential ten billion dollar deal. So OpenRouter is the you know service that enabled to switch model really easy. You can, you know, put budget, you can limit with a key, okay. You have access to specific models, so it's really really nice to see. And so but the challenge we I think we talked that in a previous part, I don't know if it was you, is that there isn't not that much incentive for the labs to basically allow I would say a pricing model that is healthy for everybody to API. So they tend to, you know, Claude Code entropy limited the usage on APIs because before you could use it with just the Claude Code plan, now they're pricing differently, because they want basically to become the platform and owns everything, right? So I'm just curious. I think open source will push pressure on them, like to kind of like lower the price on their, you know, API interface. 'Cause they have, you know, decent price on their package subscription. But I feel like if you do like even on OpenRouter, it's still I mean I I wouldn't say it's expensive if you just want to try, but at scale for a woo full enterprise, Yeah, it it would definitely be more expensive than just doing a direct deal, for example, with Anthropic. So

21:29Dumky: Yeah, of course, of course. But I do think there's there's very different use cases, right? So I I see like your personal model that you use for coding, that you use to query your database, create your reports, those kind of things. Those are very different from I think the traffic that OpenRouter is serving, where it's more about agents at scale, right? So you serve out agents or inference to other users for your application, whether that's a chatbot or like scheduled inference to analyze certain things. And there is a very good, I think, competitive advantage advantage for the end user in in using something like OpenRouter because it does indeed pit together or pit against each other these different models and these different providers. and it gives you, you know, the option to choose whether you want the cheapest one, whether you want the fastest one, et cetera.

22:29Mehdi: No, yeah, and I think also they work on open source model serving. So it's actually not true. So you could pay more a premium price per PI call for a given service compared to a subscription, like let's say, you know, Anthropic models. But then if you can switch to an open source one, you know, like Kimi or others or Qwen and have much lower price because you mostly pay for the infrastructure inference, there is no, you know, license price behind on top of it. then you can find yeah a good balance but yeah it it was just a sideway for we'll talk about where we were coming from some agent views and basically being able to do things towards multiple models and I think there is value in it because the industry is showing it you have acronuse link

23:20Dumky: Yeah, so this is a announcement from Google. it it kind of ties into the discussion we're we're having, right? So basically, there's two major figures, Demis Hassabis from from Deep Mind, and Jeff Dean from Google who are kind of moving away from Google. obviously they don't explicitly say that. Well they explicitly say for Jeff Dean. He's moving to a different venture. but Demis Hassabis is going from CEO at DeepMind to a chair at Google, which is kind of the the route that Eric Schmidt took, basically to exit Google in a way without necessarily saying that he exited Google. and this really brings in like two big discussions for me. So one of them is what's Google's position in the in the AI race, basically, in the model race, they they definitely provide a lot of the infrastructure, the data center, the TPUs, those kind of things. but it seems like their ability to retain talent to really develop next generation models, frontier models, is kind of declining here. And then the other part which kind of points to we're talking about just now is like what is actually the moat that these labs have and how how big of a trench will they be able to to dig before the they run out of cash basically so is it possible for anthropic for OpenAI for google to basically bring in enough revenue from users, from customers to justify the spending that they're doing on training these models, on opening up data centers, all these kind of things. And so the interesting point of course is that if there is so much competition from different models and from open source models, from maybe Chinese models, will you know these labs have enough of a mode to tie users to them to justify this this spending? And in this case, I think for Google it's it will be interesting to see if maybe they're not like as invested in the whole AI race. just like Apple is seems to be kind of watching from the sidelines as well. And then let's say if Anthropic and and OpenAI maybe collapse, they can jump in and and take over. so I'm not really sure what the play is here, or it could just be that they're too much of a big enterprise organization these days and they're not being able to retain talent,

25:58Mehdi: I think yeah, I think that later is most obvious. I know really good talents at Google Deep Mind and some folks from Hugging Face that were working on Lama, Heavily that moved to Deep Mind and other great talents over there. in the Deep Mind team. So and they there was also the previous I think products person at OpenAI which was there early that moved to basically the Gemini team, Logan something his family name. But anyway, there is good talents over there still. that been able to hard. I do think that what I heard is that Google Deep mind still act a bit like a startup, but you know, it's still in a corporate environment. So I think that's the biggest difference compared I would say to Anthropic or OpenAI that can move things. And I think the second reason is that you mentioned is that the moat I think is kind of like going down on proprietary model, right? because open model really start to catch up and I think they already catch up for a lot of specific tasks, right? Like why would I ask Claude to correct the grammar of a message? Doesn't make sense. Like an open one can do it.

27:17Dumky: Yeah, of course, or even summaries or those kind of things, yeah.

27:20Mehdi: Exactly. So I think it's just because the tooling around is not there yet. Like there is not an easy like yes, there is open code and so on. But like For example, for us at MotherDuck we have contracts directly with you know the labs so it's not like we can use open code and to coming back to labs not offering, you know, good pricing around the API service, you know, that that creates kind of a a friction also for for for tooling. But yeah, I would I would I would expect

27:46Dumky: But but that's also where the the the question is like what's the biggest market, right? Like is the market business users just like let's say Excel, for example, or Google Sheets or Google Workspace in this case I I guess. or is the market more for like the applications where you integrate AI or an LLM into your application and use it in in that way? Because that's a different type of market where it would be much easier and much more common to integrate something like like OpenRouter, right? Or a Vercel AI Gateway, whatever it's called.

28:20Mehdi: Yeah. No no that's true. But I saw like now that you mention I feel Google has been executing really great this past year on, you know, the Gemini theory and so on. And they P and their API. But much less on the on the integration with the rest of the Google workspace, for example. Yes, Google Meet, but I felt like Google Meet it took them forever to kind of like summarization. Like there was other company just doing that, right? Summarization on where they had already the model. So I feel indeed there is I feel a lot of corporate and barrier to break to kind of like integrate that. and so and I think it's not easy also to integrate like it's not just putting a chat bot next to you know, Excel, Google Sheet, or PowerPoint. I don't if you ever use that, they're just not working. Like I tried, it's not

29:16Dumky: No, no, I d I don't even use it, no, honestly.

29:18Mehdi: Yeah, but the problem is because you know, it's kind of like you stacking an existing hamburger and other meat on top, right? It it does just doesn't make sense. It can be a bit more tasty, maybe, you know. But it is not coherent, I would say, in terms of integration. It's not it's not easy. You need to kind of rebuild things from the ground where those major labs basically have a green field there. But yeah, it's really interesting, what you mentioned, the tension of economy. So let's see. I'm not super worried to

29:48Dumky: Yep. Yeah. And that I would say the takeaway for for people listening to this, I would say is mostly what we discussed before, right? Just make sure that you're you're not fully buying into one provider, but that you have like the possibility to exit to other providers or to to you know, take your setup and go somewhere else. Yep. Yep.

30:11Mehdi: Yeah, I think the US government gave us a good lesson a few a few weeks back. So I don't I would be surprised if anyone would give a strategy with us, you know, single lab. all right, what else do we have? let's let's switch gear and talk about data modeling. So Daniel Beach, which was actually on this pod a few weeks back, put a blog is Data Modeling Dead and it is a really nice blog because is really giving nuance on various things and taking you know, input from other people in the data ecosystem. So Andreas Kretz, Ryan Doly is there, Joe Reis of course, b because he's writing a book on, you know, data modeling. I think the key takeaway that I like on his point of view is that when we started with data on SQL Server, like modeling was everything. Like you couldn't if you couldn't model your data, you couldn't even query your data. There was a performance impact strongly to the modeling. Which there is still today, but not as big as you know, as it used before. and so I think that's one thing and the other side, so he's also looking at you know the job ads that's getting and also a couple of people have you know their their opinion, even like to name Mark Freeman, even more important with AI agents, especially conceptual data model of the the business inventory purchase go to shipped goes to delivered past return window But I think I I think what I do agree is that as I just mentioned it's changing. So it's more businessy than it used to. and you know, he's mentioning that too. so you know, I want to bring also the topic of context layers, like, you know, what if actually you have one big table or you have, you know Just data set that are I would say healthy enough in a sense like you don't have duplicate table with because there is that, right, in enterprise, which is much harder where you have like really look alike database. But let's say you've like, you know, a three hundred person company, so not that many, many various business history or data source and you just put a context layer to say, Hey, this table is all about this and this is our business and so on. So I'm I'm curious like you know, if is just good enough for having this business understanding because in time of performance it's like i i there is much less I would say impact. You can still optimize but I feel there's much less impact.

32:55Dumky: I I agree with the performance part, but I think it's even more important for data quality, right? So understanding how an ID in one table relates to an ID in another table and what the definition of that is. and I I think I just saw an example there indeed, like how do you go from Basically, how do you understand what a sale is and at which point you count something as a sale, like what if it's returned within 30 days, stuff like that, right? That's all the things that you need to incorporate into your data model to usually from different tables, to make sure it works. And if you have duplicates in there or you make the wrong join or whatever, that's those are errors that LLMs And N AI models can still make.

33:44Mehdi: Yeah, but don't you think that could be defined into a context layer? Because before you would have to define specific table, fact and dimension. But you know, maybe the model is gonna do ten times a really long query with the joints and so on, you know, because it has the context. So it's not modeling, you know, per How do you say like physical modeling? Like there is a table with facts, but he knows that and every time he's gonna create a query, you could argue it's not deterministic, so maybe the query is gonna change, right? But I'm just like yeah, I'm just curious on how, you know, how much you know context and modeling is enough today compared to to what you used before.

34:33Dumky: No, I agree. Like there's there's a strong reason to believe that that having more context will make your models work better on your data. However, I don't think it replays a d deterministic pipeline of you know, bringing in millions of records every day that need to be joined to other records, to other tables, checked for uniqueness constraints. So you need that deterministic pipeline. That being said your AI models can still help you build that deterministic pipeline. But I don't think it necessarily means we need to swap out h data modeling for context layers in that sense.

35:10Mehdi: No, okay, that's that's a good compromise. I would buy this. so next we have Is this yours?

35:21Dumky: Yeah, yeah, this was this was mine. this is Hamel Husain just an interesting guy to to follow. I think what's interesting here again is

35:32Mehdi: So can you give context just on the title there? For people listening?

35:37Dumky: Yeah, so the title it's it's hard to eval as a product smell, right? So the idea is that if you cannot run checks on something, test something, just like before before AI models, just going on your your gut feeling, it's doesn't necessarily say anything good about your product, right? So the same goes for AI models, like You have to find a way to test them in the real world, basically. And so if you have an internal agent, it's one thing to just treat it as a chatbot. but I think the more interesting, and you can see some examples of that if you if you scroll down, is that there are different UI features basically to check for quality. And so one of them is having Evals evaluations, you can run those with like test questions and then see if the outcomes are correct. But he goes into more of a style of checking things. So for example, if you have a a product like MotherDuck, right? So one thing you can do is you can have your your MotherDuck notebooks and and you can have an agent run alongside them, for example, which is kind of the approach that I think Hex is the example he he gives here, takes. It's the the the main thing is kind of this question of how do you combine the output of the model with the the kind of knowledge of the person looking at it, right? So there is a trust issue in general between what is the output of the model and what do we actually as humans believe is the correct answer. And so the model needs to have some interface to kind of validate what it's doing. Right. And a notebook could be one way to to do this. so if you have a notebook like next to your agent, then you could say like, hey, The agent's gonna help me insert those queries into the notebook and then running those queries will allow me to validate what the agent is actually doing instead of just wrapping everything into one like accordion and and a drop down box and then giving one answer as the as the output. So for different types of use cases like a medical evaluation or a kind of what's it called? A sorry, I'm blanking on the other example. yeah, like a fitness plan, kind of personal training plan, he he gives an example. That's stuff that you wanna validate as a human too. You don't just wanna see the output of the model. You want to see the process of the model and we need new kinds of interfaces to check that process. And I think that's especially true in in the data world, right? Like you need to be able to show how you got to a certain outcome. And so finding a a way to build an application that takes in both the the added value of the model, like the the speed at which you can generate insights, but then also the ability v to validate those insights and the intermediate points of getting to that insight. I think that's crucial for for us as data professionals to Make that process transparent.

38:56Mehdi: Yeah. Okay. No, I mean yeah, interaction of skills, I mean of AI tooling with with as you say like human interface is is pretty hard nowadays. I mean if like it feels like you know with MCP we had there is much more extended now in the in the specs where you can have MCP apps so you have an entire apps in your chat but it feels like for certain use case for example I've been looking forward for a pure AI video editor and people are working on that but it's pretty

39:35Dumky: Yep. Yeah. And

39:37Mehdi: hard to nail down to the experience because for example video editing is really you need a timeline you need specific thing you need the human to have specific knobs to give his inputs and depending on your your task though those knobs can look you know really really different so so yeah

39:57Dumky: Yeah, exactly. And and to me this ties in nicely to kind of the next step. The last article that I I shared was on Cloudflare OS. I do think people are using the word OS

40:05Mehdi: Yeah, I saw this one, yeah.

40:07Dumky: a little bit too much. But the general idea is that you if you start using LMs and AI in your workflows, in your processes within your organization, you need to have a safe way for people within your organization to deploy those apps and work on those apps. And so Cloudflare obviously has a lot of infrastructure to deploy apps. They've been adding like AI infrastructure on top of that as well. and so this is like a very nice sort of sandbox basically to combine all these diff different aspects and create an AI app for your organization. If you remember, Retool was like a app building platform for organizations and there's always been a need for like bespoke applications within an organization. this is kind of I would say the next version of that. So it allows you to automate processes while still, you know, keeping some kind of interaction or validation or human in the loop kind of stuff in there in a sandbox safe way scalable way because it runs on Cloudflare's infrastructure, has all the authentication and all those kind of things. So this is super interesting for me to see. Yep.

41:18Mehdi: Yeah. Yeah, I think the shareability and and how you collaborate with people as I show the demo right now. I think that's one of the big plus because a lot of people, you know, just building on their own side and deploying and I think the collaborative aspects. I think I talked we talked about that in to another part where I listened to one of the pods that was explaining. Do we have one agent that's basically work for everybody that we can interact and see? So internally, for example, at MotherDuck, we have Superduck that can go to the logs, do specific action, and sometimes in the Slack thread you can see people, you know, asking questions directly and collaborate. So it's like one agent for multiple people and there is clear visibility. Today I feel we are much more in an era where everybody has has its own thing, right? But I don't think it's it's super actually nice in terms of collaboration and real world down. can be much more creative if you can easily build on others. So yeah building those those workflows together would be nice. Yeah.

42:26Dumky: Yeah, and and this to me has been how teams have worked traditionally, right? So so a lot of teams had let's say you had a shared Excel sheet, for example, to manage signups or to manage schedules, those kind of things. Those are real requirements from teams to collaborate and and work on together. I think those are the key kind of things that can be automated, turned into applications. And just improve your workflows and processes.

42:55Mehdi: Yeah. So I have two more quick links before we close the part as we are already doing timeout. one thing I wanted just to call out, Supabase release their evals and I was I found that super interesting. It's basically how agents works with with their tools, so with Supabase here. and it's public so you can see which model works really nice. Apparently Kimi works the best with GPT-5.6 but you have also with and without skills which also is interesting to see and so I was just thinking this is really nice just for any product to basically do stores evals on like how models work really well and just have them publicly available because in the era where everybody can pick kind of like on demand what model works you could you know get into situation where basically there is a model checking out yes let me you know spawn up Supabase database actually let me switch model and then over to another one because apparently it's doing a better job than me with Supabase for some reason. so yeah I found I found this interesting I think more more company I should have it and we should have it. And the last link is

44:20Dumky: Yep, that was super interesting.

44:21Mehdi: to just do in the role. super logical, you heard about it.

44:25Dumky: I have not heard about this. What's it?

44:27Mehdi: So if from Hashimoto, so creator of HashiCorp and Ghostty, is launching a new company which is a multiplexer. So tmux stuff, basically, which is

44:39Dumky: Okay, okay, interesting. Yeah. Yep.

44:41Mehdi: yeah, which is interesting because I've spent a lot of time personally into multiplexer. I've been moving to Herder, which is a new multiplexer build a bit more for AI and so something to follow is not there yet. but there is a lot of you know interesting I would say architecture thoughts over there and how multiplexer

45:01Dumky: Yeah, and so so the idea is that this this goes hand in hand with Ghostty, like your as your terminal or yeah, yeah.

45:07Mehdi: Yeah, yeah, that would go end end to end Ghostty, but I think yeah, the for the moment there is not that much. you know, Mitchell Hashimoto just chair Really some nerd video where he explained the technical thing. but I I I found it interesting because I did I've seen much more people using a multiplexer. I use a lot tmux in my early career. you know, when I was working on an on-premise cluster with Spark because I was basically executing on remote servers super easy. And then for some reason it just got away. I didn't need that much. or I mean I was still using it, but not so easy. And since like Like a year now has been really heavy on my side. and I if you look at Reddit around tmux. it it is growing. There is a post like every hour around someone

45:56Dumky: Yeah, yeah, yeah, yeah. Yeah, yeah, no exactly.

45:59Mehdi: building a plugin to get, you know, specific integration and and so on. And so I do think if there is something that's being built from the ground up, that's something to watch. yeah. I don't know if you've been using some of them lately or not that much.

46:15Dumky: So it's it's been interesting because I I I think in this sort of AI age you need to reevaluate your tool set every now and then, right? And so at some point I I set up tmux as well with iTerm, so not Ghosty. and it worked quite well, but then my workflow kind of evolved. I was using I've been using conductor. For a long time. And so I don't need it, find myself needing it as much. but I do now find myself looking at other things. So conductor is getting a little bit away from my preferred workflow. they're they're like adding a lot of stuff on top that I don't necessarily like. so I am reevaluating my my workflows again, seeing seeing where this is going. So this is, yeah.

46:59Mehdi: Okay. Cursor cursor has changed a lot, right? Cursor has changed a lot. But what I like about multiplexer is that I mean for me the big plus is trying to have something remote and locally is, you know, is really easy. Yeah.

47:13Dumky: Yeah, exactly. So that's the part that I've been missing in my workflow as well, yeah.

47:16Mehdi: Yeah. So I have I have a Mac Mini for my home lab and so being able to just remote there, create a session, continue, you know, I have it on the go. and I want to call out, but I I'll do a a video on my channel, so this is not for for today. if you're interested in a multiplexer, there is also an app called I don't even know the name. What's the name? Jesus, you know sometimes you have an app installed and there is not even the name around. Moshi. Yes, exactly.

47:46Dumky: You just know the icon. Moshi, yeah. Yeah.

47:51Mehdi: Which is a really small app, I think, from a s a single developer. But again, it's super interesting because it's it's a mobile app designed for agent but for multiplexer. So the UI you is recognized the tmux pane and it's designed so that you can navigate and you have the chats, you don't need to, you know. So just to say that it is It is really interesting to to see how multiplexer is, you know, evolving in our day work and yeah, if it's bit from ground up, I would

48:23Dumky: I I will give a warning to that though. I recently went on on a holiday along weekend for for hike. I decided not to bring my laptop, but I hadn't realized

48:32Mehdi: yeah, I see what you're arguing.

48:34Dumky: my my Claude Code was still available on my my phone, so I ended up spending way too much time Claude coding on there. So

48:40Mehdi: Yes, that is true. That is true.

48:43Dumky: yeah, that's that's definitely dangerous to take along with you if you want to unplug.

48:47Mehdi: having having good separation. For me it's more like if there is nudge to to be think and and review on the go. But but yeah, it this this is also a good point. Anyway, that's the end of this episode. If you want all the links and all the notes of the show, you can head up to MotherDuck.com slash podcast. Everything is there, or you can look in the description if you're looking on YouTube or Spotify, whatever have you. And I'll see you in the next one where we have a special guest. So Stay tuned and Dumky have a great end of day.

49:21Dumky: Yeah, thanks, Mehdi. It was a lot of fun.