What's new in DuckDB & MotherDuck 🦆

0:12hello

0:17everybody and uh we are live uh hopefully um let me know in the comment if it's also a hot weather because I'm based in Berlin and I'm burning so if I'm start sweating during um the live stream it's uh completely normal but say hi in the chat if you are around uh with us uh this morning tonight and say where you

0:42are joining from and this is um kind of a special episode uh today because uh I'm flying solo uh might be a shorter episode than an hour U but I wanted to share with you uh some updates regarding Doug DB and mod dug but also give you some tips if you are into uh Doug DB and also any

1:06other related data content how do you uh keep up with all the information and uh how do I keep up so I'll share u a lot of resource uh today on how you can keep up with uh the different things uh around DB also just uh in data in general but please uh don't leave me alone because I'm alone for uh this live

1:27stream um so I have a hi from ell from Mexico uh ector nice and Gregory

1:36far I from France France uh so nice to to have you so we already have two people and I'm not sure why the chat uh display is not uh working but we do have people uh and that's that's beautiful anyway um so what I wanted to start with is uh basically kind of like as I said uh how

2:05do I uh keep up with Doug DB content because actually I get that question uh often uh which uh you know people are following me because I stay myself up to date and then I basically you know um

2:21kind of like uh filter the content and share with the with the others uh so uh I'll give you uh some trick and let's start first with uh I think for dug DB itself um a ga repository let

2:40me share my screen there you go so um this K up

2:48repository which is uh aome dub um is

2:52really great if you want to just have a single place with uh with all the resource uh around dgdb so you have uh specific documentation tools that's been powered by dgdb web client libraries um but also you know a knowledge to learn so around media um

3:14and uh courses blog post so that's that's also a great uh start uh it just reached 1,000 um 1,000 Stars uh this weekend uh so congrat David uh if you're uh with us uh tonight uh that's amazing and so I invite you to uh to to look at this one I'll share directly in the chat uh would

3:42it be you are on YouTube uh hi joavan so

3:47we have joavan from namia wow that's already pretty far from uh from me um so as I said uh this skub

3:58repository is a great start if you want to learn more about Dub in general is getting updated uh often you see there was one at uh thing haded yesterday another um thing that would highly recommend you to check out and of course I'm a bit biased because you know I'm working on mod deck but this is actually

4:19a community newsletter and we try to as much as possible give the voice to um the community so if you want to actually um uh part participate to the to the writing of uh this duck DB newsletter you're more than wake up you can uh ping uh ping us or ping me directly um but basically the ddb newsletter has been

4:41running for more than a year and what we do is basically gather the the top Links of the of the month uh so Blog video whatsoever we highlight the Community member uh that has been a great contributor or did um an interesting project or blog and we give you information about the upcoming events and those events are

5:04really large it's not like it's not because it's you know on the website it's really all kind of uh events related to ddb uh Talks by done by people etc etc um so this is a a really good place this was the last Edition you see you have always um a link to the blog a small comment and so we usually

5:25do uh 10 links the last one was writen by uh spy um shout out to him uh was really great to collaborate with him if you want to collaborate to this with uh with this newsleter uh let us know we had a couple of other uh guest writer for for this one so that's the second um

5:46source of information so the repository um the DU DB newsletter and why do I have uh the third one let me go to my list

6:00my mouse is getting uh shaky and I'm curious also uh if you're joining us let me know how do you keep up with DB contents or you know data content in general I'll be curious to know maybe I'm missing something uh completely obvious uh but I would be curious uh that you you share with me so um the

6:21second point is also actually on the on the mod dock website and it's uh the events page um so the events page basically uh Gather in as I said like the different events uh that we that uh we're doing um

6:39around dug DB so that both dug DB Labs or individual order company uh webinar so we try to list there uh we don't have a a submission form and we're going to fix that uh pretty soon so that if you have also an event please um reach out and there will be uh an online form for you to to to fill in so that uh

7:01basically you can also add uh your events that you want to highlight around uh dgdb so that's a third one the four one uh resource for dgdb uh I just started um I'm curious if you are using um uh

7:21daily dodev uh daily dodev is actually if I open my browser basically this is um you know what your uh you can you can see from daily. deev and I'm I'm following specific tags you know around also web development rust and ddb information and they also you know use RSS feed so you can find back in you

7:45know mod duck uh blog to if I search here and you see have some of them are vote some and there is comment so it's kind of like a community uh thing around new content uh del. Dev it's an app there has been been using for a while and so they introduce uh a squad which is basically um an OP where you regroup

8:06people at the moment I just started yet and you see I already share a couple of specific uh blogs as soon as they uh basically uh release so today I I shared this one which was released on the on the WB website done by Mark uh to explain how memory management is done in ddb and so this is also something I invite you

8:30to uh to to join so I'll actually uh

8:34share also uh that in the chat uh there

8:39the dev 30b Squad U so if you're using

8:45um a lent from India uh so if you are using a delete de again that's uh it's not a sponsored product by the way uh this is just just how personally I keep up with the content if you don't know the uh the app I would really recommend to uh to check it out uh I think it's it's it's a really

9:09interesting um uh concept and and so the

9:14the point is that you you can join the the squad and as the squad is growing other people can contribute and share uh share the links all right so that will be number four uh daily. de uh another point to keep up with content is to join uh Community slack or Discord so there is the slack uh the slack of mod deck uh

9:39and uh and there is the Discord of ddb

9:46uh if you go to the Discord of

9:51dgdb uh you're GNA have uh some Channel dedicated to uh to sharing content which is called uh and tell so let me du DB and mother duck

10:07slack and then I show you directly all right so that's sharing the chat um and so uh yeah on the Discord

10:18let me open Discord quickly you're going to have a a channel for uh sharing content we do have on our channel uh also

10:58e

11:28e

11:58e

12:28e e

13:23uh yeah so I'm not sure what happened uh but yeah my mic uh died for

13:32a second hopefully you can hear me uh you can hear me now uh yeah so it's coming back somewh yeah maybe on the

13:43end yes I do uh it just took me a glad I

13:48hope you're still around um and so the chat the the sun is is back uh just my

13:56wireless uh dying thanks for

14:06that all right so uh regarding uh

14:14the let me go back to the

14:21screen so yeah uh regarding uh the the

14:25other way to uh get updated so I was on the blogs uh I basically cover most of those things um so just to summarize you have the GI up repository you have um the bduck website

14:43with the blog and the events you have the Deb website with uh the blogs and you have also the daily dodev um uh Squad I saw that a couple of you are are already joined uh that's awesome um and and uh yeah and the Doug DB newsletter itself um so if you go to community and Doug DB news where we share monthly

15:08newsletter so yeah that's about it about how to keep uh content with um uh duck DB uh sorry with duck DB and mother duck U no I want to basically give you some update on what's actually new in uh those two and I think uh so I have a couple of points in my list um the first

15:34thing I want to talk is actually the community extension so ddb has extension right and I'm going to start actually with this project I built already year ago which is a bit broken but U that's something else so dgdb has extension which is supported by dgdb labs and they have also extension which is built by Community you can extend the extension

15:58to various Fe and um the challenge is that it's really difficult to have visibility on what people build there is no really like an App Store where people can see what kind of U uh Community extension there is and also so this cability is a problem and the other problem is that um it's not easy for people to install um and load

16:24application um in opposite of like supported extension so if you if you open the terminal and you install I don't know install ums for example this is a supported extension that you have by the way I'm just opening the DB CLI if you new to DB you can install the CLI which just reinstall on a Macos let me Zoom this uh

16:52that's better so the point is uh you can have you can see the extension um

17:00here uh that's the supported extension by the gbls and they're really to install and lo that soon as you have a community extension you need to basically be able to build this extension and make it available and then people need to download them so they need to have the URL and those start to be you you see where I'm going a bit

17:20complicated to install a community extension until now so uh ddv Labs uh

17:26released the um community extension uh website and store so how does it work is that you have a repository of uh the DU DB

17:39extensions which is uh

17:44there and I'm going to also take the time to share in the chat and to double check that

18:00yeah um sorry for that let me increase

18:04the volume that should be better and I'm going to increase also uh there yeah it's my mic that just died in the live stream of

18:27course all right uh so I was just uh mentioning the the the community extension so you can go uh over

18:38there um if you want to see

18:47uh basically

18:53um yeah uh yeah yeah I I tried my best to uh increase the volume and maybe what I can do uh it's also let let me give

19:04something I'm going to do this all right normally now that

19:11should be way

19:16loud so uh coming back to the extensions you have basically um a repository here

19:24um that where you can basically just do a PR if you have um you know build an extension that you want to add to the store and so you just follow a certain template um and basically what happen after is that you're going to have uh it's going to be available and documented on their website and so you

19:47see here we have a couple of uh uh Community extension um for example this one was built by Pedro actually from um Doug DB labs and and what it does is that you can get some data analysis including from ethereum um so here for example is uh getting information about an ethereum uh transfer um so if I want to install the

20:14extension you see it's really easy it's just install the name of the extension from community and you load it so you don't need to have what we used to have in the past where you need to have the link where the extension is hosted the loaded and then it's U loaded and also upgrade is a bit painful because if you

20:34change that URL well you need to have add the same constant to or a new realware so um yeah so let's just try it uh directly so if I do this you see the extension um is loaded I'm just uh uh checking here and

20:54um and so we can copy paste for example

20:58uh this

21:04example and we're going to see that we going to be able to uh load from a given transaction uh blocks uh a timeline so you have also uh stocks information uh you can read a bit more on the this uh this community extension but that's really a big major steps to kind of uh highlight um the the extension built by

21:26the community uh vers uh this project I did which is really passive I just scan from G up and it's really not accurate because I look for a specific extension um where if there is possibly a Doug DB extension so uh hopefully we will get more visibility over time of what uh the community is build is building and also it helped you

21:49if you want you're looking for a specific extension uh probably look that um um probably look if someone else uh as uh as build it um so yeah so that's the one major news uh I would say around uh the ddb uh

22:09extensions so the the other big news um I was a bit

22:16before um is um yeah actually it's uh a

22:22personal plug so I finished um the dgdb statch which is uh basically an endtoend uh data engineering project so if you're willing to learn Doug DB um in different steps uh so meaning you have uh you want to to to see how it works with python with the ingestion transformation with DBT uh that's a really uh good start so if you go to D

22:51DB stats you're going to have the live dashboard which is the uh the live results right um so I'm gonna share that

23:04here also in the chat if you want to uh to to check it out um but

23:13so the here uh you're going to be able to uh follow along uh there is different uh blog series and video series so if I go to the uh to the ga page you have um

23:28here uh the inje part the transform part and for each one there is a Blog and a video and this is basically the high level of the project and basically now it's live uh and so I would really recommend if you're into learning dug DB uh that's a that's a really fun store of project and what it does basically

23:47because I haven't done just shared the the the the link is that um if you go here what it

23:57provide you is the information of the downloads uh for each dgdb um python

24:05package so it's not only looking at the python library because theb has different uh uh client and you see that we passed the 1 million actually download per week now so last week was already almost 1 Point uh 1.2 million

24:21downloads and you have different information about which python version and I found this pretty interesting because you see that a lot of people are still running under uh python 3.8 which is actually going to be end of support end of life uh basically in uh a couple

24:39of months um so um so yeah that's um uh

24:44one thing interesting that you can spot with those uh data and you can do it with your own with any uh python package so if you're interested to get some stats around other python package uh you're free to do do so uh you just need you you just need basically to uh get clone the project change uh so this

25:06project change a couple of variable that specify not to have only the ddb package but your given python package running the pipeline and you get those statistic for for this with the dashboard um so yeah that's uh one big thing that happened um last week uh

25:26other than that uh on mod duck side now um so there is two thing uh we support now the um Delta L extension so if

25:37you're going into our docks into integration you have now a page regarding uh Delta leak uh which was an extension release uh supported just a like four weeks back uh by Doug DB and now it's available from other decks so if you have large table um for example Le on a tree Delta L table mother duck is a great use case because then you can

26:03use the same dug DB client but you basically leverage uh the cloud and uh as we Moder on the cloud uh the the Delta scan is going to be much faster right rather than going to uh your local network so yeah so that's um a really cool thing the other thing is that now we support the special extension as well

26:27um so I'm going to do a bit more content around this in uh the coming weeks um but so the special extension is a big extension actually uh in term of usage uh around de DB because it's really offer A New Perspective when working with geospatial data with what's actually um available uh at the moment

26:51so um I would really also recommend to try to play around the special extension with Mod deck I'm sure you you're going to have a nice ID but if you don't don't worry I'll uh cover a couple of topics and I'll also uh it's worth to note that we did a live stream around this itself

27:11um I'm going to share it it was with Matt which is kind of the expert you should definitely check out this uh YouTube channel if you're into um spal

27:24Geo dat so yes yes uh G I'm my French uh I'm

27:32actually U my mother talk is French um but I'm not speaking French on everyday basis so my French is now a bit uh

27:42pretty bad I would say getting pretty bad so um yeah this live stream uh on

27:49geospatial data uh let me share also for the LinkedIn folks I lost the page uh but if to go on also the yeah

28:08here yeah that's that's also a great resource I invite you to try it out with mother and by the way modu has has a feet here so don't be scared you can just go sign it up um it's really easy um and you won't break the the bank all right so uh that's um that's a bit uh

28:29for all the information on what's new uh I want to close on uh two things uh is the blog so what um whats up uh what

28:41blogs use I I really enjoy this past uh weeks um so the first one is um the one

28:50actually from Simon again is there um

28:54which is about loading from post grade to post grade faster using the LT I think it's really interesting because so do DB has native uh extension to connect to posg right you can easily uh query a

29:09pogress database uh just using uh

29:13basically uh de DB and a pattern an interesting pattern that we can see now is basically if you need to Adder do a migration to a different poest thinking from different database uh doing some backup are you doing some transformation in memory and you want to P it back to uh to pass um so that's all different

29:37use case which are are really interesting and so DLT help uh around

29:44that uh but there is also uh you can use mean Native uh duck uh duck DB uh but I

29:51will share with you the uh the blog here so this is the the the first blog and and uh yeah so that's one I really enjoy um and the other news that I wanted to share is that there is a new book uh from pack publishing uh getting started with Doug DB uh so it's cool because you

30:11have new author um and a new book because we have already um uh an existing book which is uh going to be released soon and you're going to you can have a free chapter by the way here um so that's the the existing

30:30one um but uh as I said you have also a getting started with dgdb from uh pack publishing uh which is uh really interesting um so yeah that was uh roughly it so we covered um kind of like

30:48um this right um what was uh how do you

30:53get uh up to dat with uh different news uh so basic basically you have the Asom G prosit you have the dougb newsl by mother doggy event page the deli. dev dgdb Squad um the Discord and uh slack

31:11uh you have the blogs you have on news we talked about the community extension uh which is pretty neat um the project theb stats if you want to learn an endtoend project using du DB um we can read from Delta L using mod duck yet there is a special extension and finally a couple of blogs and news um so yeah

31:32that's that's about it um I'm curious if you did enjoy this format I'm really sorry about the sound it's of course the karma always with me uh well my micro microphone needs to die during a live stream um but please let me know if you do did enjoy this uh this 30 minutes about uh news on Doug DB and mother duck

31:54I'm happy uh to uh do it again with maybe a better sounds without the the microphone dying and um there is uh just a question from Gregory here uh I'm wondering uh which is the best way to promote dgdb and data uh Tech Team yeah so I would say just start using it with the CLI there is a

32:16lot of stuff that you can do uh with the CLI if I reshare my screen quickly um you know the the the installation that's the cool thing with open source right it's like it's really accessible so rather than than selling it to say yeah you should use Doug DB because of this and that just give like little use case and little use case so

32:39when you have the CLI would be uh crying uh a data from par for example um converting uh a file format from CSV to par to park to CSV are doing some parsings of a a Jon file so there's a lot of like uh I would say one shot task that sometimes you need to do as a data engineer or in

33:03within a data team and I think like just people getting started by using the CLI it's a great way to start advocating to uh for for DTV yeah so that's pretty much it uh don't forget quacken code is um every other week uh again let me know in the comment if you did enjoy uh this episode and the next episode we're going to have

33:29uh someone from data breaks uh to talk about the Delta Lake uh extension and table format so if you are new to table format meaning Iceberg dctal are just like Chinese name to you um that would be uh a good place to go and uh yeah

33:48I'll look forward to that one so I'll see you in two weeks and until then uh keep