Today we are launching Dive Apps: fully custom data apps that can write-back anywhere. Build them with your agent and they run on the MotherDuck data warehouse. Use Dive Apps to replace SaaS apps, automate your operations, or bring humans into the loop to steer your agents. They are the best way to use AI to build your business.

Don’t build dashboards - build apps! You have the power of AI behind you. Anything that used to be a report should be an app instead, and can be just as easily. Connect your agent to our MCP server and an app for your whole team is just one prompt away.

Dive apps work by letting your Dive visualizations trigger Flights, our serverless Python runtime. While you can certainly update data inside MotherDuck, the flexibility of Flights means you can write data back into source systems as well. Or send emails. Or post in Slack. One of my favorites is that you can rerun machine learning models (remember when data science was the sexiest job of the 21st century?). For even more power, you can kick off a full-featured agent harness to act on your behalf.

What can you build?

Dive Apps are designed for flexibility, so there are many ways to use them. Now that custom software is so inexpensive, there are more and more places we will see ROI from building an app. If it takes an hour to speed up something your team does once a week, you see benefits immediately. These are just examples to get your creativity flowing - imagine what you will build!

Replace SaaS Apps

Completely duplicating the capabilities of a SaaS app sounds daunting, but I’m confident you aren’t using every feature of every SaaS app. Agents can replicate just the features that you rely on and customize for your business.

Still paying for project management software? Build an interface that matches your company’s exact workflow. Using a SaaS to stitch together multiple marketing and ads systems? Your agent can load that into MotherDuck and you can build a decision support system for your team. Not only can you make the right decision, but you can push that decision into the upstream systems.

AheadComputing is a semiconductor design company building on top of MotherDuck and is slashing SaaS app costs. Ben Holtzman, staff design enablement engineer, phrased it this way:

"A single license for a power analytics tool for six months could be $50,000. There are basically only three companies that create this vendor tooling. You're operating in a space with a virtual monopoly."

Instead of more licenses, they designed an even more feature rich interface on top of MotherDuck.

Automate Your Operations

Augment your domain experts with Dive Apps that bring together everything they need to make a decision. Then, put that decision into practice right in the app.

Imagine how much faster issues could be resolved if customer support specialists had a custom command center app at their disposal. It could retrieve the full context of how a user is interacting with the product, their contract details, and any similar conversations. Within the app they can send messages directly to the customer. But this is more than a vanilla chat window, this app is a command center after all! It has relevant docs automatically surfaced and best practice snippets ready to embed. As a result, your team moves faster and customers are happier!

How do you handle RevOps today for your sales team? A custom app could build forecasts based on your business’s exact subscription structure and predict based on actual usage statistics in the product.

Instead of just looking at inventory levels, you could actually send out a purchase order when stock gets low. You could game out what-if scenarios to see if same day shipping is worth it or if “priority shipping” is reliable enough to keep your production line up. These kinds of decisions need data from all kinds of source systems.

Steer Your Agents

Over time, agents will take on a larger fraction of the decisions in our businesses. However, if you’ve ever automated a complex business process, you know you don’t just flip a switch and call it a day.

The fastest way to implement automation is to make changes incrementally: first advise, then require approval, then only escalate exceptional cases, then go fully autonomous. Having an app keeps a human in the loop for as long as it is valuable. Customizing that app keeps the time-spent-per-case as low as possible and surfaces all the background someone needs to make a wise choice. Each time the domain expert correctly overrides the agent gets saved to improve the agent for next time. Eventually quality reaches 99.9% and you switch out the lights on that process and let the robots handle it.

That implementation process requires an app that gives the human the ability to write back to source systems and to document where the agent fell short. Without write-back, operators won’t be able to trust the automation or fix things when something unforeseeable occurs.

Build on the Best Infrastructure

There is plenty of noise about where you should build with AI. Why build on MotherDuck? Fundamentally, it’s because it’s incredibly simple to use, brings all the capabilities of a data warehouse, but skips the drawbacks of a legacy platform.

The Easiest Way to Centralize All Your Data

Your most important decisions need data and context from multiple source systems. If a problem could be solved with data that is already in one place, you’ve solved that problem already. MotherDuck Flights are the easiest way to pull data from anywhere into an efficient, central location. Your agent has the full power of Python available (with all its open source ETL libraries) to pull from thousands of sources in just a couple of prompts.

A Fast Query Engine That Spins Up Instantly

Once your data is centralized, you need a high performance analytical engine to make sense of it. Time is valuable, and whether it’s your operational team or executive team, they need answers quickly. Latency is critical in any app. Joining tables together or extracting meaning out of a deluge of log data are compute intensive, and analytical engines are the right tool for the job.

But you don’t want it to be a huge, always on, monolithic cluster. Likewise, you can’t wait minutes for things to initialize. Especially for data app interactions that can be variable, you want your compute to spin up just as you need it and spin down when you are done.

With MotherDuck, you can handle the scale you need without the multi-second queries and high cost baggage of a decade-old legacy warehouse architecture. In short, you need warehouse horsepower, but with the latency profile of an app DB not a batch job system. On MotherDuck, each user can have their own compute resources too, so nobody waits in line.

Flexibility is Key, Governance is Non-Negotiable

If an app-building platform locks you into cookie cutter widgets, no AI will help you build exactly what you want. Conversely, if there are no guardrails, then you can’t deploy with confidence. MotherDuck includes Single-Sign-On (SSO), Role-Based Access Control (RBAC), and now Table, Row, or Column Level Security (TLS, RLS, or CLS). That lets you share your apps with exactly who you need to and lock them down as appropriate. Data in MotherDuck is private by default, so you can opt in when it is time to share with other users.

As an integrated part of MotherDuck, you can manage your data and Dive Apps access all in the same spot. No separate dashboard permissioning system!

Store Your Context For the Long Term

If you want your agent to be able to build an app that fits your business rules, you need context. In MotherDuck Guides you can teach your agent about the structure of your data and curate best practices. That way, your agents understand your company’s data and provide more accurate answers.

Why Guides? Don’t bet on just one AI lab and put all your context in their proprietary system! What happens when the other company drops a new model that laps the rest? You want to be able to switch to the best in class LLM at a moment’s notice.

You already need a place to store your data, so store your metadata right alongside it. Data stores last forever, but the leader in the AI race changes every week…

Apps That Take Action in Any System

An app is really just a report if it doesn’t actually trigger changes in your business. If your business runs right on top of MotherDuck, write straight back to a MotherDuck database. Or if you want to take action in a source system, Flights can push data anywhere and trigger any actions needed. Your data doesn’t get stuck in any database, it can flow wherever is needed.

Actions can range from something as simple as a Slack alert to actually executing a discount on a customer’s bill. Those financial decisions are cases where the augmentation approach that keeps a human in the loop can be most effective. After the agent gets 50 right in a row, then you can move to auto-pilot with a scheduled Flight!

Conclusion

If you’re running your business just with an agent on every laptop, you’re still in the spreadsheet era of AI. Sure, you’re emailing around artifacts instead of xls files, but you’re not getting the gains that the AI labs promised.

To get the most out of AI, you want to automate entire process flows. To do that accurately, you need the right data and the right context. You need the right guardrails in place to set your organization free to innovate. Dive Apps let you move your business faster because they tie in directly with the full power of MotherDuck.

We can’t wait to see what you build! Quack on!

Aside from the absurdity of this question, it is actually a pretty decent metaphor for the technology we’ve built at MotherDuck. Whereas most of the prevailing analytical databases are designed around having a bunch of small instances working in concert, we made the conscious decision to let our ducklings get big rather than to harness more of them at once. A Snowflake 3XL, for example, is basically 64 Snowflake extra-small instances in a trenchcoat; that is, 64 fixed-sized instances that are networked together to collaborate on running queries. A MotherDuck Giga instance, on the other hand, has about the same amount of CPU and RAM as a Snowflake 3XL, but is in a single, very large machine.

One of the points I made in the blog post “Big Data is Dead” four years ago was that the line between what is considered “normal” data size and “big” data keeps moving. Above the line for “big” data means you have to use special handling, special compute, special storage to be able to use it effectively. “Normal” data is the size that just works, typically what you can process on a single machine. As the threshold for big data moves higher and higher, the number of workloads that need to be handled using “big” compute architectures goes down.

So where, exactly, is the line for Big Data? In the early days of MotherDuck, we used to describe it as “about a terabyte.” However, since then, machines with hundreds of cores and terabytes of RAM have become pretty standard. Moreover, people have gotten better at writing code for high core counts, which means you continue to get better performance as the number of CPUs in a machine increases. These days, I’d draw the line at least 10x higher, as workloads that are 10s of terabytes are now pretty straightforward to handle on a single machine.

Even now, database people tend to scoff at the idea of a single node architecture. That can’t possibly scale! So I’d like to offer an existence proof. Let’s see how MotherDuck scales to workloads of 10s of terabytes. And let's see how it lines up against more conventional data warehouses with distributed architectures.

Giga Duckling vs the World

Our largest instance at MotherDuck is called a “Giga.” We don’t usually share the precise hardware specifications, but it uses the better part of an instance with 192 cores and 1.5 TB of RAM. I wanted to see how this instance stacks up against other data warehouses as we start to toe the line into big data territory. Just how big can you go before the single node architecture starts to be a liability?

It is dangerous to write blog posts about performance because people tend to accuse you of cheating; either you optimize your system behind the curtain to make it look better or you fail to optimize the competitor’s system to make it look worse. What makes things even more complicated is that many vendors have “no benchmark” clauses in their terms of service that threaten to do nasty things to you if you publish benchmarks.

My solution to this is that I’m only referring to numbers that other people have published. Then I don’t have to worry about violating their terms of service. The downside is that some of them are a little bit old; that would be a deal-breaker if I was trying to say one service was faster than another. But I’m really just trying to show that MotherDuck can keep up.

The other thing that can be somewhat sketchy is when vendors report their own numbers. That’s why a lot of people get third parties to do their benchmarking. But that tends to take a lot of time to do, and I’m taking a slightly different approach; I’m just asking Claude to run the benchmarks, with no special instructions. I also inspect the output to make sure it doesn’t do something weird.

I picked the TPC-DS benchmark because it is the hardest data warehouse benchmark and one that scales to large sizes. It has 99 different queries, with some fiendishly difficult joins. You can generate the benchmark data to run at any scale. A scale factor of 1 corresponds to about 1 GB of data stored and scans about 17 GB of data during the course of the benchmark.

To score the benchmark results, we report the geometric mean of the query times. This is in keeping with the other results that were reported, so we can compare apples to apples. It is also generally considered best practice to use geomean when running database benchmarks because some of the queries have very different complexities and expected runtimes. If you used the total duration or the average (arithmetic mean) duration to compare timings, the results would be dominated by a handful of the longer running queries and the short ones would barely count.

The first terabyte

To start out with, I looked at the TPC-DS 1 TB size (scale factor 1,000). MotherDuck ran this benchmark with a geomean of 690 milliseconds. How does that stack up against other “big data” warehouses? Well, I found a blog post from Fivetran in 2022 that has Redshift at 4.2 seconds, Snowflake 2XL at 4.9 seconds, Databricks 2X at 6 seconds, and BigQuery at 6.1 seconds (data here). This is a few years old, and all of those other systems have likely gotten a lot faster since then, but even so, it doesn’t look like MotherDuck struggles at this data size.

We also ran this benchmark using the smaller MotherDuck Mega instance, which clocked in at 2 seconds, which is still faster than the competitive systems. So you don’t need the biggest duck in the pond to handle the 1 terabyte data size.

Tiny Terabytes

There are times when one terabyte just won’t do. What about ten? Surely the single node DuckDB architecture starts to founder at that point, no? Why don’t we try TPC-DS at scale factor 10,000? This is about 10 Terabytes on disk, and scans 168 Terabytes in the 99 queries of the benchmark. To put this in context, running this benchmark on BigQuery with their default pricing model costs more than $1,000.

The MotherDuck result is now 5.6 seconds geomean. To compare with other data warehouses, I found a blog post from Oracle, also in 2022, that benchmarked Redshift, Snowflake, BigQuery, and Azure Synapse, with instance sizes that are all price-wise just a little bit more expensive than the MotherDuck Giga. The results are pretty close together, with Redshift the fastest of the non-MotherDuck lot at 8 seconds flat.

To be clear, the goal here really isn’t to claim that MotherDuck is faster than any other engine. I’m just saying that at scales that many people consider to be “pretty big,” MotherDuck, with the single-node architecture, is competitive. It isn’t just for toy data sizes.

On one hand, these results are a few years old, and every engine on the list has almost certainly gotten faster in that time. On the other hand, I’m also running an older version of DuckDB. DuckDB 2.0 came out last week and has some very significant performance improvements, particularly for larger instances. These benchmarks were done against DuckDB version 1.5.6 because we haven’t upgraded to version 2.0 yet in MotherDuck (we’re waiting for the first patch version to come out next month). So I may not be showing other engines in their best light, but I’m also not showing DuckDB in its best light either.

Ducks, Ducks, and More Ducks

These results show that scaling up can work well to handle large workloads that scan 10s of Terabytes of data. That scale includes the vast majority of workloads at the vast majority of organizations, and the results vindicate the decision by the DuckDB team to focus on making a super fast single node engine.

We bet heavily on DuckDB, as not only a robust engine but also one that was going to get better quickly. This has borne out well because DuckDB has continued to improve at a rapid pace; over the last year the performance on these benchmarks has largely doubled, and we expect that the recently released DuckDB 2.0 is likely going to significantly improve performance again for these workloads.

At MotherDuck we’ve also gotten very good at running DuckDB at scale. Our FlashCache storage engine creates a distributed tiered store underneath DuckDB that provides great performance even on cold starts. On this benchmark, MotherDuck scales as well as competitive systems at a similar or larger price point. We’d go further and say that if a workload can be handled on a Snowflake XL, it would be surprising if you couldn’t do it in MotherDuck faster and less expensively; after all, a MotherDuck Giga has a lower price point and has more CPU and RAM with a lower latency architecture.

We have a long-standing bet going with one of our investors about whether we would need to make MotherDuck run in distributed mode. There are still some workloads that do struggle to scale up, and I’d expect we’d want distributed support before long. In the age of AI, adding distributed support gets easier and easier (I know of three ongoing projects to do so). That said, the number of workloads that need distributed mode is pretty small, and gets smaller over time as machines get bigger. In the meantime, make way for giant ducklings!

Appendix: A note on the Methodology

We ran the standard TPC-DS (99 queries) query sets, unmodified, using the query text that ships in DuckDB's tpcds extension.The queries ran against MotherDuck instances (Mega or Giga) running DuckDB version 1.5.6. The Scale Factor 1k dataset was generated with DuckDB's built-in dsdgen and stored in a MotherDuck share; TPC-DS 10k scale factor set came from Amazon's public dataset (
s3://redshift-downloads/TPC-DS/2.13/10TB) . The queries were fun from a Python DuckDB client (also version 1.5.6) on EC2 in us-east-1, with the parameter attach_mode=single which avoids loading databases other than the ones being used in the benchmark. Each time is client wall-clock and includes fetching the results. The first run is cold, and we report the best of three iterations per query. Results are summarized as the geometric mean of hot query times. The blog posts we compared with also reported ‘hot’ query times, and the approach taken was similar to how snowflake ran TPCH in their own recent blog post.

I hope you're doing well. I'm Simon, and I am happy to share another monthly newsletter with highlights and the latest updates about DuckDB, delivered straight to your inbox.

In this October issue, I gathered the usual 10 updates, starting with Jev, the new "System One" model and how to use it with DuckDB and MotherDuck, the new PEG parser and faster S3 reads in DuckDB 2.0, a maintenance strategy for DuckLake, querying OneLake in the browser, and a native Mac app. Lots of great articles from the community as always.

Data Outpost is happening November 4-5 in SF, with speakers from DoorDash, Perplexity, Stripe, Canva, Vercel, and Fivetran + dbt Labs on agentic data infrastructure, the semantic layer, and data stacks after agents.

Introducing prompt_jev(): bringing Jev to DuckDB/MotherDuck SQL

TL;DR: Jev, TypeSafe AI's recently announced "System One" model, returns a typed answer (a yes/no probability, one label from a fixed list, or an ordinal score) instead of generated text and can now be called from SQL through the jev DuckDB community extension and MotherDuck's prompt_jev().

I combined different articles for this paragraph, in the first, Geertjan and Gábor walk through INSTALL jev FROM community; and the jev(), jev_prob(), jev_choice() and jev_score() functions. They note that a cheaper predicate such as age > 40 in the same WHERE clause runs first, so rows it rejects are never sent to the API. Then Hamilton, Till and Jacob report 89% accuracy on 100k AG News rows in 40 s for $0.50, against 83% in 17m49s for gpt-5-nano and 88% in about 32 minutes and $37.58 for GPT-5.6 Terra. Runs at 1M and 10M rows performed similarly, and the post includes SQL that reproduces the benchmark from hf://datasets/fancyzhx/ag_news/....

Mehdi classified 100k CFPB complaints in 82.2 s into a struct with request.choice and request.confidence fields. On high-confidence rows (≥ 0.8), Jev matched the label GPT-5 and Claude Opus agreed on 96.9% of the time. In Part 2, he swapped a remote-work regex (22 hand-tuned alternatives plus exclusion lists) for a single choice := question. On 2,000 postings it ran in 1.7 s, versus 37.0 s for prompt(). The regex was still faster at 0.2 s, but Jev only needs label definitions to maintain. There's also the video version with Hamilton.

DuckDB v2.0: Your Database Deserves a Better Parser

TL;DR: DuckDB v2.0 replaces its PostgreSQL-derived LALR parser with a PEG-based parser to simplify grammar evolution.

Daniël explains that shifting to a Parsing Expression Grammar (PEG) allows DuckDB to avoid traditional Bison shift/reduce conflicts by using ordered choices and packrat parsing. This memoization technique prevents exponential backtracking on malformed queries; for instance, parsing 19 unmatched parentheses improved from 10.6 seconds to 0.001 seconds.

The new architecture allows extensions to register custom grammar rules at runtime, such as BigQuery-style pipe syntax (|>), by hooking into extension points like select_atom_rule. This enables developers to reuse existing internal rules like GroupByClause or WhereClause within their own syntax transformations.

Kelsey Hightower using DuckDB

TL;DR: Even Kelsey Hightower (known for his work with Kubernetes & cloud computing) is using DuckDB , and says:

I've been playing with DuckDB and I gotta say the docs are amazing. The built-in web shell is a really nice touch. You can play with DuckDB right inside your web browser. This might be my SQLite replacement.

DuckLake in Production: Maintenance Strategy

TL;DR: Hoyt details automated snapshot expiration and file cleanup routines to optimize storage and query performance.

Technical implementations focus on managing the DuckLake metadata catalog and S3 storage. DuckLake optimizes small writes by inlining data directly in the catalog, which may eventually require ducklake_flush_inlined_data to move rows to Parquet. After attaching the database, engineers should execute CALL ducklake_expire_snapshots(..., older_than => now() - INTERVAL '7 days') to enforce time-travel policies. This must be followed by ducklake_cleanup_old_files to purge physical data from S3 and ducklake_delete_orphaned_files to remove untracked Parquet files.

Analysis-Ready OpenStreetMap

TL;DR: Mark highlights how the Layercake project's spatially-sorted Parquet files enable high-performance OpenStreetMap analysis with DuckDB via metadata-based row-group skipping.

Mark demonstrates how DuckDB's Parquet reader efficiently processes the 39 GB buildings dataset. A count query with GROUP BY 2 on 657 million rows consumed only 2.2 MiB of bandwidth, showcasing projection pushdown and column-level statistics. The data is updated weekly and hosted on Cloudflare. He also details integrating these datasets into QGIS using the GeoParquet Downloader plugin, which utilizes an internal DuckDB environment to perform bounding box queries against remote files.

Falling Into Databases: The DuckDB Story with Hannes Mühleisen

TL;DR: Hannes explains how DuckDB grew from an accidental career at the CWI research lab into a from-scratch in-process analytical database.

The team first tried retrofitting MonetDB, but usability requirements leaked into engine design, so they restarted despite a roughly three-year setback. Adoption came from fixing concrete pain points: a robust CSV parser, Parquet reading without Hadoop, and SQL conveniences like trailing commas (SELECT a, b, FROM t). Another one is that operators spill to disk instead of crashing when intermediate results exceed memory. DuckLake later stored lake metadata in a database rather than Avro files. Quack is an HTTP-based protocol with DuckDB on both client and server, so queries can run locally, remotely, or split between them.

Serverless, in-browser SQL over your OneLake

TL;DR: The OneLake Studio project offers a SQL interface for querying OneLake data directly using DuckDB-WASM, eliminating the need for backend infrastructure or data copying.

The application is a static web app. It integrates with Azure Storage via delegated permissions (user_impersonation) to access data using an Entra ID token and queries data through OneLake's Iceberg REST catalog, processing tables via Parquet files. For efficient data retrieval, it utilizes HTTP range requests and implements a service worker (site/sw.js) for caching responses (up to 512MB).

Data files are registered as URLs, allowing DuckDB to fetch only required row groups and columns, with limitations including a 200,000-row cap for query results to prevent browser tab crashes. Other file formats like CSV, JSON, and SQLite are handled, with SQLite files being copied and converted into DuckDB tables. Web Demo

Bedevere-wise: In-browser SQL data viewer for SAS, SPSS, Stata, Parquet, Excel, and CSV files

TL;DR: bedevere-wise is a local-first viewer that utilizes DuckDB-WASM to query and visualize files like Parquet, SAS (statistical software packages), and SPSS without data leaving the device.

The technical implementation relies on a DuckDBService wrapper and a DataProvider interface to drive canvas-rendered grids and Vega-Lite charts. Integration is straightforward; developers can await duck.initialize() and then execute queries like CREATE OR REPLACE TABLE table_name AS SELECT * FROM read_parquet('file.parquet'). The project also features an EmbedSqlEditor with a PostgreSQL-extended dialect. Web app, Latest updates

DuckDB 2.0 and S3 File Read Performance

TL;DR: DuckDB 2.0 introduces native asynchronous I/O for file reads, delivering a ~67% performance improvement for querying large datasets on AWS S3.

Matt benchmarked the 2.0 Alpha against version 1.5.6, processing thousands of JSON files. By utilizing read_json_auto() combined with UNNEST, the test showed that the async engine reduces wait times on high-latency storage. For a workload of 10,000 files, the execution time decreased from approximately 297 seconds to 99 seconds.

I built a native Mac app for DuckDB

TL;DR: Haris developed Datapuddle, a DuckDB client optimized for columnar data rendering with minimal memory overhead.

Haris implemented a custom grid view utilizing CGContext and CTLineDraw to read cells directly from DuckDB's columnar chunks at draw time, bypassing row-object instantiation. This maintained flat memory usage while processing a 41-million-row Parquet file. For Mac users, this provides a low-latency environment for ad-hoc exploration of large datasets without the resource footprint of Chromium-based alternatives. AppStore (not free though), Reddit post.

Behind the Agent: How Hazel Built an AI Coworker for Consumer Brands

2026-10-21. h: 09:30 PT. Online

DuckLake Performance Optimization: Faster Queries From Your Lakehouse

2026-10-27. h: 10:00 PT. Online

The Data Team's New Tools (MotherDuck at Helsinki Data Week)

2026-10-28. h: 15:00. Helsinki Bryggeri Brewhouse, Helsinki, Finland

Data Outpost 2026

2026-11-05. h: 08:30. Convene 100 Stockton, San Francisco, CA

DuckDB 2.0: How to Speed Up Your Workloads

2026-11-10. h: 09:00 PT. Online

Today, with TypeSafe's Jev 1.14, prompt_jev also takes images. Store them in a BLOB column, ask your questions, and the answers come back as columns you can filter, join and GROUP BY with the rest of your data.

And yes, we also asked seven models the only question that really matters: hot dog or not hot dog? On hotdogbench.com, Jev isn't the top dog, but it's the first one to the bun.

Try it in one query

Hugging Face hosts lots of public image datasets as Parquet, with each image stored as bytes. Read them with hf:// and hand the bytes straight to Jev:

SELECT image.path AS file,
       prompt_jev([{name: 'dish', image: image.bytes}], 'Is `dish` a dessert?') AS is_dessert
FROM 'hf://datasets/ethz/food101/data/validation-00000-of-00003.parquet'
LIMIT 5;

| file | is_dessert | | ----------- | ---------: | | 1827892.jpg | 0.72 | | 2802124.jpg | 0.90 | | 2192983.jpg | 0.90 | | 1062952.jpg | 0.95 | | 3157161.jpg | 0.93 |

Each row gets a probability between 0 and 1. These five photos are beignets, so they score high.

How fast, how accurate, how cheap?

We used the same dataset: Food-101, labeled photos of 101 dishes. One question per photo, "Which dish is shown?", with all 101 dishes as the allowed answers, asked to Jev, Gemini, GPT and Claude on the same 1,010 photos (10 per dish).

| Model | Rows/s | Accuracy | Cost for the 1,010 photos | | ---------------------------------- | -----: | -------: | ------------------------: | | Jev (prompt_jev in MotherDuck) | 44 | 90.2% | ~$0.06 | | Gemini 3.8 Flash, low thinking | 10 | 96.8% | $1.14 | | Gemini 3.8 Flash, default thinking | 7 | 97.0% | $1.43 | | GPT 6.1 Sol | 7 | 93.4% | $1.89 | | Claude Opus 5.5 | 9 | 95.7% | $6.60 |

We can score accuracy here because Food-101 has the right answer for every photo. On your own data you won't have that, so the real question is which answers you can trust.

The big difference between Jev and a standard LLM is the confidence score. Every answer comes with one, and on this benchmark it reflects how often Jev is actually right. If you keep only the answers at 0.8 or more (WHERE jev.confidence >= 0.8), Jev is right 97.7% of the time, on 80% of the photos.

| Min. confidence threshold | Photos kept | Accuracy on kept photos | |---|---:|---:| | (keep everything) | 100% | 90.2% | | 0.5 | 94% | 92.7% | | 0.7 | 86% | 95.8% | | 0.8 | 81% | 97.7% | | 0.9 | 73% | 98.5% | | 0.95 | 65% | 98.9% | | 0.99 | 50% | 99.2% |

This of course begs the question - what if we combine Jev with another model to a cheap first pass and then a secondary cleanup loop?

| Models to use | Accuracy | Cost per 1,000 photos | |---|---:|---:| | Gemini 3.8 Flash alone | 97.0% | $1.40 | | Jev + Gemini 3.8 Flash | 96.5% | ~$0.45 | | Jev + Gemini Flash-Lite | 95.3% | ~$0.18 | | Jev alone | 90.2% | $0.08 |

Notice how we can achieve big gains in accuracy for relatively small increases cost compared to using a System Two LLM for the entire job.

Getting your images in

If your images sit in S3, one query loads them into a BLOB column (with an S3 secret in place):

CREATE TABLE images AS
SELECT filename AS path, content AS image
FROM read_blob('s3://my-bucket/photos/*.jpg');

Jev needs each side of an image between 280 and 1,568 pixels. Bigger images need a resize first: a Flight (Python that runs on MotherDuck compute) with Pillow does it. The docs have a ready-to-use example that reads from S3, shrinks anything too big and writes the result back as a BLOB.

Or skip the code and ask the MotherDuck MCP server from Claude or your agent of choice:

Load the JPEGs under s3://my-bucket/photos/ into a new table photos.main.images, one row per file with the image as a BLOB. Resize anything longer than 1568px on its longest side, skip images under 280px on their shortest side, and run it as a MotherDuck Flight. Tell me how many images landed and how many were skipped.

Five centuries of art in one GROUP BY

We took the Metropolitan Museum of Art's open access collection, half a million CC0 catalog records, and kept every public-domain painting, print, drawing and photograph with an image: about 77k works, loaded as 600px BLOBs. Then we asked 14 questions about every image in one SQL query. Here are 4 of them:

SELECT object_id,
       prompt_jev(
         [{name: 'art', image: image}],
         questions := {
           smile: {type: 'choice',
                   instructions: 'If a human face is clearly visible in `art`, what is the expression of the most prominent face?',
                   criteria: ['no_clear_face', 'neutral_or_serious', 'closed_mouth_smile', 'smile_showing_teeth', 'laughing']},
           dog: {type: 'noul', instructions: 'Is there a dog anywhere in `art`?'},
           dog_box: {type: 'box', instructions: 'the dog in `art`'},
           skull: {type: 'noul', instructions: 'Does `art` show a human skull or skeleton?'}
           -- plus 10 more: main subject, weather, sunset, drink, food, cat, ...
         }) AS jev
FROM images;

Once the answers are stored as columns, they join with the Met's catalog like any other data. When did people start smiling in art? One GROUP BY:

SELECT (floor(o.begin_year / 50) * 50)::INT AS half_century,
       round(100 * count(*) FILTER (WHERE f.smile IN ('closed_mouth_smile', 'smile_showing_teeth', 'laughing'))
                 / count(*) FILTER (WHERE f.smile <> 'no_clear_face')) AS pct_smiling
FROM features f  -- Jev's answers, one column per question
JOIN objects o USING (object_id)
WHERE o.begin_year BETWEEN 1400 AND 1949
  AND o.end_year - o.begin_year <= 25  -- only works the Met dates within 25 years
  AND NOT f.rejected
GROUP BY ALL
ORDER BY half_century;

Almost nobody smiles before 1600! Many reasons for that, but that's for another blog.

Explore the full dive below, or open it in the dive gallery:

Get started

Images are now just another column in MotherDuck. Load your product photos, scans, screenshots or camera-trap pictures as BLOBs, ask your questions, and join the answers with the rest of your data.

Everything you need is in the prompt_jev() docs.

Many data teams don't get to pick their cloud. Their company standardized on Microsoft years ago, so identity runs through Entra ID, the data is on Azure Blob Storage, and the reports run through Power BI. Hundreds of those teams have asked us for MotherDuck on Azure. Today they query across clouds from MotherDuck on AWS, and Azure bills them for egress on every query.

So we're bringing MotherDuck to Azure. We're building it to be the same product, running in Azure instead of AWS.

What MotherDuck is

MotherDuck is a serverless data warehouse built on DuckDB, where every person and agent gets its own compute on the same data.

Most warehouses give each team or workload its own cluster, sized by an admin. That works when a few analysts run reports. It breaks down when every person, and every person's agents, ask questions at the same time. Sizing each cluster for the peak leaves capacity that is mostly idle, and sizing it for the average means people wait at the peak.

Instead, MotherDuck gives every user, every agent and every customer its own compute, called a Duckling. Each one connects to the same shared data and is sized independently. A heavy agent doesn't slow anyone else's work. These Ducklings scale to zero when they're idle, so you pay for what runs, not for the peak.

On top of that architecture, we've built the rest of the platform for agents.

Built for agents

• An MCP server for your agents. Connect Claude, ChatGPT, Cursor, Copilot and other MCP clients to your MotherDuck databases, and let the LLM write the SQL query. There are both read-only and read-write MCP tools depending on your needs.
• Dives: data apps, built by agents. Ask a question and get back an interactive, live app. Share it with your team, or embed it in your own product for your customers.
• Guides: business context in one place. Write down how your company defines revenue, which table to trust and what's changed recently, so every person and agent works from the same definitions.
• Flights: pipelines next to the data. The agent writes Python jobs for ingestion, enrichment and transformation of data, with a built-in cron for scheduling.

The data team uses these to bring in the data and write down what it means. Everyone else asks questions on top, each person and agent on its own Duckling, so a busy agent never slows anyone else down.

What it looks like in practice

AheadComputing, a startup designing a RISC-V CPU, built its first data platform on Azure Postgres. Once the team started using it, transformations on small data sets took over an hour. It moved to MotherDuck and was in production within a week. Hardware engineers who don't write SQL now ask Claude Code, which queries MotherDuck through the MCP server. 74% of the company's engineers were onboarded in the first weeks. A dashboard request that used to take three months now takes about 90 minutes, because engineers build their own Dives.

Ready for Microsoft

The Microsoft tools your company already uses connect to MotherDuck today.

• Entra ID. Your team signs in to MotherDuck with the Microsoft identity it already uses.
• Copilot Studio. Add MotherDuck as an MCP tool on your agent. Each person can sign in with their own MotherDuck account, or the agent can use a shared service account. For an agent you publish to a wide audience, a read-scaling token keeps it read-only.
• GitHub Copilot in VS Code. Install the MotherDuck MCP server, and its tools appear in Copilot's Agent mode.
• Power BI. Connect through Power BI's native PostgreSQL connector and the MotherDuck Postgres endpoint. You don't need a custom connector or a driver install.

The MCP setup docs and the Power BI guide have the detailed steps, and today your Copilot agent and your Power BI reports reach MotherDuck on AWS. With Azure regions, the compute will sit in Azure, right next to your data.

In addition to handling data warehousing workloads, MotherDuck can sit beside the other parts of the ecosystem. Fabric, Synapse and Databricks can handle portions of the work, with MotherDuck serving the interactive layer: dashboards, ad hoc analysis, customer-facing analytics and agents.

Help decide which region we start with

We haven't picked our first Azure region yet, and we want feedback from customers and prospects to decide. Regions will open in order of demand, and early access within each region goes out in waitlist order. We'll email you when yours opens, and we'll share timing, pricing and what early access includes before then. Joining doesn't commit you to anything.

In the meantime

MotherDuck on AWS can already read Parquet and CSV files in Azure Blob Storage:

CREATE SECRET IN MOTHERDUCK (TYPE azure, CONNECTION_STRING '<your connection string>');

SELECT * FROM 'az://my-container/events/*.parquet' LIMIT 10;

The Azure Blob Storage docs have the details.

Join the waitlist and tell us which Azure region you need.

Self-serve analytics finally works. Hurray! Agents plus the MotherDuck MCP mean anyone can query the warehouse without ever having written a lick of SQL in their life. And while that’s all incredibly exciting to me and to everyone else who feels this huge shift in velocity compared to 18 months ago, it’s also… a little terrifying.

As Chief of Staff at MotherDuck, I own our operating plan and all the reporting and rituals related to how we’re executing against that plan. My job is to help MotherDuck understand itself, and make sure everyone is on the same page so that we can make the right decisions for the future. The problem is that self-service analytics, when left to its own devices, doesn’t tend to converge on a single view of the truth. It fragments into a dozen private versions of it, all with a slight bend towards how the end user perceives the world.

We are LIVING this at MotherDuck right now. Everyone here is spinning up Dives and Flights. A salesperson who has never written a SELECT statement in their life can open the MotherDuck MCP server and build a view of net revenue retention (NRR) joined against external demographic data across their book of accounts, without ever talking to the data team. That is incredible. It’s a world that people have been dreaming about for years, and there’s a graveyard of startups from the past two decades of people who have tried and failed to make it happen.

I think Ian Macomber from Ramp said it really well on a recent podcast: In this brave new world, the scarce resource is no longer access to the data. It's consensus.

Alternative facts

Here's the thing about that salesperson. If they don't compute NRR the way the rest of us agreed to compute it (same grain, same account exclusions, same treatment of partner revenue) they walk into our weekly business review with a number that doesn't match mine. Best case, two people who don’t know how to read or write SQL burn twenty minutes doing forensic accounting. Worse case, their dashboard looks so cool that three other reps clone it, and before anyone notices, the entire revenue org is running its book off the wrong definition of retention.

Now multiply that by every person and every metric in the company. Terrifying.

In the pre-AI world, the data team was the gatekeeper. You didn't make your own dashboard, you asked Kenan to make you one, and you complained to your boss about how slow it was. But, hidden in that bottleneck was doing something we all took for granted: it meant someone was full-time responsible for being right. The data team obsessed over modeling, freshness, and lineage, and the rest of us relaxed a little, knowing there was a single throat to choke when a number was wrong (and it wasn’t ours).

The self-serve revolution removed that gatekeeper overnight. Everyone is feeling the velocity gain, and getting totally addicted to it. Fewer people are noticing consensus degrading. People trust the dashboard they built more than the one someone handed them… which is human and reasonable and also exactly the problem. Left alone, self-serve doesn't drift toward the truth. It fragments into everyone privately trusting their own reality.

To be clear: I don't think that the state we’re in is a net-bad thing, and even if I did, it’s too late to put the genie back in the bottle. The question I’ve had to reoriented myself around is: how do we get all these newly agent-empowered people to land on the same version of reality?

The answer is context.

What's a human actually for?

I felt this problem emerging almost immediately when we started duck-fooding the early version of Dives at MotherDuck. In a pre-AI world, I’m not sure I, a non-technical person, could have owned this on my own. In a post-AI world, this is entirely possible. Here’s how we approached it.

I started with getting a small tiger team together to agree on definitions for our most common metrics. We bootstrapped it the lazy way: we ran a pretty basic prompt on top of the warehouse and asked it to build a “big dumb document”, basically a data dictionary reverse-engineered from what the LLM could see. It did a decent job. An agent can introspect your schema: table and column names, types, foreign-key relationships, whatever comments you were disciplined enough to leave behind. From that it reconstructs the stuff everyone already agrees on. What's the canonical revenue table? What does a contraction event look like? I would say the big dumb document came back about 50% right.

The other 50% is the hard part, and where a human needs to get involved. No amount of column-name introspection tells you all of the nuances of your business.

Let’s take what counts as “churn” in a usage-based revenue business as an example. Is it a canceled contract? 30 consecutive days of zero compute? But what if they come back on day 31?

For the complicated or controversial metrics, write down all of the nuances and complexities onto paper, and lay out the different ways you could calculate the metric depending on what you care about. Go pressure test those with one or two people who interact with the inputs to that metric every day. For churn, that was customer success (affectionately called Hatchery at MotherDuck) and sales. Get them to poke holes in your assumptions and call out things you missed.

You probably have decent instincts about the churn patterns in your customer base, so overlay those on your actual data to see if the patterns you care about are getting captured in this definition. For something like churn, you can even go so far as to pull up the actual list of customers who counted as “churned” in the last week or month and ask yourself whether those are indeed churned customers in the spirit of the definition. If you see someone who actually is just experiencing some seasonality and doesn’t really feel like a lost customer, great, use that to tweak your definition. Do this 2-3 times max. You could get caught in the edge cases forever, and you don’t want perfect to get in the way of good enough.

Use what you learn to come to the table with a clear recommendation and a sample run of the data with the new definition. People tend to give better feedback when they have something concrete to react to vs. asked about a topic generally, with no guidelines.

This isn't a post on how to reach exec alignment (my quick tips: be discerning about which metrics you govern centrally, take input but understand the definition will not be decided democratically, and metrics trees with owners are a genuinely useful tool for figuring out what's worth fighting over). But, it’s a call to action for you, the non-technical human, to focus your time and energy on these sorts of things, and let the LLM handle everything downstream. Turning “here's how we define NRR, at monthly grain” into correct, performant SQL is precisely the kind of work LLMs are great at. The highest-leverage thing that a human (technical or not) contributes to this process is not writing the query. It's manufacturing the consensus that the query executes against. You are responsible for the what. The LLM handles the how.

Where should context live

A friend I used to work with at dbt is now on the data team at an AI company where everyone internally builds agents. His team did all the beautiful analytics engineering work that dbt pioneered, modeling every metric cleanly in their BI tool. They're confident that if your agent connects there, it comes out with the right answer. I’ve heard data teams at other companies say something similar, whether it’s the BI tool, a data catalog (or whatever we’re calling the third party context layer these days), or something else.

The problem is that most people don't connect through some intermediary. LLMs want to take the path of least resistance, which means 9 times out of 10 it’s just going to query the data warehouse directly. It’s hard to enforce making a stop along the way, especially if not all people (or agents) know that stop exists. My friend knows this, and he says it’s one of the biggest things keeping him up at night.

This is why I, a non-technical human, have become deeply convinced that context has to live as close to the raw data as possible. If your definitions sit in a modeling layer up the stack, they only protect the agents that bother to route through that layer. And in an AI-native company, you cannot enforce that routing. A context layer that can be easily bypassed isn't actually a context layer. At that point, it’s just a museum artifact.

Guides!

All of this is why I'm SO excited about Guides. A Guide is a plain markdown document, stored in the MotherDuck warehouse, that agents read before they write SQL. It captures everything your schema can't tell an agent on its own, like how you define a metric, which tables to join, which columns to exclude and why.

Some highlights that I’m especially excited about:

It's agent-native. Guides load automatically through the MotherDuck MCP. On connect, the agent gets an index of guide paths. It sees that core-metrics/revenue/revenue-retention.md is the one it wants, reads that single file, and only then generates SQL. By the time it touches your data, it already knows to exclude the right accounts and compute at the right grain.

You write it in plain markdown. Author it in the UI, dictate it from an agent chat, or push it from your data repo in CI. It reads like a short doc: purpose, key tables, the canonical definition, join patterns, known pitfalls. The workflow I'd recommend is what we did internally: let an agent bootstrap the V1 off your schema to get something that’s 50% there, then spend your human time on the 50% that requires discernment and (dare I say it) taste.

Org-wide or personal. Admins publish the definitions everyone shares. A power user who wants their own spin can layer a personal Guide on top, but the org version supersedes it. You give people flexibility to define things that aren’t centrally governed (usually things that are very domain specific, or changing quickly) and enforce the most important metrics that everyone needs to have right.

Versioned and auditable. Every save is a new version with a change comment. When a number moves next month and someone asks why, you can point at which definition was live when the query ran.

There's a real push-and-pull in AI-native circles about determinism versus letting an agent be free and creative. My current mental model is that you want to be deterministic about the what (the definition, the filters, the grain) and let the agent do its thing on the how (ex. whether it uses a CTE or a window function, how it orders the joins…even if you have no clue what all that means).

Don’t be afraid of the fog bank

Some people have described AI as a fog bank rolling in on a long stretch of highway: you used to be able to see for miles, and now you can't see more than a few feet ahead. As tempting as it is to slow down and wait for it to clear, I really don’t think that’s an option in this environment. Like I said, regardless of whether you find this new reality of self-serve analytics exciting or terrifying, you aren’t going to be able to put the genie back in the bottle. I think taking control of the context layer is one of the most high leverage things a non-technical domain expert can do right now. We’re in a moment where something that may have felt intimidating 18 months ago, is (1) totally possible for a nontechnical person to own, and (2) can actually be codified into workflows that allow everyone to function better.

My advice: just try it. You’ll be surprised at what you can accomplish with the tools we have today. And if you want a data infrastructure solution that brings this all together in one platform, check out MotherDuck! We’ve got a free trial you can sign up for here.

We go into spiritual territory, where he feels in his gut whether a pipeline will work. Then we finish with where every data engineer or software engineer eventually wants to end up, as the field is evolving so fast: on a farm with beautiful flowers and no screen. This is interview #5. I hope you enjoy it.

This article is structured into four parts: (1) how Kubernetes and disk abstractions are key for his work, (2) how he works with the models and agents, and how much the model actually matters, (3) what tooling he uses and how agents ease his life at work, and (4) whether data engineering is automated away with agents.

Introducing the Guest: #5 Josh Wills

Josh Wills is a software and data engineer with 25 years of experience. At Google, he worked on the ad auction system and led the analytics infrastructure behind Google+. As head of data science at Cloudera, he founded the Apache Crunch project and co-authored O'Reilly's Advanced Analytics with Spark, and he went on to build and lead Slack's data engineering team. He's also the original creator of the dbt-duckdb adapter.

Since 2024, Josh has been a Member of Technical Staff at DatologyAI, which helps teams train better models, faster and smaller. There he builds tools that automatically select the best data for training deep learning models, along with the distributed pipelines that process it at trillion-token pretraining scale. We go into more detail on what it takes to work with datasets that gigantic.

DuckDB Yes, but also Kubernetes Disk Abstractions

As many of you might know, Josh was the initial creator of dbt-duckdb, so I asked him if he still uses dbt and, more importantly, how much DuckDB.

His answer was that everyone uses it: "Everybody uses DuckDB in some capacity. I feel like anyone who knows about DuckDB uses DuckDB for something. Everyone has their own specific use case for it". He is using it less for his specific pipelines at work, as they have huge datasets, text data, not classical datasets: e.g., 30 TB is tiny. He uses Spark or Ray. He does much of the processing on text and multimodal data (time series, images, etc.), not much classical relational data in his day-to-day.

The use case for DuckDB he uses a lot is reading Parquet metadata that sits on S3, as the above data pipelines "throw off enormous amounts of Parquet metadata, which is still substantial but much smaller, on the order of 100 GBs". Information about runs, when a dataset was last used to train a model, how much he pays to store it in S3 for indefinite periods. He usually asks his agents to read that on his dev box on EC2, which he runs these commands on, to cut down on transfer times, just for speed.

"We Lost a Whole Generation" of Data Infra People, Retired 2022, AI Pulled Him Back

When talking about what has changed since he worked at Google as a staff engineer (2007–2011) and led data engineering at Slack (2015–2019), and how data engineering workloads differ today with AI, he said Snowflake is great. Still, we do not have many people anymore who know how the data is laid out on disk.

But not for his size of workload. Josh works with such big data that we normies can barely fathom, and that's why he uses lots of Kubernetes jobs and workloads. He likes Apache Celeborn (Pronunciation: spelled Celeborn, pronounced "Keleborn"), which abstracts the disks (not needing to know where they physically come from), because in Kubernetes you still do that manually, as Kubernetes is not really cloud-abstracted, he says:

Kubernetes is fantastic in that it's in theory abstracted away from an individual cloud, but in practice it's not, because where does your disk come from? If you're on AWS or GCP you can use an EBS-like service, but if you're running bare metal, that doesn't exist the same way.

Celeborn has something called push-based shuffle write, where mapper nodes skip local disk storage, ideal for cloud-native architectures, which lets you separate storage and compute, so

The Spark jobs don't need to worry about where their disk comes from anymore, from a shuffle perspective. There's the shuffle service, no matter how much data you're going to shuffle. You have your local disk, and then you have your shuffle service for all your other disk needs.

How Much Does the Model Matter to You?

When asked about the models and how much they matter, Josh said for him it's like the iMessage blue bubbles vs. green bubbles. Android phones are fantastic, but he wants the blue bubbles. And Josh's prediction is that all the models will become competent, so the real question becomes:

which one you like and enjoy hanging out with

Source: Josh's tweet

Above, Josh explains how he disliked the Opus 5 model, and continues to do so. In this interview, he added that it's also the way it communicates. An interesting insight, he said, as someone who works on building models all day:

the fact that Opus 5 is so bad, it deeply indicates to me that we literally have no idea what we're doing at all when it comes to training these models.

The good thing: changing might be easier than ever1, and he disliked the previous 5.0 so much that he changed his workflow and switched to OpenAI. He commented that:

with no drop in productivity, it's kind of insane.

How to Code with Agents These Days

Next, we talked about working with agents. He jokingly came back to a Tweet where he wants to start a reinforcement learning (RL) environments company, where he does the teaching:

Josh says:

[..] a joke that I want to start my own RL environments company where it's just me teaching the models how not to write absolutely garbage data pipelines. Because I use agents to write all of my code, with very, very few exceptions. And yet I still find myself deeply unhappy with the data pipelines it writes.

How to Get Fast Feedback Loops with Agents?

Josh says AI wins where there's a large training model and fast responses, because if it's not in the training data, it's slow and it needs to do a lot of stuff. He says: "Any sort of environment in which the models have highly reliable training data with fast feedback mechanisms, they're just going to dominate. Absolutely going to dominate. You see this with math, you see this with front-end engineering, any RL environment where there is a fast feedback mechanism."

And that's exactly what's missing in large-scale data pipelines today:

What's tricky about data pipelines, at least at a large scale, is that you don't actually have that fast feedback mechanism available to you, because running a giant pipeline can take a long time and a lot of computers.

Fast feedback loops are also what Josh was already referring to almost four years ago, before any of the AI hype. He liked fast feedback loops, and quoted Erik Bernhardsson:

Feeling the Tuples Flowing through the Pipelines, a Gut Feeling

Josh, with his years of experience, has built a gut feeling:

Through years and years of suffering and writing gigantic pipelines, I can look at a pipeline and it kind of feels like something to me. I can feel the tuples flowing through it, because I have a mental model of how this is going to go.

He explains how SQL databases have a great fast feedback mechanism with their query explain plans. They tell you if it's a good or a bad query. But for a Spark job, it's not yet exactly that.2

Josh describes an aesthetic aspect of data pipelines:

I have, weirdly, a kind of aesthetic sensibility when it comes to data pipelines. A data pipeline can be beautiful to me. It can be elegant, well designed, thoughtful. That makes me want to look, in the way other people would visit an art museum, or read math proofs because they find them beautiful.

That's where he still sees a lot of "ugliness" in what the agents write. They were not yet beautiful when we talked a couple of weeks back. Josh said, "Some of the best programmers I've worked with were musicians by training". And he continues:

One distinguished engineer at Google, his master's degree is in the violin, and he was the best coder I think I'd ever seen, because his code was just like music. It was beautiful.

I have a similar affinity for writing text, and good writing has a musical quality to it that certain people can hear. You develop an ear for good writing after a while. It sounds like something in your head, like poetry or prose. Josh says that "This is something we have not figured out how to RL in any kind of way".

The Cost Aspect, Literally Lighting Money on Fire

When talking about cost, Josh says that:

"the models might waste time on the most pedantic, stupid checks for errors that are simply not possible given my knowledge of the structure of the upstream data sources." "Literally lighting money on fire"

He continues that the pipelines are huge and compute costs a lot. And models might write a pipeline for average-sized data, but not have the knowledge or the big training data on "gigantic pipelines": "It will work, it's correct. It's just stupidly cost inefficient".

He has seen this problem at companies getting rid of their data teams, as LLMs could vibe code a job in Databricks. He said it like this: "You can easily vibe code a job in Databricks that costs like two hundred thousand dollars a day to run, that a competent data engineer, or at least one who supervised it, would never have written."

And it's not the model's fault, but a function of the training data that has not investigated or run a lot of "gigantic data pipelines for multiple hours across hundreds of machines".

That's just not the way the reinforcement learning environments are structured right now:

It's this fast feedback loop: tiny data, is it correct? Tiny front end, does it render the CSS correctly?

Josh makes a great science analogy:

People talk a lot about how science has the same problem. Labs, biology, physics, these are all real things where you literally have to wait for the culture to grow. Data engineering has a lightweight version of this. We are not as immediately amenable to this kind of feedback style right now.

He found the models to be fantastic at SQL back when we held the interview. "They're really, really good," he said. But for his kind of work (the raw Ray pipeline, the Spark pipeline), there is no explain plan. You're still building the thing from scratch, as he said.

The Job of 'Emergency Exception Handler'

He mentions that in his job, because the models haven't learned everything yet, he is basically like a pilot who usually flies on autopilot, but in case of emergency, he is there to fix it or take over.

What Is the Tooling He Uses & How Agents Ease His Life at Work

As initially pointed out, he disliked Opus and switched to Codex, so now he is using Codex all the time, but he used Claude for a long time.

When asked how he works or reviews code, he said: mostly giving feedback, like a staff engineer does. His life as a senior technical IC has been trending towards that for a very long time, he says. He primarily gives feedback. That's his primary job. Interviewing people, reviewing design documents, and less hands-on, with very few exceptions.

He likes to work on high-demand, high-urgency work that forces him to create. This is back to the 'emergency exception handler':

If there's a broken-ass data pipeline and it has to run right now, because a customer has a training job coming up and the data has to be ready for it. That's the primary value I provide.

What Josh Likes about Agents: No Feelings and the Caffeine Dial

Josh also revealed that he likes working with agents because agents have no feelings. He can give agents feedback on demand, force things to be created, he says:

I don't have to be very nice to the agents.

He can just say things:

I don't have to care about their feelings or their career development, because they're going to cease to exist when I'm done with this coding session. I can be like: this is bad and you should feel bad. Ctrl-C, you're dead, I'm going to start over with someone else.

As someone who calls himself "a fairly mediocre manager, because I don't generally like caring about these things", this works well for him 3.

The other interesting thing he says:

I can dial up and down the work on demand as I see fit. If I'm in a moment where I need five agents going, I can do it. If it gets to be too much, I dial it back. In the same way that I try to maintain a certain level of caffeine at any given moment of the day, I can maintain a certain pace to match my energy, to match what I'm up to right now. Am I super excited, am I kind of bored? That's a virtue of this new world that was not true before.

It's like the caffeine dose.

No More Unit Tests!

He also likes to automate most of the boring work. Who wants to do one more database migration? Or fix "flaky unit tests"? He likes to do the higher-level strategic stuff, which is how people end up becoming investors or product managers, he says.

Even more so:

I have an agent whose whole job is just to watch for flaky tests and fix them. That's it, that's all it does. Database migrations? Absolutely not. I have an agent that does database migrations. I have automated away the shittiest, lowest-value, most unpleasant parts of my life, and I am here for it.

For him, that's a massive quality increase. And he can focus very hard on things he likes, and where the leverage is high, to make hard decisions.

Decisions Made Are What Is Left, and Communicating Why Is Key

In regard to code review, one thing he is trying to be militant about in the world where agents do everything is:

For every project, I'm focusing really hard on the decisions I am making.

He mentions a concrete example from Datology where he's building an index over datasets to figure out which parts of a dataset are reusable, wholesale or partially, someplace else:

• Option A: query the metadata catalog at read time and figure out the useful assets.
• Option B: write an explicit index up front: "this asset exists in this index and it is usable for this purpose"

He says:

Either choice is valid, there are pros and cons. I'm making an explicit trade-off here, and those are the decisions I need to communicate. The details of how the index works are much less important.

He makes a great point that:

It's really the decisions we need to communicate. The decisions build the shared understanding, the choices we make, the trade-offs. That's still the meat of engineering work.

I also think this very thing is the most important, and it can turn into a bad spiral if you are not careful, as someone vented in this tweet. Plus, in my opinion, agents force you to focus on architecture, and that's why they're handier for seniors.

Is Data Engineering Automated Away with Agents?

Josh says clearly that data engineering is not going away. For the same reason it hasn't gone away for so long: the fundamental problems won't go away. We're still solving the problem of materializing views and caching!

I don't see data engineering going away in any way, shape or form, even in this world. Aside from the fact that the RL environment problem makes it unusually difficult for the agents to be good at running these very long pipelines, the fundamental problem of data engineering is the same problem it's been my whole career, and 25 years before I started working. It's the materialized view problem.

The questions are: "What data should I pre-process and create now to make queries faster later on, and what stuff is not worth doing, what should I just do in the moment?" These are the fundamental problems.

He says it will be a "problem for the superintelligence", because:

It's not a fixed thing, it's going to continuously move, and the right trade-off is going to change all the time.

Josh says, aside from reinforcement learning:

It's the two hard problems in computer science, right? Cache invalidation and naming things. And the agents still suck at them because they're legit hard.

"Data engineering is fundamentally the cache invalidation problem", he continues, "just done in a batch-processing context".

Humility and Burnout

Lastly, we talked about humility and how not to get burned out.

It helps to stay humble, as he says he is very good at it, telling his family:

I am simultaneously an idiot and also among the best in the world at what I do.

I say that we all probably feel some kind of ambivalence, as the AI can do part of our job and we feel unneeded. That's where humility helps us. Josh also adds that he is kind of grateful that he can still be at the keyboard and program. It's "tremendous fun" for him, and he loves to be here for this moment in time.

I asked if he does not get burned out when agents do not generate code the way he does it. Or does he split into "code I don't mind" vs "code I want perfect"? He says that his advantage is having done it for a "really, really long time":

Old me tolerates things that younger me would never, ever, ever tolerate at all. I am used to cycles of feedback with somewhat "stubborn" people.

He said that once, the agent couldn't find a problem he was telling it about: "And I could not, for the life of me, get the agent to write the pipeline in the way it would do this". He was telling the agent "you cannot use caching this way", and it couldn't help itself. He had to take over.

This again is the perfect example of the exception handling situation. It's "kind of my job these days," Josh says, to "identify these situations where, for whatever reason, the model literally isn't capable of doing what I am asking". But he says that the models are getting better every day:

The reality is, I don't encounter that many of these problems anymore.

Conclusion

One main takeaway from this conversation is that data engineering is the cache invalidation problem, just done in a batch-processing context. That's important: when done right, agents dominate because of a fast feedback loop, like SQL with its explain plans or math. Gigantic Spark and Ray pipelines that run for hours across hundreds of machines don't have that cycle yet. That's why Josh still ends up as the emergency exception handler, and why others can vibe code a Databricks job that costs $200k a day.

What's interesting is how little the model itself matters to him. It's more about design and feel, e.g., blue bubbles vs. green bubbles: pick the one you enjoy hanging out with, switch when you don't like it anymore, or hand off the flaky tests and database migrations to an agent. What's left is what was always the hard part of data engineering: making the decisions and communicating the trade-offs.

We got a bit spiritual, with Josh feeling the tuples flow through a pipeline, and ended where many of us do as the field moves this fast: dreaming of a farm with beautiful flowers and no screen . Josh admits he periodically flirts with it too: "Maybe I'm just going to retire again and just go play chess. I'm not made of stone".

I hope you enjoyed this episode of the interview series. Check out the others, starting with Mark Freeman, Chris Riccomini, Wes McKinney and Maxime Beauchemin.

If you read this far and liked it, also check out the talk with Mehdi about the "Coupling Problem in Data Engineering" or check Josh's recent talk Josh Wills on the End of Coding at Pebblebed.

He also has no shame about hopping back when the model changes. With the latest model, he went back to Opus 5.5. The same impression someone had on Reddit: "agents are absolutely dogshit AI at optimizing Spark jobs (or any sort of high complexity pipeline). They are laughably bad, in my experience." I haven't worked under him, but I'm sure that he is a great manager nonetheless!

Jev is a new type of decision model. It picks its answer from a closed list you define (yes or no, a label, a score) and returns a typed column with a confidence, fast enough to run on every row. On MotherDuck, you call it directly from SQL with the prompt_jev() function.

In this post, I'll go through 5 practical use cases on real datasets, with real results and shares you can attach to play around yourself.

1. AI agent logs: which turns worked, and which took the long way?

If you run an agent, your trace logs already say what happened: every model step, tool call, token and dollar. What they don't say is whether the turn did what the user asked, or whether it took the long way to get there. Making that kind of judgment call is where prompt_jev() shines.

motherduck.com/try lets anyone try MotherDuck by chatting with an agent that queries data, runs Flights and builds Dives through the MotherDuck MCP server. Every turn emits OpenTelemetry spans and writes them into a MotherDuck table.

Open a turn and you get its full history, step by step, with the judge's scores on top:

We run a daily pipeline (Flight) which grades every production turn with a handful of questions, in SQL, next to the spans:

| question | asked on | Jev returns | |---|---|---| | completed: did the agent do what the visitor asked? | every turn | score: fails, partially, fully | | failure_mode: why did it fall short? | turns that scored below 1.5 | choice: budget exhausted, tool or SQL error, wrong answer, ... | | error_cause: what broke this tool call? | each failed tool call | choice: model wrote invalid SQL or Python, wrong guess about the data, platform error, tool bug | | efficiency: how many tool calls were unnecessary? | turns with tool calls | score: many, a few, none wasted | | main_waste: where did the waste come from? | turns that scored below 1.2 | choice: repeated failure, redundant read, polling, unneeded exploration, tool gap |

efficiency is the tricky one. Some calls look useless but are required by the product (reading a guide before using its tools, for example), so the instructions list them as never waste. Jev reads the request, the timeline of what the model wrote and called, and the start of the final answer:

prompt_jev(
  'USER REQUEST: ' || left(request, 1500) || chr(10) ||
  'TIMELINE (what the model wrote and the tool calls it made, in order):' || chr(10) || left(timeline, 8000) || chr(10) ||
  'FINAL ANSWER (start): ' || left(coalesce(answer, ''), 1200),
  questions := {
    efficiency: {
      type: 'score',
      instructions: 'How many tool calls in the TIMELINE were unnecessary to answer the USER REQUEST?
        These are required by the product and never waste:
        - reading each guide once before first using its tools
        - view_dive right after save_dive
        - one wait_for_flight_run after each run_flight
        A failed call followed by one corrected retry is not waste.',
      criteria: ['many wasted (4 or more)', 'a few wasted (1 to 3)', 'none wasted']
    }
  }
)

Two interesting design choices:

• A cheap score on every turn, and the "why" questions only on the turns that score low. That keeps the cost down.
• Facts and judgments never mix: exact counts (repeated calls, extra waits) come from plain SQL on the spans. Jev only answers what needs judgment.

The same prompt_jev() judge grades our eval runs, where every case has an expected behavior. On the latest run, with 12 cases per model:

| model | cases passed | judge score | cost per turn | |---|---:|---:|---:| | Claude Opus 5.5 | 10 of 12 | 0.90 | $0.21 | | GPT-6 Luna | 12 of 12 | 0.88 | $0.005 | | Claude Sonnet 5.5 | 12 of 12 | 0.87 | $0.12 | | GPT-5.6 Luna | 9 of 12 | 0.86 | $0.011 |

12 cases per model is still small, so treat this as a reason to run a bigger eval before switching models ;) But that's the decision table you want: quality next to cost, graded the same way every time, in the same database as the traces.

2. Social listening: what does Hacker News think of each AI lab?

Sentiment analysis is another big use case for text. It's hard to get right because people say the same thing in a hundred ways (hello, sarcasm). With Jev, you ask a simple question, "is this comment negative about X?", and get a clean classification back.

Tech adds its own problem: a lot of product names are ambiguous. "Claude" is also Claude Shannon, and "Gemini" is also a zodiac sign, an internet protocol, a NASA program and a crypto exchange. Good news: a prompt solves this one too.

Let's take Hacker News. MotherDuck hosts all of it in a share, refreshed daily: about 50M stories and comments since 2006. Attach it and you can rerun everything below:

ATTACH 'md:_share/hacker_news/daa0cc99-d20c-4f5c-abd5-f7b22a1a1da9' AS hacker_news;

I looked at four AI labs: OpenAI (ChatGPT and the GPT models), Anthropic (Claude), Google (Gemini, formerly Bard) and DeepSeek.

The first step is the one a regex would do: find every item that names one of them, with patterns like \b(anthropic|claude)\b. That's 488,483 mentions since 2019. Too many to classify for a blog post, so I took a stratified random sample: up to 1,000 mentions per lab per quarter, 65,635 in total, each weighted back to its quarter's full count. Every percentage below is weighted.

Then the prompt handles the name confusion. The date goes into the input, because Google's Gemini and Bard chatbots and Anthropic's Claude models didn't exist before 2023:

-- input = 'DATE: 2020-06' || chr(10) || 'NAMED PRODUCT: Google Gemini (formerly Bard)' || chr(10) || 'TEXT: ' || comment
SELECT id, lab,
       prompt_jev(input,
         'Does the TEXT use that name to mean the NAMED PRODUCT, the AI company or its AI models or chatbot (even if only in passing)? Use the DATE: Google''s Gemini and Bard chatbots and Anthropic''s Claude models did not exist before 2023. Answer no when the name means something else, such as the Gemini internet protocol, the NASA Gemini program or the zodiac sign, Claude Shannon or another person named Claude, the anthropic principle, a bard as in a poet, or GPT disk partitions.') AS refers
FROM mentions;

So how often was the regex wrong? Here's the share of regex matches that Jev says are about something else (refers below 0.5):

| lab | regex matches 2019-22 | about something else | regex matches 2023-26 | about something else | |---|---:|---:|---:|---:| | OpenAI | 22,623 | 2.0% | 261,266 | 1.1% | | Anthropic | 795 | 89% | 144,131 | 0.8% | | Google | 3,085 | 76% | 40,788 | 3.6% | | DeepSeek | (didn't exist yet) | | 15,795 | 1.0% |

So before 2023, 89% of the HN items matching claude or anthropic weren't about the AI lab at all, mostly Claude Shannon and the anthropic principle. Once the Claude models shipped, that dropped to 0.8%. Gemini is the messiest one: still 3.6% wrong since 2023, because the Gemini protocol, the NASA program and the crypto exchange keep showing up. Yes, that's a lot of Geminis out there.

The cool thing is that we can ask 4 questions in one call per comment: the name check above, whether the comment is really about the lab, whether it's negative, and what it's about. The instructions of prompt_jev() must be constants, so the lab goes into the input:

SELECT id, lab,
       prompt_jev(input, questions := {
         refers:   {type: 'noul', instructions: '...the name check above...'},
         relevant: {type: 'noul', instructions: 'Is the TEXT about the NAMED PRODUCT itself (the AI company, its models or its assistant), rather than only mentioning it in passing?'},
         negative: {type: 'noul', instructions: 'Does the author express a negative experience with, or a negative opinion of, the NAMED PRODUCT?'},
         topic:    {type: 'choice', instructions: 'What aspect of the NAMED PRODUCT is the TEXT mainly about?',
                    criteria: ['answer quality and hallucinations', 'coding and developer tools', 'pricing and usage limits', ..., 'not about the product']}
       }) AS r
FROM mentions;

Each comment goes through the four questions, and the answers decide where it ends up:

| comment (trimmed) | lab | r.refers | r.relevant | r.negative | r.topic.choice | counted as | |---|---|---:|---:|---:|---|---| | "Crypto exchange Gemini reveals lower revenue and wider loss in US IPO filing" | Google | 0.06 | 0.11 | 0.14 | not about the product | dropped: wrong Gemini | | "Earthquake Scatter Plot Santorini (done with some help from ChatGPT)" | OpenAI | 0.94 | 0.11 | 0.07 | coding and developer tools | dropped: only in passing | | "Tested out Gemini-2 Flash [...] It still hallucinates like crazy compared to GPT-4o." | Google | 0.97 | 0.97 | 0.97 | answer quality and hallucinations | negative | | "A nice thing about Deepseek is that it is so cheap to run." | DeepSeek | 0.94 | 0.91 | 0.04 | pricing and usage limits | not negative |

Jev gives us a confidence score, so we can filter on it to make the output more reliable. For instance, "clearly negative" means r.negative >= 0.8, counted only on comments that are really about the lab (refers >= 0.5, relevant >= 0.8, and a topic other than "not about the product").

The usual caveats apply. Hacker News is a loud, developer-heavy crowd where coding tools loom large, so use your human judgment: don't read this as market share :)

Here's the full dive, also available on the dive gallery.

The sample and the Jev labels are in a public share if you want to run your own queries:

ATTACH 'md:_share/hn_ai_labs_public/3b78bda2-5bf1-4e0f-9e4b-f4ecb168405b' AS hn_ai_labs;

3. Sales calls: which conversations match the use cases we serve?

It's never been easier to record a meeting and get the transcript: tons of tools do it natively. At MotherDuck, I use it to understand what our customers ask for and which walls they hit in their developer experience and onboarding.

For the demo here, though, I'll take a made-up sales case.

Typically, the CRM knows the account size and the contact's title. The call transcript knows what they're trying to do. Joining the two is where lead scoring gets interesting (and that is how sales decides which leads to call first).

Say sales_calls has call_id, account_id and transcript, and accounts holds your firmographic enrichment. Ask a narrow yes/no question about a documented product fit, for example whether the prospect needs to analyze lots of customer conversations or reviews. Then use the answer as one signal in a review queue:

-- Illustrative schema and product-fit question.
WITH calls AS (
  SELECT call_id, account_id,
         prompt_jev(transcript,
           'Does the prospect describe a need to analyze many customer conversations or reviews?')
           AS text_analytics_fit
  FROM sales_calls
)
SELECT c.call_id, a.segment, a.contact_role, c.text_analytics_fit
FROM calls c
JOIN accounts a USING (account_id)
WHERE c.text_analytics_fit >= 0.8;

A sample of the shortlist (illustrative):

| call_id | segment | contact_role | transcript snippet | text_analytics_fit | |---|---|---|---|---:| | 4812 | mid-market | Head of Data | "we have 2M support chats a year and nobody reads them" | 0.94 | | 4830 | enterprise | Analytics Engineer | "our NPS comments sit in a varchar column in Snowflake" | 0.88 |

Again, treat the output as a shortlist for a human to check. A high score doesn't make anyone a "good lead". If title or seniority matters, use the CRM field you already have instead of asking a model to guess it from the transcript. And if your product covers several use cases, ask one well-defined question per use case and validate the thresholds on labelled calls.

4. Job postings: which cloud does each market actually run on?

Which cloud provider is the best?

OK, wrong question. What I actually want to know is which cloud provider wins, and where.

Job postings are a decent proxy: a company hiring Azure data engineers is betting on Azure for the next few years. Five years ago, I wrote a blog about the most requested skills in 35k data job postings I scraped. Parsing them was hard work: I had to train a small model just to classify a few things in each posting.

The catch is that titles don't say which cloud, and keyword counts lie. Plenty of postings list "AWS, Azure or GCP" as a nice-to-have, and "AWS is a plus" doesn't mean the stack runs on AWS. So instead of counting keywords, I asked Jev one question per posting.

The data is 650k data job postings that mention at least one cloud. Most are Google Jobs postings for 21 European countries and the US, from Luke Barousse's Data Nerds dataset, plus 2025 LinkedIn postings for eight cities from my colleague Dumky's data_jobs_2025 sample dataset. To keep tokens low, I didn't send the full description: the input is the job title plus only the sentences that name a cloud, about 750 characters instead of 4,200.

SELECT source, job_id, country_code,
       prompt_jev(snippet,
         'Which cloud platform is this data job''s stack primarily built on?',
         choice := ['aws', 'azure', 'gcp', 'ovhcloud', 'scaleway', 'ionos', 'stackit',
                    'hetzner', 'open telekom cloud', 'oracle cloud', 'ibm cloud',
                    'alibaba cloud', 'multi-cloud, no clear primary',
                    'cloud only mentioned in passing']) AS r
FROM jev_candidates;

Watch the last two labels. They're the escape hatches: without them, Jev is forced to pick a cloud even when the posting doesn't really have one.

| title | excerpt sent to Jev | r.choice | confidence | |---|---|---|---:| | Data Engineer (DE) | "Du verwaltest und optimierst Datenpipelines in MS Azure Data Factory ... Datenmodelle ... in Synapse Analytics" | azure | 1.00 | | AI Data Engineer (DE) | "General knowledge of AWS cloud services like AWS Batch, ECS Fargate, S3, RDS" | aws | 1.00 | | Alternance - Data Engineer (FR) | "Google Cloud (BigQuery, Cloud Storage, ...)" | gcp | 1.00 | | Sr Data Engineer (US) | "Leverage cloud platforms like AWS, GCP, or Azure to build and manage data solutions" | multi-cloud, no clear primary | 0.95 | | Senior Data Engineer (US) | "AWS cloud experience is a plus" | cloud only mentioned in passing | 0.50 | | Data Scientist (US) | "Cloud Certification (AWS/Azure preferred)" | multi-cloud, no clear primary | 0.57 |

Also cool: I didn't translate anything first. The question is in English, and the postings are in German, French, Italian and more (welcome to Europe!).

For the map, I only count a posting when all of these are true:

• it wasn't posted by a cloud vendor (AWS hiring for AWS says nothing about the market)
• it names a single cloud
• Jev picked a real cloud, not an escape hatch, with a confidence of at least 0.8

SELECT country_code, r.choice AS cloud, count(*) AS postings
FROM classified
WHERE posted_by_provider IS NULL          -- drop postings by the cloud vendors
  AND n_clouds_mentioned = 1              -- one cloud named in the posting
  AND r.confidence >= 0.8
  AND r.choice NOT IN ('multi-cloud, no clear primary', 'cloud only mentioned in passing')
GROUP BY ALL;

Of 396k postings from 2024-25, 213k survive these filters. And the answer: Europe runs on Azure, which leads in 18 of 21 countries. The US is split: Azure leads more states (25 against 22 for AWS), but the AWS states carry three times more postings.

Play with it yourself: switch between Europe and the US, focus on a single cloud, and hover any country or state for the full split.

The full Dive is also on the dive gallery. The aggregates behind the map are public too:

ATTACH 'md:_share/cloud_regionality_public/10d8fd77-a5f7-42e5-95d5-6cd999bf8535' AS cloud_regionality;

5. Skip the hand-written parser: is this job remote?

This one is more technical, but it pays off in maintenance and speed. To turn a text column into a category, you write a parser. Usually that's a CASE WHEN (or worse, a custom UDF) with a few regexes that someone wrote once and keeps patching.

So let's take a sample of the job postings above and ask: is the job remote, hybrid, on-site, or doesn't the posting say?

You can follow along by attaching this database:

ATTACH 'md:_share/jev_use_cases/6a306d41-76e6-4cca-828b-84f412a28009' AS jev_use_cases;

The classic regex version typically looks like this:

CASE
  WHEN description ILIKE '%hybrid%' THEN 'hybrid'
  WHEN description ILIKE '%remote%' THEN 'remote'
  WHEN regexp_matches(description, '(?i)\bon-?site\b|\bin[- ]office\b') THEN 'on-site'
  ELSE 'not stated'
END

Then you hit the obvious false positives, like "remote sensing" or "hybrid cloud". You keep patching and end up with something that kind of works, but is hard to maintain (my version two has 22 hand-tuned alternatives and exclusion lists).

The Jev version is one question, with the definitions written down:

SELECT job_id,
       prompt_jev('TITLE: ' || title || chr(10) || 'LOCATION: ' || location || chr(10) ||
                  'DESCRIPTION: ' || left(description, 8000),
         'Where does this job expect the person to work?',
         choice := [
           {label: 'remote',     description: 'Fully remote: work from home or anywhere all the time, possibly limited to a country or time zone'},
           {label: 'hybrid',     description: 'A mix of home and office: some days a week in the office, or remote work, home office or télétravail offered as a regular option or benefit'},
           {label: 'on-site',    description: 'The posting says the work is at the office, a site or a client site, with no regular remote work'},
           {label: 'not stated', description: 'The posting does not say whether the work is remote, hybrid or in the office'}
         ]) AS work_mode
FROM jev_use_cases.job_postings_sample;

How does it compare with the regex and a standard LLM call (prompt())? On the same 2,000 postings:

| approach | time on 2,000 postings | what you have to maintain | |---|---:|---| | regex, version two | 0.2 s | 22 hand-tuned alternatives and exclusion lists | | prompt() with an ENUM return type | 37.0 s | the prompt | | prompt_jev() | 1.7 s | the label definitions |

prompt_jev() is again about 20 times faster than the LLM call, and asking the LLM for JSON didn't help much (31.1 s, before the cleanup). The regex is still the fastest, but Jev is fast enough to replace it without the fragility, and it skips the slow LLM detour.

So should you use Jev everywhere you have text?

Well, a regex is still nice for structure: things that have a format (IDs, emails, URLs, dates, a known product code). Use Jev for meaning: is this about X, is it negative, which one is the main one. And use both together: a cheap ILIKE or regex to build the shortlist, then prompt_jev() to decide.

A good first experiment

Pick a text column you already have and a business question you couldn't answer from it (or only the hard way). Write down the answers you'd accept. Label a small sample by hand (or with prompt() on MotherDuck or with a standard LLM call), run prompt_jev() on it, and look at both the winners and the uncertain rows. Once the question holds up, materialize the result so your dashboard doesn't pay to classify the same text twice.

Keep the limits in mind: Jev gives you a bounded choice, a yes/no probability or an ordered score. It won't write a summary or invent a new category. Test your confidence threshold on your own data, and check your data-handling rules before sending customer text to an AI function. The MotherDuck guide and the prompt_jev() reference cover the syntax and availability.

For now, the duck-sized test is simple: what question is hiding in a text column you already own?

Reading about agents is fun. Building one with your own hands is better! On Wednesday, November 4, Data Outpost kicks off with a full day of hands-on workshops in San Francisco. You'll build agents, wire up the data they need, and learn what it takes to run them in production. Bring a laptop, because most of these sessions end with something running on it.

There are 11 workshops across 3 time slots (9:15 AM, 11:00 AM, and 2:30 PM), and they run in parallel, so you'll pick 1 per slot. They're at Convene 100 Stockton, the same venue as the main day of talks and panels on Thursday, November 5. (Here's what you'll learn on day 2.)

Sound good? Register for Data Outpost here! Need help choosing? Read on!

Here's what you'll build in each workshop, plus my take on each one.

Build Your Own Agent

4 workshops where you walk out with an agent of your own: one you can message on Slack, one that builds a whole data stack, one that lives in your BI tool, and an MCP server that reserves you a gift.

Build a Data Agent with eve

Colton will walk through the fundamentals of the eve framework and some popular use cases, then help you build your own personal data analytics agent. You'll connect it to the context sources in your enterprise (such as MotherDuck) and talk to it wherever you already work: Slack, iMessage, or embedded in your website.

My take: I love that this is open source! They have thought through the tough problems in the space: durable execution, sandboxing, and more.

The Next Data Stack Has No UI: Building End-to-End Analytics with Agents and Ducks

What if your next data platform is a conversation instead of a pile of dashboards and config screens? You bring an agent (Claude Code or similar), Mehdi will bring some raw data and coach you through the process. You'll point your agent at those files with DuckDB, then let it profile the data, design a schema, ingest it, write the transformations (complete with data quality checks), and publish to MotherDuck.

Along the way you'll break the data on purpose to see how to build a resilient system, and talk through where the DuckDB ecosystem shines for agents and where it falls apart: state, context, reproducibility, permissions, and observability. You'll leave with a working end-to-end stack, a repo of prompts and guardrails, and the experience of building a modern platform from scratch.

My take: If you use AI to speed up just 1 small part of a process, you don't see much benefit. This is all about applying AI to the entire process end to end, and it will speed things up dramatically. In my experience, finding and ingesting data takes the longest, and this will help!

See it live: a hands-on Sigma + MotherDuck lab

Workshop: Zalak Trivedi (Product Leader, Sigma) and Tyler Johnson (RVP, Solutions Manager, Sigma) · 2:30 to 4:30 PM. On the agenda

Most BI demos show you a finished dashboard. In this lab you build one, with Sigma connected directly to a live MotherDuck warehouse. The first half is classic analysis: drag fields into tables and build a pivot table from scratch.

In the second half you'll put Sigma's AI to work. Ask Sigma Assistant questions in plain language, then build your own Sigma agent and watch it take action on your behalf.

You'll leave with a working workbook connected to MotherDuck and a clear sense of what your team could build in an afternoon.

My take: My favorite part of this is the "take action" piece. That's where the benefits of agents really compound.

Build an MCP Server and Unwrap a Mystery Gift with FastMCP and Horizon

Workshop: Edward Park (Engineering Director, Prefect) and Radhika Gulati (Product Marketer, Prefect) · 2:30 to 4:30 PM. On the agenda

Yes, there is an actual mystery gift! You'll use FastMCP and Horizon (Prefect's MCP hosting platform) to build an MCP server, expose tools and resources, and take it from local development to a hosted deployment. Then you'll connect it to a shared gift service, so an agent can check which numbered gifts are still available and reserve one for you.

You'll leave with a working hosted MCP server, hands-on experience connecting agents to a shared service, and a gift to take home.

My take: Building an MCP is a great way to scale your expertise across the company.

Get Data Agent-Ready

An agent is only as good as the data it can reach. These 5 workshops get your data in shape: migrated from your current warehouse, stored in an open lakehouse, streamed fresh from a live database, synced from a custom source, and pulled out of messy documents.

What to move, what to fix, what to leave behind: migrating to MotherDuck

Thinking about moving to MotherDuck, but dreading peeling back the duct tape on your current setup? Dumky will help you build your own system for migrating data: an inventory built from query history, logs, and access patterns, the right strategy for each workload, and the data movement, transformations, and consumption layers to match.

It applies whether you're coming from Snowflake, BigQuery, Postgres, or something else, and whether your data feeds dbt, Hex, Power BI, or a customer-facing web app. You'll leave with the runbook to migrate with confidence.

My take: The hardest part of a migration is always in the details. Making sure you handle all workloads and can validate that the data is correct will really speed things up!

Lakehouses for Robots: Scaling DuckLake and Iceberg for Agent Workloads

Workshop: Alex Monahan (MotherDuck) and Matt Martin (Staff Engineer, State Farm) · 9:15 to 11:00 AM. On the agenda

Data lakehouses are open, scalable, and interoperable, but most are not ready for agents. We’ll show you how to query Iceberg and DuckLake using both DuckDB and MotherDuck with the agent of your choice. You’ll also learn how to take full advantage of your local laptop so you can build even faster.

Whether you are new to lakehouses or a seasoned expert, you’ll end the session with a working lakehouse example ready for the flood of agents.

My take: I'm a bit biased - I'll be leading this one with Matt! I really like the lakehouse architecture, so we'll build one that's agent ready.

From Scheduled Syncs to Real Time Data Using Change Data Capture: A Hands-On Lab

Stale data used to be tolerable, because the person reading the dashboard knew it was a few hours behind. Agents query constantly and act on what they find, and a partially arrived transaction looks exactly like an arrived one. That makes stale data a correctness problem - your agent will draw the wrong conclusion!

In this guided lab you'll capture updates from a live source database as they happen, materialize them into your warehouse in a transactionally safe way, and answer how current your data actually is.

My take: Nobody wants to point agents at their production database. Replicating your data to a safer and faster environment for agents is a great plan, and Estuary can help make that seamless!

Turning Your Data Into AI Context, Powered by Airbyte

Go from a raw source to an agent that answers questions about your data. You'll build a custom source in the Airbyte Cloud connector builder, use the Airbyte MCP to sync it into MotherDuck, then build an agent on top that explores the data in natural language and puts a dashboard on it.

Then comes the important part: structuring the agent's context so its answers are consistently correct. You'll leave with a working connection, a dashboard on your synced data, and an agent whose accuracy you've actually measured.

My take: I've pulled data out of all kinds of legacy data systems (in a past role), so it is really valuable to be able to customize your data ingestion. I'm certain Airbyte + agents make this way more fun than it used to be!

From Unstructured Documents to Agent-Ready Data

Workshop: Palak Agarwal (DevRel Lead, Reducto) and Josh Nkoy (Founding Solutions Engineer, Reducto) · 2:30 to 4:30 PM. On the agenda

PDFs, scans, images, and all kinds of messy documents go in, and a structured, searchable archive comes out. You'll use Reducto's MCP to build a curator, then pick your own collection and create a mini-exhibition: a poetry show made from drafts and annotations, a moment in history told through letters and newspapers, or a championship season rebuilt from box scores and scouting reports.

You'll leave with a working document pipeline and an exhibition you can share.

My take: A scanned in PDF is the final boss of messy data, but sometimes that's where the core nuggets of organizational context live.

Run Agents in Production

Shipping an agent is the start. These 2 workshops cover what comes after: finding out what your agent actually does in production, and what it takes to serve lots of long-running agents at once.

The Agent Loop: From Traces to Patterns to Better Agents

Workshop: Izzy Hurley (Eval Engineer, Braintrust) and Spencer Seale (Solutions Engineer, Braintrust) · 11:00 AM to 1:00 PM. On the agenda

Your agent is in production. Now you need to understand what keeps going wrong. Agents rarely fail in obvious ways, and no team can manually review every production trace.

You'll run the full evaluation loop twice: first with a physical building challenge that makes the ambiguity tangible, then with a real AI agent. You'll explore production traces, uncover recurring patterns, build an evaluation dataset and scorer, and test an improvement.

You'll leave with a repeatable workflow for turning real agent behavior into more reliable AI.

My take: If you want your company to trust AI, you need evals. The folks at Braintrust have a ton of expertise here!

LLM Inference for the World of Agents

Learn how your agent works, down to the details, so you can build or customize your own.

An agent might run for minutes, make 100 tool calls, and generate tens of thousands of tokens before it's done. That's a very different workload from the single prompt and response that most examples describe.

Zain will take apart a coding agent (think Claude Code, Codex, or Cursor) into its model, harness, context, tools, and memory, then follow the tokens into the inference engine: tokenization, continuous batching, prefill and decode, KV-cache management, prefix caching, and more.

You'll leave with one continuous mental model of the entire agent stack.

My take: We used to design software, but I think that we will increasingly be designing the systems we use to build software. This is a great way to get comfortable with the agent-centric future we're headed to.

See You at the Outpost!

Every workshop on this list ends with something you can keep using: an agent, a pipeline, a runbook, or a mental model. And they share the big idea from the main day: agents get better when the data underneath them is done well.

A quick heads up: workshops run in parallel, so pick 1 per time slot. Then stick around for Thursday, November 5, for a full day of talks and panels on the data stack after agents, AI at scale, semantic layers, and more.

Register for Data Outpost and bring your laptop.

See you in San Francisco!

AI is rebuilding how companies work, but we all know that slapping a chatbot on top won't cut it. Data systems are the way that even the most powerful frontier model knows anything about your business. The intersection of the two is filled with complexity: what does it mean when agents are "part of the team"? What does it take for an AI system to be trustworthy? How can we build insight-generating systems that scale?

Data Outpost is an AI and data conference where you'll learn from down-to-earth experts and visionaries alike. It's a small format event, so you get to actually talk to these folks and collaborate with them.

Workshops are on Wednesday, November 4th, and the main day of talks and panels is Thursday, November 5, 2026, both at Convene 100 Stockton in San Francisco.

Sound good? Register for Data Outpost here! Need more convincing? Read on!

We'll cover a quick tour of what you'll learn from our sessions plus my take about what I'm most excited about for each!

Read-Only Was Nice While It Lasted

Agents used to read your data and hand back answers. Now they act on it, and once they do, someone has to decide what they're trusted with.

Data + AI Product Leadership: The Data Stack After Agents

Panel: Tristan Handy (President & Co-Founder, Fivetran + dbt Labs, moderator), Stefanie Tignor (Head of Data, Clay), Jake Hannan (VP, Data & Revenue Engineering, Sigma), Leanne Fretz (Head of Data, Mercury), and Alexis Weill (Head of Data Science, Perplexity). On the agenda

dbt, Clay, Sigma, Mercury, and Perplexity take on what a data stack looks like when agents write, respond, and assemble their own context at a pace far faster than we humans do. If your agents are starting to do more than read, you'll hear how 5 companies are tackling that challenge and the benefits they are seeing from it.

My take: Data and AI are linked together now, just like software and AI. Agents writing data makes a lot of sense as the next step to me.

Finding a Seat at the Table for Your Agent

Michael's team built a capable data engineering agent, but found that creating it was only half the challenge. They wanted to make the agent a "real" team member.

Fitting an agent into their workflows raised harder questions around trust and accountability. You'll leave with the framework his team uses for agent oversight.

My take: If you have questions about the details, so do I! This approach is definitely at the frontier and I want to know more.

Getting It to Work Is the Easy Part

This is the most AI-flavored theme of the day. Even here, the hard problems show up after the demo.

AI at Scale Inside Real Companies

Panel: Tomasz Tunguz (General Partner, Theory Ventures, moderator), Sudeep Das (Head of New Verticals AI, DoorDash), and Tosh Rayadhurgam (Head of Advanced AI, Stripe). On the agenda

Getting an AI feature to work is the easy part. Keeping it working for millions of users, under real latency and cost constraints, is decidedly harder.

The heads of AI at Stripe and DoorDash will share how they evaluate models, decide what to build versus buy, and structure teams around systems that are never quite finished. If you have a working prototype and a roadmap that says "production," this is your panel.

My take: Scaling AI systems is absolutely an in-demand skillset. Demos are one thing, but it's a lot tougher to actually build a full platform.

What Does "Active Customer" Mean, Anyway?

If you want AI to give people the right answers, it first needs to understand what the business means. Both talks here are about getting that meaning in front of the AI.

Automating the creation of the semantic layer

Do you have 3 definitions of "active customer" and just one expert who knows the difference? You are not alone.

Sam at Canva tackled this problem with semantic leaking, a novel technique that uses LLMs to deliberately leak context between technical artifacts and the language of the business. It surfaces the terms people actually use, generates realistic questions for each domain, and finds the concepts and relationships your models need to support.

Then Sam turns that same context against the system. Ambiguous, adversarial, and deliberately unanswerable questions test whether the AI answers what it can, refuses what it can't, and explains the difference.

My take: Agents should answer most questions, but sometimes the agents really ought to escalate to the data folks! Teaching them how to decide is powerful.

Delivering Self-Serve Intelligence in a Composable System of Record

Picture an operator asking why an order is late. The answer is in the ERP somewhere, but getting to it means knowing the data model, stitching together reports, and asking a technical team for help.

Doss is building a system of record whose schema reflects how each customer actually runs their business, with interfaces and agents on top. Arnav will show how they connect operational data, business context, permissions, and agent-driven exploration, where that flexibility helps, and how to make a system of record actually useful.

My take: I'm a self-service-maximalist, so a talk about serving customers who know their business better than their data is right up my alley!

The Harness Around the Model

Which matters more for AI-written SQL: the model, or everything around it? Julia's talk has a clear answer.

The SQL Agent Harness We Didn’t Mean to Build

Natural language to SQL rests on an appealing assumption: the model is what makes the query correct. Julia argues the result depends far less on the model than on the harness around it, meaning what the model can see, verify, and run.

Her team built SQL tools in Positron (a data science IDE from the team behind RStudio) for people: schema browsing, completions, an interactive console, and a data explorer. Then they pointed an AI assistant at that same environment and realized they had already built the harness.

You'll learn what a good SQL authoring harness requires, and why bolting on an "AI mode" is so often the wrong shape for this kind of work.

My take: For SQL, a more advanced harness beats a more expensive LLM any day. Learning what a good harness needs is a very valuable skill!

Understand (and Own) Your Data First

AI can turn documents into tables faster than ever. These 2 talks cover what has to be true about your data for that speed to pay off.

Understanding comes before modeling: a data engineer's philosophy for the age of AI extraction

Thanks to agents, regulatory documents, insurance claims, and bills can become tables in a heartbeat. By itself, that won't amplify your team.

Lucie built the data platform at Opto Investments on 1 principle: you don't model what you don't understand yet. You keep the data, study it, and let it show you what it is before you impose structure. That is how you reach scale.

When AI extraction lands on a well-understood data model, every new document and source compounds. Skip that work, and every new thing breaks something.

You'll see the infrastructure behind that approach, including what it looks like when the data keeps surprising you and there's no playbook.

My take: Documents are the final boss of unstructured data. Lucie is getting real value out of them, and it takes both data and AI expertise.

Your Compliance Team Was Right: Why Regulated Firms Must Own Their Data+AI Stack

For banks, insurers, healthcare organizations, and law firms, sending sensitive data to third-party AI tools can be more than risky - it can be illegal. But sitting out AI isn't an option.

Upal has deployed document intelligence pipelines inside regulated Fortune 500 environments. He'll cover the rules driving this (privilege, HIPAA, GLBA, and data residency) and the architecture patterns that make it feasible for lean data teams.

He'll also show where to draw the line between building and buying self-hosted components, so compliance, legal, and engineering all sign off. You'll leave with a concrete framework for bringing AI to your data instead of shipping your data to AI.

My take: For something as critical as security but as new as AI, you want to learn from someone who has been there.

See You at the Outpost!

AI gets better when the data work underneath it is done well, whether that's a context layer, a SQL harness, or a data model that fits the domain. And that data work gets more valuable once agents can use it.

Stay tuned, because a few more sessions are still landing! Jordan Tigani (Co-Founder & CEO, MotherDuck) kicks off the day with the opening keynote. Joe Reis (co-author of O'Reilly's Fundamentals of Data Engineering), Ameya Bhatawdekar (VP, Field CTO, Braintrust), and Benn Stancil (former Co-founder and Chief Analytics Officer at Mode) are on the agenda too, with topics coming soon. Plus, Ben Holtzman (Head of Data, AheadComputing) will share real life stories of building an AI-centered self-service data platform. Keep an eye on the agenda for the details.

Want to get your hands dirty first? The day before, November 4, has 11 hands-on workshops where you can build a data agent, scale DuckLake and Iceberg for agent workloads, stream real-time data with change data capture, and even build an MCP server that reserves a real mystery gift!

If these abstracts read like your team's whiteboard, come ask your questions in person. Register for Data Outpost, and come a day early for the workshops if you want to get hands-on.

See you in San Francisco!

Let's take a dataset with 100,000 complaints. Each row has a narrative (someone explaining what went wrong with a bank, a lender, or their credit report) and how the company responded. The column I actually want is missing: what was the person asking for?

It sounds like a job for an LLM. But running an LLM over every row just to pick one of seven known answers is like using a cannon to kill a fly. It's slow on a big table, so it scales badly, or it just burns your credit card.

prompt_jev() classified the 100,000 rows in 82.2 seconds, with no LLM generating the per-row answers. On the same complaints, gpt-5-nano through prompt() needed 6 minutes 9 seconds for only 10,000 rows (Jev: 12.5 seconds). On cost, the launch benchmark put Jev at $0.50 per 100,000 rows, against $1.58 for gpt-5-nano and $37.58 for GPT-5.6 Terra (API prices). That's about 1% of the frontier model's bill.

Faster and cheaper? Let's look at what Jev does, run it on the dataset, and see where I would (and wouldn't) trust the result.

Already convinced and looking for ideas for your own data? I wrote a follow-up with 5 Jev use cases on real datasets, from AI agent logs to Hacker News and job postings, with shares you can attach.

What Jev does

TypeSafe calls Jev a "System One" model, borrowing from Kahneman's Thinking, Fast and Slow. System 1 is the quick, intuitive answer. System 2 is the slow, deliberate one. A classic LLM is System 2: it reasons and writes its answer token by token. Jev is System 1: it reads the text once and scores the answers you allowed.

| | System 2: a classic LLM | System 1: Jev | | --- | --- | --- | | How it answers | Writes text, token by token | Scores a fixed list of answers in one pass | | What you get back | Free text (you parse it, or force a schema) | A typed label, probability, or score | | Can invent a new answer | Yes | No, only the options you give it | | Tells you how sure it is | Not directly | A probability per option, plus a confidence | | 100k rows benchmark | 18 to 32 minutes, $1.58 to $37.58 | 40 seconds, $0.50 | | Good at | New taxonomies, summaries, explanations | The same bounded decision over many rows | | Example: customer request: "Please refund the two fees you charged me." | Answer is something like "The customer is asking for a refund of the two fees..." (a sentence to parse) | {choice: 'refund', confidence: 1.0} (real output, 3 labels) |

For SQL, think of it as a smart decision function: you give it one row of text, a question, and the allowed answers. You get back a typed decision, plus a number that tells you how sure it is.

There are three question shapes. Here they are on the same real complaint (10306039, someone charged low balance fees on an account they had closed):

| Shape | You ask | You get back | Real answer | SQL use | |---|---|---|---|---| | Choice | "What outcome is the consumer asking for?" + 7 labels | one label + confidence | refund_or_charge_reversal, 0.95 | GROUP BY | | Noul | "The complaint is about fees the bank charged." | a probability, 0 to 1 | 0.97 (and 0.02 for "reports identity theft") | WHERE | | Score | "How urgent is this?" on low / medium / high | a position on the scale + confidence | 1.62 (0 = low, 2 = high), 0.43 | ORDER BY |

This walkthrough uses Choice, since each complaint should land in one request category.

How does that compare to what you already use?

• An LLM is the right tool for a new taxonomy, a summary, or an explanation. A schema can hold it to your labels, but it still writes each one out token by token. Jev picks from your list, so you always get back one of your strings.
• CASE WHEN or a regex only works when the rule is literal. People write "reverse the charge" instead of "refund", and mentioning money isn't always asking for it.
• A trained classifier scales well but is painful to run (labelled examples, a model to maintain). With Jev, you describe the categories in the query and try them right away.
• Parsing rules (JSON paths, address parsers) are static and break when the format changes. Jev works from what the text says, so messy input hurts it less.

Jev can only return one of the options you gave it, so poor options get you a neatly typed poor answer. Garbage in, garbage out.

About the dataset: complaints from CFPB

The complaints come from the U.S. Consumer Financial Protection Bureau (CFPB's archive), the agency that collects complaints about banks, lenders, credit bureaus, and other financial services.

My table complaints_100k has (spoiler alert) 100,000 rows with complaint_id, narrative, company_response, and a few other fields. Here's what four rows look like:

| complaint_id | product | narrative (trimmed) | company_response | |---|---|---|---| | 9906201 | credit reporting | "This account is incorrectly being reported as a charged off account with a balance due. Please provide proof of the charge off and update the balance to {$0.00}..." | Closed with explanation | | 10306039 | bank account | "I closed my Chase checking account in XXXX by sending Chase a letter via mail. Chase conveniently ignored the account closure request so they can charge low balance fees..." | Closed with monetary relief | | 6084737 | debt collection | "I have no outstanding debts owed that I am aware of. Americollect has called numerous times almost daily..." | Closed with explanation | | 2042216 | credit card | "I paid my credit card with a money order. They did not apply the payment to my account..." | Closed with monetary relief |

Let's get the gold out of that narrative column.

ATTACH 'md:_share/jev_complaints/24f0bb33-43ef-4fe3-b41c-f37f3882995c' AS jev_complaints;
USE jev_complaints;

complaints_100k is the before: the raw complaints, ready for prompt_jev(). complaint_requests is the after: the Jev labels for all 100,000 rows, so you can jump straight to the GROUP BY. Tip: try prompt_jev() on a LIMIT 1000 first, since every call uses AI units.

Step 1: define the labels, then classify

Before classifying 100,000 rows, we need the categories. I sampled 40 narratives and asked MotherDuck's prompt() function, running gpt-5-mini, to suggest seven requested-outcome categories. That took 7.6 seconds. How you sample matters (and you can take a bigger one), but that's a business question more than a technical one.

Then Jev classifies every row against these categories, in less than 2 minutes:

.dd20-sim{width:100%;height:auto;margin:32px 0}design the labels once · apply them to every rowonce · 40 sampled rowsPlease note the above referenced accounts were...I paid my credit card with a money order. They...I just understand how my scores went from 840s...+ 37 more narrativesprompt()gpt-5-mini7.6 sFix credit reportIdentity theftRefund / reversalOtherExplain decisionStop contactTransaction error7 labels + a description eachevery row · 100,000 of themsame 7 labels, as constantsThis account is incorrectly being reported as a...I closed my Chase checking account in XXXX by s...I have no outstanding debts owed that I am awar...+ 99,997 moreprompt_jev()scores 7 labels82.2 s totalFix credit report1.00Refund / reversal0.95Stop contact1.00request.choice · confidence

Jev requires the question and the choices to be constants, so the simplest way is to paste the taxonomy straight into choice :=. Here's the classification step, with the same labels and descriptions as the measured run:

CREATE TABLE complaint_requests AS
SELECT complaint_id, company_response,
       prompt_jev(
         narrative,
         'What outcome is the consumer asking for?',
         choice := [
           {label: 'refund_or_charge_reversal',
            description: 'Consumer requests money returned, reversal of fees/charges, or reimbursement.'},
           {label: 'correct_or_remove_credit_report_entry',
            description: 'Consumer asks to fix, update, or remove inaccurate items on their credit report.'},
           {label: 'investigate_and_credit_transaction_error',
            description: 'Consumer requests investigation and correction of a transaction or deposit error and credit to their account.'},
           {label: 'stop_harassment_or_contact',
            description: 'Consumer asks for communications or collection calls/texts/emails to stop or follow rules.'},
           {label: 'address_identity_theft_or_fraud',
            description: 'Consumer requests action to resolve identity theft, fraudulent accounts, or unauthorized inquiries.'},
           {label: 'explain_decision_or_account_status',
            description: 'Consumer asks for clarification or explanation of a decision, account action, or why charges occurred.'},
           {label: 'other',
            description: 'Any requested outcome that does not fit the categories above.'}
         ]
       ) AS request
FROM complaints_100k;

.dd20-sim{width:100%;height:auto;margin:32px 0}prompt_jev() · one label per complaintMotherDuckbatches the rowsTypeSafe · Jevwaitingscoring 7 labelsdone · no text outnarratives · 1 qnarrativeproductrequest.choiceconfidenceThis account is incorrectly being reported as a charged...This account is incorrectly being reported as a charged off account with a balance due. Please provide proof of the charge off and update the balance to {$0.00} or delete the negative payment status on this account.credit reportingFix credit report1.00I closed my Chase checking account in XXXX by sending C...I closed my Chase checking account in XXXX by sending Chase a letter via mail. Chase conveniently ignored the account closure request so they can charge low balance fees. They have twice charged me low balance fees on an account that should be closed.bank accountRefund / reversal0.95I have no outstanding debts owed that I am aware of. Am...I have no outstanding debts owed that I am aware of. Americollect has called numerous times almost daily and has now begun calling me outside of the legal time frame before XXXX and after XXXX.debt collectionStop contact1.00Please note the above referenced accounts were opened F...Please note the above referenced accounts were opened FRAUDULENTLY ; as a result of Identity Theft. My information was compromised & fraudulently used without my permission.credit reportingIdentity theft0.96I paid my credit card with a money order. They did not...I paid my credit card with a money order. They did not apply the payment to my account. I sent conformation/copies from the money order company that the money order had been cashed by the credit card company.credit cardTransaction error0.99I just understand how my scores went from 840s to the 7...I just understand how my scores went from 840s to the 770s with NO ACTIVITY on my reports??????? I am very clueless and I need an explanation from each of the Creditors.credit reportingExplain decision0.98

request is a struct. request.choice is the picked label, and request.confidence and request.probabilities help you inspect the decision. Here's what came back for the Chase fees complaint (10306039):

{
  choice: 'refund_or_charge_reversal',
  confidence: 0.95,
  probabilities: [
    {value: 'refund_or_charge_reversal', probability: 0.96},
    {value: 'other', probability: 0.03},
    {value: 'explain_decision_or_account_status', probability: 0.01},
    ... -- the other 4 labels at 0.00
  ]
}

Bonus: keep the labels in a table

Pasting 7 labels into a query is fine once. If the taxonomy sticks around (versioned, shared across queries, edited by the business), you'd rather keep it in a table. A subquery in choice := doesn't work ("choice" parameter must be a constant value), but a DuckDB variable does, because it's resolved before the query runs:

SET VARIABLE request_labels = (
  SELECT list({label: label, description: description} ORDER BY label)
  FROM request_taxonomy
);

SELECT complaint_id,
       prompt_jev(narrative, 'What outcome is the consumer asking for?',
                  choice := getvariable('request_labels')) AS request
FROM complaints_100k;

Why bother: one source of truth for the labels, easy to version (add a taxonomy_version column), and the business can review the labels without reading SQL.

Step 2: don't trust Jev blindly

Jev can't invent a label, but it can still pick the wrong one.

The first thing to look at is confidence. Jev scores every label, and confidence says how concentrated those scores are: 1.0 when one label takes everything, low when two or three labels share the probability. Across the 100,000 rows, the mean confidence is 0.83, 65.8% of rows are at 0.8 or above, and 10.9% are below 0.5.

SQL gives you the review queue:

SELECT complaint_id, request.choice, request.confidence
FROM complaint_requests
WHERE request.confidence < 0.8
ORDER BY request.confidence;

.dd20-sim{width:100%;height:auto;margin:32px 0}request.confidence · keep it or review itcomplaint 10306039"I closed my Chase checking account in XXXX bysending Chase a letter via mail. Chaseconveniently ignored the account closure request ..."request.probabilitiesRefund / reversal0.96Other0.03Explain decision0.01request.confidence0.950.8keepcomplaint 8834942"I urge you to act swiftly in removing theseunauthorized inquiries from my credit report. Theyare causing significant distress, and I have ..."request.probabilitiesIdentity theft0.60Fix credit report0.40Other0.00request.confidence0.530.8review queue

0.8 is an example, and a row above it can still be wrong. Send uncertain rows to a human or a bigger model, and keep the rest in the analytics table once you've validated that cut on your data.

The second check is to compare against another model on a sample. On 300 rows, I labelled them twice: with GPT-5 through prompt() (the newest model MotherDuck's prompt() offers today) and with Claude Opus 5.5, which read each narrative blind, without seeing any other label. Here's what came out:

| Comparison on 300 rows | Agreement | |---|---:| | GPT-5 vs Claude Opus 5.5 | 72.7% | | Jev vs GPT-5 | 77.3% | | Jev vs Claude Opus 5.5 | 74.0% | | Jev, on the 218 rows where GPT-5 and Opus agree | 90.4% | | Jev, on those rows with confidence >= 0.8 (161 rows) | 96.9% |

The two big LLMs agree with each other less often than Jev agrees with either of them. So the task itself is fuzzy: most disagreements sit on the boundaries we already saw (identity theft vs credit-report fix, "other" vs a specific label).

Step 3: ask the question (the good boring part)

Once the requested outcome is a typed column, it's plain SQL:

SELECT request.choice AS requested_outcome,
       count(*) AS complaints,
       round(100.0 * avg(
         (company_response = 'Closed with monetary relief')::INT
       ), 1) AS pct_monetary_relief
FROM complaint_requests
GROUP BY 1
ORDER BY complaints DESC;

It runs in 0.9 seconds on MotherDuck. The 82.2 seconds of classification happen once, and every question after that costs what any other GROUP BY costs.

.dd20-sim{width:100%;height:auto;margin:32px 0}GROUP BY request.choice · 100,000 complaintscomplaintsgot money backFix credit reportFix credit report: 59,098 complaints59,098Fix credit report: 0.2% got money back0.2%Identity theftIdentity theft: 16,149 complaints16,149Identity theft: 1.5% got money back1.5%Refund / reversalRefund / reversal: 7,140 complaints7,140Refund / reversal: 19.9% got money back19.9%OtherOther: 6,878 complaints6,878Other: 2.5% got money back2.5%Explain decisionExplain decision: 5,173 complaints5,173Explain decision: 4% got money back4.0%Stop contactStop contact: 3,116 complaints3,116Stop contact: 0.8% got money back0.8%Transaction errorTransaction error: 2,446 complaints2,446Transaction error: 13.6% got money back13.6%

When I would use this pattern

Use prompt_jev() when the same decision repeats across many rows and you can write the possible answers down: complaint intent, product family, "did they ask for a refund?", a severity rubric. Use an LLM when you need new categories, generated text, or an explanation a human will read. Or use both, like here: the LLM helps design the question, Jev applies it at scale.

The MotherDuck launch benchmark classified 100,000 short news articles in 40 seconds. My 82.2-second complaint run is a different task (longer narratives, seven described choices), so I wouldn't promise either number on your table. Test on a sample first :)

prompt_jev() is available on MotherDuck paid plans. The text is sent to TypeSafe for inference, so check your data-handling requirements before passing sensitive customer data. The guide and function reference have the full syntax.

DuckDB is small. The data it queries doesn't have to be. So let's try something big. Common Crawl has been crawling and publishing web pages since 2008, and every crawl lands on S3 for anyone to read. Coding agents have turned a lot of people into web developers, so here is the internet-sized question for today: how much has personal, platform-hosted publishing actually grown?

GitHub, Vercel, Lovable and the rest all hand you a subdomain for whatever you just built, a dating app for ducks included. The obvious guess is that those subdomains are multiplying fast. Let's check. First some history before we get our hands on those 100 billion rows, so cue the old man yelling at the cloud.

Common Crawl, HTTP Archive, Wayback Machine: same same, but different?

If you've ever wanted to see the history of a page or find a site that has since been deleted, you may be familiar with the Wayback Machine, part of the Internet Archive. It is a great database of internet history, and while it is perfect for finding historical artefacts it doesn't lend itself well to analysis across sites. The HTTP Archive is a Google-backed project that cares less about the content than about the technology of the web, and it even loads JavaScript, partially, when it crawls. The downside is that it mostly captures the homepage of a site and nothing else.

Our dataset of choice, Common Crawl, does not load JavaScript, so it might miss parts of a JavaScript-powered single-page application, but it does go wide and deep, following all the links it finds. That makes it perfect to find, for example, github.io pages, which usually have the structure <github-username>.github.io/<repo-name>.

In short: Common Crawl is a broad net for all page content, HTTP Archive is a lab measurement of a fixed number of sites, and the Wayback Machine is a time series of individual URLs.

| | Common Crawl | HTTP Archive | Wayback Machine | | --- | --- | --- | --- | | Size / scope | ~2.2B pages and ~150 TiB per monthly crawl since 2008 | ~16.2M websites per crawl (15.4M mobile, 12.2M desktop), sourced from Chrome UX Report (2025 methodology), since 2010 | >1 trillion URL captures, ~99 PB unique, ~150 TB/day (blog.archive.org/trillion). Internet Archive since 1996, Wayback Machine launched 2001 | | Home page vs. paths | Link-following crawl, so /v2/category/... style URLs are there, though coverage per site is a sample, not exhaustive | Home pages primarily, plus some secondary pages in recent years | Whatever anyone or any crawler ever requested | | What's stored | Raw WARC (headers + HTML), plus WAT (metadata) and WET (plain text), plus a columnar Parquet index and a hyperlink/PageRank graph | HAR files, Lighthouse audits, technology detection | Full WARC captures of the original bytes, replayable in a browser | | Access | s3://commoncrawl + data.commoncrawl.org, CDX API, Parquet index, queryable straight from DuckDB | BigQuery (httparchive.crawl.*), monthly partitions | CDX API + web.archive.org/web/<timestamp>/<url>; no bulk dump | | Best for | Text corpora (LLM training), link graphs, domain inventories | Web-performance, Core Web Vitals, framework/tracker share over time | Point lookups and history of a specific page |

How much of the Common Crawl dataset do you actually need?

Common Crawl collects about 150 TiB per crawl. Every crawl is a sample, so I want several crawls per year and a few years of history. That sounds like querying 15 petabytes. It isn't. Most of the time "Big Data" just means a poorly designed dataset. This one is designed well, and it all sits on S3 for us to query and slice as we need.

So let's look at what's in the dataset and what we actually need. Common Crawl looks roughly like this.

• Each crawl is a folder, e.g. https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-34/
• The folder contains both the data in different formats and pointers to the data
  • WARC contains the full payload
  • WAT contains the metadata: HTTP headers, meta tags, outbound links, etc.
  • WET only contains the plain text
  • CDX URL index: the 300 or so files with just the index to point you to the right WARC file. You can try it out directly with e.g. https://index.commoncrawl.org/CC-MAIN-2026-34-index?url=motherduck.com&output=json
  • Columnar URL index: 900 Hive-partitioned Parquet files.
• Every 3 months crawls are combined into a web graph. This dataset records how domains connect to each other and ranks them with several algorithms. It produces a list for both domains (e.g. github.io) and hostnames (duckfan67.github.io). For each of these there is:
  • A list of all nodes, that is the host or domain name with their IDs.
  • A list of all edges, that is which ID connects to which ID.
  • A ranking based on harmonic centrality and PageRank.

So we can take a shortcut and query the web graph's hostname list instead of the crawls it was built from. One release of that list is between 4.6GB and 11GB gzipped and holds every hostname the crawl saw across three months: 247 million of them in June 2026, spread over 121 million registered domains. Sixteen releases, one per quarter back to 2021, comes to 103GB of gzip. That drops us from 15 petabytes to 103 gigabytes without giving up a single hostname. And I do want the hostnames, because a count you can't drill into is a count you can't check.

The hostname list isn't quite enough, though. Remember the shape of a github.io URL. The hostname tells you who published, and how many repositories they published lives on the path. So we'll read both: the hostname list for breadth, the columnar URL index for paths.

Two ways to query Common Crawl with DuckDB

First we need to know what to query.

1. Query Common Crawl on S3

All Common Crawl data sits in S3, free to read, and it is Hive partitioned, so querying several crawls at once costs you one glob.

from read_parquet('s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-*/subset=warc/*.parquet', union_by_name = true)
where url_surtkey like 'com,motherduck)%'
limit 10000

That returns every MotherDuck page in the 2026 crawls in about 7 seconds. url_surtkey is the sort key, so DuckDB reads the Parquet row-group statistics and skips nearly every file. No cluster to start, no pool of servers, just a small DuckDB instance. The full scan back to 2008 takes three to four minutes.

Note that you will need a (free) AWS account with an S3 access key to make this work. If you don't want to do that you can also query directly over HTTP. For the rest of the article I'll use S3.

2. Query Common Crawl over HTTP

If you don't want to use S3, you can still query Common Crawl. DuckDB uses HTTP range headers to download only the parts of the Parquet file it needs. With HTTP you cannot glob using the * as you would with S3, so you have to get all the actual paths of the Parquet files first.

-- store all Parquet files as a list
set variable index_files = (
  select list('https://data.commoncrawl.org/' || path)
  from read_csv([
      'https://data.commoncrawl.org/crawl-data/CC-MAIN-2025-38/cc-index-table.paths.gz',
      'https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-34/cc-index-table.paths.gz'
    ],
    header = false,
    columns = {'path': 'STRING'})
  where path like '%subset=warc%'
);

-- access all Parquet files in the list
from read_parquet(getvariable('index_files'), union_by_name = true)
where url_surtkey like 'com,motherduck)%'
limit 10000;

The extraction: pulling hostnames out of the web graph

First, the list of platforms to look for. Swap in any you like. I picked the ones I could name off the top of my head, plus a reference group I'll come back to in the analysis.

create or replace table your_db.commoncrawl.platforms as
select * from (values
  ('base44.app',      'app.base44',      'Base44',                      'platform'),
  ('bolt.host',       'host.bolt',       'Bolt',                        'platform'),
  ('deno.dev',        'dev.deno',        'Deno Deploy',                 'platform'),
  ('firebaseapp.com', 'com.firebaseapp', 'Google Firebase',             'platform'),
  ('web.app',         'app.web',         'Google Firebase',             'platform'),
  ('fly.dev',         'dev.fly',         'Fly.io',                      'platform'),
  ('framer.app',      'app.framer',      'Framer',                      'platform'),
  ('framer.website',  'website.framer',  'Framer',                      'platform'),
  ('github.io',       'io.github',       'GitHub Pages',                'platform'),
  ('gitlab.io',       'io.gitlab',       'GitLab Pages',                'platform'),
  ('glitch.me',       'me.glitch',       'Glitch',                      'platform'),
  ('herokuapp.com',   'com.herokuapp',   'Heroku',                      'platform'),
  ('lovable.app',     'app.lovable',     'Lovable',                     'platform'),
  ('neocities.org',   'org.neocities',   'Neocities',                   'platform'),
  ('netlify.app',     'app.netlify',     'Netlify',                     'platform'),
  ('onrender.com',    'com.onrender',    'Render',                      'platform'),
  ('pages.dev',       'dev.pages',       'Cloudflare',                  'platform'),
  ('workers.dev',     'dev.workers',     'Cloudflare',                  'platform'),
  ('railway.app',     'app.railway',     'Railway',                     'platform'),
  ('replit.app',      'app.replit',      'Replit',                      'platform'),
  ('replit.dev',      'dev.replit',      'Replit',                      'platform'),
  ('streamlit.app',   'app.streamlit',   'Streamlit',                   'platform'),
  ('surge.sh',        'sh.surge',        'Surge',                       'platform'),
  ('v0.app',          'app.v0',          'Vercel (v0)',                 'platform'),
  ('v0.dev',          'dev.v0',          'Vercel (v0)',                 'platform'),
  ('vercel.app',      'app.vercel',      'Vercel',                      'platform'),
  ('webflow.io',      'io.webflow',      'Webflow',                     'platform'),

  -- the old guard, as a sanity check that the crawl itself isn't just growing
  ('blogspot.com',    'com.blogspot',    'Blogspot',                    'reference'),
  ('medium.com',      'com.medium',      'Medium',                      'reference'),
  ('myshopify.com',   'com.myshopify',   'Shopify',                     'reference'),
  ('shopify.com',     'com.shopify',     'Shopify',                     'reference'),
  ('squarespace.com', 'com.squarespace', 'Squarespace',                 'reference'),
  ('substack.com',    'com.substack',    'Substack',                    'reference'),
  ('tumblr.com',      'com.tumblr',      'Tumblr',                      'reference'),
  ('weebly.com',      'com.weebly',      'Weebly',                      'reference'),
  ('wixsite.com',     'com.wixsite',     'Wix',                         'reference'),
  ('wordpress.com',   'com.wordpress',   'WordPress.com',               'reference')
) as t(domain, domain_rev, platform, kind);

Note the middle column. The web graph stores every name reversed, so github.io becomes io.github and duckfan67.github.io becomes io.github.duckfan67. Everything under one platform therefore sorts together, and the platform a hostname belongs to is its first two labels. That is the join key we'll build in a moment. But first: the actual data. We'll create the table, then go through each release one by one and insert them into the created table.

create or replace table your_db.commoncrawl.platform_hosts (
  release           varchar,
  release_end_date  date,
  domain_rev        varchar,
  host_rev          varchar,
  harmonicc_pos     bigint,
  harmonicc_val     double,
  pr_pos            bigint,
  pr_val            double
);

insert into your_db.commoncrawl.platform_hosts
with hosts as (
  select
    -- io.github.duckfan67 -> io.github, the reversed registered domain
    split_part(host_rev, '.', 1) || '.' || split_part(host_rev, '.', 2) as domain_rev,
    host_rev, harmonicc_pos, harmonicc_val, pr_pos, pr_val
  from read_csv(
    's3://commoncrawl/projects/hyperlinkgraph/cc-main-2026-apr-may-jun/host/cc-main-2026-apr-may-jun-host-ranks.txt.gz',
    delim = '\t',
    header = true,
    columns = {
      'harmonicc_pos' : 'bigint',
      'harmonicc_val' : 'double',
      'pr_pos'        : 'bigint',
      'pr_val'        : 'double',
      'host_rev'      : 'varchar'})
)
select
  'cc-main-2026-apr-may-jun', date '2026-06-30',
  domain_rev, host_rev, harmonicc_pos, harmonicc_val, pr_pos, pr_val
from hosts
where domain_rev in (select domain_rev from your_db.commoncrawl.platforms);

The join key has to be derived, which is the one bit of thinking in that query. Every platform on the list is a two-label domain, so the first two labels of a reversed hostname are the reversed domain, and DuckDB can hash-join that against the platform list instead of testing 37 patterns per row.

One release takes three to six minutes and lands 10 to 14 million rows, scaling with the size of the gzip. Sixteen of them, run one after another: 103GB read, 64 minutes wall clock, 190,685,102 rows kept. That is the entire compute bill for this post, because the actual analysis queries in MotherDuck are counted in milliseconds rather than minutes.

Backfilling the web graph with a MotherDuck Flight

Now that we have the model for backfilling the 3-month spanning web graphs, we won't be running 16 separate queries ourselves. Because the same job runs once per release, this is a natural fit for a Flight, a small Python job that runs on MotherDuck compute on a schedule. The where release not in (select release from ...) guard makes it idempotent, so you can re-run it and it only fetches what's missing. The full flight code is in the MotherDuck cookbook.

cc-main-202* seems like an easy one-liner to get all releases, but since 2024 Common Crawl publishes the web graph as a rolling three-month window, released monthly. cc-main-2025-jul-aug-sep, cc-main-2025-aug-sep-oct and cc-main-2025-sep-oct-nov share two thirds of their input. Glob them all and you download 40 files, 250GB of gzip, for maybe 13 independent data points.

We'll pick one release per quarter and pin the names to make our backfill consistent. Some sets leap over into the next year (cc-main-2022-23-sep-nov-jan, for example); we'll take those as a good enough representation of the underlying quarter.

cc-main-2021-feb-apr-may     cc-main-2024-apr-may-jun   cc-main-2025-jul-aug-sep
cc-main-2021-jun-jul-sep     cc-main-2024-jul-aug-sep   cc-main-2025-oct-nov-dec
cc-main-2021-22-oct-nov-jan  cc-main-2024-oct-nov-dec   cc-main-2026-jan-feb-mar
cc-main-2022-may-jun-aug     cc-main-2025-jan-feb-mar   cc-main-2026-apr-may-jun
cc-main-2022-23-sep-nov-jan  cc-main-2025-apr-may-jun
cc-main-2023-mar-may-oct
cc-main-2023-24-sep-nov-feb

If you'd rather not fetch all sixteen releases yourself, attach the share I prepared:

ATTACH 'md:_share/commoncrawl_platforms_public/96fe0b63-45b0-4775-be4e-9bfcff677311' AS commoncrawl_platforms;
from commoncrawl_platforms.main.platform_hosts limit 10;

The analysis: growth relative to the old free-hosting web

We cannot assume that every crawl covers the exact same parts of the internet. Even though the web graph already covers 3 months of crawls, we'll have to account for the fact that there is still sampling. Instead of looking at absolute numbers, then, we can analyse the relative presence of the tools. That is what the reference rows in the platform list are for. WordPress, Blogspot, Wix and friends are the boring, established part of the free-hosting web. If the crawl gets bigger, they get bigger too. If a platform grows relative to them, something real is happening.

The SQL for our analysis is relatively simple:

• take all the different platforms
• group by platform and crawl date
• count the number of hosts
• compare the reference platforms to the 'new' platforms

with ranked as (
  select
    p.platform,
    p.kind,
    r.release_end_date,
    count(*) as hosts
  from your_db.commoncrawl.platform_hosts r
  join your_db.commoncrawl.platforms p using (domain_rev)
  group by all
),
baseline as (
  select release_end_date, sum(hosts) as reference_hosts
  from ranked where kind = 'reference'
  group by all
)
select
  platform,
  release_end_date,
  hosts,
  round(1000.0 * hosts / reference_hosts, 2) as hosts_per_1k_reference,
  round(100.0 * hosts / first_value(hosts) over (
    partition by platform order by release_end_date), 1) as indexed_to_first
from ranked
join baseline using (release_end_date)
where kind = 'platform'
order by platform, release_end_date;

The first findings: how fast the vibe-coded web is growing

To make our analysis a bit easier, I've built a MotherDuck Dive that compares the old free-hosting web (WordPress.com, Blogspot, and the rest of the gang) to the new platforms. I've grouped them into four categories.

• Visual site builders like Webflow or Framer
• The full AI experience: Lovable, Replit, v0, Base44, Bolt
• Frontend and edge hosting: Cloudflare, Vercel, Netlify
• Developer-focused platforms: GitHub Pages, GitLab Pages, Streamlit, Heroku

Every category is up and to the right, and the visual site builders lead, at 55.6 hostnames per 1,000 reference hostnames in June 2026. Webflow is most of that. The more recent vibe-coding trend shows up in frontend and edge hosting, which went from 2.6 to 47.3 over the same five years, and the AI app builders are the newest line on the chart: flat at zero until 2025, then 4.7 by mid-2026.

Common Crawl data has its downsides: it only counts crawled pages and pages with inbound links, not total sites built. It also doesn't crawl domains that deny access to bots, which more of them have done over the last few years. That explains the dip in crawled WordPress.com hostnames, for example. Even so, I think it's a reasonable proxy for how fast these platforms are growing.

Deselect the categories and pick individual platforms, and the story sharpens. Webflow is the line at the top. GitHub Pages grows steadily as you'd expect from the default answer of the last decade. Vercel grows steadily too, but Cloudflare Pages is the one nobody wrote about: from roughly nothing in 2023 to a peak above Vercel in late 2025, before settling back. Lovable barely registers on this scale, which is the sober version of the AI-builder story: plenty of new hostnames, none of them linked to by much yet.

Now switch the Dive to Absolute (hostnames) and add the established platforms back in, because in absolute numbers these platforms still pale compared to the big names.

WordPress.com alone fell from 6.0M crawled hostnames to under 2.0M over the five years, which is a bigger move in absolute terms than everything the new platforms added put together. Most of the relative growth in the first chart is the old web shrinking. What do you make of the difference in growth and decline? You now have the numbers and tools to answer your own questions.

One user, many projects: querying 100 billion rows of URL index

We have forgotten one thing though. github.io pages are not one user, one project. You can have many different projects based on different repositories in your GitHub account. I've written before about what the growth of GitHub and package registries looks like during the AI boom. But we're not here to be lazy, we're here to query the internet. So let's query some paths.

We'll scan 100 billion rows in a couple of minutes, but did you know that for a long time, the fastest way to transfer 100 petabytes of data was with an actual truck?

So let's drop the web graph dataset and go back to the columnar URL index, because that is the only place where the actual path of a page is captured. And let me tell you, this is what we call going out with a banger. We are querying:

• 5 years of data (2022-2026)
• 12,300 Parquet files
• 109,431,785,857 rows. That is MORE THAN 100 BILLION ROWS!

Those two numbers come straight out of the Parquet footers, which is why counting 109 billion rows takes half a minute rather than half a day:

Of course reading all of it would take forever, but url_surtkey is the sort key of the index, so DuckDB reads the Parquet row-group statistics and skips almost every file without opening it. Parquet lets us cut the other way too: we only ask for ten of the index's columns, and a columnar format can skip the rest without reading a byte of them (as opposed to, say, Postgres or MySQL). This is also the reason why friends don't let friends select *.

create or replace table your_db.commoncrawl.crawl_githubio as
select
  url,
  url_host_name,
  url_host_name_reversed,
  url_path,
  fetch_time,
  fetch_status,
  content_digest,
  content_mime_type,
  content_languages,
  crawl
from read_parquet([
    's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-*/subset=warc/*.parquet',
    's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2023-*/subset=warc/*.parquet',
    's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2024-*/subset=warc/*.parquet',
    's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2025-*/subset=warc/*.parquet',
    's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-*/subset=warc/*.parquet'
  ], hive_partitioning = true, union_by_name = true)
where url_surtkey like 'io,github,%';

That is 16,353,324 rows kept out of 109 billion, across 41 crawls, and it takes a couple of minutes. The first path segment is the repository, and everything under it is one project no matter how deep the crawler went:

create or replace view your_db.commoncrawl.githubio_projects as
select
  crawl,
  url_host_name,
  case
    -- /myrepo/docs/api.html -> myrepo; /about.html -> the user site itself
    when url_path = '/' or url_path not like '/%/%'
      then '(user site root)'
    else split_part(url_path, '/', 2)
  end as project
from your_db.commoncrawl.crawl_githubio
where fetch_status = 200;

Absolute counts still can't be compared across crawls, because the crawls themselves differ in size. But we can take the total row count per crawl as a reference point, and that one is cheap for the same reason: it only reads the metadata in the footer of each Parquet file, not the file itself.

create or replace table your_db.commoncrawl.crawl_sizes as
select crawl, count(*) as total_pages
from read_parquet(
    's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2025-*/subset=warc/*.parquet',
    hive_partitioning = true)
group by all;

One year of footers comes back in a few seconds. Run it five times, once per year, and you have the size of all 41 crawls. They sum to 109,431,785,857, which is the cheapest possible check that you globbed what you meant to glob. The crawls shrink as they go, from 2,986,392,030 pages in CC-MAIN-2022-05 to 2,139,617,681 in CC-MAIN-2026-34.

Two crawls per quarter, and always two

One crawl is one sample, and for this question it is a thin one. Common Crawl ran two crawls in the third quarter of 2026. Each of them found about 47,179 distinct user.github.io/repo pairs. Pool the two and you get 79,976, a factor of 1.70. Two crawls three weeks apart overlap far less than you would guess, so a second crawl buys most of a second crawl's worth of projects rather than a rounding error. That is the whole argument for pooling: a quarter sees more of the web than any crawl in it.

The catch is the release schedule. Common Crawl ran six crawls in 2022, five in 2023 but twelve in 2025. A quarter pooled from one crawl and a quarter pooled from three is a different type of measurement. We're not doing double blind testing here, but we do want some semblance of a meaningful statistic. So to accommodate we take exactly the two earliest crawls of every quarter that has two.

create or replace view your_db.commoncrawl.crawl_pairs as
select
  crawl,
  total_pages,
  -- CC-MAIN-2026-34 means ISO week 34 of 2026
  year(crawl_start) || '-Q' || quarter(crawl_start) as quarter
from (
  select *,
    date_trunc('week', make_date(cast(split_part(crawl, '-', 3) as int), 1, 4))
      + to_days((cast(split_part(crawl, '-', 4) as int) - 1) * 7) as crawl_start
  from your_db.commoncrawl.crawl_sizes
)
qualify count(*) over (partition by quarter) >= 2
   and row_number() over (partition by quarter order by crawl) <= 2;

That leaves 14 quarters, 2022-Q3 to 2026-Q3. 2022-Q1, 2022-Q2, 2023-Q1 and 2024-Q1 drop out with a single crawl each, and Common Crawl ran nothing at all in the third quarter of 2023. Now pool:

with per_user as (
  select
    c.quarter,
    p.url_host_name,
    count(*)                                                             as urls,
    count(distinct project) filter (where project <> '(user site root)') as n_projects
  from your_db.commoncrawl.githubio_projects p
  join your_db.commoncrawl.crawl_pairs c using (crawl)
  group by all
),

effort as (
  select quarter, sum(total_pages) as pages_crawled
  from your_db.commoncrawl.crawl_pairs
  group by all
)

select
  quarter,
  pages_crawled,
  count(*)                                        as user_sites,
  round(1e6 * count(*) / pages_crawled, 2)        as sites_per_million,
  round(1e6 * sum(n_projects) / pages_crawled, 1) as project_sites_per_million,
  round(1.0 * sum(n_projects) / count(*), 2)      as projects_per_user_site
from per_user join effort using (quarter)
group by quarter, pages_crawled   -- not `group by all`: pages_crawled sits inside an aggregate
order by quarter;

Weeks below are the two ISO weeks pooled into that quarter, so 2022-Q3 is CC-MAIN-2022-27 plus CC-MAIN-2022-33. Every row is two crawls, so every row is the same instrument, and only the per-million columns are comparable down the table.

| Quarter | Weeks | Pages crawled | User sites | User sites per million | Project sites per million | | --- | --- | ---: | ---: | ---: | ---: | | 2022-Q3 | 27 + 33 | 5,698,299,922 | 40,008 | 7.02 | 20.9 | | 2022-Q4 | 40 + 49 | 6,558,375,936 | 42,559 | 6.49 | 16.7 | | 2023-Q2 | 14 + 23 | 6,259,207,381 | 45,369 | 7.25 | 16.8 | | 2023-Q4 | 40 + 50 | 6,799,057,161 | 52,894 | 7.78 | 17.3 | | 2024-Q2 | 18 + 22 | 5,496,678,032 | 44,185 | 8.04 | 18.5 | | 2024-Q3 | 30 + 33 | 4,862,556,811 | 43,336 | 8.91 | 16.2 | | 2024-Q4 | 42 + 46 | 5,183,002,501 | 47,030 | 9.07 | 17.7 | | 2025-Q1 | 05 + 08 | 5,710,984,393 | 44,117 | 7.72 | 14.7 | | 2025-Q2 | 18 + 21 | 5,224,501,712 | 43,087 | 8.25 | 13.9 | | 2025-Q3 | 30 + 33 | 4,865,073,985 | 38,163 | 7.84 | 12.1 | | 2025-Q4 | 43 + 47 | 4,911,269,769 | 40,645 | 8.28 | 12.9 | | 2026-Q1 | 04 + 08 | 4,496,622,477 | 41,410 | 9.21 | 14.2 | | 2026-Q2 | 17 + 21 | 4,356,077,177 | 54,850 | 12.59 | 20.6 | | 2026-Q3 | 30 + 34 | 4,288,619,137 | 52,573 | 12.26 | 18.6 |

Normalised for crawl size, distinct github.io user sites are up 75% over the four years, 7.02 per million crawled pages to 12.26, and about half of that arrives in the last two quarters. Distinct project sites per million barely move, 20.9 to 18.6. More people, the same amount of stuff. Which points straight at the number in the last column of the pooled query, and that one needs a much harder look.

Note that the actual sites crawled on Github Pages is much lower than the total number of Github Pages sites in the web graph. The web graph also takes into account links, but not every link is crawled every time.

Final results

The platform web went from 28 hostnames per 1,000 of the old free-hosting web in mid-2021 to 177 per 1,000 in mid-2026. 6.3x in five years and it keeps rising quarter-over-quarter including the quarters where the absolute platform count fell, because the crawl shrank faster than the platforms did. Normalised for crawl size, distinct github.io user sites are up 75% in four years. The drop in projects per user is the crawler taking fewer URLs per host, not people publishing less: hold the crawler's effort constant and the number is flat.

More people are putting things on the internet. Whether that is the AI boom or just what happens when publishing gets cheap enough is a question for a different post, but the trend itself is unambiguous and all it took was a few SQL queries.

The part you should take away though is that you can build your own billion row analysis in no time. Common Crawl publishes roughly 15 petabytes across the crawls in this analysis, but the answer came out in gigabytes. The power of partitioned Parquet files and DuckDB means that you can do this analysis on your laptop instead of a cluster of servers.

Reproduce it yourself

Everything above is four tables, a few SQL queries and a lot of Parquet files.

1. platforms, the hand-written list of 27 platforms and 10 reference domains.
2. platform_hosts, one insert per pinned web-graph release, three to six minutes each. The Flight in the cookbook runs all sixteen and skips the ones you already have.
3. crawl_githubio, one read_parquet over 12,300 Parquet files of the columnar URL index.
4. crawl_sizes, one metadata-only count(*) per year, seconds each, and the denominator for everything normalised.

The charts are the Dive embedded above, so you can switch between relative and absolute, pick your own platforms, and watch the numbers change when a new web-graph release lands.

If you want the numbers without the wait, attach the share. All four tables are in it, with crawl_githubio published as githubio_pages:

ATTACH 'md:_share/commoncrawl_platforms_public/96fe0b63-45b0-4775-be4e-9bfcff677311' AS commoncrawl_platforms;
from commoncrawl_platforms.main.githubio_pages limit 10;

And if you want it to keep updating itself as new web-graph releases land, take the Flight from the cookbook and give it a monthly cron. Swap my platform list for yours, point the surtkey filter at a different domain, and the same two queries answer a different question about the internet.

So why dbt Charts? Charts provide a declarative syntax for dashboards. Sure those are fun words (SQL is declarative too after all!), but why does that help? It makes it easy for agents to write and humans to interpret at a glance. You can manage your dashboards in git just like your SQL, so you can update your visuals in the same PR that your data changes to keep things nicely in sync. Plus even though they are declarative, they still provide powerful interactivity.

How did we build this so quickly? In a word: Claude. In a few more words: the MotherDuck platform is so flexible, you can build just about anything. MotherDuck Dives are full React apps. We automatically host them for you in a sandbox with direct access to your MotherDuck data. That means they support a superset of what dbt Charts can provide and give you a blank canvas to build on. Customers create full apps in Dives, but they still work great for dashboards too.

As an example of a dbt board running on MotherDuck, have a look at this Dive:

I apologize for the pie chart - avert your eyes! This was one of their examples…

Deploying in your projects

Want to give it a dbt build? First, connect your dbt project up to MotherDuck. Then add a link to this git repo in your packages.yml file and set save_charts_as_dives: true in dbt_project.yml.

That’s it!

The next time you build your project, your Charts will be published as MotherDuck Dives.

How does the dbt package work?

All of the complexity is nicely abstracted away thanks to dbt’s extensible package approach, but here is a bit of detail about what is going on under the hood.

Three core components across SQL, Python, and TypeScript

There are 3 main pieces to the puzzle: SQL to kick things off in an on-run-end hook, Python that runs in MotherDuck’s serverless Flights runtime, and the TypeScript that powers the Dive visualization. There are a few other files in the package, but all are helpers for these 3 core tasks. For example, the Vega Lite plotting library needs to be bundled up before it is loaded into the TypeScript powering the Dive. All of that is nicely abstracted away by the dbt package.

This architecture mirrors MotherDuck’s architecture as well and is part of why this was straightforward to implement. Core data processing in MotherDuck tends to happen in SQL on our compute ducklings. Orchestration and integrating data from a variety of sources (including running dbt!) happens in Python with Flights. Visualization is TypeScript territory, and Dives use React to make that incredibly simple yet flexible. In an AI era where converting between languages is easier than ever, you can pick the right tool for the job all within the MotherDuck platform.

Let’s dig (Dive?) a little deeper into the flow of what happens when you kick off a dbt build.

What happens at runtime?

First, your dbt DAG runs to update your data. When that run is about to complete, dbt kicks off an on-run-end hook to start the rest of this process.

That hook runs a SQL statement that ships in the dbt package. It loads your chart configurations into a MotherDuck table, along with the logic that is needed to transform those charts into Dives. That allows all of the logic to be encapsulated within the dbt package to make it easy to set up. Once the configurations are in MotherDuck, the on-run-end hook uses several other SQL statements to create and run a MotherDuck Flight. A Flight is a Python script that is run in a serverless fashion.

That Flight uses the Python logic within dbt-charts itself to convert your YAML configs into Vega Lite configs. The Flight then creates one Dive for each dashboard and publishes them to MotherDuck using our Dive management SQL functions.

Once the Flight is finished, the dbt job detects its completion and ends.

Then, any authorized user can visit each Dive in their browser and see your dashboards, freshly populated with the latest data.

The beauty of this approach is that it reuses the good work done by the fine folks behind dbt Charts - especially the compilation from YAML to a full dashboard.

dbt Charts MotherDuck Dives

So if you want a dbt-governed approach to charts, we’ve got you covered! Your dashboard configuration can be as simple as a YAML file and can be version controlled in git alongside your dbt models. The same dbt build command that runs your pipeline will update your MotherDuck Dive visualizations as well. You can run those dbt models anywhere, including right on MotherDuck with a Python Flight. Thanks to Dives, you get the added benefit of live data refreshes.

Quack on! Jaffles look good on a dashboard too…

TypeSafe describes the state of AI today as "databases before SQL”, where we first need to understand the database before we can build the universal language on top. Their pitch for Jev is a frontier-intelligence function call: unstructured state goes in, typed probabilistic decisions come out. We read that and thought, well, this fits right into the universal language of SQL. It can be framed as a function that takes a text column and hands back a label, a score, or a yes/no with a confidence score - exactly how scalar functions today. There is no need to parse the response, and so it's immediately ready to filter, join, and aggregate in the same statement.

This combination of speed, accuracy, and cost together is what makes new workloads viable. LLMs can already classify text, but running it across a million rows is slow enough and expensive enough it is hard to see the value. The alternative, training an encoder like BERT, means collecting labeled examples and maintaining your own model. This is a big bet for something that may not work until it’s close to prod ready. prompt_jev() is configured with a few sentences, runs at analytics speed, and is extremely cost effective for the rows it labels. Below, it pulls the main complaint out of a table of support transcripts:

SELECT
    conversation_id,
    prompt_jev(
        transcript,
        'Identify the customer''s main complaint',
        choice := [
{label: 'billing', description: 'Payments, invoices, and refunds'},
{label: 'technical', description: 'Errors, outages, and integrations'},
{label: 'sales', description: 'Pricing and upgrades'},
{label: 'account', description: 'Cancellations and account administration'}
	  ]
    ) AS classification
FROM customer_conversations;

We think this is a very big deal for analytics. A database is, after all, an excellent place to store your million customer conversations. But it’s still really hard to figure out what your customers are complaining about. Before you can track which problems are getting worse, someone or something has to read those conversations and label them. Encoder models like BERT can efficiently classify over a large dataset, but first you have to collect labeled examples, fine-tune the model, and improve the model as new data comes in. This takes patience and expertise. On the other hand, LLMs enable you to just prompt for what you want instead, but running one across a meaningfully large dataset is slow, expensive, and error-prone.

prompt_jev() gives you LLM-style ergonomics with encoder-style efficiency, all at analytics scale. Here are some comparisons that made us excited:

| model | rows/s | accuracy | Retail cost/100k | wall time at 100k rows | | :---- | ----: | ----: | ----: | ----: | | Jev | 2,484 | 89% | $0.50 | 40s | | gpt-4o-mini | 84 | 80% | $1.93 | 19m 45s | | gpt-5-nano | 94 | 83% | $1.58 | 17m 49s | | gpt-5.6-luna | 61 | 84% | $3.53 | 27m 25s | | gpt-5.6-terra | 52 | 88% | $37.58 | 31m 59s |

We benchmarked on 100,000 articles sampled from the training split of AG News, the four-class news topic dataset introduced by Zhang, Zhao & LeCun (2015), scoring each model against the ground truth.

What really excited us was that it exceed existing models by >25x across cost, accuracy, and speed dimensions. Furthermore, tests at 1m and 10m rows yielded similar performance (and in some cases even faster than our baseline presented above).

prompt_jev() is available on all paid MotherDuck plans. Pick a question you’ve been putting off and try it on your own data!

Dive into Jev

Use our guide to learn how to use MotherDuck and Jev together for classifying text. Or play around with the Dive below to see what is possible.

Further Reading

• MotherDuck docs on prompt_jev
• Guide: Classifying text with Jev on MotherDuck

-- 1) Load AG News (train split) straight from Hugging Face and draw the 100k sample
CREATE TABLE ag_train AS
SELECT row_number() OVER () AS id, text, label
FROM 'hf://datasets/fancyzhx/ag_news/data/train-00000-of-00001.parquet';

CREATE TABLE sample_100k AS
SELECT * FROM ag_train USING SAMPLE 100000 ROWS (reservoir, 43);

-- 2) Classify every row with prompt_jev
CREATE TABLE preds AS
SELECT id, label,
       prompt_jev(text,
                  'Classify the topic of this news article.',
                  choice := ['World', 'Sports', 'Business', 'Sci/Tech']) AS result
FROM sample_100k;

-- 3) Score against the dataset labels
CREATE MACRO ag_name(l) AS ['World', 'Sports', 'Business', 'Sci/Tech'][l + 1];

-- overall accuracy (NULLs reported separately, never scored as wrong)
SELECT count(*)                                              AS n,
       count(*) FILTER (WHERE result.choice IS NULL)         AS nulls,
       round(avg((result.choice = ag_name(label))::INT), 4)  AS accuracy,
       round(avg(result.confidence), 3)                      AS mean_confidence
FROM preds;

-- per-class precision, recall, F1
WITH c AS (
  SELECT ag_name(label) AS truth, result.choice AS pred FROM preds WHERE result.choice IS NOT NULL
), k AS (SELECT unnest(['World', 'Sports', 'Business', 'Sci/Tech']) AS class)
SELECT class,
       count(*) FILTER (WHERE truth = class AND pred = class) AS tp,
       count(*) FILTER (WHERE truth <> class AND pred = class) AS fp,
       count(*) FILTER (WHERE truth = class AND pred <> class) AS fn,
       round(tp / (tp + fp), 4) AS "precision",
       round(tp / (tp + fn), 4) AS recall,
       round(2 * tp / (2 * tp + fp + fn), 4) AS f1
FROM k CROSS JOIN c GROUP BY class ORDER BY class;

-- confusion matrix
PIVOT (SELECT ag_name(label) AS truth, result.choice AS pred FROM preds)
ON pred USING count(*) GROUP BY truth ORDER BY truth;

I hope you're doing well. I'm Simon, and I am happy to share another monthly newsletter with highlights and the latest updates about DuckDB, delivered straight to your inbox.

In this September issue, I gathered the usual 10 updates, with the big two being the acquisitions (DuckLabs joins AWS, MotherDuck acquires Tower) and the upcoming 2.0 release, plus gains across the ecosystem, including table functions in pure Java, bulk loads into SQL Server, and Zarr for n-dimensional arrays, and the usual fun side project, such as an infinite canvas for your data.

Acquisitions: DuckLabs to Join AWS; MotherDuck Acquires Tower

TL;DR: The two biggest ones first: AWS acquires DuckLabs, and MotherDuck acquires Tower.

MotherDuck acquired Tower.dev to integrate its runtime infrastructure into MotherDuck Flights, enabling AI agents to perform data engineering tasks and expose data as APIs. DuckLabs, the company behind DuckDB, joins Amazon Web Services (AWS), with its core open-source projects (DuckDB, DuckLake, Quack) remaining under the MIT license and stewarded by the DuckDB Foundation.

The HN consensus is a mix of concern and cautious optimism, given the independent foundation and MIT-licensed code. Read Jordan's comment on how MotherDuck thinks about it.

Try DuckDB v2.0-alpha, and Upcoming Features

TL;DR: DuckDB v2.0-alpha (code-named Cyanoptera) is available, featuring a feature freeze, bug fixing, and preparation for an October release.

Key updates include the quack extension moving to v1.0, promising higher throughput for both server-to-client and client-to-server communication. The CLI and Python library have alpha releases available.

As a recap, the Preview of DuckDB v2.0 introduces a client/server mode via Quack and CONNECT, a new PEG-based SQL parser, a revised C API for extensions, and a new default storage format. Check out Mehdi's take on Why DuckDB 2.0 is faster (or video): async I/O for S3, a rewritten recursive CTE engine, and a more efficient VARIANT data type.

sql.garden: An infinite canvas for your data

TL;DR: sql.garden is an open-source, Wails-based infinite canvas for data exploration that leverages an in-process DuckDB instance and the Model Context Protocol (MCP) for autonomous AI interaction.

Tony built it with a Go backend and Vue 3 frontend, querying Parquet, CSV, and JSON files at native speed alongside Postgres and MySQL connections. It imports data directly via s3:// URIs, offers a WebAssembly build for browser-based sandboxing, and ships an embedded MCP server. Try the demo in the browser.

DuckDB Table Functions in Java

TL;DR: The DuckDB Java client now supports registering table functions in pure Java, allowing developers to query any JVM-accessible data source directly via SQL without native C++ extensions.

Geertjan and Alex show heterogeneous joins between remote systems like MongoDB and local files using the DuckDBFunctions.tableFunction() builder. The implementation follows a vectorized lifecycle: bind defines the output schema, init establishes the connection state, and apply streams data into a DuckDBDataChunkWriter in 2048-row batches. Source-side predicate pushdown works by passing native filters, such as FROM mongo_query('coll', '{ "status": "shipped" }', ...), directly to vendor SDKs. Because functions run in-process, they eliminate intermediate export steps, giving engineers a high-performance path to integrate proprietary data sources or SOAP/REST services.

I put DuckDB behind the file viewer in my IDE: a 1B row, 29 GB Parquet opens in about a second

TL;DR: A new JetBrains IDE plugin leverages DuckDB's engine to provide near-instant visualization of billion-row Parquet, Avro, and ORC files with full schema support.

Less-Entertainer on Reddit built the Parquet, Avro & ORC Viewer because existing viewers render nested types or string columns as raw byte arrays. A 29 GB Parquet file with 1B rows opens in about 1039ms, and paging to the end takes roughly 800ms. Version 0.1.1 added cell editing, but an edit rewrites the whole file, 17 minutes on that 29 GB one. A not-yet-shipped branch has DuckDB rebuild only the affected row group and splice it back in, bringing that down to 1.8 seconds. JetBrains user? Check out the plugin.

Optimizing Bulk Loads into SQL Server with DuckDB

TL;DR: The DuckDB MSSQL extension v0.2.3 optimizes bulk loading into SQL Server through parallel writers and schema-aware target shaping, reducing ingestion times for 38M rows from 933s to 96s.

Vladimir's (our featured member this month) primary bottleneck is the "PLP tax": unsized strings default to NVARCHAR(MAX), forcing data through a slower LOB path. Explicitly casting to MSSQL_NVARCHAR(200) gives a 2.45× speedup. SET mssql_copy_parallel_writers = 8 scales ingest across sessions, and the mssql_copy_flush_rows default of 102,400 lets rows bypass the delta store for direct compressed rowgroup entry. Raising mssql_tds_packet_size to 16,384 reduces client CPU usage by 27%.

The small-file problem gets worse with CDC

TL;DR: Vignesh describes Streambed's shift from Apache Iceberg to DuckLake to solve the small-file problem in Postgres-to-DuckDB CDC pipelines.

Frequent small commits from the Postgres WAL create metadata overhead and high object counts in Iceberg. DuckLake stores catalog metadata in a database like DuckDB or SQLite rather than in object-store files, which slims down the Postgres WAL → Streambed → DuckDB write path. Benchmarks for 1M rows with 100k updates at flush=1,000 showed DuckLake with DuckDB finishing in 6,576 ms (405 objects) versus Iceberg COW's 269,344 ms (806 objects). Less commit work per batch means fresher analytical replicas.

duckdb-zarr: Query Zarr stores with SQL directly from DuckDB

TL;DR: duckdb-zarr is a Rust-based extension enabling DuckDB to query Zarr stores (an open format for chunked, compressed, N-dimensional arrays) directly using SQL, supporting both local files and remote object storage like S3 and GCS.

Alex and Dave implemented read_zarr(), read_zarr_metadata(), and read_zarr_groups(). Users can select specific arrays with array_path='0' or define dimensions with dims=['time','lat','lon'], and S3/GCS credentials come from DuckDB's secrets manager. Multi-dimensional scientific datasets become relational tables without an external query engine, which simplifies spatial and climate data analysis.

DuckDB for Apache Iceberg

TL;DR: Rob explains how DuckDB v1.5.3 enables full read/write lifecycle management for Apache Iceberg tables through REST catalogs and native V3 support without requiring a distributed cluster.

The release unlocks MERGE INTO, ALTER TABLE, and Iceberg V3 features like VARIANT types and TIMESTAMP_NS precision. Integration goes through ATTACH 'uri' AS my_lake (TYPE ICEBERG), supporting REST, Glue, and S3 Tables. Performance relies on vectorized execution and metadata-driven pruning, but merge-on-read introduces "delete-file debt" and small-file I/O bottlenecks that degrade latency. DuckDB starts in milliseconds, yet query speed is dictated by table layout, so compaction and table maintenance still have to run outside DuckDB.

SQLite vs DuckDB on the same $16 box: every cliff moved 100x

TL;DR: Jovan benchmarked DuckDB against SQLite for observability data, finding that DuckDB moves "read cliffs" 100x further while increasing write throughput on identical $16 hardware.

As Traceway's embedded storage engine, DuckDB reached 254,242 metrics/sec (4x) and 75,225 logs/sec (15x). DuckDB's columnar engine served metrics dashboards at 100M rows with a 3.0s median latency, while spans and logs hit their "cliff" at 10M rows. The benchmark ran with a 4 GB DUCKDB_MEMORY_LIMIT, while the compose file ships 2 GB. The takeaway: for single-node self-hosting, DuckDB is the better choice for high-volume logs and metrics.

Big Data London

2026-09-23. h: 09:00. Olympia, London, UK

Duck In, Duck Out: London Edition

2026-09-23. h: 19:00. London, UK

Virtual Workshop: Build a Data Agent in 60 Minutes

2026-10-01. h: 09:00 PT. Online

Ship, Scale, Sail - #SFTechWeek

2026-10-05. h: 18:15. San Francisco Bay, CA

Data Outpost 2026

2026-11-05. h: 08:30. Convene 100 Stockton, San Francisco, CA