Text-to-SQL, Data Modeling for LLMs, MCP, and Dives with Jacob Matson

Text-to-SQL accuracy is a data modeling problem, not an LLM problem

Jacob walks through his research on the Bird Bench text-to-SQL benchmark. The big finding: give an LLM a clean, well-named data model and accuracy goes way up. On simple schemas, models like Claude can hit 90%+ accuracy on text-to-SQL benchmarks. A lot of the remaining errors in popular benchmarks actually come from annotation mistakes in the benchmarks themselves. Jacob found roughly a 20-30% error rate in the Bird Bench mini dev set.

The practical takeaway: if your column and table names make sense to a human, they'll make sense to an LLM. If they confuse a person, they'll confuse AI too. Jacob suggests thinking about an "AI-ready" data layer where you flatten joins and use descriptive names, even if it offends your inner star-schema purist.

Context matters more than you think

When adding context to help LLMs write SQL, Jacob says to focus on things the model can't figure out on its own. Definitions of fiscal years, business-specific metric logic, domain terminology - that's the high-value stuff. Statistical metadata like min/max values and enum lists help too. But restating what the LLM can already infer from column names just wastes context window space. He compares it to a Sudoku puzzle: each confirmed golden question-answer pair constrains the space of possible wrong answers.

The MotherDuck MCP server changes how you work with data

Jacob and the host both use the MotherDuck MCP server daily. The workflow is straightforward: ask a question in natural language, get SQL back, review it, run it. Internally at MotherDuck, a pattern has emerged where people use traditional BI dashboards as a source of truth for known metrics, then use the MCP to slice and explore beyond what those dashboards cover. Non-technical team members can self-serve their own data questions for the first time.

Dives turn SQL results into shareable interactive apps

MotherDuck Dives are interactive data visualizations built from SQL queries. Jacob explains that Dives started as a way to share analysis with non-technical stakeholders but have turned into lightweight data apps. You can create Dives through the MotherDuck MCP server, which means you can go from a natural language question to a shareable interactive visualization without opening a separate BI tool. The host runs the Super Data Brothers business backend on them.

What's ahead for MotherDuck

Jacob is honest about the uncertainty: nobody knows whether AI progress keeps compounding or levels off. Near-term, MotherDuck's roadmap includes enterprise features like finer-grained role-based access control, row-level security, and higher-scale partition handling. On the AI side, the team is exploring whether Claude and similar tools become the primary way people interact with data apps, which is part of why Dives were built to work through the MCP in the first place.