- 17 min read
BY- 17 min read
BYIt's hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still need to coordinate and understand what the agents produced, potentially test it, which makes it even harder to keep up.
Luckily, Maxime Beauchemin, the creator of Airflow and Superset and the person who defined what "data engineer" meant for a decade (more on him below), joins us to show how he uses agents and what he's built for working with them. I tried to extract the patterns behind how he actually uses AI in his data work today. This is the fourth interview in 'How to use AI with DE'.
In this article, we go into four parts: (1) How to balance quality with messy data warehouse work, and how to manage agents with Figma for agents. (2) We elaborate on the future of the context layer and the return to semantics, (3) how Okta for Agents is needed for security, and (4) how the future of agentic workloads can be done in teams, whose yap-to-ship ratio is best, and why Amdahl's law still counts.
Our guest in this interview is Max Beauchemin, the creator of Airflow and Superset. He's known as one of the OGs of defining how data engineering worked back in 2017, and founded Preset, the company behind Superset, and currently serves as its CEO.
He is heavily involved in the AI workflow, which is another reason I wanted to interview him for this series, but he has also been building in the space himself: Agor (Ag: AI agent + Or: orchestration), earlier tooling like claudette-cli2, and db-agents, an experiment to embed agent context directly inside databases. We'll get into it all.
Max and I talked about many things, among them how to use AI in data engineering, how security plays a role, how shared, secure, context-rich agent workspaces work within teams, and how he uses AI assistants to run his business and ease his life as a CEO.
Max is a true open-source enthusiast, and he wants open source to win. Everything we discuss here is somewhere on GitHub, which I have happily linked throughout the interview.
Before we start using Figma for Agents, coordinating them on canvas, we need to ask why we need coordination and orchestration in the first place.
That's where we started, with the challenge of messy data warehouse environments that most people find themselves in. I asked how he balances quality and quantity, aiming for high quality.
Max says that the new models, Opus 4.5 or 4.61, are not making many errors anymore and are very clever when they get the right context as above with all the database schemas of tables and data types, and even querying it with MCP. He says they almost run in self-serve mode, but he still prefers that users know what they are doing and can either read the generated code or verify the generated numbers on a dashboard or chat results.
But the setup is critical. With these three prerequisites, the agents handle almost all queries really well:
agents.mdThe only problem, and always has been, is the messy structure and sources that most organizations have, growing from an initial small project into a certain stage. There are always obscure tables or strings, timestamps not aligned, or hidden information that is not encoded in code or written down. Or there's the hidden knowledge, like that a certain table shouldn't be used anymore or has bad data, which is known to the people using it but might not be to agents.
When he recently saw the power of agentic coding, Max went all in and has been building the Figma for agents ever since. Something he can use to collaborate with agents within his company, instead of everyone running the same prompts locally and needing to sync with each other manually. That's when Agor was born.
Agor stands for Ag: agent and Or: for orchestration. As the creator of Airflow and CEO of a data company, he knows exactly how a tool needs to improve his workflow. He also called it:
The goal is to automate most of the automatable CEO-stuff
The board: branches as cards, zones as regions, agent sessions, and teammates present live. See full demo Agor [Agent Orchestration] Demo.
Agor was Max's answer to "how he uses AI beyond a research tool", but doing data modeling, writing data pipelines, even legal or HR roles he added later to Agor, so you can give company-wide roles to agents that can be fed with dedicated documents and context, and triggered by any employee internally. In contrast, others see the jobs and avoid asking the same questions, reusing the output for new queries—Andrej Karpathy's concept of an LLM-maintained shared team wiki—which builds an internal knowledge base.
Initially, when we first chatted, Agor had already changed how he worked as a CEO, but since then, Agor has gone even further. Agor can replace high-level tasks while still being very hands-on by working closely with the code via git worktrees (branch cards in Agor's UI) and verifying the code in the PRs it produces. Max also added OpenClaw-like features around memory and identity via dedicated Markdown files, such as AGENTS.md, SOUL.md, MEMORY.md, so that Agor's agents can learn from recent runs and carry a purpose and clear instructions. This led to role-based agents called Assistants (use agor-assistant as a template to build your own).
Example of different Agor assistants: Saul for legal, or OpEx for observability and so forth. | From the Webinar Anatomy of Our Internal Data Agent
Inspired by OpenClaw's agent loop, Assistants became first-class citizens, persistent AI companions with memory, identity, skills, and scheduled tasks integrated into Agor's canvas, multiplayer workflows, and reachable directly from Slack, for example. Additionally, Agor adds features beyond OpenClaw, such as better multi-user support, RBAC, one-click, full session inspection, and many more.
Asked about the goal of Agor, Max said:
The initial premise was to remove DevOps and set up time for other members of the company. Instead of people needing to connect all the MCPs or CLIs to add API keys, set permissions, or integrate with Slack, the prompt window with the needed context is there and ready to start.
OpenClaw (formerly ClawdBot) is an open-source agent framework built around a persistent agent loop: a serialized cycle that turns a message into actions, using file-based identity (SOUL.md) and layered memory (MEMORY.md). Agor's Assistants adopt this pattern and extend it with multiplayer boards, RBAC, and canvas-level orchestration.
When you do a lot of agent work, it's really hard to keep up with all of it. That's where Agor's visual and spatial overview really helps and is unique in its approach.
It brings the local and private session to a server, where everybody can see and work together on the same queries, and use the insights from other results, as dashboards are built for. So instead of keeping output locally, others can source the artifacts generated by agents, stored as Artifacts within Agor, ready to use by anyone, with no integration or deployment needed.
Example Artifacts such as AI Ops Command Center, Tool Log triage, these live directly in Agor based on an Agor session
Or how Agor tracks its own spending across sessions:
See demo at Live Talk: Anatomy of Our Internal Data Agent at Preset (ft. Agor)
Such as an assistant needing access to all pipelines' metadata as illustrated data stack.
Or the data needs an analytics agent to support, such as self-serve, the data team, and extras such as memory, skills, documentation, etc.
Max also believes that we are going back to the Semantic layer, or using it for AI as agents benefit from structured information - helping with the data model and SQL part, to make sure it's correct, especially with the needs of AI agents and the persistent challenge of providing trustworthy self-service analytics.
With the shift of semantics outside of the BI tool, versioned, testable, portable, it's a chance for better integration between business domain experts and data engineers. His thinking has evolved since he wrote the article, and Max told me:
I see two different semantics: the semantic layer and the YAML. There are the hard constraints — not every area needs that strictness — and then the softer ones with Markdown and Agentic Skills, good for 80-90% but with no guarantees.
Based on that idea, Max created an experiment to bring the AGENTS.md convention into the database. DB-AGENTS reserves a dedicated schema and table, _agents._agents, that holds agent-oriented documentation at different scopes (global, domain, schema, table, and even column). You write the docs locally as markdown files with YAML frontmatter, and a small CLI (dba) deterministically syncs them into that table — since databases don't let you drop files into them, the table becomes the file. Agents then query it at session start the same way they'd read an AGENTS.md, making it a natural companion to INFORMATION_SCHEMA: one holds structure, the other holds meaning. Max calls it a "soft semantic layer", which maps directly onto the hard-vs-soft split he described above. Check out the repo at db-agents.
With context being key for agents to understand what we humans know, Agor also added a context layer called knowledge. Agor Knowledge acts as a central place where humans and agents can store, organize, connect, and find the context that makes work compound over time, with Slack quickly becoming the main interface to many of the team's agents.
Another big topic is security when agents have so much access to powerful CLIs, sometimes root access to systems or databases containing private keys, or just downloading random skills from the internet that may contain hidden secret messages.
Max coined the idea of Okta for Agents, which I found super interesting, and something I believe will become ever more important if we want to find a healthy way of working with agents in enterprises or with sensitive data. Okta for Agents means working around identity, scoped delegated permissions, leases, and audit logs.
When asked how he's managing security, verifying what Agor or the agents are doing, Max responded:
I let the workers run in
god mode3, but using dedicated environments/sandboxes, hooked to a dedicated git worktree repo, it can run autonomously and solve problems on an initial prompt, visualized in a shared canvas style.
I asked how he sees Okta for Agents being implemented. We desperately need it, he said, granting agents permissions like impersonation. Delegating the permission is an OAuth. With the roles, we can scope permissions strongly. E.g., the sales agent only has access to sales documents.
When asked at what level to integrate the Okta security layer, Max said it hasn't been solved yet. Still, he sees it as the same question: whether we have 50 agents or 50 users who use a platform, both need a security layer.
Likewise, Max shared:
I trust agents the same way as I would an employee.
Okta is the identity layer companies put in front of their tools: it authenticates who you are, decides which systems you can open, and logs what you did. Max's point is that agents need the same layer. E.g. Clawdbot, when he tried it, was effectively a DIY IAM manager for agents, config hell and all.
The twist is that it adds a whole new dimension to RBAC. It's not just "the bot gets an email account" — it's "the bot gets an email account, but can only read mine, and via MCP rather than as a real user." Not so different from onboarding a human personal assistant, except this assistant can help with nearly everything, so the blast radius is much bigger.
Max's own example: he saw a 1Password skill and immediately backed off then reconsidered, wondering whether the bot should have its own 1Password account with only safe credentials shared into it. Which is exactly the problem: you can be strict on paper, but the moment the agent has your email, Slack, and calendar, it can leak private things all day. (Full discussion at this post)
How I Dropped Our Production Database and Now Pay 10% More for AWS, or another one, or when 13-hour AWS outage reportedly caused by Amazon's own AI tools. Or also just hacks by getting attacked via GitHub PRs or How secret instructions injected into skills.
Related to security is the deterministic, repeatable behavior of data sets with the same input. Agents are the opposite: probabilistic. I was curious to hear from Max, who initially defined the functional data engineering paradigm for deterministic and idempotent batch data processing, what he thinks about the non-deterministic outcomes of agents, specifically with large language models.
Max said, regarding declarative definitions, that he finds a claude.md is usually sufficient for most tasks that have a git repo, more context, and an issue or PR to work with, given the initial prompts come from users who know what they are doing.
Regarding reliability, Max thinks about using good methodology references. Agents get it and understand it. E.g., data modeling practices such as Kimball are still valid, or the approach shared by him with Entity-Centric Data Modeling (ECM), he says, and when prompted to model in those patterns, agents follow them well (either via research or provided).
The other part is that some non-dangerous work can have vibe data pipelines, and there's no danger. And there are cognitive-depth tasks, such as a complex Spark cluster, where you can't just debug quickly with large data sets.
Also, the field varies: not every area is getting agentic-piled as fast. E.g., platform demands go through the roof (see GitHub outages), so we have 10-20x the platform needs, but at the same time, the work is critical to be correct. So it depends.
When asked about how Agor has changed how they at Preset develop products (if at all?), or made them more effective, Max said:
There are more agents than humans nowadays. Everyone has a Claude Max plan, and agents handle almost all code writing.
And on a personal level:
I haven't written a function by hand for a long time, and I might not anymore — except when I feel nostalgic.
He also thinks that the Yap-to-Ship Ratio, a metric that he announced half-jokingly on LinkedIn, describing people's velocity by just getting stuff done without involving others at every step, will be very low-yap for 10x engineers, as they solve the problem and ship a solution without much back and forth.
They deploy it somewhere for others to use, not only for human consumption, but as a solution or CLI that other agents can use to discover further and solve their problems. A high Yap-to-Ship ratio would mean lots of human interaction, which is the clear new bottleneck.
As human code review gets bottlenecked, I asked how he does the review. He said he uses Codex with sub-agents to review, ensuring everything is DRY (Don't Repeat Yourself) and that all expected callbacks are made.
You can also ask the operator assistant agents if you are not sure whether an implementation is correct.
One bottleneck is still Amdahl's Law. We can speed up tooling by using extremely fast agents, but unless the end-to-end workload is sped up, we only increase by a 2-3x factor, not 10 or 100 as any one tool does. This also overlaps with Max's Yap-to-Ship ratio: if PRs need the human in the loop to review many of them, the overall speed at which we build is not faster.
Another side effect is that it takes a lot of context switching. Max said he has ten active sessions in Agor. He is good at context switching (maybe also learned through recent Agor workflow? 🙂).
It's hard to predict the future with AI, but Max took a stab and shared his predictions for 2026 and categorized them into wired and tired:
Find the full talk at Webinar.
We end this interview with Max's setup for working with agents, since we didn't have time to go into full details on the call. I'm sharing the one he shared four months ago. I'm sure it changes almost daily. Still, it helps us get a good overview of his software engineering stack for the team at Preset, as well as his personal local computer stack.
For software engineering and data engineering:
agor-openclaw framework, agent is pushing projects across a kanban-type layout: tons of new automation there. Agent checks on agents, prompts them, moves worktrees to "needs human review" zone if/when neededHEARTBEAT.md with pseudocode to make a decision for coding project (worktree) on that boardAnd his new "Personal assistant" local instance of Agor (brand new/sensitive):
We've learned how to use Figma for agents with Agor and to collaboratively work as a team, using shared prompts and creating artifacts. We've seen how Okta for Agents is needed but really hard to implement, and how the future of AI is mostly about context and how to integrate it well. Plus, we learned how Max automates many tasks as a CEO with dedicated AI assistants and still produces low-level code with the same assistants for both his personal and company-wide needs.
I hope you enjoyed this fourth interview with Max. Huge thanks to Max for taking the time to speak with me (twice!) and for sharing his experience with all of us. Follow him on LinkedIn, GitHub, or on Preset Blog, where he shares his distilled thoughts on the ecosystem, and obviously, if you want to know more about Agor, check it out at Agor GitHub repo.
Max shares a lot of his ideas and thoughts online. Here are some further articles and interviews to read/watch:
More interviews are coming out, so please share feedback, questions you might want to ask, or your experience working with AI in the data space. We're all in this together, figuring it all out.
2026/07/20 - Ryan Boyd
Data Outpost is a new conference for data and AI builders, coming to San Francisco November 4-5, 2026. The next evolution of Small Data SF: two days of workshops, talks, and real time with the people building the data layer AI depends on.
2026/07/29 - Hamilton Ulmer, Till Döhmen, Jacob Matson, Garrett O'Brien
Introducing Guides, MotherDuck's context layer for AI agents. More accurate queries, lower token spend, and distributed context for your entire organization.
