Building Reliable AI Agents with Seth Rosen: Checkpoints, Not Workflows

Show notes

Seth Rosen joins Jacob Matson and Mehdi Ouazza to explore how teams can trust AI agents without prescribing every step they take. Seth is a co-founder of ThruWire, where he works on checkpoint-based agent systems. He starts with a familiar piece of data-team history: the story behind his “can you just quickly pull this data?” meme. Agents can now assemble much of that work themselves, but deciding whether the result is correct remains a human and organizational problem.

The discussion of software factories and GitHub’s internal analytics agent brings that problem into focus. An agent that can answer a question in a private chat is useful, but work that changes permissions or affects other people needs shared visibility and review. Seth describes checkpoints as a way to require evidence at important moments while leaving the agent free to choose its route. A low-risk change and a security-sensitive change may need different checkpoints, even when the agent uses the same tools.

As agents cross the traditional boundaries between ingestion, modeling, querying, and visualization, data teams need to make their conventions and definitions available beyond a single interface. The group discusses who documentation is written for when agents increasingly read it on people’s behalf, and how new tools compete with the patterns already familiar to models.

Becoming AI-native also changes how people work together. Giving everyone access to a coding agent does not automatically align a team. Jacob, Mehdi, and Seth discuss making ideas discoverable, creating space for collaboration, and learning where AI-generated communication adds friction. Their “slop cannon” stories connect experimentation with the need to keep responsibility for what gets sent to colleagues.

The final discussion treats agent traces and evals as a new source of data for analytics teams. Seth distinguishes observing an agent after the fact from asking it to produce structured intermediate artifacts that can be validated during the work. Those receipts can help steer the agent as well as explain what happened. The episode closes with why data modeling experience is useful here: defining goals, dependencies, and evidence without over-constraining how the work gets done.

Key takeaways

• Define what an agent must prove at important checkpoints, then give it freedom between them.
• Shared approval and review paths matter when agent actions affect customers, permissions, or teammates.
• Make data definitions and working conventions usable across the interfaces agents choose.
• AI adoption requires changes to collaboration and organizational design as well as tools.
• Use traces and evals to understand how agent behavior affects the actual task.
• Structured receipts can both explain intermediate work and help steer it toward the intended result.