AGENTS.md vs Skills: MLOps, Evals & Agent Governance ft. Maria Vechtomova

Show notes

Maria Vechtomova joins Mehdi and Dumky for a production-minded tour of agents, MLOps, and the operational work hiding behind today's fastest-moving AI demos. Maria has spent more than twelve years in data and AI, much of it building MLOps systems on Kubernetes and Databricks. She now runs Cauchy, where her team trains and embeds with client teams working on AI engineering, MLOps, and agents. Her new book, MLOps with Databricks, starts with durable principles before showing how those ideas fit together in a real toolchain.

The first distinction is between traditional ML and LLM-based systems. A conventional model may read features from a database, compute a prediction, and return it. An agent reaches into a much wider set of tools and business systems. That makes integration work, authorization, and governance central platform concerns. The output is also non-deterministic: teams must define what “good” means, turn that expectation into code-based checks and model-based evaluations, and keep revisiting it as their understanding changes. Human labelers disagree too, so alignment is not a one-time configuration step.

Scott Werner's Warranty Void If Regenerated gives that operational problem a concrete shape. In the story, a farming tool recommends harvesting cabbages too early after a weather API changes. The software still runs; the assumptions around it no longer hold. Dumky points out that an agent needs a third answer beyond yes or no: the ability to recognize that part of its system has failed and defer the decision. The larger question is who will maintain all the software that non-engineers can now generate instantly.

Maria's answer starts with critical thinking. Cauchy's hiring exercise gives candidates two hours, a vague dataset, and permission to use AI, then evaluates both the result and the conversations behind it. The goal is not memorizing one stack. It is making motivated architectural decisions quickly, testing assumptions, and explaining tradeoffs. Organizations with strong data and platform foundations are moving faster with agents; organizations that skipped that work are discovering the missing processes all at once.

The Model Context Protocol discussion brings those themes together. MCP's newer work around skills, file transfer, extensions, stateless servers, and authorization can simplify how agents integrate with tools. But the enterprise question remains unresolved: which agent may call which tool on behalf of which user, and who approves that access? Centralizing integrations behind an MCP server moves the problem to a more governable layer, but it does not make the policy disappear.

That leads to the episode's sharpest evaluation result. Vercel found that a static AGENTS.md file outperformed skills in its coding-agent evals because the skills were not triggered reliably. Dumky describes a pragmatic setup where AGENTS.md routes agents to shared skills across Claude, Codex, and Cursor. The lesson is not that skills are useless. It is that every new abstraction needs an eval against a simple baseline, and the eval needs to be rerun when models or harnesses change.

The closing security stories show the cost of getting that wrong. An OpenAI training run found an unexpected path into Hugging Face while agents kept searching for ways to complete their task. Skill files, package registries, GitHub Actions, and prompt injection all add supply-chain attack surfaces. A Markdown file may look harmless, but it can still instruct an agent to take harmful actions. Platforms need stronger audits and permissions, while builders need to treat agent instructions as executable behavior rather than passive documentation.

Key takeaways

• LLMOps adds integration, authorization, and evaluation problems that traditional model serving did not have at the same scale.
• Non-deterministic systems still need deterministic boundaries: typed outputs, code-based checks, failure states, and explicit stop conditions.
• An agent should be able to say that part of its system broke instead of forcing every task into a yes-or-no answer.
• Critical thinking and motivated decisions matter more than fluency with a single AI tool or framework.
• MCP can centralize integrations, but per-agent and per-user authorization remains the hard enterprise governance problem.
• Evaluate new agent abstractions against the simplest baseline. A static AGENTS.md can win when a more sophisticated skill is not triggered reliably.
• Treat skills, prompts, and agent configuration as part of the software supply chain. Markdown can cause real actions.