# Agent observability overview
> Why agent traces belong in a data warehouse, how OpenTelemetry spans reach MotherDuck, and what to watch for when you model and analyze them.
An AI agent run is a tree of operations: the agent loop, each model call, and each tool call. OpenTelemetry records that tree as spans, and its [generative AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) give agent frameworks shared span names and attributes for it. A tracing tool shows you one run at a time. The questions that decide what to fix next cover thousands of runs: which tool started failing after a vendor change, whether a cheaper model raised the retry rate, what a resolved case costs once human handoffs are counted. Those are SQL questions over millions of spans joined with business data, and MotherDuck, a serverless cloud data warehouse built on DuckDB, is built for them.

MotherDuck also brings AI-native tooling to that data. You build the dashboard as a [Dive](/key-tasks/dives/), load and alert with [Flights](/concepts/flights/), classify error messages and outcomes in SQL with [`prompt_jev`](/sql-reference/motherduck-sql-reference/ai-functions/prompt-jev/), and let agents query their own traces through the [MotherDuck MCP server](/key-tasks/ai-and-motherduck/mcp-workflows/).

## Explore agent traces in a Dive

The following Dive reads 6.4 million synthetic agent runs, 31.8 million spans, from Brightholm Mutual, a fictional insurer whose agents take claims, triage and settle them, quote policies, and service existing policies. The data has incidents planted in it: a vehicle valuation tool that starts timing out on February 10, a payment tool that quietly queues payments for manual handling, and a cheaper model that sends invalid tool arguments. Start on the **Tools** tab, pick `vehicle_valuation`, and open a failing call to see its trace as a waterfall.

Embedded Dive: **Agent observability**.
Open it in the Dive Gallery: https://motherduck.com/dive-gallery/dives/agent-observability/

You can also open it in the [Dive Gallery](https://motherduck.com/dive-gallery/dives/agent-observability). The data is an unrestricted share, so you can attach it and query it from your own account:

```sql
ATTACH 'md:_share/dg_insurance_brightholm_agent_ops_m_1e02507e_r1/838d91b9-658d-4f11-ae51-d5beddb91950' AS agent_ops;
```

## What an agent trace looks like

Every span carries a trace ID, its own span ID, its parent's span ID, start and end times, and a status of `OK`, `ERROR`, or `UNSET`. The root span covers the whole agent run, and model calls and tool calls hang off it. A run that retries a timed-out tool looks like this:

```mermaid
flowchart TD
    Root["invoke_agent claims_triage<br/>12.4 s"]:::green
    Chat1["chat tier-large<br/>1,830 input tokens"]
    Lookup["execute_tool policy_lookup<br/>OK"]
    Timeout["execute_tool vehicle_valuation<br/>ERROR: timeout after 10 s"]:::watermelon
    Retry["execute_tool vehicle_valuation<br/>OK on attempt 2"]
    Chat2["chat tier-large<br/>final answer"]
    Root --> Chat1
    Root --> Lookup
    Root --> Timeout
    Root --> Retry
    Root --> Chat2
```

The attributes that matter most for analysis are few:

| Attribute | Found on | Holds |
|---|---|---|
| `gen_ai.operation.name` | Every span | `invoke_agent`, `chat`, or `execute_tool` |
| `gen_ai.agent.name` | Agent spans | Which agent ran |
| `gen_ai.request.model` | Model calls | The model the agent asked for |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Model calls | Token counts, the basis of cost |
| `gen_ai.tool.name`, `gen_ai.tool.call.id` | Tool calls | Which tool ran, and the call the model requested |
| `error.type` | Failed spans | The error class, such as `timeout` |

:::note
The generative AI conventions have development status in OpenTelemetry, so attribute names still change between versions, and some instrumentation libraries use names of their own. Keep every raw attribute when you load spans, so you can derive columns again after a rename.
:::

Agent traces get read two ways. Debugging reads one trace by its ID. Analysis aggregates across all of them: error rate and p95 latency for one tool by week, or the cost of each resolved case. A columnar engine reads only the columns a query touches, so both patterns stay fast at this scale. On the share above, this query finds the week the valuation tool started failing:

```sql
SELECT
    date_trunc('week', start_time) AS week,
    count(*) AS calls,
    round(100.0 * avg((status_code = 'ERROR')::INTEGER), 1) AS error_pct,
    round(quantile_cont(duration_ms, 0.95)) AS p95_ms
FROM agent_ops.spans
WHERE gen_ai_operation_name = 'execute_tool'
    AND gen_ai_tool_name = 'vehicle_valuation'
    AND start_time >= TIMESTAMP '2026-01-19'
    AND start_time < TIMESTAMP '2026-03-02'
GROUP BY ALL
ORDER BY week;
```

The error rate goes from about 1.5% to 10% and above in the week of February 9, and p95 latency climbs to the tool's 10-second timeout.

## How a typical pipeline works

Most teams already run an OpenTelemetry Collector between their applications and their tracing backend. Adding MotherDuck means adding one more destination to it, and a scheduled load on the MotherDuck side. The Collector batches spans, can tail-sample and redact them, and writes each batch to object storage as an OTLP JSON file. A Flight loads the new files every few minutes, flattens them into one row per span, and rebuilds a table with one row per agent run that most charts read. Files make a good buffer: agents never wait on the warehouse, and you can replay a day by loading its files again.

```mermaid
flowchart LR
    App["Agent application<br/>OpenTelemetry SDK"]:::green
    Collector["OpenTelemetry Collector<br/>batch, sample, redact"]
    Files[("Object storage<br/>OTLP JSON files")]:::yellow
    Load["Flight every few minutes<br/>INSERT ... SELECT"]:::watermelon
    Tables[("MotherDuck<br/>spans, agent_runs")]:::yellow
    Use["Dives, MCP server,<br/>prompt_jev"]
    App -- OTLP --> Collector
    Collector --> Files
    Files --> Load
    Load --> Tables
    Tables --> Use
```

[Load OpenTelemetry agent traces into MotherDuck](/key-tasks/ai-and-motherduck/load-opentelemetry-agent-traces/) sets this pipeline up step by step: the Collector configuration, the tables, and the Flight that loads new files.

## What to watch for

### Batch every write

An agent fleet can emit thousands of spans a second. Sending each one as its own `INSERT` pays a network round trip and a transaction per span, and it's the most common way a trace pipeline falls behind. Batch at every stage instead: the SDK's batch span processor, the Collector's `batch` processor, and the load itself, which writes each batch in one statement. Bigger batches load more efficiently, so scheduled loads and backfills should write at least 100,000 rows at a time and finish each transaction in under a minute. [Loading data best practices](/key-tasks/loading-data-into-motherduck/considerations-for-loading-data/) explains the numbers, and [data loading patterns](/key-tasks/loading-data-into-motherduck/loading-patterns/) covers incremental loads and deduplication.

### Promote the attributes you filter on

A `MAP` column keeps every attribute, but a filter on a map entry reads the whole map for every row. Give the attributes your charts filter and group by a column each (operation, agent, model, tool, tokens, and error type) and leave the long tail in the map. When an attribute name changes, add a new column from the map instead of reloading the spans.

### Expect late and repeated spans

Spans arrive out of order. A child span can land after its parent, and a retry from the Collector can deliver the same batch twice. Deduplicate on trace ID and span ID when you load, and rebuild recent runs from their spans instead of appending each run once, as the [loading guide](/key-tasks/ai-and-motherduck/load-opentelemetry-agent-traces/) does.

### Status OK isn't success

A tool call can end with status `OK` and still fail the user: a payment queued for manual review, a declined identity check, a search that returned nothing. The span status reports whether the call raised an error, not whether it did its job. Record the tool's result as an attribute of its own and chart it next to the error rate. In the share above, silent failures outnumber thrown errors about ten to one for the payment and identity tools:

```sql
SELECT
    gen_ai_tool_name AS tool,
    tool_result_status,
    count(*) AS calls
FROM agent_ops.spans
WHERE gen_ai_operation_name = 'execute_tool'
    AND status_code <> 'ERROR'
    AND tool_result_status NOT IN ('success', 'none')
GROUP BY ALL
ORDER BY calls DESC;
```

### Join traces to business outcomes

Token counts become cost when you join them to a price table for each model, with the dates each price applied. Cost per agent run isn't the number that matters, though. Cost per resolved case is, and it depends on tables your tracing tool never sees, such as the case and the human who took over. In the share above, agent work costs cents per case, and a case that needs a human costs about $190 of human time:

```sql
WITH per_case AS (
    SELECT
        case_id,
        sum(llm_cost_micros + tool_cost_micros) / 1e6 AS agent_usd,
        sum(human_cost_cents) / 100 AS human_usd
    FROM agent_ops.agent_traces
    GROUP BY case_id
)
SELECT
    c.final_status,
    count(*) AS cases,
    round(avg(p.agent_usd), 4) AS agent_usd_per_case,
    round(avg(p.human_usd), 2) AS human_usd_per_case
FROM agent_ops.cases AS c
JOIN per_case AS p USING (case_id)
WHERE c.final_status IN ('resolved_stp', 'resolved_with_human')
GROUP BY ALL
ORDER BY c.final_status;
```

### Treat message content as sensitive

The generative AI conventions make prompt and completion content opt-in, because it often holds customer data. If you capture it, keep it in a separate table or database from the span metadata, and grant access to it with [roles](/concepts/roles-and-access-control/). Most dashboards need only the metadata.

### Classify text with prompt_jev

Error messages, user feedback, and final answers are free text, so you can't group by them. [`prompt_jev`](/sql-reference/motherduck-sql-reference/ai-functions/prompt-jev/) turns each one into a typed label with a confidence, in SQL. For example, it can sort failed tool calls into timeouts, invalid arguments, authentication errors, rate limits, and upstream errors, so the failure chart groups by cause instead of by message. The [loading guide](/key-tasks/ai-and-motherduck/load-opentelemetry-agent-traces/#label-failed-tool-calls-with-prompt_jev) labels each new failure once and stores the result.

`prompt_jev` is billed in AI Units by input tokens: a `choice` question over 50-character messages runs on roughly 260,000 rows per AI Unit. `prompt_jev` is in preview, and its requests are processed by TypeSafe in the United States, so check what your traces contain before you send their text to it.

## Where MotherDuck fits

As an analytical database, MotherDuck covers both sides of debugging agents: finding the one trace that went wrong, and seeing the pattern behind it before your users report it. How fresh the data is depends on how often you load it. A scheduled load every few minutes suits most dashboards. To query runs within seconds of them ending, write each batch as the Collector flushes it, with an exporter process instead of a scheduled load.

With every trace in SQL, you can debug proactively instead of one request at a time:

- Alert from a [Flight](/concepts/flights/) when a tool's error rate or p95 latency crosses a threshold.
- Compare runs before and after a model or prompt change.
- Join failures to the cases and customers they affected.
- Label failures and outcomes with `prompt_jev`.
- Let agents query their own history through the MCP server.

## Related

- [Load OpenTelemetry agent traces into MotherDuck](/key-tasks/ai-and-motherduck/load-opentelemetry-agent-traces/)
- [Creating visualizations with Dives](/key-tasks/dives/)
- [Loading data best practices](/key-tasks/loading-data-into-motherduck/considerations-for-loading-data/)
- [`prompt_jev` reference](/sql-reference/motherduck-sql-reference/ai-functions/prompt-jev/)
- [Using the MotherDuck MCP server](/key-tasks/ai-and-motherduck/mcp-workflows/)


---

## Docs feedback

MotherDuck accepts optional user-submitted feedback about this page at `GET https://motherduck.com/docs/api/feedback/agent`.
For agents and automated tools, feedback submission should be user-confirmed before sending.

URL-encode query parameter values and send a GET request:

```text
GET https://motherduck.com/docs/api/feedback/agent?page_path=%2Fgetting-started%2Fagent-observability%2F&page_title=Agent%20observability%20overview&text=<url-encoded user feedback, max 2000 characters>
```

Optionally append `&source=<url-encoded interface identifier>` such as `claude.ai` or `chatgpt`.

`page_path` and `text` are required; `page_title` and `source` are optional. Responses: `200 {"feedback_id": "<uuid>"}`, `400` for malformed query parameters, and `429` when rate-limited.