# Monitoring and observability

> Follow Flight runs, see which step each run reached, read logs, and debug failures.

## Included pages

- [Monitoring and debugging Flights](https://motherduck.com/docs/key-tasks/flights/observability/monitoring-and-debugging): Read run status, step-level progress, and logs, triage common failure patterns, and inspect historical Flight versions.
- [Watch Flight run progress with the Flight Explainer](https://motherduck.com/docs/key-tasks/flights/observability/explain-flight-runs): See a live progress view of each Flight run, step by step, including which step ran, failed, or was skipped, and how long each took.

Source: https://motherduck.com/docs/category/monitoring-and-observability

---

## Monitoring and debugging Flights

Source: https://motherduck.com/docs/key-tasks/flights/observability/monitoring-and-debugging

> Read run status, step-level progress, and logs, triage common failure patterns, and inspect historical Flight versions.

A Flight that runs unattended needs to be observable. This page covers reading runs, seeing which step a run reached, reading logs, common failure patterns, and inspecting historical versions when something went wrong on a past run.

## Listing runs

Every Flight tracks its run history. Get the recent runs newest-first:

### MCP / AI agent

Ask your AI agent:

> "Show me the last 10 runs of the heartbeat Flight."

The agent calls `list_flight_runs` and formats the result.

### CLI

```bash
motherduck flight list-runs my_flight --limit 10
```

### SQL

```sql
SELECT run_number, status, flight_version, created_at
FROM MD_LIST_FLIGHT_RUNS(flight_id := '<flight_id>')
ORDER BY run_number DESC
LIMIT 10;
```

Each run records whether the schedule triggered it (`is_scheduled`).

## Seeing which step a run reached

When a Flight declares a chart and prints `@flight-run` progress lines, every run shows its steps with their status, rows, duration, and errors, so a failed run shows the step that failed without reading the log. Open the **Explainer** tab on the run in the UI. See [Watch Flight run progress with the Flight Explainer](/key-tasks/flights/observability/explain-flight-runs) to instrument a Flight.

## Reading logs

Logs are the combined stdout and stderr captured during the run. Use them to verify what your Flight printed and to read tracebacks from failed runs.

### MCP / AI agent

```text
get_flight_run_logs(flight_id, run_number)
```

The response includes the run record and log content. When the log is large, the response is the tail; pass `max_bytes` to control the size cap.

### CLI

```bash
motherduck flight logs my_flight --run 42
motherduck flight logs my_flight --run 42 | grep -i error
```

### SQL

```sql
SELECT line
FROM MD_GET_FLIGHT_LOGS(flight_id := '<flight_id>', run_number := <n>)
ORDER BY line_number;
```

## Common failure patterns

Most failed runs fall into a handful of categories. Read the log first, then match the symptom:

| Symptom in the log | Likely cause | Fix |
|---|---|---|
| `ModuleNotFoundError: No module named '<x>'` | The package isn't in `requirements.txt`, or the name is misspelled. | Add or correct the entry, then `update_flight` (produces a fresh version). |
| `InvalidInputException: ... CreateShortLivedToken: Unable to resolve user` | The Flight was created with an explicit `access_token_name` that doesn't exist in the environment the Flight is talking to (often a staging vs prod confusion). | Verify the token exists with `SELECT token_name FROM md_access_tokens()` in the MotherDuck UI SQL editor or DuckDB CLI against the same MotherDuck environment. |
| `duckdb.duckdb.IOException: ... Could not find file` or `Catalog Error: ... does not exist` | The Flight uses a database it can't reach with the permissions of the identity it runs as, or the data isn't there. | Check the privileges of the user or service account the Flight runs as and whether the source files or shares are attached. |
| Schedule didn't fire | The Flight has no `schedule_cron`, or the cron expression is in local time when it should be UTC. | Check `schedule_cron` on `MD_LIST_FLIGHTS()` and convert the intended time to UTC. |
| Run uses "old" source after an update | The run started before the update landed and locked to the previous version. | Trigger a fresh run; subsequent runs use the latest version. |
| `pip` install errors during startup | A requirement can't be resolved (typo, yanked version, missing platform wheel). | Check PyPI for the exact package and version, and pin to a version with a Linux wheel. |
| Run cancelled unexpectedly | Someone clicked Cancel in the UI, or the run hit a timeout. | Check the trigger and the duration in the run record. |
| Run fails near the end of a large build with little or no log; the run record shows a non-zero `exit_code` | The runtime hit its memory ceiling and was OOM-killed. | Lower peak memory: push heavy compute into SQL, process in bounded chunks, and lower dbt `--threads`. See [runtime limits](/key-tasks/flights/packages-and-runtime#runtime-limits). |

If the failure doesn't match any of these, capture the full traceback from the log and the run record before re-running — that's the information support needs.

## Inspecting a specific version

If a past run misbehaved and the current Flight source has moved on, you can still see exactly what that run executed.

List versions:

```sql
SELECT version, created_at, source_code
FROM MD_LIST_FLIGHT_VERSIONS(flight_id := '<flight_id>')
ORDER BY version DESC;
```

Or get a single version with full content:

```sql
SELECT *
FROM MD_GET_FLIGHT_VERSION(flight_id := '<flight_id>', version_number := <n>);
```

Through the MCP server, the equivalent calls are `list_flight_versions` and `get_flight` with a `version` argument.

The `flight_version` column on `MD_LIST_FLIGHT_RUNS` tells you which version a given run used. Combine the two to read the exact source and requirements that produced a failed run.

## When the log isn't enough

Some failures don't surface much stdout/stderr — for example, an OOM kill or a process the runtime terminated. Start with the run record from `MD_LIST_FLIGHT_RUNS`: `status`, `flight_version`, the `started_at`/`ended_at` timings, the `config` the run used, and the process `exit_code`:

- `RUN_STATUS_FAILED` with no useful log usually means the process failed before it could print the traceback, or the runtime terminated it.
- `RUN_STATUS_FAILED` with a short log ending during dependency installation usually points to package resolution, platform wheels, or startup limits.
- `RUN_STATUS_CANCELLED` with a manual trigger is someone clicking the button or calling the cancel API.

Match the pattern, add observability to your `main()` (a log line at each milestone, a `print` of memory usage if you suspect OOM), and re-run. [`@flight-run` progress lines](/key-tasks/flights/observability/explain-flight-runs) are a structured way to add those milestones: the last step that reported `running` is where the process stopped.

## Getting help

For issues that aren't in the failure table above, gather:

- The Flight's name or ID.
- The failing run number.
- The full log content (use `max_bytes` set high enough to capture the whole thing).
- The `flight_version` the run used (from `MD_LIST_FLIGHT_RUNS`).

Then reach out through your usual MotherDuck support channel.

---

## Watch Flight run progress with the Flight Explainer

Source: https://motherduck.com/docs/key-tasks/flights/observability/explain-flight-runs

> See a live progress view of each Flight run, step by step, including which step ran, failed, or was skipped, and how long each took.

The **Flight Explainer** is a live progress view of a Flight run: each step in your pipeline lights up as it runs, with its status, row count, duration, the tables or endpoints it touched, and any error. Use it when a Flight has several steps and you want to know which one failed or where the time went, without reading the raw log.

![The Flight Explainer for a succeeded run: each step with its progress readout, a timeline bar, and its duration, above the run log and run info](img/flight-explainer-run.png)

You instrument a Flight once, in two parts:

1. **A chart** that declares the Flight's steps, as a Mermaid `flowchart`.
2. **Progress lines** the Flight prints while it runs, one `@flight-run` line each time a step changes state.

MotherDuck joins the two by step ID, so every node in the chart needs an explicit ID that its log lines repeat exactly. Every run of an instrumented Flight then renders in the **Explainer** tab of the run in the MotherDuck UI.

## Prerequisites

- A Flight you own. See [Running Python with Flights](/key-tasks/flights/) to create one.
- To add the Flight Explainer with AI, an AI agent connected to the [MotherDuck MCP Server](/sql-reference/mcp/).

:::note
Progress lines are captured as the run executes, so a run that has already finished can't be visualized after the fact. Add the chart and progress lines first, then trigger a new run.
:::

## Add a chart and progress lines with an AI agent

The fastest way to instrument an existing Flight is to have an AI agent do it. The agent reads the Flight's source, writes a chart that matches its steps, adds the `@flight-run` lines, and saves the result as a new version.

1. Open the Flight from the **Flights** page and select a run.
2. Select the **Explainer** tab in the run pane. A Flight without a chart shows **No Flight Explainer yet**.
3. Click **Add Flight Explainer support with AI**. The dialog contains a prompt that names the Flight and describes the convention. Copy it into an AI agent connected to the MotherDuck MCP Server.
4. When the agent saves the new version, run the Flight again and open the **Explainer** tab on the new run.

## Add a chart and progress lines by hand

### Step 1: Write the chart

Declare one node per step, each with an explicit ID, a label, and a **kind** class that says what the step does:

```text
flowchart TD
  extract[Pull orders from API]:::extract --> normalise[Flatten and cast]:::transform
  normalise --> load[Write to warehouse]:::load
  load --> verify{Row counts match?}:::check
  verify -->|ok| publish[Promote to prod]:::load
  verify -->|mismatch| quarantine(Park in staging)
```

| Construct | Form | Meaning |
|---|---|---|
| Box node | `id[label]` | A step labeled `label`. |
| Decision node | `id{label}` | A step that routes, marked with a diamond. |
| Edge | `a --> b` | `b` follows `a`. |
| Labeled edge | `a -->\|label\| b` | `b` follows `a` on the branch named `label`. |
| Inline class | `id:::name` | Applies one class to the node. |
| Class statement | `class a,b name` | Applies a class to several nodes. Use it to give a node both a kind and a meter. |

IDs start with a letter, contain letters, digits, `_`, or `-`, and are case-sensitive. Rounded `id(label)` and bare `id` nodes also work. `classDef`, `style`, and inline HTML are ignored with a warning.

**Kind classes** set each step's icon and are checked against what the run reports:

| Class | The step… |
|---|---|
| `:::query` | Runs SQL against MotherDuck. |
| `:::extract` | Reads from an external source, such as an API, a file, or another database. |
| `:::load` | Writes rows into a table. |
| `:::transform` | Computes in-process, with no I/O. |
| `:::check` | Validates data. A failed check means the data didn't pass, which the Flight can handle. |
| `:::switch` | Picks one outgoing path. A failed switch is a real error. |
| `:::wait` | Blocks on an external condition, such as a poll, a sleep, or an approval. |
| `:::flight` | Runs another Flight. Name the child run on its lines, for example `flight=daily_dims run=142`. |

A node without a kind class is a generic step.

**Meter classes** control how a step's progress reads: `:::fraction` shows `812 / 1000` from `done=` and `total=`, `:::percent` shows `81%` from `progress=`, and `:::fanout` shows a row per instance from `key=`. A meter's readout stays on the node after the step ends.

An unrecognized kind or meter class is reported as a warning, so typos are easy to spot.

### Step 2: Declare the chart in the source

Put the chart at the top of `main.py`, as comment lines between `# @flight` and `# @end-flight`:

```python
# @flight
#   flowchart TD
#     extract[Pull orders from API]:::extract --> normalise[Flatten and cast]:::transform
#     normalise --> load[Write to warehouse]:::load
#     load --> verify{Row counts match?}:::check
#     verify -->|ok| publish[Promote to prod]:::load
#     verify -->|mismatch| quarantine(Park in staging)
# @end-flight
```

The chart is part of the Flight's source, so saving it creates a new Flight version.

### Step 3: Print progress lines

Print one `@flight-run` line each time a step changes state. Each line must end in a newline and be flushed, so it lands in the log as its own event:

```text
@flight-run <step_id> <running|ok|failed|skipped> [field=value ...]
```

For a sequential Flight with a handful of steps, plain `print()` calls are enough:

```python
from datetime import datetime, timezone

def ts():
    return datetime.now(timezone.utc).isoformat()

def main():
    print(f"@flight-run extract running ts={ts()}", flush=True)
    rows = pull_orders()
    print(f"@flight-run extract ok rows={len(rows)} ts={ts()}", flush=True)

    print(f"@flight-run load running ts={ts()}", flush=True)
    written = write_orders(rows)
    print(f"@flight-run load ok rows={written} effect=table:analytics.main.orders ts={ts()}", flush=True)

    print(f"@flight-run verify running ts={ts()}", flush=True)
    if written != len(rows):
        print(f'@flight-run verify failed error="{written} written, {len(rows)} pulled" ts={ts()}', flush=True)
        raise SystemExit(1)
    print(f"@flight-run verify ok ts={ts()}", flush=True)

if __name__ == "__main__":
    main()
```

Fields you can add to a `@flight-run` line:

| Field | Meaning |
|---|---|
| `rows` | Row count, as an integer. |
| `error` | Error message. |
| `ts` | ISO-8601 timestamp. Add it to the `running` line and the final line of a step to get per-step durations and the timeline column. |
| `effect` | What the step touched, as `type:detail`. Repeatable. Types are `table:`, `file:`, `net:`, and `other:`, for example `effect=table:analytics.main.orders` or `effect=net:api.example.com`. |
| `key` | Instance name for a fan-out step. Defaults to a single instance. |
| `progress` | A value from `0` to `1`, for a `:::percent` meter. |
| `done`, `total` | Integers, for a `:::fraction` meter. |
| `branch` | The label of the outgoing edge this step took. |

Wrap any value that contains a space, `"`, or `=` in double quotes, for example `error="timeout after 30s"`. A line with an unknown status or no step ID is dropped with a warning.

To attach free-text messages to a step, print `@flight-log <step_id> <message>`. The message is kept verbatim.

You don't need a line for the run itself: the Flight Explainer takes the run's outcome from its status and exit code. To report it explicitly, print `@flight-run @flight ok exit=0` or `@flight-run @flight failed exit=1`.

### Step 4: Run the Flight and open the run

Trigger a run, select it in the run history, and open the **Explainer** tab. The tab updates live while the run executes. Before a Flight's first run, its page shows the chart for the current version, so you can check the chart before running anything.

## Read a run

In the Flight Explainer, each step shows:

- **Status**: running, ok, failed, skipped, or stalled. Once a run succeeds, steps it never started show as skipped. In a failed or cancelled run, steps it never reached stay pending, because the run stopped before it got to them. See [Unexpected or missing progress lines](#unexpected-or-missing-progress-lines) for the details.
- **Readout**: the row count, or a meter's progress such as `812 / 1000`.
- **Timing**: a duration and a timeline bar, when the step reported `ts=`.
- **Effects and errors**: the tables, files, and endpoints it touched, and any `error=` text.

Edges light up as the run takes them. **Run info** shows the run's config and timing, and **Run log** shows the log.

In a failed run, the failed step is highlighted in red and shows its error. Expand a fan-out step to see which instance failed. A step whose branch the run didn't take is marked as not taken, and a warning icon flags a step whose log disagrees with the chart:

![The Flight Explainer for a failed run: the Write to warehouse step is red, its ap-south instance shows the error connection reset after 30s, and Promote to prod is marked as not taken](img/flight-explainer-failed-run.png)

## Use cases and examples

### Find the failing step in a nightly pipeline

Without the Flight Explainer, a failure means scrolling a long log for the traceback. With it, the failed step is highlighted in red and shows an error. Each later step shows that it never ran, so you know what did and did not complete.

### Spot the slow shard in a partitioned load

When one step processes many partitions, declare it as a fan-out and report each partition with `key=`. The Flight Explainer lists one row per partition with its own status, row count, and timeline bar, so the long pole stands out.

```text
flowchart TD
  plan[List partitions]:::query --> load[Load partition]:::load
  load --> verify{All partitions loaded?}:::check
  class load fanout
```

```python
for day, files in partitions.items():
    print(f"@flight-run load running key={day} ts={ts()}", flush=True)
    try:
        n = load_day(day, files)
        print(f"@flight-run load ok key={day} rows={n} ts={ts()}", flush=True)
    except TimeoutError as e:
        print(f'@flight-run load failed key={day} error="{e}" ts={ts()}', flush=True)
```

A fan-out step's overall status is determined by its child instances: it is shown as running while any of its individual child instances are still running, then failed if any instance fails.

### Gate a publish on a data-quality check

Use a `:::check` decision node with one labeled edge per outcome, and report the outcome with `branch=`. When the counts match, report `ok` with `branch=match`. When they don't, report `failed` with the reason and `branch=mismatch`. The check step shows as failed because the data failed, but the Flight continues down the quarantine path and the run can still finish successfully.

```text
flowchart TD
  load[Write to staging]:::load --> verify{Row counts match?}:::check
  verify -->|match| publish[Promote to prod]:::query
  verify -->|mismatch| quarantine(Park in staging)
```

```python
print("@flight-run verify running", flush=True)
if staged == expected:
    print("@flight-run verify ok branch=match", flush=True)
    print("@flight-run publish running", flush=True)
    promote()
    print("@flight-run publish ok", flush=True)
else:
    print(f'@flight-run verify failed branch=mismatch error="{staged} staged, {expected} expected"', flush=True)
    print("@flight-run quarantine running", flush=True)
    quarantine()
    print("@flight-run quarantine ok", flush=True)
```

In a run that caught bad data, `verify` is red with its error, the `mismatch` edge and `quarantine` are lit, the `match` edge is marked as not taken, and `publish` shows as skipped.

Reporting `branch=` is more reliable than letting the Flight Explainer infer the path from which steps finished, especially for retry loops and branches that rejoin.

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| **No Flight Explainer yet**. | The Flight version has no chart. | Add a chart and run the Flight again. |
| **No Flight Explainer for this run**. | The run used a version without a chart. A newer version may have one. | Run the Flight again to visualize the current version. |
| Every step stays pending. | The Flight prints no `@flight-run` lines, or the IDs don't match the chart. | Check that each line's step ID matches a node ID exactly, including case. |
| A warning icon on a step. | The chart and the log disagree. For example, a log line names an undeclared ID, a `:::transform`, `:::check`, or `:::switch` step reported a `table:` or `file:` effect, a `branch=` names no edge out of the step, a meter class never got its fields, or a `:::flight` step never reported `flight=` and `run=`. Hover the icon to see what the chart declared and what the run logged, such as `transform step, which should not write` and `a write to table analytics.main.features`. | Fix the chart or the `print` call so they agree. A step that writes a table is a `:::load`, not a `:::transform`. |
| Later steps read as pending after a very long log. | The Flight Explainer reads the first 5,000 log lines. | Print less free text, so the `@flight-run` lines fall inside that range. |

### Unexpected or missing progress lines

The Flight Explainer doesn't reject progress lines that arrive out of order. It applies these rules instead:

- **A step logs again after it finished.** The latest line for a step (and `key=`) wins. A `running` line after `ok` shows the step as running again, and an `ok` line after `failed` shows it as ok. That's how a retry looks: log `running` again, then the final status. Nothing flags an out-of-order status, so if a step's status looks wrong, read its lines in the run log.
- **The run ends while a step is still `running`.** The step takes the run's outcome: ok if the run succeeded, failed if it failed, and stalled if it was cancelled. A process that's killed mid-step, for example for running out of memory, fails the run, so that step shows as failed even though it never printed a `failed` line.
- **A step never starts.** In a run that succeeded, it shows as skipped. In a failed or cancelled run, it stays pending, unless every way into it came from a branch the run decided against, in which case it shows as skipped.
- **A line is malformed.** A line with an unknown status or no step ID is dropped with a warning. A line for a step ID that isn't in the chart is reported as a warning, so you can fix the chart or the `print` call.

## Related

- [Monitoring and debugging Flights](/key-tasks/flights/observability/monitoring-and-debugging) — run history, logs, and common failure patterns.
- [Running Python with Flights](/key-tasks/flights/) — create, run, and schedule a Flight.

---