Data observability
Data observability is the continuous, automated monitoring of data pipelines and datasets to detect freshness, volume, schema, and quality problems before they reach downstream users.
Overview
Data observability applies the same idea as observability in software engineering—instrumenting systems so problems are visible automatically, rather than discovered by users after the fact—to data pipelines. Instead of waiting for someone to notice a dashboard looks wrong, an observability platform continuously tracks metrics about the pipeline itself and alerts when something deviates from the expected pattern.
The discipline is usually described around five key pillars, sometimes called the "five pillars of data observability":
- Freshness: is data arriving as recently as expected?
- Volume: is the row count in an expected range, or did a source suddenly send 10x or 0.1x the usual data?
- Schema: did a column get added, removed, renamed, or change type unexpectedly?
- Distribution: are values within expected ranges (no sudden spike of nulls, no impossible values)?
- Lineage: when something breaks, what's upstream and downstream, so impact and root cause can be identified quickly?
How it differs from testing
Testing (like dbt tests) checks specific, known rules that someone wrote in advance—"this column must never be null." Observability is broader: it establishes a baseline from historical patterns and flags anomalies automatically, catching problems nobody thought to write an explicit test for, like a source that normally sends 50,000 rows a day suddenly sending 500.
Implementing basic observability with SQL
Many observability checks can be expressed as straightforward SQL run on a schedule, even without a dedicated platform. A freshness and volume check against a DuckDB table might look like:
Copy code
SELECT
max(loaded_at) AS latest_load,
now() - max(loaded_at) AS staleness,
count(*) AS row_count
FROM raw_orders
WHERE loaded_at::date = current_date;
Comparing today's row_count against a rolling average of the past N days is a simple, effective volume-anomaly check that doesn't require specialized tooling—just a query, a threshold, and something to run it on a schedule and alert on the result.
Dedicated tooling
Purpose-built platforms (Monte Carlo, Metaplane, Bigeye, and similar) automate this pattern at scale: connecting to a warehouse, learning normal patterns for every table automatically, and surfacing anomalies without requiring each check to be hand-written. Some of these tools integrate with orchestrators and lineage graphs so an alert immediately shows which downstream dashboards or models are affected.
Related terms
Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for the purposes it's used for—dashboards, models, and operational decisions.
Data freshness →Data freshness measures how up to date a dataset is relative to when the underlying real-world events actually occurred, and is a core signal for whether downstream data and dashboards can be trusted.
Data lineage →Data lineage is the traceable record of where a piece of data came from, what transformations it passed through, and where it's used downstream.
Data contract →A data contract is a formal, agreed-upon specification between a data producer and its consumers that defines the schema, format, semantics, and guarantees (like update frequency or backward-compatibility rules) of a dataset.
Data orchestration →Data orchestration is the automated coordination, scheduling, and monitoring of interdependent data tasks—such as extracts, transformations, and loads—so they run in the correct order and recover cleanly from failure.
CHECK constraint →A CHECK constraint is a rule attached to a table column (or the table as a whole) that requires every row to satisfy a given boolean expression.
FAQS
They're related but distinct: data quality is the property being protected (accurate, complete, consistent data), while data observability is the practice of continuously monitoring pipelines to detect quality problems automatically and quickly.
Not necessarily to start—basic freshness and volume checks can be implemented as scheduled SQL queries with simple alerting. Dedicated platforms add value at scale by learning baselines automatically across many tables and integrating with lineage to speed up root-cause analysis.
