Data freshness
Data freshness measures how up to date a dataset is relative to when the underlying real-world events actually occurred, and is a core signal for whether downstream data and dashboards can be trusted.
Overview
Data freshness answers the question "how current is this data right now?" It's typically measured as the lag between when an event happened (or when a source record was last updated) and when that data is available and correct in a downstream table or dashboard. A dashboard showing yesterday's numbers as "today's revenue" because a pipeline silently stopped running is a data freshness failure, even though the numbers themselves might be accurate for the day they actually represent.
Freshness is one of the core dimensions of data quality (alongside accuracy, completeness, and consistency), and unlike some quality dimensions, it's usually straightforward to measure automatically because it just requires comparing timestamps.
Why it matters
Stale data is dangerous specifically because it often looks fine—the numbers are internally consistent and plausible, just old. A sales dashboard that hasn't updated in three days because an upstream API started failing will still render correctly; it just quietly shows the wrong reality. Freshness checks catch this class of failure automatically, without needing to know anything about whether the values themselves are "correct."
How freshness is checked
Freshness checks generally compare two timestamps: the most recent timestamp in a table, and the current time (or the time the check runs), then alert if the gap exceeds an expected threshold.
dbt has built-in source freshness checks configured in YAML:
Copy code
sources:
- name: raw
tables:
- name: orders
loaded_at_field: _loaded_at
freshness:
warn_after: {count: 6, period: hour}
error_after: {count: 24, period: hour}
Running dbt source freshness checks the max _loaded_at value against the current time and raises a warning or error if the data is older than the configured thresholds.
The same idea can be expressed directly in SQL, which works the same way whether the table lives in DuckDB or any other engine:
Copy code
SELECT
max(_loaded_at) AS latest_load,
now() - max(_loaded_at) AS staleness
FROM raw_orders
HAVING now() - max(_loaded_at) > INTERVAL '24 hours';
A query like this returning any rows signals a freshness violation and can be wired into an orchestrator or monitoring tool to trigger an alert.
Related terms
Data observability is the continuous, automated monitoring of data pipelines and datasets to detect freshness, volume, schema, and quality problems before they reach downstream users.
Data quality →Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for the purposes it's used for—dashboards, models, and operational decisions.
Data contract →A data contract is a formal, agreed-upon specification between a data producer and its consumers that defines the schema, format, semantics, and guarantees (like update frequency or backward-compatibility rules) of a dataset.
Great Expectations →Great Expectations (GX) is an open-source Python framework for defining, running, and documenting automated data quality checks called Expectations.
FAQS
They're closely related; latency usually refers to the delay in a specific pipeline step (how long extraction or transformation takes), while freshness is the end-to-end measure of how current the final data is relative to real-world events, which is what actually matters to a data consumer.
Base it on how the data is used, not on what's technically achievable. A real-time fraud detection feed might need freshness measured in seconds, while a monthly financial report might tolerate a day of lag—setting thresholds tighter than necessary just creates alert fatigue.
