Data quality

Overview

Data quality describes whether data can be trusted for its intended use. It's usually broken down into several concrete dimensions:

• Accuracy — does the data correctly reflect reality?
• Completeness — are required fields and records missing?
• Consistency — do the same facts agree across tables and systems?
• Timeliness / freshness — is the data current enough to be useful?
• Uniqueness — are there unintended duplicate records?
• Validity — does data conform to expected formats, types, and ranges?

Poor data quality is expensive precisely because it's often invisible until it causes a bad decision—a dashboard with silently duplicated rows overstates revenue, a null-heavy join undercounts customers, and nobody notices until finance asks why the numbers don't reconcile.

How data quality is enforced

Most teams enforce data quality with two complementary approaches: automated tests that run as part of the pipeline, and profiling/monitoring that surfaces anomalies over time.

Testing checks specific, known rules—dbt's built-in and custom tests are a common example:

Copy code
models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: amount
        tests:
          - not_null

Profiling looks at the actual shape of the data to catch problems you didn't think to test for—unexpected nulls, a min/max outside a sane range, or a cardinality that suddenly changed.

Profiling with DuckDB

DuckDB's SUMMARIZE command gives an instant profile of a table's columns—type, min, max, approximate unique count, average, standard deviation, quantiles, and null percentage—without writing any custom SQL:

Copy code
SUMMARIZE SELECT * FROM orders;

This is a fast first check for the kinds of quality issues that plain row counts miss, like a column that's unexpectedly 40% null or an amount column with a suspicious negative minimum:

Copy code
SELECT count(*) AS total_rows,
       count(*) FILTER (WHERE amount IS NULL) AS null_amounts,
       count(*) FILTER (WHERE amount < 0) AS negative_amounts
FROM orders;

Where it fits with governance and observability

Data quality overlaps with, but isn't identical to, data governance (policy and ownership) and data observability (automated, ongoing monitoring of pipeline health). Quality is the property being managed; governance and observability are the processes and tooling that manage it.