Data lineage

Overview

Data lineage answers two questions about any given dataset or field: where did this come from, and what does it feed into? Upstream lineage traces a table back through every transformation to its original source (a production database, an API, a file drop). Downstream lineage traces forward to every dashboard, model, or table that depends on it. Together they form a map of how data flows through an organization.

Lineage matters for three practical reasons: debugging (a number looks wrong—which upstream table or transformation is responsible?), impact analysis (before changing a table's schema, what breaks downstream?), and compliance (regulators or auditors need to know exactly how a reported figure was derived, or where a piece of personal data flows).

How lineage is captured

There are two broad approaches:

• Static/parsed lineage: tools parse SQL or transformation code (like dbt's ref() and source() functions) to build a dependency graph without running anything. dbt automatically generates lineage graphs this way, since every model's dependencies are explicit in its code.
• Runtime/observed lineage: tools watch actual query execution (via query logs or metadata APIs) to infer lineage from what really ran, which can catch dependencies that aren't obvious from code alone, like an analyst's ad hoc query feeding a spreadsheet that later gets treated as a source of truth.

dbt's dbt docs generate command produces an interactive lineage graph directly from a project's model dependencies, showing sources, staging models, marts, and exposures (like dashboards) as connected nodes.

Column-level vs. table-level lineage

Table-level lineage shows that table B depends on table A, but not which specific columns. Column-level lineage tracks dependencies at the field level—showing, for instance, that a customer_ltv column is derived specifically from orders.amount and orders.customer_id, not the whole orders table. Column-level lineage is more useful for precise impact analysis (can I safely drop this one column?) but is harder to compute automatically, especially across complex SQL with joins and aggregations.

Why lineage matters for governance

Lineage is foundational to data governance and to handling regulated data like PII: you can't answer "everywhere does this customer's email address exist in our systems" without a working lineage graph, and that question is exactly what regulations like GDPR's right-to-erasure require an organization to answer.