PII (personally identifiable information)

Overview

PII refers to any information that can identify a specific person, either directly (a name, a passport number, an email address) or indirectly, by combining multiple pieces of otherwise-ordinary data (a ZIP code, birth date, and gender can together uniquely identify most people in a population). Data engineers deal with PII constantly, because most operational systems—CRMs, billing platforms, support tools—are built around records of individual people.

Common categories of PII include:

• Direct identifiers: full name, email, phone number, government ID/SSN, physical address
• Indirect/quasi-identifiers: ZIP code, date of birth, gender, IP address—individually not identifying, but potentially identifying in combination
• Sensitive PII: health data, financial account numbers, biometric data—subject to stricter regulatory handling in most frameworks

Regulatory context

Regulations like the EU's GDPR and California's CCPA/CPRA impose specific obligations around PII: data minimization (collect only what's needed), purpose limitation, the right for individuals to access or delete their data, and breach notification requirements. These regulations are a major reason data teams need working data lineage—you can't fulfill a deletion request without knowing everywhere a person's data lives.

Technical handling in pipelines

Common techniques for reducing PII risk in a data pipeline, roughly from least to most protective:

• Masking: replacing part of a value for display, e.g. showing ***-**-1234 instead of a full SSN.
• Tokenization: replacing a real value with a non-reversible or separately-stored reference token, so the token can be used for joins without exposing the underlying value to most consumers.
• Hashing: applying a one-way function so the same input always produces the same output (useful for deduplication or joining without exposing the raw value), but note that hashing alone is not always sufficient protection if the input space is small enough to brute-force.
• Encryption: reversible protection that requires a key to decrypt, appropriate when the original value needs to be recoverable by authorized systems.

DuckDB includes built-in hash functions like md5() and sha256() that can be used to pseudonymize a column for analytics use cases where the raw value isn't needed downstream:

Copy code
SELECT
    md5(email) AS email_hash,
    order_id,
    amount
FROM orders;

This lets analysts join and aggregate by customer without ever seeing the raw email address, though for genuinely sensitive fields, dedicated tokenization or encryption is generally preferred over hashing alone.

Access control and classification

Beyond technical transformation, most governance frameworks require PII columns to be explicitly tagged/classified in a catalog, with row- or column-level access controls restricting who can query the raw values versus a masked or aggregated version.