Data skew
Data skew is an uneven distribution of data across partitions or keys in a distributed system, causing some worker nodes to process far more data than others and become bottlenecks.
Overview
Distributed processing engines like Apache Spark and Hadoop split work across many nodes by partitioning data, often by a key such as customer ID or country. Data skew occurs when that key distribution is uneven — for example, one customer accounts for 40% of all rows — so the partition (and the worker processing it) ends up with a disproportionate share of the data. While most workers finish quickly, the skewed worker becomes a straggler, and the whole job waits on it.
Where skew shows up
- GROUP BY / aggregations: a few extremely common keys dominate a partition, e.g., aggregating web events by
user_idwhen a handful of bot accounts generate millions of events. - Joins: joining on a skewed key (e.g., joining orders to a
countrydimension when most orders come from one country) sends a disproportionate volume of rows to a single join task. - Partitioned storage: writing files partitioned by a skewed column produces wildly uneven file sizes, hurting both write and read performance.
Mitigation techniques
- Salting: append a random suffix to skewed keys to spread them across more partitions, then aggregate in two stages (per-salted-key, then across salts).
- Broadcast joins: for skewed joins against a small dimension table, broadcast the small table to all workers instead of shuffling the large table by the skewed key.
- Adaptive query execution: modern engines (e.g., Spark's AQE) can detect skewed partitions at runtime and automatically split them into smaller tasks.
- Repartitioning: explicitly repartition on a more evenly-distributed key, or increase partition count so skewed keys are spread more thinly.
DuckDB and skew
DuckDB is a single-node, embedded analytical database rather than a distributed cluster, so the classic distributed-systems failure mode of skew — one node stalling a whole cluster because of an uneven shuffle — doesn't apply in the same way. A single machine still has to do the work for a heavy key, so a very skewed aggregation or join can still be slower than a balanced one, but there's no cross-node coordination overhead or straggler-task problem to manage; DuckDB's query optimizer and vectorized execution handle uneven key cardinalities within a single process.
Related terms
Partition pruning is a query optimization that skips reading entire partitions of a dataset when a query's filters make it impossible for those partitions to contain matching rows.
Apache Spark →Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
Apache Hadoop →Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.
partitions →Partitions in data systems refer to the logical or physical division of large datasets into smaller, more manageable segments.
Composite key →A composite key is a primary key made up of two or more columns whose combined values uniquely identify a row, used when no single column is unique on its own.
Surrogate key →A surrogate key is a system-generated identifier — typically a sequential integer or UUID — assigned to a row that has no business meaning of its own, used as the primary key in dimension and fact tables.
FAQS
Data skew is caused by an uneven distribution of values in the column used to partition or shuffle data — for example, a small number of keys accounting for a disproportionate share of rows. This is common with things like power-law distributed user activity, a dominant country or category, or null-heavy join keys.
Common fixes include salting skewed keys to spread them across more partitions, using broadcast joins for skewed joins against small tables, enabling adaptive query execution to auto-split skewed partitions at runtime, and repartitioning data on a more evenly distributed key.
Not in the same way it affects distributed clusters. There's no cross-node shuffle or straggler task to worry about, since everything runs in one process. A very skewed key can still make a specific aggregation or join take longer, but it doesn't stall other workers the way it does in a distributed system.
