---
title: "Data skew"
description: "Data skew is an uneven distribution of data across partitions or keys in a distributed system, causing some worker nodes to process far more data than others and become bottlenecks."
canonical: "https://motherduck.com/glossary/data-skew/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "Read Scaling | MotherDuck Docs"
    url: "https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/read-scaling/"
  - title: "How to Build Robust Data Pipelines with AI | MotherDuck"
    url: "https://motherduck.com/videos/robust-data-pipelines-ai/"
gated_asset:
  title: "Postgres Is Full: A Field Guide to Analytics at Scale"
  url: "https://motherduck.com/lp/postgres-analytics-guide-full/"
---

# Data skew

> Data skew is an uneven distribution of data across partitions or keys in a distributed system, causing some worker nodes to process far more data than others and become bottlenecks.

## Overview

Distributed processing engines like Apache Spark and Hadoop split work across many nodes by partitioning data, often by a key such as customer ID or country. Data skew occurs when that key distribution is uneven — for example, one customer accounts for 40% of all rows — so the partition (and the worker processing it) ends up with a disproportionate share of the data. While most workers finish quickly, the skewed worker becomes a straggler, and the whole job waits on it.

## Where skew shows up

- **GROUP BY / aggregations**: a few extremely common keys dominate a partition, e.g., aggregating web events by `user_id` when a handful of bot accounts generate millions of events.
- **Joins**: joining on a skewed key (e.g., joining orders to a `country` dimension when most orders come from one country) sends a disproportionate volume of rows to a single join task.
- **Partitioned storage**: writing files partitioned by a skewed column produces wildly uneven file sizes, hurting both write and read performance.

## Mitigation techniques

- **Salting**: append a random suffix to skewed keys to spread them across more partitions, then aggregate in two stages (per-salted-key, then across salts).
- **Broadcast joins**: for skewed joins against a small dimension table, broadcast the small table to all workers instead of shuffling the large table by the skewed key.
- **Adaptive query execution**: modern engines (e.g., Spark's AQE) can detect skewed partitions at runtime and automatically split them into smaller tasks.
- **Repartitioning**: explicitly repartition on a more evenly-distributed key, or increase partition count so skewed keys are spread more thinly.

## DuckDB and skew

DuckDB is a single-node, embedded analytical database rather than a distributed cluster, so the classic distributed-systems failure mode of skew — one node stalling a whole cluster because of an uneven shuffle — doesn't apply in the same way. A single machine still has to do the work for a heavy key, so a very skewed aggregation or join can still be slower than a balanced one, but there's no cross-node coordination overhead or straggler-task problem to manage; DuckDB's query optimizer and vectorized execution handle uneven key cardinalities within a single process.