Database sharding
Database sharding is a horizontal partitioning technique that splits a large database into smaller, independent pieces called shards, each hosted on a separate server, to scale storage and throughput beyond a single machine.
Overview
As a database grows beyond what a single server can efficiently store or serve, sharding splits its data horizontally: rows are distributed across multiple independent database instances (shards) based on a shard key, such as customer ID or geographic region. Each shard holds a distinct subset of the rows and typically runs on its own server, so total storage capacity and write throughput scale roughly with the number of shards.
This is different from vertical partitioning (splitting a table by columns) or from read replicas (copies of the full dataset used to scale reads). Sharding specifically addresses cases where a single server can't hold or serve the entire dataset, even for a single table.
Sharding strategies
- Range-based: rows are assigned to shards based on ranges of the shard key (e.g., customer IDs 1-1M on shard A, 1M-2M on shard B). Simple to reason about, but can create hotspots if activity isn't evenly distributed across ranges.
- Hash-based: a hash function maps the shard key to a shard, spreading rows more evenly and avoiding range hotspots, at the cost of making range queries across shards harder.
- Directory-based: a lookup service maps keys to shards explicitly, offering flexibility to rebalance shards but adding a dependency on that lookup service.
Tradeoffs
Sharding solves a real scaling problem but adds significant application and operational complexity: cross-shard joins and transactions become expensive or impossible without additional coordination, rebalancing shards as data grows is nontrivial, and schema changes must be applied consistently across every shard. Because of this, teams typically only shard once vertical scaling (bigger servers) and read replicas are no longer sufficient.
Where DuckDB fits
DuckDB is an embedded, single-node analytical database — it doesn't shard data across servers itself. It's commonly used as a fast query engine on top of already-partitioned data, for example running analytical queries over Parquet files that are organized (partitioned) by date or region in object storage. This is a different problem than transactional sharding: it's about organizing files for efficient scanning, not distributing write load across independent database servers.
Related terms
Database replication is the process of copying and synchronizing data from one database (the primary) to one or more other databases (replicas), used to improve availability, read scalability, and disaster recovery.
Throughput →Throughput is the amount of work a system completes per unit of time — such as queries per second, rows processed per second, or requests handled per minute — measuring capacity rather than the speed of any single operation.
SQLite →SQLite is a lightweight, serverless database engine that runs as an embedded part of an application rather than as a separate server process.
analytical database →An analytical database, also known as an Online Analytical Processing (OLAP) database, is designed to efficiently handle complex queries and data analysis…
Massively Parallel Processing (MPP) →Massively Parallel Processing (MPP) is a database architecture in which many independent nodes, each with their own CPU, memory, and storage, cooperate to execute a single query in parallel across a cluster.
relational database →A relational database is a structured collection of data organized into tables with rows and columns.
FAQS
Partitioning is the general concept of splitting data into smaller pieces, which can happen within a single database instance (e.g., table partitions by date). Sharding specifically refers to partitioning data across multiple separate servers or database instances, so each shard is an independent piece of infrastructure.
Sharding splits a dataset into distinct, non-overlapping subsets distributed across servers, primarily to scale storage and write throughput. Replication copies the same full dataset to multiple servers, primarily to scale read throughput and provide redundancy. Large systems often use both together.
