---
title: "Dask"
description: "Dask is a Python library for parallel and distributed computing that scales pandas-like DataFrame, NumPy-like array, and general task-graph workloads across multiple cores or a cluster."
canonical: "https://motherduck.com/glossary/dask/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "Data-based: Going Beyond the Dataframe | MotherDuck"
    url: "https://motherduck.com/videos/going-beyond-the-dataframe/"
  - title: "DuckLake | MotherDuck Docs"
    url: "https://motherduck.com/docs/concepts/ducklake/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# Dask

> Dask is a Python library for parallel and distributed computing that scales pandas-like DataFrame, NumPy-like array, and general task-graph workloads across multiple cores or a cluster.

## Overview

Dask is a flexible parallel-computing library for Python. It provides dataframe, array, and general task-scheduling APIs that mirror pandas and NumPy, but execute the underlying work as a graph of tasks spread across multiple CPU cores on a single machine or across many machines in a cluster. Dask is commonly reached for when a dataset or computation no longer fits comfortably in memory on one machine, or when a workload needs to scale out horizontally.

## How it works

A Dask DataFrame is a collection of smaller pandas DataFrames (partitions) that Dask operates on lazily, building up a task graph. Nothing computes until you call `.compute()`, at which point the Dask scheduler executes the graph in parallel.

```python
import dask.dataframe as dd

ddf = dd.read_csv("events-*.csv")
result = ddf.groupby("user_id").amount.sum().compute()
```

Dask also offers `dask.array` for chunked NumPy-style arrays and `dask.delayed` for parallelizing arbitrary Python functions.

## Dask versus a single-node engine like DuckDB

Dask's strength is horizontal scaling across a cluster for workloads that genuinely exceed one machine's resources or need distributed scheduling. DuckDB, by contrast, is a single-node, in-process OLAP engine that is heavily optimized for out-of-core execution — it can often process datasets much larger than RAM efficiently on one machine using columnar storage, vectorized execution, and spilling to disk, without the coordination overhead of a distributed scheduler. In practice, many workloads that people historically reached for Dask to handle (because pandas ran out of memory) can run faster and with far less operational complexity on a single machine with DuckDB. Some teams use both: Dask for cluster-scale distributed pipelines, and DuckDB for fast local exploration, ad hoc SQL, and feeding curated Parquet output back into a Dask or pandas workflow.