connectorx

Overview

connectorx (imported as connectorx, commonly aliased cx) is a library designed to solve one specific problem well: getting data out of a SQL database and into a Python DataFrame as quickly as possible. Traditional approaches using drivers like SQLAlchemy or pandas.read_sql extract rows one at a time in Python, which is slow for large result sets. connectorx instead partitions a query, extracts data in parallel using a Rust core, and writes results directly into the destination format (pandas, Polars, Arrow, or Dask) with as few copies as possible, following a "load once, copy never" design.

Basic usage

Copy code
import connectorx as cx

df = cx.read_sql(
    "postgresql://user:pass@host:5432/db",
    "SELECT * FROM orders WHERE order_date >= '2026-01-01'",
    partition_on="order_id",
    partition_num=4,
)

Specifying a partition_on column lets connectorx split the query into ranges and pull them concurrently, which is where most of its speedup comes from on large tables.

Where it fits with DuckDB

connectorx and DuckDB solve adjacent but different problems: connectorx is focused purely on fast extraction from source databases (Postgres, MySQL, SQL Server, and others) into a DataFrame, while DuckDB is a fast in-process analytical query engine for transforming and analyzing that data once it's local. A common pattern is to use connectorx to pull data out of an OLTP database quickly, land it as a Polars or pandas DataFrame (or write it to Parquet), and then run DuckDB SQL directly against that in-memory DataFrame or file for aggregation, joins, and analysis — combining connectorx's fast extraction with DuckDB's fast local querying rather than relying on the source database's own (often single-threaded) query execution for analytical workloads.