Ibis
Ibis is a Python dataframe library that lets you write a single portable API and execute it against 20+ backends, with DuckDB as its default local engine.
Overview
Ibis is a Python library that provides a deferred, expression-based dataframe API: you write Python code that looks like pandas or SQL, and Ibis compiles it into the native query language of whichever backend you're targeting. The same Ibis code can run against DuckDB, Snowflake, BigQuery, Postgres, Spark, Trino, Polars, and many others without rewriting the query, which makes it useful for portable analytics code and for prototyping locally before pointing at a production warehouse.
Why DuckDB is the default backend
DuckDB is Ibis's default backend for local, in-memory execution. When you call ibis.connect() or ibis.duckdb.connect() with no other backend configured, Ibis spins up an in-process DuckDB database. This pairing works well because DuckDB is fast, embeddable, and has no server to manage, so Ibis expressions can be developed and tested instantly on a laptop and later pointed at a distributed backend with the same code.
Copy code
import ibis
con = ibis.duckdb.connect() # in-memory DuckDB
t = con.read_parquet("orders.parquet")
result = (
t.filter(t.status == "completed")
.group_by(t.customer_id)
.aggregate(total=t.amount.sum())
.order_by(ibis.desc("total"))
)
df = result.to_pandas()
Because Ibis expressions are lazy, nothing executes until you call a method like .to_pandas(), .execute(), or .to_pyarrow() — Ibis pushes the whole computation down into DuckDB's query engine rather than pulling rows into Python row by row.
Practical use
Ibis is popular for building backend-agnostic analytics libraries, for interactively exploring large tables without loading them fully into memory, and for teams that want pandas-like ergonomics with SQL-engine performance and scale.
Related terms
A DataFrame is a two-dimensional data structure that organizes data into rows and columns, similar to a spreadsheet or database table.
Great Expectations →Great Expectations (GX) is an open-source Python framework for defining, running, and documenting automated data quality checks called Expectations.
PyArrow →PyArrow is a Python library that provides a high-performance interface for working with columnar data structures, particularly those defined by the Apache…
relational API →The relational API in DuckDB provides a fluent, Pythonic interface for constructing and executing SQL queries programmatically.
Polars →Polars is a high-performance data manipulation library written in Rust, designed to handle large datasets efficiently.
DEFAULT value →A DEFAULT value is a value a database column automatically takes on when an INSERT statement doesn't explicitly specify a value for it.
FAQS
No. Ibis's default DuckDB backend runs in-process, so you can start working with local files (CSV, Parquet) or in-memory data immediately without setting up a server.
Not exactly — Ibis expressions compile down to SQL (or another backend's native plan) under the hood. It gives you a composable, type-checked Python API as an alternative to writing raw SQL strings.
