PyIceberg
PyIceberg is the official Python implementation of Apache Iceberg — a pure-Python library for reading, writing, and managing Iceberg tables and catalogs without requiring Spark or a JVM.
Overview
PyIceberg is a Python library, maintained under the Apache Iceberg project, that implements the Iceberg table format natively in Python. Before PyIceberg, working with Iceberg tables typically meant going through Spark, which pulls in a JVM dependency even for simple read or metadata operations. PyIceberg lets Python code load a table, inspect its schema and snapshots, plan a scan, and read the resulting data as a pandas DataFrame or PyArrow Table, all without a Spark cluster.
What it's used for
Common uses include lightweight table inspection and maintenance scripts, small-to-medium ETL jobs that don't need distributed compute, and building custom tooling or services on top of Iceberg tables. PyIceberg supports the major Iceberg catalogs — REST, AWS Glue, Hive Metastore, and SQL catalogs — and can also perform basic writes (appends, and increasingly more complex operations) directly from Python.
Typical workflow
Copy code
from pyiceberg.catalog import load_catalog
catalog = load_catalog("my_catalog", **{
"uri": "https://catalog.example.com",
"type": "rest",
})
table = catalog.load_table("sales.orders")
df = table.scan(row_filter="order_date >= '2026-01-01'").to_pandas()
Using PyIceberg alongside DuckDB
Because PyIceberg can materialize a table scan as a PyArrow Table, it's straightforward to hand that result to DuckDB for further SQL analysis in the same process:
Copy code
import duckdb
arrow_table = table.scan().to_arrow()
duckdb.sql("SELECT region, SUM(amount) FROM arrow_table GROUP BY ALL").show()
For simpler cases, DuckDB's own iceberg extension can query Iceberg tables directly with SQL — via iceberg_scan() on files, or by attaching a REST catalog — without going through PyIceberg at all, which is often the more direct path when the goal is SQL analytics rather than programmatic table management.
Related terms
Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to large analytic tables stored as files in a data lake.
Data catalog →A data catalog is a centralized inventory of an organization's datasets, tracking metadata like schema, location, lineage, and ownership so people and systems can discover, understand, and access data.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Data lakehouse →A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
Apache Spark →Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
PyArrow →PyArrow is a Python library that provides a high-performance interface for working with columnar data structures, particularly those defined by the Apache…
FAQS
PyIceberg is a pure-Python client for Apache Iceberg, used to read, write, and manage Iceberg tables and catalogs (REST, Glue, Hive Metastore) without needing Spark or a JVM.
Not necessarily. DuckDB's own iceberg extension can query Iceberg tables directly with SQL. PyIceberg is more useful for programmatic table management, custom catalog integrations, or Python-native ETL, and its results can also be handed to DuckDB as a PyArrow table for further SQL analysis.
No. PyIceberg is a native Python implementation of the Iceberg table format and does not depend on Spark or the JVM, unlike earlier ways of working with Iceberg tables.
