Practical Postgres 🐘 CDC with StreamkapLivestream July 15

START FREE
← Back to Glossary

data load tool (dlt)

Data Load Tool (dlt) is an open-source Python library that automates extracting data from sources and loading it into destinations like DuckDB and MotherDuck, with automatic schema inference and incremental loading.

Data Load Tool (dlt) is an open-source Python library that simplifies building data pipelines. It automates the tedious parts of data ingestion — extraction, schema management, type conversion, and incremental loading — so you write pipeline logic instead of boilerplate. dlt supports a wide range of sources (REST APIs, databases, files) and destinations (data warehouses, data lakes, and local engines), presenting a consistent interface across all of them.

What dlt does for you

  • Automatic schema inference: dlt reads your source data, infers a schema, and creates matching tables in the destination — evolving them as the source changes.
  • Incremental loading: load only new or changed records on each run instead of reprocessing everything, using cursor fields or merge/upsert strategies.
  • Normalization: nested JSON is automatically unpacked into relational tables with keys linking them together.
  • Consistent EL(T): the same code shape works whether you load into DuckDB locally or a cloud warehouse, which makes dlt a natural front end for ELT workflows.

dlt and DuckDB

dlt ships with a first-class DuckDB destination, which makes the pair a popular, zero-infrastructure way to build and test pipelines locally. A minimal pipeline that loads records into DuckDB looks like this:

Copy code
import dlt

pipeline = dlt.pipeline(
    pipeline_name="github_events",
    destination="duckdb",
    dataset_name="raw",
)

data = [{"id": 1, "type": "push"}, {"id": 2, "type": "pull_request"}]
load_info = pipeline.run(data, table_name="events")
print(load_info)

dlt creates the DuckDB database, infers the events table schema, and loads the rows — no manual DDL required.

Incremental loading

On subsequent runs, incremental settings let dlt load only new records rather than reloading the whole table:

Copy code
@dlt.resource(primary_key="id", write_disposition="merge")
def events(updated=dlt.sources.incremental("updated_at")):
    yield from fetch_events(since=updated.last_value)

Here dlt tracks the updated_at cursor between runs and merges on id, so each run pulls and upserts only changed rows.

From local DuckDB to the cloud

Because dlt abstracts the destination, a pipeline you prototype against local DuckDB can be pointed at MotherDuck by changing the destination connection string — letting you develop locally and load into a serverless cloud warehouse in production without rewriting pipeline code. Paired with a transformation tool like dbt, dlt covers the extract-and-load half of a modern ELT stack.

Related terms

ELT →

ELT (Extract, Load, Transform) is a modern data integration process that reverses the order of traditional ETL (Extract, Transform, Load) workflows.

ETL →

ETL (Extract, Transform, Load) is a data integration process that combines data from multiple sources into a single destination, typically a data warehouse…

data build tool (dbt) →

dbt is an open-source command-line tool that enables data analysts and engineers to transform data in their warehouses more effectively.

data pipeline →

A data pipeline is a series of interconnected processes that extract data from various sources, transform it into a usable format, and load it into a…

data ingestion →

Data ingestion is the process of importing raw data from various sources into a system where it can be stored and analyzed.

DuckDB →

DuckDB is an embeddable SQL database management system designed for analytical workloads.

FAQS

dlt is an open-source Python library for building data pipelines. It automates much of data ingestion — extracting from APIs and databases and loading into warehouses or lakes — so you can focus on pipeline logic instead of boilerplate.

Yes. dlt ships with a first-class DuckDB destination, so you can load extracted data straight into a local DuckDB database with no infrastructure. Because dlt abstracts the destination, the same pipeline can later target MotherDuck in production by changing the connection.

It infers schemas, creates and evolves destination tables, handles data-type conversions, normalizes nested JSON into relational tables, and manages incremental loading for efficient updates — all with minimal code.

Building robust, maintainable EL(T) pipelines with a consistent interface across many sources and destinations. It covers the extract-and-load stage of a modern data stack, often paired with a transformation tool like dbt.

READ NEXT