# Packages and recommended libraries > Manage Python dependencies, choose Flight loading patterns, and use dlt and dbt for ingest and transformation. A Flight runs your Python with the packages you list in `requirements.txt`. This page covers how to declare dependencies, how to choose a loading pattern for Flight ingestion, and the two libraries we recommend for the most common workloads. ## requirements.txt is plain pip syntax Pass package specifications one per line, the same as a regular pip requirements file: ```text duckdb==1.5.3 dlt==1.27.0 httpx==0.28.1 pandas==2.2.3 ``` You can use any version specifier pip supports: `==`, `>=`, `~=`, extras (`some-package[extra]`), and so on. The one dependency worth special attention is **DuckDB**: pin it to the version MotherDuck's server ships. Find that version in the [MotherDuck release notes](/about-motherduck/release-notes), or run a quick query against MotherDuck: #### Check the MotherDuck DuckDB version Database: `docs_playground` ```sql SELECT version(); ``` ## The runtime environment Before `main()` runs, the runtime installs the packages from `requirements.txt` into the Flight's Python environment. A few properties of that environment are worth knowing up front: - **Declare every dependency in `requirements.txt`.** Dependencies are installed once, before `main()` starts; there's no interactive `pip` step inside the run. To run a tool's command-line interface (dbt, dlt), call its console script with `subprocess` — for example `subprocess.run(["dbt", "build"], check=True)`. The console scripts are on `PATH` after install, so you don't need `python -m`. - **Most system binaries aren't preinstalled.** The runtime is a base Debian image with `git` available, so `git+https://` requirements work without extra setup. Other tools, such as `ffmpeg` or Playwright, aren't present until you install them with `apt-get` at the start of `main()` (see [Beyond Python](/concepts/flights#beyond-python)). ### Runtime limits A Flight is sized for orchestration and basic processing, not for crunching large tables in the runtime memory. Two limits commonly bite first: - **Definition size.** The Flight's `source_code` is capped at 200 KB, and `requirements.txt` at 20 KB. Don't embed reference data or large fixtures in the source — load them from object storage or an external URL at run time instead. - **Memory.** The runtime has a fixed memory ceiling of 16 GB. Heavy in-memory work can be OOM-killed, often with little in the log. Keep heavy compute in SQL so MotherDuck does the work, process in bounded chunks, and when running dbt lower `--threads` to cap peak memory. See [Monitoring and debugging](/key-tasks/flights/monitoring-and-debugging#common-failure-patterns) for the OOM symptom and fix. - **Maximum runtime per run.** A single run can execute for up to 1 hour on Lite, or up to 8 hours by default on Business and Enterprise plans. See [Availability and plan limits](/concepts/flights#availability-and-plan-limits). :::warning `CAST(timestamptz AS VARCHAR)` renders in the **session time zone**. The same row hashed on a laptop (local time zone) and in a Flight (UTC) produces different strings, so md5 or row-hash recipes built on string-cast timestamps disagree across environments and can trigger a false full re-import. Pin the session time zone (`SET TimeZone = 'UTC';`) wherever determinism matters, or hash an epoch value (`epoch_ms(ts)`) instead of a string cast. ::: ## Choose a loading pattern Flights often start with Python variables: API responses, scraped rows, JSON objects, or files written under `/tmp`. The slow path is to send one row at a time to MotherDuck. Pick a bulk pattern before the data grows. | Source shape | Use this pattern | Why | |---|---|---| | A few hundred control rows | Direct `INSERT` or `executemany` is acceptable. | The code stays simple and the round-trip overhead is small enough. | | API pages already in Python memory | Build batches with PyArrow, Polars, or Pandas, then `INSERT INTO ... SELECT` from the registered table. | Keeps the load as a bulk operation. PyArrow and Polars give better type control than plain Python objects. | | Larger scrape or API pull without cloud storage | Write CSV, Parquet, or a local DuckDB file under `/tmp`, then load in chunks. | Keeps memory bounded. Parquet is typed and compressed; CSV is easy when you control both write and read. Clean up `/tmp` at the end of the run. | | Files already in S3, or data you want to replay and backfill | Write Parquet to S3 and load with `read_parquet()` or `INSERT INTO ... SELECT`. | Best fit for large, partitioned, or shared datasets. It requires cloud credentials, but gives you durable staging and easier retries. | | Schema-evolving API or app data | Use `dlt[motherduck]` and make the loader format explicit with `loader_file_format="parquet"`. | dlt handles state, schema evolution, and merge logic while avoiding row-wise remote inserts. | As a rough rule, direct inserts are only for tiny control tables. For Flight ingestion, aim to flush batches rather than individual rows. Batches in the 10-100 MB range are usually easier to reason about than one huge load, and they leave room for retries, logging, and memory headroom. :::tip If you already have files in object storage, keep them there and let MotherDuck read them. If the data exists only inside the Flight process, batch it locally first; only write to S3 when you need durable staging, replay, backfills, or larger parallel reads. ::: ## Recommended libraries Two libraries cover most of what teams build with Flights. ### dltHub's dlt for ingest [dlt](/integrations/ingestion/dlt), from [dltHub](https://dlthub.com/), is the recommended Python library for moving data **into** MotherDuck. It handles schema evolution, incremental loading, retries, and state tracking, and it ships a MotherDuck destination out of the box. ```text duckdb==1.5.3 dlt[motherduck]==1.27.0 ``` A minimal ingest from a REST API into MotherDuck: ```python import dlt import httpx def main(): pipeline = dlt.pipeline( pipeline_name="github_stars", destination="motherduck", dataset_name="github", ) response = httpx.get("https://api.github.com/repos/duckdb/duckdb", timeout=30) response.raise_for_status() pipeline.run( [response.json()], table_name="repo_stats", loader_file_format="parquet", ) if __name__ == "__main__": main() ``` Use the MotherDuck destination, not the generic DuckDB destination pointed at `md:`, for remote MotherDuck loads. The MotherDuck destination uses Parquet and `COPY` for data loading; the generic DuckDB destination has different defaults. Passing `loader_file_format="parquet"` in Flight examples makes the intended loading path explicit. See the [dlt MotherDuck destination docs](https://dlthub.com/docs/dlt-ecosystem/destinations/motherduck) for the full setup. ### dbt for transformation [dbt](https://docs.getdbt.com/) with the [`dbt-duckdb`](https://github.com/duckdb/dbt-duckdb) adapter is the recommended way to run transformation graphs against MotherDuck data. ```text duckdb==1.5.3 dbt-duckdb==1.10.1 ``` Run a dbt project from a Flight: ```python import os import subprocess def main(): cwd = os.path.dirname(os.path.abspath(__file__)) subprocess.run(["dbt", "build", "--target", "prod"], cwd=cwd, check=True) if __name__ == "__main__": main() ``` If your dbt project pulls in dbt packages from git (for example, `dbt-utils` declared in `packages.yml`), call `dbt deps` before `dbt build`. `git` is available in the runtime, so no `apt-get` step is needed: ```python import subprocess def main(): subprocess.run(["dbt", "deps"], check=True) subprocess.run(["dbt", "build"], check=True) if __name__ == "__main__": main() ``` ### Use a dlt verified source dlt's [verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources/) cover many SaaS applications, including Stripe, HubSpot, and Shopify. The usual way to get one is `dlt init motherduck`, which copies the connector into a local project so you can edit it. That scaffold spans several modules, and a Flight's `source_code` is a single file, so you can't paste it in. Install the connector as a dependency instead. The [`verified-sources`](https://github.com/dlt-hub/verified-sources) repository is a buildable Python package, so pip can install it straight from a GitHub archive: ```text dlt[motherduck]==1.30.0 dlt-verified-sources @ https://github.com/dlt-hub/verified-sources/archive/3957506893a7da821dbcc6acd51c7ca4475d1f53.tar.gz ``` The connectors then import from the `sources` namespace: ```python from sources.hubspot import hubspot from sources.shopify_dlt import shopify_source from sources.stripe_analytics import stripe_source ``` Four things to know before you rely on this: - **Pin a commit SHA.** The package isn't published to PyPI and its tags trail the default branch, so a SHA in the archive URL is the only meaningful pin. `master.tar.gz` also works, but a Flight reinstalls its dependencies on every run, so an unpinned URL can change the connector between two runs of a Flight you haven't touched. Move the pin forward deliberately instead, and rerun the Flight once on demand after you do. - **Add each source's extra dependencies yourself.** The repository declares them in `[dependency-groups]`, which pip doesn't install. Stripe needs `stripe`, for example, or `sources.stripe_analytics` fails with `ModuleNotFoundError: No module named 'stripe'`. HubSpot and Shopify need nothing beyond dlt. - **Expect a heavy install.** The package depends on `dlt[bigquery, duckdb]`, which pulls in the Google Cloud libraries on every run. It doesn't include the MotherDuck destination, so declare `dlt[motherduck]` yourself. - **You get upstream behavior only.** A `dlt init` scaffold exists so you can modify the connector. As a dependency you can't, so if you need to change how a source extracts data, keep it in a local project and run it elsewhere. The repository's `sources/*_pipeline.py` demo scripts import as `from hubspot import hubspot`, which assumes the `dlt init` layout. Read them for the resource names and arguments, then write your own `main()` against the `sources.` import path. Pass credentials with [Flight secrets](/key-tasks/flights/flights-authentication-config-and-secrets#secrets-sensitive-environment-variables), and create the secret before the Flight that references it: `MD_CREATE_FLIGHT` rejects an unknown name with `user_secret not found`. Because a secret's keys are injected under their bare names with case preserved, naming a key after dlt's own config variable lets dlt resolve it with no glue code in your Flight. For a source's non-secret settings, use the Flight's `config` map with the same naming: ```text SOURCES__HUBSPOT__API_KEY SOURCES__STRIPE_ANALYTICS__STRIPE_SECRET_KEY SOURCES__SHOPIFY_DLT__PRIVATE_APP_PASSWORD ``` That bare-name trick is for values dlt itself reads. For anything your own code reads, prefer the namespaced `_` form, which is unambiguous when two secrets define the same key. dlt attaches its destination database but never creates it, so run `CREATE DATABASE IF NOT EXISTS` once at the start of `main()` or the first load fails with `Catalog with name "" does not exist`. See the [Stripe](/integrations/ingestion/stripe), [HubSpot](/integrations/ingestion/hubspot), and [Shopify](/integrations/ingestion/shopify) integration pages for complete Flights built this way. --- ## Docs feedback MotherDuck accepts optional user-submitted feedback about this page at `GET https://motherduck.com/docs/api/feedback/agent`. For agents and automated tools, feedback submission should be user-confirmed before sending. URL-encode query parameter values and send a GET request: ```text GET https://motherduck.com/docs/api/feedback/agent?page_path=%2Fkey-tasks%2Fflights%2Fpackages-and-runtime%2F&page_title=Packages%20and%20recommended%20libraries&text= ``` Optionally append `&source=` such as `claude.ai` or `chatgpt`. `page_path` and `text` are required; `page_title` and `source` are optional. Responses: `200 {"feedback_id": ""}`, `400` for malformed query parameters, and `429` when rate-limited.