# Data Warehousing How-to
> Data Warehousing How-to guides
This guide is tailored for those building a production data warehouse for organizational use. It covers the decisions and trade-offs involved rather than step-by-step mechanics, with each section linking out to detailed documentation.

If you want to explore MotherDuck first, check out our [getting started tutorials](/getting-started/). Alternatively, if you're using MotherDuck to provide customer-facing analytics through an app or SaaS platform, read our [customer-facing analytics guide](/key-tasks/customer-facing-analytics/) for specific considerations.

## Introduction to MotherDuck for data warehousing

MotherDuck is a serverless cloud data warehouse built on [DuckDB](https://duckdb.org/docs/sql/introduction), a fast, in-process analytical database. While DuckDB provides the core analytical engine capabilities, MotherDuck adds cloud storage, sharing, and collaboration features, as well as built-in, AI-powered data pipeline and visualization tools.

MotherDuck is built for analytical workloads and batch operations. It's not optimized for high-frequency small transactions the way an OLTP database is, so keep your transactional system where it is and replicate your data into MotherDuck for analysis. Streaming sources should be buffered, through Kafka, Kinesis, SQS, or similar, and loaded as micro-batches rather than written row by row.

Diagram summary: the MotherDuck ecosystem. Sources and ingestion tools load data into MotherDuck; BI tools, notebooks, and applications query it back out; transformation, orchestration, and data quality tools work on it. Each lane below is a sample, not the full list.

Loading data in:
- **Databases**: PostgreSQL, MySQL, SQL Server, Snowflake, BigQuery, MongoDB (/integrations/databases/)
- **Files and object storage**: Amazon S3, Cloudflare R2, Azure Blob, Parquet, Apache Iceberg, DuckLake (/integrations/cloud-storage/)
- **Ingestion**: dltHub, Fivetran, Airbyte, Estuary, Sling, Apache Kafka, Salesforce, Stripe (/integrations/ingestion/)

Reading data out:
- **Business intelligence**: Power BI, Tableau, Metabase, Hex, Omni, Superset, Rill Data, Evidence (/integrations/bi-tools/)
- **Data science and AI**: Python, Jupyter, Marimo, Google Colab, LangChain, LlamaIndex (/integrations/data-science-ai/)
- **Apps and developer tools**: Node.js, Go Driver, JDBC, Rust, Streamlit, Retool, Cloudflare Workers, DBeaver (/integrations/language-apis-and-drivers/)

Working on the warehouse:
- **Transformation**: dbt core, dbt cloud, SQLMesh, Paradime (/integrations/transformation/)
- **Orchestration**: Airflow, Dagster, Prefect, Kestra, Bacalhau, Astronomer (/integrations/orchestration/)
- **Data quality**: Great Expectations, Monte Carlo, Soda, Secoda (/integrations/data-quality/)

What MotherDuck brings in the middle:
- **Ducklings**: Compute, one per account (/concepts/hypertenancy/)
- **Storage**: Databases, snapshots, catalog (/concepts/storage-lifecycle/)
- **Dual Execution**: Query cloud and local data together (/concepts/architecture-and-capabilities/#dual-execution)
- **Flights**: Python on a schedule (/concepts/flights/)
- **Dives**: SQL and React data apps (/key-tasks/dives/)
- **Guides**: Context your agents read (/key-tasks/guides/)
- **MCP server**: Agent access to your data (/sql-reference/mcp/)
- **Shares**: Read access without copying (/key-tasks/sharing-data/)

The full list is at /integrations/.

MotherDuck integrates with the tools you already use across [ingestion](/integrations/ingestion/), [transformation](/integrations/transformation/), [orchestration](/integrations/orchestration/), and [BI](/integrations/bi-tools/), as shown above. The sections below cover choosing between them.

## MotherDuck basics: concepts to understand before you start

MotherDuck separates storage from compute. Data lives in a managed storage layer, while queries run on a "Duckling", our term for an isolated, single-node compute instance. There is no cluster to coordinate or spin up, so queries start immediately.

Every user, service account, and agent gets its own Duckling, a model called [hypertenancy](/concepts/hypertenancy/). Because compute is allocated per account rather than shared, workloads on separate accounts never compete for resources and each account can be sized independently. It also means that each database is limited to one write user, but it can have many read users. This concept underpins how you architect a data warehouse on MotherDuck, and we'll come back to it later.

```mermaid
flowchart LR
    AN{{"Analyst"}}:::green
    SA{{"Service account"}}:::green
    AG{{"AI agent"}}:::green

    subgraph MD["MotherDuck"]
        subgraph Compute["One Duckling per account"]
            D1["Duckling"]:::yellow
            D2["Duckling"]:::yellow
            D3["Duckling"]:::yellow
        end
        STORE[("Managed storage")]:::yellow
    end

    EXT["External sources<br/>S3, Iceberg, DuckLake"]

    AN --> D1
    SA --> D2
    AG --> D3
    D1 -->|"read"| STORE
    D2 -->|"write"| STORE
    D3 -->|"read"| STORE
    Compute -->|"query in place"| EXT
```

MotherDuck also runs a [Dual Execution model](/concepts/architecture-and-capabilities/#dual-execution): a single query can read data stored in MotherDuck alongside local files and external cloud storage, with the planner intelligently deciding whether to process data locally, in the cloud, or using a hybrid approach. You don't have to load everything into MotherDuck before you can query it, which matters when you're deciding what to ingest and what to leave in place.

## Prerequisites: getting set up on MotherDuck

### Create a MotherDuck organization

If you haven't already, create a [MotherDuck organization](/key-tasks/managing-organizations/) to house your data warehouse. Billing, region selection, and role-based controls are all configured at the organization level.

Your organization is created in a single [cloud region](/about-motherduck/cloud-regions/), chosen at sign-up. Databases are scoped to that region, and moving an organization to a different one means creating a new organization and reloading your data. Pick the region closest to your source systems and your users, and check it satisfies any data residency requirements you have before you start loading data.

### Authentication and connecting to MotherDuck

You and your organization's users can [authenticate to MotherDuck](/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck/) manually through the UI, with access tokens for anything run outside it (Python, CLI, clients, and service accounts), or through [Single Sign On](/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck/sso-setup/) on Business and Enterprise plans.

### Assign user roles

MotherDuck uses [role-based access control](/concepts/roles-and-access-control/), with three pre-configured roles that control what users can do in the platform:

| Role | Purpose | Typical users |
| --- | --- | --- |
| **Admin** | Govern the organization: billing, users and roles, SSO and SCIM, and service accounts, plus all Builder capabilities. | Platform owners, usually 1-3 people. |
| **Builder** | Produce data: databases, shares, Ducklings, service accounts, Flights, and Dives. | Data engineers and analytics engineers. |
| **Explorer** | Consume data: run SQL, attach shares, manage secrets, and create their own Dives and Flights. | Analysts and business users. |

Whoever creates the organization holds the Admin role by default and manages everyone else's. Be selective about granting it: most people working with your warehouse need only Builder or Explorer. [Custom roles](/key-tasks/managing-organizations/managing-roles/#create-a-custom-role) are available on the Business and Enterprise plans.

### Create service accounts

A service account is a non-human identity for workloads that connect to MotherDuck without a person's credentials: scheduled pipelines, BI connections, and backend services. Each has its own access token and its own Duckling, so different service accounts don't compete for compute.

[Creating a service account](/key-tasks/service-accounts-guide/create-and-configure-service-accounts/) requires the Admin or Builder roles, both roles can also [impersonate service accounts](/key-tasks/service-accounts-guide/impersonate-service-accounts/) to collaborate on the databases and objects it owns. Create production databases with a service account, and run scheduled or shared workloads as one rather than as a person, so that databases and core workloads survive staff changes.

### Connect the MCP server

The [MotherDuck MCP server](/sql-reference/mcp/) lets AI clients query and explore your databases using the Model Context Protocol, and works with any MCP-compatible client. Connect it early: it's the primary interface for creating both Flights and Dives, and several sections below assume you have an AI client connected.

[MotherDuck Skills](/key-tasks/ai-and-motherduck/agent-skills/) give agents procedural knowledge for working with MotherDuck, and [Guides](/key-tasks/guides/) capture the domain knowledge that isn't visible from a schema, such as what columns to join on. Develop your Guides as the warehouse evolves rather than treating them as one-time setup.

## Designing your MotherDuck data warehouse

MotherDuck follows a familiar hierarchical structure with databases containing schemas and tables. Databases serve as the primary unit of organization and access control, while schemas help logically group related tables together. This structure provides a clean way to organize data while maintaining compatibility with common [SQL patterns](https://duckdb.org/docs/sql/introduction) and tools.

Going back to the concepts we described earlier, there are two properties of our hypertenant architecture that drive how you design a warehouse:
- **Each database has a single write user**, the database owner.
- **Compute is provisioned on a per-user basis**, with every user and service account having their own configurable Duckling.

Together these mean that one service account per workload isolates your pipeline stages at both layers. Each writes only to the database it owns, and runs on its own compute. For a typical warehouse:

| Service Account | Owns database | Holds | Reads from |
|---|---|---|---|
| `raw_sa` | `raw` | Source data landed by ingestion | External sources |
| `transform_sa` | `transform` | Models built by transformation | `raw` |
| `marts_sa` | `marts` | Curated tables for consumers | `transform` |

Each stage reads from the one below it and writes only to its own database, so a heavy transformation run can't slow down ingestion or starve a BI query.

If one component of a workload is significantly heavier than the others, you can isolate it by creating a dedicated service account and database. A single high-volume source, like clickstream or event data, could get its own service account and database rather than sharing `raw_sa` with everything else. That lets you size its Duckling for the job it actually does, and stops it slowing down the rest of your ingestion while it runs.

The same data and compute isolation approach extends to [environment management](/key-tasks/data-warehousing/environment-management/): separate dev, staging, and production service accounts and databases mean a development run can't write to production data or compete with it for compute.

### Compute scaling

As each user and service account gets its own Duckling, its compute can be [scaled](/concepts/scaling-patterns/) to match the workload it runs. Each [Duckling size](/about-motherduck/billing/duckling-sizes/) has different performance characteristics and billing implications. As a starting point, we recommend [Pulse](/about-motherduck/billing/duckling-sizes/#pulse) for spiky, read-only workloads, and [Standard](/about-motherduck/billing/duckling-sizes/#standard) or larger for ingestion and transformation. You can rescale a Duckling at any time, so this is something you can experiment with to get the best results.

Heavy concurrent read workloads, like BI tool querying, can be horizontally scaled using [read scaling](/key-tasks/authenticating-and-connecting-to-motherduck/read-scaling/), which can also be adjusted at any time.

## Ingesting data into MotherDuck

MotherDuck provides multiple strategies for [ingesting data](/key-tasks/loading-data-into-motherduck/) into your data warehouse, including [Flights](/concepts/flights), our built-in tool for running Python on a schedule inside MotherDuck. Following the ownership pattern above, we recommend that scheduled ingestion jobs are run using a service account. Any cloud storage secrets it needs should be created by that account.

Which approach fits depends on what the source is, where it resides and whether you need to ingest it on a regular cadence:

| Approach | Fits when |
|---|---|
| **Flights** | Sources are bespoke or need custom logic, you want the flexibility of Python, you want to be able to quickly develop and iterate using an agent, or you want to avoid deploying and maintaining additional infrastructure. |
| **Direct queries** | Data already sits locally or in cloud storage in a queryable format, needs little or no transformation, or only needs to be refreshed on an ad-hoc basis. Can handle a wide range of file formats and dataframes. |
| **Third-party tools** (Fivetran, Airbyte, Estuary) | Sources are commodity SaaS applications with existing connectors, and you don't want to handle the maintenance. |

These aren't exclusive. Most warehouses end up combining them, with a managed connector for commodity SaaS sources, Flights for anything bespoke and direct querying for ad-hoc analytics on local files.

### Flights
A [Flight](/concepts/flights/) is a Python program that runs on MotherDuck, on demand or on a recurring schedule. There's no runner to provision and no infrastructure to host: MotherDuck runs the Python alongside your databases rather than shipping data out to a separate runtime, and injects the access token so you never embed credentials in the pipeline.

Flights are quick to iterate on. Describe the job in natural language to an agent through the [MCP server](/sql-reference/mcp/), and it writes the Python, sets the schedule, and runs it. You can then review, edit, and manage the result as code. You get the full flexibility of Python with it: any PyPI package, any custom logic, and [DuckDB community extensions](https://duckdb.org/community_extensions/list_of_extensions) that aren't available on MotherDuck by default.

For ingestion, Flights pull from a wide range of external sources (Postgres, BigQuery, Snowflake, S3, APIs) and write to MotherDuck tables. [dltHub](/integrations/ingestion/dlt/)'s dlt library is the recommended ingest library: it gives you a declarative pipeline with schema evolution, incremental loading, and a MotherDuck destination. Flights aren't limited to ingestion though, and can do whatever Python can. See the detailed [guide](/key-tasks/flights/) or the [Flights Cookbook](/cookbook/?feature=flights) for more.

### Loading local files
MotherDuck supports loading data from your [local machine](/key-tasks/loading-data-into-motherduck/loading-data-from-local-machine/) using common file formats like CSV, Parquet, and JSON. The [MotherDuck UI](/getting-started/interfaces/motherduck-quick-tour/) provides an intuitive interface for uploading files directly, the [MotherDuck CLI](/getting-started/interfaces/motherduck-cli/) lets you drive from your terminal, and the [Python client](https://duckdb.org/docs/api/python/overview) enables programmatic loading using DuckDB's native functions. For example, you can use [`read_csv()`](https://duckdb.org/docs/data/csv), [`read_parquet()`](https://duckdb.org/docs/data/parquet), or [`read_json()`](https://duckdb.org/docs/data/json) to efficiently load data files while taking advantage of DuckDB's parallel processing capabilities.

Local [DuckDB databases](/key-tasks/loading-data-into-motherduck/loading-duckdb-database/) can be attached and imported efficiently using the CLI since MotherDuck is built on DuckDB's core engine.

### Interacting with cloud storage (S3, GCS, etc)
[Cloud storage integration](/key-tasks/loading-data-into-motherduck/loading-data-from-cloud-or-https/) lets you directly query and load data from major providers including [AWS S3](https://duckdb.org/docs/guides/import/s3_import), [Google Cloud Storage](https://duckdb.org/docs/guides/import/gcs_import), [Azure Blob Storage](https://duckdb.org/docs/stable/extensions/azure), and [Cloudflare R2](https://duckdb.org/docs/guides/import/s3_import). Using SQL commands like `SELECT FROM read_parquet('s3://bucket/file.parquet')`, you can seamlessly access cloud data.

Cloud storage credentials are stored as secrets in MotherDuck using DuckDB's `CREATE SECRET` syntax. Secrets are scoped to the user that creates them, so create them under the service account that runs your ingestion.

### Replicating from transactional databases
MotherDuck supports importing data from OLTP databases like [PostgreSQL](https://duckdb.org/docs/guides/import/query_postgres.html) and [MySQL](https://duckdb.org/docs/guides/import/query_mysql). You can directly connect to these sources using database connectors and execute queries to extract and load data. See our [guide](/key-tasks/loading-data-into-motherduck/loading-data-from-postgres/) for more detail.

### Third-party ingestion tools
Tools like [dltHub, Fivetran, Airbyte, and Estuary](/integrations/ingestion/) integrate with MotherDuck to provide automated, reliable data pipelines. These tools handle complex ETL workflows, data validation, and transformation while offering features like scheduling, monitoring and error handling that simplify ongoing data operations.

For near real-time data needs, MotherDuck works with streaming partners like [Estuary](https://docs.estuary.dev/reference/Connectors/materialization-connectors/motherduck/) to enable continuous data ingestion. While DuckDB is optimized for batch operations, these integrations allow you to build streaming pipelines that buffer and load data in micro-batches for near real-time analytics.

### Unstructured data integrations
When working with unstructured data like documents, emails or images, tools like [Unstructured.io](https://motherduck.com/blog/effortless-etl-unstructured-data-unstructuredio-motherduck/) can pre-process and structure the data before loading into MotherDuck. This lets you analyze unstructured data alongside your structured data warehouse tables.

### Loading performance notes
Batch size, data types, and sort order all affect load performance and cost. Load data in reasonably sized batches to balance memory usage and throughput. See our full guide on [recommended practices for loading data](/key-tasks/loading-data-into-motherduck/considerations-for-loading-data/), and monitor recent queries using [`MD_INFORMATION_SCHEMA.QUERY_HISTORY`](/sql-reference/motherduck-sql-reference/md_information_schema/query_history/) during large loads.

## Data transformation: shaping your data for analysis

Data transformation is a critical step in the data warehousing process that converts raw data into analysis-ready formats. MotherDuck provides powerful SQL capabilities inherited from DuckDB for transforming data directly within the warehouse. You can leverage DuckDB's rich library of SQL functions to clean, reshape, and model your data through operations like filtering, joining, aggregating and pivoting.

### Transformation tools
**[Flights](/concepts/flights)**
- Describe the transformation in natural language and MotherDuck generates the Flight.
- The runtime for scheduled transformations: dbt projects can [run from a Flight](/key-tasks/flights/run-dbt-transformations-from-a-flight/).
- Handles transformations SQL can't express on its own using any PyPI package.
- Quick to stand up for development or ad-hoc work, and the same Flight can be promoted to a scheduled production job once it's reviewed.

**[dbt Core](/integrations/transformation/dbt/) & [dbt Cloud](/integrations/transformation/dbt-cloud/)**
- [`dbt-duckdb`](https://github.com/duckdb/dbt-duckdb) adapter for seamless integration.
- Enables version controlled, modular SQL transformations.
- Supports testing, documentation and lineage tracking.
- Recommended for complex transformation workflows.
- See our [blog post](https://motherduck.com/blog/duckdb-dbt-e2e-data-engineering-project-part-2/) for detailed examples.

**[SQLMesh](https://sqlmesh.readthedocs.io/en/stable/integrations/engines/motherduck/)**
- [Compatible with MotherDuck](/integrations/transformation/sqlmesh/) through DuckDB support.
- Provides data pipeline and transformation management.
- Enables incremental processing and scheduling.

**[Paradime](https://docs.paradime.io/app-help/documentation/settings/connections/scheduler-environment/duckdb)**
- [Integrates](/integrations/transformation/paradime/) by connecting dbt development environments with MotherDuck.
- Offers collaborative development environment.
- Includes version control and deployment tools.

## Orchestration: automating your data pipelines

Keeping your warehouse current means running ingestion and transformation on a schedule. How much of your pipeline lives inside MotherDuck determines which tool fits. Whichever you choose, the pipeline should authenticate as the service account that owns the database it writes to, using an access token stored in your external secret manager rather than a personal credential.

For orchestrating MotherDuck workflows, you have several options:

Flights run Python on a cron schedule inside MotherDuck, with no runner or infrastructure to host. If your pipeline is contained within MotherDuck, a [scheduled Flight](/key-tasks/flights/scheduling-and-runs/) is usually the simplest path. Flights schedule individual programs rather than managing dependencies between tasks, so pipelines with fan-in or cross-system dependencies are better served by a dedicated orchestrator.

For simple scheduling outside MotherDuck, cron jobs or [GitHub Actions](/key-tasks/data-warehousing/orchestration/github-action-cron/) can effectively orchestrate data pipelines, and are a good fit when the schedule already lives alongside your code in CI.

Popular workflow orchestration platforms like [Airflow, Dagster, Kestra, Prefect and Bacalhau](/integrations/orchestration/) provide robust scheduling, dependency management and monitoring capabilities. Reach for these when you have dependencies spanning multiple systems, need backfills and retries as first-class concepts, or want lineage across your whole pipeline.

Some ingestion and transformation tools also schedule their own runs (dbt Cloud, Fivetran, and Airbyte), which may be enough if your pipeline is a single tool end to end.

When orchestrating MotherDuck pipelines, follow these best practices:
- Design idempotent jobs that can safely re-run without duplicating or corrupting data.
- Implement proper error handling and retries to gracefully handle temporary failures.
- Set up logging and monitoring to maintain visibility into pipeline health and performance.

## Consuming your data

Once your data is modelled, consumers need read access to it. BI tools and applications connect as service accounts with read access, while people and agents query through their own accounts. Shares are the mechanism for granting that access, including to recipients outside your organization.

### Data sharing and collaboration
MotherDuck implements a data sharing model through [SHARES](/key-tasks/sharing-data/), which provide read-only access to specific databases. To [create a share](/key-tasks/sharing-data/sharing-within-org/), use the `CREATE SHARE` command or the UI, and specify the databases you want to share. You can further restrict data access to specific tables or schemas using [table-level security](/key-tasks/sharing-data/table-level-security/). Recipients can then access the shared data through their own MotherDuck account while maintaining data isolation.

### Agentic and natural language querying
Once connected, the MCP server is how people and agents query MotherDuck in natural language. We provide a set of [MotherDuck Skills](/key-tasks/ai-and-motherduck/agent-skills/) and an in-warehouse context layer ([Guides](/key-tasks/guides/)) to help agents deliver the best results. Our hypertenant, single-node architecture is particularly well suited to agentic querying as agents query in rapid bursts of small, exploratory reads. Each agent gets its own Duckling, so there's no cluster to spin up and no queue behind other users.

Together, natural language querying, hypertenancy, and share-level access control make it practical to open the warehouse to business users. New users get their own compute and their own space to work in, so they can't slow anyone down or write into your production databases.

### Dives
[Dives](/key-tasks/dives/) are interactive visualizations you create with natural language using the MotherDuck MCP server, directly on top of your data in MotherDuck. Ask your AI agent a question, and MotherDuck generates a persistent, interactive component that lives in your workspace alongside your SQL. Dives are React and SQL based data apps, so they can act as more than just a BI tool. We have a wide range of examples in our [Dives Gallery](https://motherduck.com/dive-gallery/) to get inspired and see what's possible. You can also read about [how we replaced our BI tool](https://motherduck.com/blog/replacing-our-bi-tool-with-dives/) with Dives.

Any user with access to data in the warehouse and an MCP connection can create a Dive, and Dives can be shared across your organization, provided recipients have read access to the underlying data. [Dive Statuses](/key-tasks/dives/dive-statuses/) let you flag which versions are work in progress, ready to use, and trusted, and Dive development can be version tracked or [managed as code](/key-tasks/dives/managing-dives-as-code/).

### External BI tools
Most BI tools connect to MotherDuck through DuckDB's JDBC or ODBC drivers, or through a native connector where one exists. Tableau, Power BI, Omni, Metabase, Preset/Superset, and Rill are all supported — see [BI tool integrations](/integrations/bi-tools/) for connection details for each.

Three things are worth deciding before you connect:
- **Which account it connects as.** A BI tool should authenticate as a service account with read access to your marts, not as a person. That keeps the connection working when people join or leave, and gives dashboard queries their own, scalable compute.
- **Whether to scope the connection.** By default, a connection attaches every database in the workspace, which some BI tools handle badly. Appending [`?attach_mode=single`](/key-tasks/authenticating-and-connecting-to-motherduck/attach-modes/) scopes it to one database and avoids the problem.
- **How many people query at once.** A single Duckling serves queries sequentially, so concurrent dashboard users can queue behind each other. [Read scaling](/key-tasks/authenticating-and-connecting-to-motherduck/read-scaling/) spreads those queries across a pool of Ducklings on read-only tokens, which lets you scale horizontally along with concurrency demands.

### Data science and ML
Python connects to MotherDuck through the [DuckDB Python package](/getting-started/interfaces/client-apis/python/), which loads the motherduck extension automatically when you open an `md:` connection string. There's no separate client to install, so anything that works with DuckDB works against your warehouse — pandas and Polars dataframes, Arrow, and the wider Python data stack.

That covers notebooks like Jupyter, Colab, and Marimo directly, and AI frameworks such as LangChain and LlamaIndex can use MotherDuck as a data source for retrieval and agent workflows. See [data science and AI integrations](/integrations/data-science-ai/) for the full list.

Data scientists benefit from per-account compute in the same way analysts do: an exploratory notebook running a heavy query doesn't compete with production pipelines, and anything written out lands in the data scientist's own database rather than production.

### Embedded and customer-facing analytics
MotherDuck can serve analytics inside your own product, including running DuckDB in the browser through [WebAssembly](https://duckdb.org/docs/lts/clients/wasm/overview) for interactive queries with near-zero latency. Dives can also be embedded in a web app. Customer-facing analytics involves a different set of trade-offs around tenancy, concurrency, and per-customer isolation — see the [customer-facing analytics](/getting-started/customer-facing-analytics/) guide for those.

## Running your warehouse

### Query performance and optimization in MotherDuck

MotherDuck inherits DuckDB's query optimization capabilities. You can analyze query performance using the `EXPLAIN` and `EXPLAIN ANALYZE` commands to view execution plans, surface detailed execution statistics, and identify bottlenecks. Execution plans also show which parts of a query ran locally and which ran in the cloud, which is how Dual Execution becomes visible.

While DuckDB doesn't use traditional [indexes](https://duckdb.org/docs/current/guides/performance/indexing), it automatically creates statistics and metadata to optimize query execution with row groups. As a result, sorting the data on insert is a very effective way to improve query performance.

If queries are slow across the board for a given user or service account after optimization, the cause is more likely Duckling sizing. We recommend that any ingestion, transformation or complex analytical workloads are run using a standard Duckling or larger.

### Monitoring and observability

The Duckling overview (**Settings** -> **Duckling overview**) provides a view of activity across every Duckling in the organization over the past 24 hours. For anything more specific, several system-defined views within [MD_INFORMATION_SCHEMA](/sql-reference/motherduck-sql-reference/md_information_schema/introduction/) provide metadata about your MotherDuck objects:

- [RECENT_QUERIES](/sql-reference/motherduck-sql-reference/md_information_schema/recent_queries/) provides a view of running and just-completed queries. Also visible in the UI (**Settings** -> **Running queries**).
- [QUERY_HISTORY](/sql-reference/motherduck-sql-reference/md_information_schema/query_history/) provides a consolidated view of all queries run (the Duckling overview is built using this).
- [STORAGE_INFO](/sql-reference/motherduck-sql-reference/md_information_schema/storage_info/) provides a comprehensive view of storage information for all your databases.

These are also what you'd use for periodic governance checks: reviewing who is querying what, and keeping an inventory of active shares so access granted for a one-off doesn't quietly persist.

You can create a custom observability Dive using the functions above if you need any additional always-on monitoring reporting beyond that provided on the [Settings](https://app.motherduck.com/settings) page.

### Cost tracking

You can [manage billing](/about-motherduck/billing/managing-billing/) and keep track of your costs from the [Billing page](https://app.motherduck.com/settings/billing) in the MotherDuck UI. Because compute is provisioned per account, costs attribute cleanly to the service account or user that incurred them.

Beyond right-sizing Ducklings, the main levers for cost optimization are storage and query efficiency: consider [data lifecycle](/concepts/storage-lifecycle/) policies to archive or delete old data, and monitor query patterns for unnecessary data processing.

## Further guides

## Included pages

- [GitHub Actions](https://motherduck.com/docs/key-tasks/data-warehousing/orchestration/github-action-cron): Schedule MotherDuck SQL and dbt jobs with GitHub Actions as a lightweight cron-based orchestrator.
- [PostgreSQL](https://motherduck.com/docs/key-tasks/data-warehousing/replication/postgres): Replicate PostgreSQL tables to MotherDuck using DuckDB and the PostgreSQL extension.
- [Dagster](https://motherduck.com/docs/key-tasks/data-warehousing/orchestration/dagster): Orchestrate an incremental S3-to-MotherDuck data loading pipeline with Dagster and Python.
- [SQL Server](https://motherduck.com/docs/key-tasks/data-warehousing/replication/sql-server): Replicate SQL Server tables to MotherDuck using Python and dataframes.
- [Environment management](https://motherduck.com/docs/key-tasks/data-warehousing/environment-management): Set up development, staging, and production data environments in MotherDuck with workload isolation, read-only access to production data, and dbt.
- [Flat Files](https://motherduck.com/docs/key-tasks/data-warehousing/replication/flat-files): Load CSV, Parquet, and JSON files into MotherDuck from local storage or cloud sources.
- [Excel and Google Sheets](https://motherduck.com/docs/key-tasks/data-warehousing/replication/spreadsheets): Load Excel and Google Sheets data into MotherDuck using the DuckDB CLI or HTTPS CSV export URLs.

### Further resources
To deepen your understanding of data warehousing with MotherDuck, explore our [concepts](/concepts/), which cover architectural principles and best practices. For hands-on examples, the free [DuckDB in Action eBook](https://motherduck.com/duckdb-book-brief/) provides real-world scenarios and solutions.

Please do not hesitate to **[contact us](https://motherduck.com/customer-support/)** if you need help along your journey.


---

## Docs feedback

MotherDuck accepts optional user-submitted feedback about this page at `GET https://motherduck.com/docs/api/feedback/agent`.
For agents and automated tools, feedback submission should be user-confirmed before sending.

URL-encode query parameter values and send a GET request:

```text
GET https://motherduck.com/docs/api/feedback/agent?page_path=%2Fkey-tasks%2Fdata-warehousing%2F&page_title=Data%20Warehousing%20How-to&text=<url-encoded user feedback, max 2000 characters>
```

Optionally append `&source=<url-encoded interface identifier>` such as `claude.ai` or `chatgpt`.

`page_path` and `text` are required; `page_title` and `source` are optional. Responses: `200 {"feedback_id": "<uuid>"}`, `400` for malformed query parameters, and `429` when rate-limited.