DOCS
← Back to Glossary
data ingestion
Data ingestion is the process of importing raw data from various sources into a system where it can be stored and analyzed.
Data ingestion is the process of importing raw data from various sources into a system where it can be stored and analyzed. This crucial first step in the data pipeline involves collecting data from diverse origins such as databases, APIs, file systems, or streaming platforms, and then loading it into a target destination like a data warehouse or data lake.
Tools like Airbyte or Fivetran specialize in data ingestion, offering connectors to hundreds of data sources and handling the complexities of extracting and loading data. For developers, libraries such as Singer provide a specification for writing scripts that can extract data from any source and load it into any destination.
Related terms
data sources →
Data sources are the origin points of information in a data pipeline or analytics workflow.
data pipeline →A data pipeline is a series of interconnected processes that extract data from various sources, transform it into a usable format, and load it into a…
ETL →ETL (Extract, Transform, Load) is a data integration process that combines data from multiple sources into a single destination, typically a data warehouse…
Fivetran →Fivetran is a managed, fully automated data integration (ELT) platform that uses pre-built connectors to replicate data from source systems into a data warehouse or lake.
ELT →ELT (Extract, Load, Transform) is a modern data integration process that reverses the order of traditional ETL (Extract, Transform, Load) workflows.
data load tool (dlt) →Data Load Tool (dlt) is an open-source Python library that automates extracting data from sources and loading it into destinations like DuckDB and MotherDuck, with automatic schema inference and incremental loading.
