DOCS
← Back to Glossary
pipelines
Data pipelines are automated workflows that move and transform data from various sources to one or more destinations.
Data pipelines are automated workflows that move and transform data from various sources to one or more destinations. They typically consist of interconnected steps or tasks that extract data from its origin, apply transformations or cleansing operations, and load the processed data into a target system for analysis or storage. Modern data pipelines often leverage tools like Apache Airflow, Dagster, or Prefect to orchestrate these workflows, ensuring data flows smoothly and reliably through an organization's data infrastructure. Pipelines can handle batch processing of large datasets or facilitate real-time streaming of data, depending on the requirements. They play a crucial role in maintaining data quality, consistency, and timeliness across different systems and applications within a data ecosystem.
Related terms
data pipeline →
A data pipeline is a series of interconnected processes that extract data from various sources, transform it into a usable format, and load it into a…
data sources →Data sources are the origin points of information in a data pipeline or analytics workflow.
ELT →ELT (Extract, Load, Transform) is a modern data integration process that reverses the order of traditional ETL (Extract, Transform, Load) workflows.
ETL →ETL (Extract, Transform, Load) is a data integration process that combines data from multiple sources into a single destination, typically a data warehouse…
Apache Airflow →Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows, where pipelines are defined as directed acyclic graphs (DAGs) in Python.
Dagster →Dagster is an open-source data orchestration platform designed to help data engineers and scientists build, test, and monitor data pipelines.
