Meltano

Overview

Meltano is an open-source data integration platform for building ELT pipelines using the Singer specification. It started in 2018 as an internal project at GitLab aimed at a broader end-to-end analytics platform, then narrowed its focus to ELT and spun out as an independent open-source project and company around 2021. Meltano's core idea is "data pipelines as code": sources (Singer taps), destinations (Singer targets), and transformations (typically dbt) are declared in version-controlled YAML files and run through a CLI, fitting naturally into existing software engineering practices like code review and CI/CD.

Core concepts

• Tap: a Singer-spec extractor that pulls data from a source and emits it as a stream of JSON records.
• Target: a Singer-spec loader that writes those records into a destination.
• Plugin: Meltano's term for any tap, target, or transformer it manages, discoverable via MeltanoHub, a public registry of Singer taps and targets.
• Project: a Meltano project directory containing meltano.yml, which declares plugins, pipeline schedules, and environment configuration.

Example

Copy code
meltano add extractor tap-postgres
meltano add loader target-jsonl
meltano run tap-postgres target-jsonl

Copy code
# meltano.yml (excerpt)
plugins:
  extractors:
    - name: tap-postgres
      config:
        host: db.internal
        dbname: app_prod
  loaders:
    - name: target-jsonl
schedules:
  - name: daily-sync
    interval: "@daily"
    job: tap-postgres-to-jsonl

Why it matters

Because it's built on Singer's open, JSON-over-stdout protocol, Meltano can use any conforming community tap or target without depending on a single vendor's connector catalog, and teams can write their own taps/targets when one doesn't exist. Meltano itself is orchestration-agnostic for scheduling at scale: its own scheduler suits simple cases, while production deployments commonly trigger meltano run from an external orchestrator like Airflow or Dagster. A common downstream pattern lands Meltano's output as files or a database that DuckDB or MotherDuck then queries directly for analysis.