Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data

LakeFS

LakeFS is one of the first tools to bring git-like versioning to object-storage-based data lakes. Its core approach is a metadata layer over object storage with immutable data and logical-to-physical address mapping on top of an object store such as a data lake, hence "lake" as part of the name.

It segregates data data/ with random physical addresses from its metadata _lakefs/, which includes range files, meta-range files, and commit information.

When you upload allstar_games_stats.csv to branch main, lakeFS generates a random physical address like s3://bucket/data/gp0n1l7d77pn0cke6jjg/cg6p50nd77pn0cke6jk0. This ensures immutability and files are never overwritten.

LakeFS operates as an S3-compatible gateway, intercepting read/write operations and managing versioning transparently. Applications interact with it like normal object storage while getting full Git semantics underneath.

The system implements a layered architecture:

1. Graveler: Core versioning engine managing branches, commits, and merges
2. Storage Adapter: Interfaces with S3/GCS/Azure
3. Hooks: Pre-merge and post-commit validation

LakeFS Architecture overview

Creating a branch from the CLI is as simple as this:

Copy code
lakectl branch create lakefs://quickstart/denmark-lakes --source lakefs://quickstart/main

The UI supports creating pull requests, or branches, literally like GitHub but for data. LakeFS interface, here an example of a Pull Requests

Check out their GitHub repo, documentation, or a practical example of Implementing a Write-Audit-Publish (WAP) Pattern for much more information.

Nessie

Nessie came out of Dremio and is another early adopter that has been doing this for a long time. Its core approach is a transactional catalog with Git-like versioning for Apache Iceberg and Delta Lake tables.

Rather than versioning data files, Nessie versions the catalog metadata, the registry of tables and their locations.

This separation enables zero-copy branching where branches share table metadata pointers, multi-table transactions with atomic commits across multiple tables, and Git semantics such as branch, tag, merge, and cherry-pick operations.

Nessie leverages the immutability of modern table formats with Iceberg:

1. Iceberg snapshots are immutable: Each table change creates new metadata.
2. Nessie tracks which snapshot each branch points to.
3. Branching copies pointers, not data or metadata files.
4. Merging updates pointers to replay changes from source to target.

Example workflow:

Copy code
# Create branch
catalog.create_branch('experiment', 'main')

# Modify table on experiment branch
spark.sql("INSERT INTO catalog.experiment.orders VALUES (...)")
# This creates new Iceberg snapshot, Nessie updates experiment pointer

# Main branch unchanged - still points to original snapshot
spark.sql("SELECT * FROM catalog.main.orders")  # Original data

Nessie runs as a REST service with pluggable backends including metadata storage such as PostgreSQL, DynamoDB, or RocksDB, data lake integration that works with any Iceberg-compatible engine (Spark, Trino, Dremio), and version control with a Git-like commit graph with branches and tags.

Nessie doesn't touch your data files. It's a lightweight coordination layer that brings Git semantics to your lakehouse by versioning the catalog. This makes it complementary to tools like lakeFS (which versions data) and ideal for multi-table transactional workflows. Read more on GitHub.

Bauplan

Similar to LakeFS, Bauplan calls itself the programmable data lake and is a code-native platform for versioned pipelines, built on Apache Iceberg and initially optimized for ML. It's not open source. Bauplan is built on a Python-first serverless lakehouse and is rather new.

Bauplan treats your data lake as a Git repository where:

• Data branches are first-class citizens, not just pipeline configs.
• Every pipeline execution is a commit with full lineage.
• All tables use Apache Iceberg format (Delta Lake compatible).

Architectural overview from Bauplan Website

Creating an isolated branch with new snapshots of Iceberg tables from the CLI is as simple as this:

Copy code
client.create_branch('experiment')  # Instant, zero data copying

It supports merging verified using Alloy model checking:

Copy code
client.merge_branch(source='experiment', target='main')

The way it works is that it integrates a commit's changes into another branch and uses Alloy, a lightweight model checker, to stress-test the core logic behind merging (also used for checking branching and commits).

The merge operation tries to detect conflicts at the table level, performs three-way merges for compatible changes, and creates merge commits preserving lineage. Find more info on Git-for-Data Semantics: Safe Branching & Merging at Scale or their implementation of the WAP pattern.

Bauplan brings Git's full semantic model with branch, merge, commit, and revert to lakehouse data while maintaining compatibility with standard Iceberg tables accessible from MotherDuck, Snowflake, Databricks, or Trino.

Supabase

Supabase's core approach is full instance branching. Each branch is a completely isolated Postgres database with the entire Supabase stack (Auth, Storage, Realtime, Edge Functions).

Supabase branches create separate environments that spin off from your main project, allowing you to test changes like new configurations, database schemas, or features without affecting production.

It works by creating a Git branch and opening a pull request. Supabase automatically launches a Preview Branch and runs migrations from the repository's migrations directory. Each branch gets a dedicated Postgres instance with a unique connection string and APIs, isolating them from production and other branches.

Creating a branch via GitHub integration:

Copy code
# Automatic with GitHub integration enabled
git checkout -b feature/new-reports
git push origin feature/new-reports
# Supabase automatically creates preview branch when PR is opened

Or via the CLI:

Copy code
supabase branches create feature-branch --project-ref your-project

When merging, migrations in the repository's migrations folder run incrementally on each commit, allowing you to verify schema changes on existing seed data. When you merge the PR, those migrations automatically apply to production.

As each branch is a new Postgres instance created from scratch, the approach is conceptually simple but requires branches to be seeded (manually populated with test data since production data isn't copied) with data since they start empty. Each branch incurs its own compute and storage costs. Read more on Branching Supabase Docs.

Ideal for full-stack development where you need the entire backend stack (database + auth + storage + functions) to test features end-to-end.

Neon

Neon is a serverless Postgres platform (now part of Databricks) whose core approach is copy-on-write storage-level branching. Unlike Supabase which spins up a full new instance, Neon branches at the storage layer, making them instant regardless of database size and including the actual data.

Each branch is a new timeline in Neon's custom storage engine. No data is physically copied. The branch simply starts from a pointer to the parent's state at a specific LSN (log sequence number). Pages only diverge when writes happen, so you're billed only for the delta.

Copy code
# Create a branch from the CLI
neon branches create --name feature/user-auth

# Branch from a specific point in time
neon branches create --name recovery --parent 2025-01-15T10:00:00Z

Neon also supports snapshots (named, immutable point-in-time saves, like git tags) and rollback via finalize_restore: true, which restores a snapshot onto the active branch in-place while preserving the stable connection string. There's no reconfiguration needed. For safe experimentation, finalize_restore: false creates a temporary preview branch instead.

The key limitation: Neon has no merge support. Branches diverge but can't be reconciled automatically. Changes are applied back to production using standard migration tools.

Ideal for database-focused workflows where you want instant, full-data branches with production-like data out of the box, and don't need the full backend stack.

Dolt: Git + MySQL

Dolt is a SQL database that you can fork, clone, branch, merge, push, and pull just like a Git repository. It's a MySQL-compatible database and is fully open-source. Dolt's core approach is a SQL database where every row is versioned, combining Git's commit graph with MySQL's query interface.

Dolt stores data in a content-addressed graph using Prolly Trees, a novel data structure that enables cell-level version history, efficient structural sharing between versions, and fast diffs and merges.

Every database operation can be committed with:

Copy code
INSERT INTO employees VALUES (1, 'Alice', 50000);
SELECT DOLT_COMMIT('-am', 'Add Alice to payroll');

The commit creates a snapshot of the entire database state at that moment, stored in the commit graph just like Git. Unlike traditional databases, you can diff any two versions:

Copy code
-- See what changed between commits
SELECT * FROM DOLT_DIFF('main', 'feature-branch', 'employees');

-- Show cell-level changes
SELECT * FROM DOLT_COMMIT_DIFF_employees WHERE from_commit='abc123' AND to_commit='def456';

This enables cell-level audit trails with diffs showing which rows were added/deleted/modified, which cells changed with their before/after values, and who made the change via commit metadata.

Dolt implements Git commands almost literally. You can run dolt with any of these commands: branch feature-123, checkout feature-123, add ., commit -m "Add new customers", push origin feature-123, checkout main, merge feature-123.

You can even push/pull to DoltHub (like GitHub for databases) or run Dolt as a MySQL replica for existing applications.

Dolt uses copy-on-write with structural sharing where unchanged rows are shared between branches via pointers, and modified rows create new leaf nodes in the Prolly Tree.

This means cloning isn't "free" like with lakeFS, but it provides true database semantics with ACID transactions.

There's much more. Read more on their GitHub.

MotherDuck

MotherDuck, as a cloud data warehouse, implements versioning differently from dedicated Git-for-data tools, prioritizing operational convenience over full version control semantics. With the addition of named snapshots, it gets even closer to Git-like semantics.

It offers two types of snapshots. Automatic snapshots: Created continuously in the background (roughly every minute when no writes are active). These are governed by SNAPSHOT_RETENTION_DAYS. These are configurable up to 90 days on the Business plan, defaulting to 7 days. They provide point-in-time recovery without any manual intervention.

And named snapshots that you create explicitly with CREATE SNAPSHOT. These are not subject to garbage collection as they persist indefinitely, even if the source database is deleted. Think of them as Git tags for your database, a permanent bookmark of a known-good state you can always return to.

The git analogy maps well:

1. CREATE SNAPSHOT → git tag: bookmark a known-good state
2. CREATE DATABASE ... FROM → git checkout -b: isolated environment from a snapshot
3. ALTER DATABASE SET SNAPSHOT TO → git reset --hard: roll back to a previous state
4. UNDROP DATABASE → recovering a deleted branch

Combined with zero-copy cloning and database sharing, this enables practical git-like workflows. While MotherDuck doesn't support Git-style merging, COPY FROM DATABASE (OVERWRITE) acts as a replace, somewhat like a merge without conflict resolution. Combined with snapshots and zero-copy clones, this gives you a practical branch-modify-promote workflow:

Copy code
-- 1. Snapshot production before changes (persists indefinitely)
CREATE SNAPSHOT 'pre_release_v2' OF production;

-- 2. Clone from that named snapshot to an isolated dev database (instant, zero-copy)
CREATE DATABASE dev_branch FROM production (SNAPSHOT_NAME 'pre_release_v2');
-- Or clone from a point in time: (SNAPSHOT_TIME '2026-01-28 08:00:00')

-- 3. Make and validate changes on dev_branch
-- ... run transforms, test queries ...

-- 4. Promote: overwrite production with dev_branch (instant, metadata-only)
COPY FROM DATABASE dev_branch (OVERWRITE) TO production;

-- 5. If something goes wrong, restore from snapshot
ALTER DATABASE production SET SNAPSHOT TO (SNAPSHOT_NAME 'pre_release_v2');

This operates purely at the metadata layer and is nearly instantaneous. It's not a true merge (it's a full replacement, not a diff-based reconciliation), but for many data workflows where you want to validate changes in isolation before promoting them, it covers the key use case.

DuckLake

DuckLake is the open lakehouse format that uses a SQL database as its metadata catalog instead of JSON/Avro manifest files. DuckLake is relatively new (with 1.0 around the corner and its first release in May 2025), so you could use other mature open table formats like Apache Iceberg, Delta Lake or Apache Hudi.

But DuckLake has its relevancy for git-like workflows because:

1. Snapshots are Git commits: Every DuckLake change creates a snapshot with author, commit message, and changeset tracking. This is the closest to actual Git semantics in the data lake world.
2. SQL-native metadata: Uses DuckDB/PostgreSQL/MySQL as catalog, so metadata operations are standard SQL transactions. No manifest file scanning or compaction storms like Iceberg.
3. Millions of snapshots: Snapshots are just a few rows in the catalog DB. No need to proactively prune snapshots (a major operational burden with Iceberg).
4. Time travel + change feed: Query any table at any version, track insertions/deletions between versions.

With MotherDuck (fully managed):

Copy code
-- Fully managed DuckLake on MotherDuck
CREATE DATABASE my_lake (TYPE DUCKLAKE);

-- Or bring your own S3 bucket
CREATE DATABASE my_lake (TYPE DUCKLAKE, DATA_PATH 's3://my-bucket/lake/');