The Problem We Have

When you have multiple stages in your data engineering architecture, from stage -> core -> data marts, potentially a cube on top, and multiple data pipelines running in parallel, the problem is rolling back an error consistently across the data stack.

How do we do that? We can't just revert one table, the sales table for example, because it will not work with all related customers, as they might have changed in the meantime, or the products, or their location, or gone out of business.

Data might be stored on a data lake, maybe on a database, or a key-value store like Redis. The data might be huge, containing the full CRM or all sales transactions over the last years. So the question is, how do we revert or test things consistently based on production-like data in terms of both quality and size?

That's where Git for data came from, and where tools such as LakeFS, Nessie, Bauplan and approaches like branching found a way to do it for a dedicated spot or across the stack.

The Goal: What to Expect from a Git-like Workflow for Data?

We want to explore how Git can be integrated with data storage solutions like data lakes and databases to enable branching, cloning, and other Git-based functionality, which is what we already use for code in the realm of data engineering.

For this, we need to investigate how the full data stack, such as orchestration or transformation tools in the data workflow, can leverage Git-based versioning and branching. We need a strategy for scaling the Git-based data workflow to handle large production datasets without the extra work of copying or backup processes of existing databases + code + environment variables manually as we do today.

We want Git for data management, similar to how it is used for code, to facilitate deployment and testing. As we have the code packaged, we can package it into a Docker file and run it on that set of data.

Imagine if we could have Git management end-to-end on data analytics, a fully integrated Git workflow. If we only have the data, that already helps because then we can run a set of code. The end-to-end integration would be nice, but not the most important. Data is the hard part here. So if we are able to achieve that, we already win big by avoiding all the test cycles and CI/CDs we need to wait for or gain stability by quickly testing before deploying to production.

The hard part is scaling Git for data, especially in large-scale production environments where production data is really large. Copying this data and even adding some additional jobs with tests or setting up environments and integrations on top might take hours, or even the full night, if no errors occur.

Let's find out what existing tools in the market figured out and what efficient ways there are to scale Git for data without the downsides.

Why not Plain "Git"?

Literally using Git doesn't work well for data. We get line-level conflict resolution (not cell-level). Git has no concept of schema. The files need to be sorted the same way to get useful diffs, and it has a 100MB GitHub file limit.

Git isn't made for data itself, but for small, text-based code changes. On top of that, in data work we differentiate between these two categories:

• Data pipeline versioning -> transformations or code
• Versioned databases -> the actual data

How Does Git Work?

To understand Git for data, we need to understand how branching with Git works, so we can apply it to data.

For example, Git branching holds all metadata and changes of the code from each state. This is handled through hashes. But Git is not made for data because it was designed with code versioning in mind, not large binary files or datasets. As Linus Torvalds himself noted, as the creator of Git, large files in Git were never part of the intended use case. The system's architecture of storing complete snapshots and computing hashes for everything works well for text-based code but becomes unwieldy with large data files. But as data practitioners, we actively want to work with data, with state, which is always harder than just code.

Git and Git-like solutions (alternatives are Tangled and Gitea) work. But which of these features do we want for data? And which specific ones do we need more compared to versioning code?

Git has concepts like versioning, rollback, diffs, lineage, branch/merge, and sharing. On the data side, which we get into more later, we have concepts such as files vs tables, structured vs unstructured, schema vs data, branching, and time travel.

For data, we need a storage layer or a way optimized for large data, schemas, and column types without necessarily duplicating the data. We also need to be able to revert the code and state easily. For example, revert the data pipelines that put production in an incorrect state.

If we look at The Struggle of Enterprise Data Integration, we can see that lots of what enterprises struggle with in data is change management and managing complexity. So hopefully, Git for data will help us with this?

How Does It Work with Data?

Data works differently. We need an open way of sharing and moving data that we can then version, branch off to different versions easily, and roll back to older versions.

Source: Git for Data - What, How and Why Now?

Branching is the right word, also what Git is doing:

Copy code
                 E---F---G  experiment-spark-3
                /
           C---D  dev-testing
          /
main  A---B---H---I  production
                 \
                  J---K  hotfix-corrupted-sales

We start with a version, and then diverge into different versions, and potentially merge back. Merging different branches is one option we won't need for data compared to code. With code, different features can be developed independently and then merged into the main branch at the end. With data, it's more about testing prod data on dev and then rolling out the code changes to prod, but not merging the "test" branch with the prod branch; otherwise we change, duplicate, or corrupt data.

The LakeFS solution (more on how it works later down) and its implemented Git-like features: Source: Git for Data - What, How and Why Now?

Tigris's new Fork capabilities solve some of these challenges with fractal snapshots:

You can instantly create an isolated copy of your data for development, testing, or experimentation. Have a massive production dataset you want to play with? You don't need to wait for a full copy. Just fork your source bucket, experiment freely, throw it away, and spin up a new one — instantly.

Their timelines diverge from the source bucket at the moment of the fork. It's the many-worlds version of object storage.

The key is that every object is immutable. Each write creates a new version, timestamped and preserved.

That immutability allows Tigris to version the entire bucket, and capture it as a single snapshot.

This is interesting. Rather than single Delta or Iceberg tables, it versions the full bucket with the help of the versioning capabilities of these open table formats. Tigris says further, "Each object maintains its own version chain, and a snapshot is an atomic cut across all those chains at a specific moment in time."

A more comprehensive example with two different tables and different isolations that helps understand these processes in a data lake example with open table format tables stored on object storage:

Important to know: a snapshot is an atomically consistent version across all those chains at a specific moment in time, and when retrieving a snapshot, Tigris, for example, returns the newest version ≤ snapshot timestamp of each table. For example, Snapshot T3-dev would contain Customer Table v4-dev and only Sales Table v5-dev (not v4-dev).

One technology used behind this is called Prolly Tree, also known as Merkle Trees: Image from Prolly Trees on Dolt Documentation

Data Movement Efficiency Spectrum

Before we get into the architectural decisions and the tools, let's observe the data movements when we implement Git for data, and let's categorize them by the amount of data movement required, ordered from most to least efficient:

The most efficient approach uses metadata/catalog-based versioning. Catalog pointers that just point to the same files multiple times (lakeFS and Iceberg are using this) create multiple logical versions of datasets without any physical duplication. No data movement involved.

The next best approach is zero-copy or data virtualization technologies. Tools like Apache Arrow enable data sharing between processes and systems without serialization overhead. You avoid the costly conversion between formats—no deserializing from source format to an intermediate representation and back again.

When changes occur, delta-based approaches are the best way. Rather than copying the entire dataset, you only store what has changed in new files. If you need to roll back, you simply revert the pointer to the previous file and state while keeping the changed files. This requires data management to manage changes.

The least efficient but simplest approach is full 1:1 data copying. Traditional methods like ODBC transfers, CSV exports, or database dumps require serializing data from the source format, moving it entirely, and deserializing it at the destination (e.g., from MS SQL to Pandas). But also, just creating a copy on S3 while keeping the same format is an expensive transaction, even more so with bigger datasets.

This works best for small datasets where the overhead doesn't matter, and offers the convenience of true isolation and easy rollback without complex change tracking.

We can say we work from metadata → zero-copy → delta → full copy. Let's investigate how lakeFS and other tools solved that problem and which approach they have chosen.

Zero-Copy and Cloning

Zero-copying is important as we want fast creation of a new state. Zero-copying and cloning are the solution to that initial fast copy of an existing dataset. You can think of cloning a "production" database or lake.

Both zero-copy and cloning are related but not quite the same. For example, something can support cloning but it's NOT zero-copy (e.g., Dolt). It uses copy-on-write with structural sharing. We can say that the difference is:

• Cloning = Can you create a copy? (the capability)
• Zero-copy = Does it duplicate data or just use metadata pointers? (the implementation technique)

In the best case, we have both cloning and zero-copy: cloning production or a set of data without the need to duplicate the data, therefore zero-copy.

We can also borrow an analogy from Linux with a Symlink. You can have multiple pointers at different places pointing to the same file. You can read, open, and change, but the data is only stored once. Instead of moving data, we just create one new or many new pointers.

The result is creating new datasets instantly, as it's just a metadata process, and not an actual data transfer. We change the pointer without moving data.

Branching with Metadata Catalogs

Branching is implemented through metadata catalogs, systems that use pointers to track different versions of data without duplicating it, just like Git does. This is the most efficient way of versioning, as it's just a metadata process.

As mentioned above, this is the best way of versioning, as it's just a metadata process. Most modern tools implement this approach, though not all mean the same thing. Let's conceptually explore what we mean by branching data.

Branching is when you freeze the current state in an atomic and consistent way across multiple tables. Instead of focusing on one singular table, we do it for a full data warehouse layer or bucket.

Snapshotting is one of the approaches we use as part of our Data Engineering Toolkit. Here we snapshot each table based on recurring date-time, e.g., every end of month. Because we do that for all tables in our data warehouse, it's also what I'd call the same approach we are doing with newer branching capabilities.

But generally, branching allows a snapshot or fork across tables and data assets. It can also be used to integrate a Write-Audit-Publish (WAP) workflow, where you write into a temporary state, audit the quality and integrity of data, and only then publish (merge) it into production. This shows that branching solves the problem of having consistency to test a certain feature in isolation before merging all changes coherently, or none at all.

With additional features of merging (with some tools) or having a detailed commit log for what's happening, especially in combination with AI agents, this provides strong support to steward these autonomous agents and gatekeep and verify through humans.

Hybrid Approaches

In reality, we often combine multiple techniques to get the best of all worlds. For example, you might use open table formats (Iceberg/Delta Lake) for their built-in time travel capabilities, layered with lakeFS for branch-based isolation across your entire data lake.

Or pair MotherDuck's zero-copy cloning with scheduled snapshots to create comprehensive version history beyond the default 7-day window. The key is matching your data versioning strategy (metadata, zero-copy, or delta) with your orchestration and transformation tools, supporting branch deployments that clone both code and data together for true isolated testing environments.