The Large Data File Nightmare in Git
Anyone working on AI projects, video processing, or database administration has likely faced this scenario: you accidentally commit an 8GB model checkpoint or a 15GB zip archive into your repository. Within just a few commits, the .git directory balloons to hundreds of gigabytes. As a result, every time a teammate runs git clone, connections constantly time out and the Git server throws overload errors.
Git was designed to track plain-text source code. When you push heavy binary files into commit history, Git compresses and stores a complete new version every time a file changes. Your repository size surges linearly with the size of these files.
Usually, Git LFS (Large File Storage) or using DVC with Git is the first solution that comes to mind. However, LFS operates on a centralized model that is tightly coupled with your Git server. Take GitHub as an example: a free tier includes a mere 1GB of storage and 1GB of download bandwidth per month. Exceed that limit, and you’ll have to pay $5/month per data pack (50GB). For teams dealing with multi-terabyte datasets, these recurring monthly costs escalate quickly.
Moreover, if your team already has a 40TB office NAS or a private MinIO bucket, Git LFS makes it difficult to leverage these resources flexibly. That is why many engineers are switching to git-annex.
How Does git-annex Work?
The core difference lies in its distributed (peer-to-peer) architecture.
Instead of forcing every large file onto a central server, git-annex treats each computer, server, or external drive as an independent node. When you add a file to git-annex:
- The actual file content is moved into an internal object store at
.git/annex/objects/and locked as read-only to prevent accidental modifications. - In your working directory, git-annex creates a symlink or pointer file pointing to that object. Standard Git only commits this pointer file, which is just a few dozen bytes in size.
- git-annex tracks the physical location of each file via an underlying branch named
git-annex. This branch records metadata: which file hash X currently resides on a developer’s laptop, a GPU server, or an office NAS.
Quick Comparison: git-annex vs. Git LFS
| Criteria | Git LFS | git-annex |
|---|---|---|
| Architecture | Centralized Client-Server | Distributed Peer-to-Peer |
| Storage Infrastructure | Tightly coupled with Git forge (GitHub, GitLab) | Rsync, SSH, S3, MinIO, NAS, external drives… |
| Data Ownership | Server retains the central master copy | Configurable: choose which node holds which files |
| Operational Costs | Costly storage and egress bandwidth fees | Virtually zero by leveraging existing infrastructure |
Hands-on: Installing and Using git-annex
1. Installation
On Ubuntu or Debian:
sudo apt update
sudo apt install -y git-annexOn macOS via Homebrew:
brew install git-annex2. Initializing a Repository and Tracking Large Files
Suppose you need to track weights for an AI model:
mkdir ai-project && cd ai-project
git init
git annex init "workstation-laptop"
# Create a dummy 500MB binary file for testing
fallocate -l 500M model_weights_v1.bin
# Track the file with git-annex instead of a regular git add
git annex add model_weights_v1.bin
# Commit metadata into Git
git commit -m "feat: track model_weights_v1.bin via git-annex"Running ls -l shows that model_weights_v1.bin points to a symlink inside .git/annex/objects/. Git only tracks this symlink, keeping your Git repository ultra-lightweight.
3. Configuring a Special Remote via SSH/rsync
The true power of git-annex lies in its special remotes. You can turn a Synology NAS or an internal backup server into a storage backend:
# Initialize an rsync remote pointing to an internal server
git annex initremote office-nas type=rsync \
[email protected]:/volume1/annex-data \
encryption=none4. Syncing and Reclaiming Disk Space
Once model training is complete, push the data to the NAS:
# Push file contents to the NAS server
git annex copy model_weights_v1.bin --to=office-nas
# Check the current locations of the file
git annex whereis model_weights_v1.binThe output will show that the file is present on both workstation-laptop and office-nas.
Running low on laptop SSD space? Reclaiming disk space is straightforward:
git annex drop model_weights_v1.bingit-annex features built-in safety checks. If the file has not been safely backed up to at least one other remote (governed by the numcopies setting, which defaults to 1), the drop command will refuse to delete it. You never have to worry about losing files to an accidental keystroke.
When a teammate clones the repo and needs that specific file to run tests:
git annex get model_weights_v1.bin5. Editing File Content
By default, files tracked by git-annex are read-only to prevent unintended SHA-256 hash mismatches. To edit a file, unlock it first:
# Unlock write permissions for the file
git annex unlock model_weights_v1.bin
# Append new data or modify the file
echo "new checkpoint data" >> model_weights_v1.bin
# Stage and commit the new version
git annex add model_weights_v1.bin
git commit -m "fix: update model weights checkpoint"Conclusion
git-annex breaks through the storage limits of traditional Git without locking you into expensive Git LFS pricing tiers. By flexibly distributing assets across local machines, SSH servers, and cloud buckets, you retain complete ownership of your data. Try it out on your next data science project to experience the difference.
