The Nightmare of “Git Pushing” Heavy Data
When I first started in Machine Learning, I made a “classic” mistake: throwing everything into Git. From few-KB Python code files to a 5GB image dataset and an 800MB model file (.pth). The result was a git push command that ran for an hour without finishing. Eventually, GitHub flat out rejected it because the files exceeded the 100MB limit, leaving me with a bloated and sluggish repository.
I tried to fix it with git-lfs. However, storage costs started skyrocketing, and syncing between team members was painfully slow. Even worse, when I needed to revert to an old data version for debugging, I completely lost track of which code went with which data, which is one of the many Git disasters developers face. Git was born to manage code (text); it wasn’t designed to carry gigabytes of binary data.
Why is Git “Afraid” of Big Data?
The problem lies in the snapshot mechanism. When you edit a text file, Git only saves the difference (diff). But with binary files like images or model weights, changing just 1 byte causes Git to save the entire new file. Your repo will bloat rapidly, though you can supercharge Git performance using modern tools. Imagine git clone-ing a project for 30 minutes just because of redundant data from the past – it’s a real nightmare that often necessitates using the BFG Repo-Cleaner to shrink the repository.
Many people often choose “survival” methods like:
- Compressing data and tossing it onto Google Drive (very easy to mix up versions).
- Using Git LFS (expensive and server-dependent).
- Ignoring data versioning altogether (extremely dangerous in AI work).
DVC (Data Version Control) – The Escape Route for AI Folks
After 6 months of applying DVC to real projects, I can confirm it’s the perfect “missing piece.” DVC doesn’t store data directly in Git. It only creates lightweight meta-files (.dvc) – just a few KB – to act as placeholders. The actual data is pushed to separate storage like S3, Google Drive, or local servers via SSH.
How DVC Works:
- Actual Data: Stays on your machine and is automatically added to
.gitignore. - .dvc File: Contains a unique hash code for the data; this file is managed by Git.
- DVC Remote: Where the actual data is stored (like a warehouse for heavy items).
Implementing DVC from Scratch
Let’s get started with setting up a real project.
1. Tool Installation
You can install it quickly via pip. Don’t forget to install the plugin corresponding to the storage you use:
pip install dvc
# If using S3
pip install "dvc[s3]"
# If using Google Drive
pip install "dvc[gdrive]"2. Initializing the Environment
In your project’s root directory, run the following command:
dvc init
git commit -m "Initialize DVC"At this point, DVC will create a .dvc/ configuration directory. Everything is ready.
3. Managing Your First Data
Suppose you have a data/raw/ folder that’s 2GB. Instead of using regular Git commands, use:
dvc add data/raw/DVC will create a data/raw.dvc file. Simultaneously, it automatically adds data/raw/ to .gitignore. Now, you just need to commit the meta-file to Git:
git add data/raw.dvc .gitignore
git commit -m "Add raw dataset via DVC"4. Configuring Remote Storage
You can use your personal Google Drive as storage. Here’s how to set up a local directory to simulate a server:
dvc remote add -d myremote /path/to/dvc_storage
git commit .dvc/config -m "Configure remote storage"To push data to the storage, you only need one command: dvc push.
Seamless Team Collaboration
When a colleague git pulls the code, their data/raw/ folder will be empty. But don’t worry, they just need to type:
dvc pullDVC will automatically read the hash in the .dvc file and download the correct data version from the remote. Download speeds are usually much faster than Git LFS because it optimizes large file transfers.
Real-world Experience: Managing Model Weights
I once spent an entire morning just trying to find the file model_v2_final_fix_v3.pth. With DVC, everything is much more professional. You just need to tag your Git commit to mark important milestones.
For example, when you finish training a model that reaches 95% accuracy:
dvc add model.pth
git add model.pth.dvc
git commit -m "Model reached 95% Acc with LR 0.001"
git tag -a "v1.0-prod" -m "Model to production"
dvc pushLater, no matter how much the code changes, just run git checkout v1.0-prod and then dvc checkout, and you’ll have the exact code and model from that specific time. No more mismatching code and data versions.
Crucial Lessons Learned
- Always remember dvc push: Git commit only saves the “shell.” If you forget to push data to the remote, your colleagues will have nothing to run.
- Prioritize SSH: If working in a company, use SSH to connect to a shared server. Speed will be much more stable compared to Google Drive.
- Automate with Pipelines: Look into
dvc run. It helps you connect Code -> Data -> Model into an automated chain, ensuring results are always reproducible.
Conclusion
If your project only has a few lightweight CSV files, Git is enough. But when you start working with thousands of images, videos, or model files weighing hundreds of MBs, DVC is a must. It keeps your Git Repo clean, professional, and makes team collaboration easier than ever.
