2 AM Nightmare: When Git Submodule Betrays You
2 AM, the phone vibrates incessantly. The company’s microservices system is down. We needed to urgently rollback 15 services to last week’s stable version. At that time, the team was still using Git Submodule to manage sub-repositories within a main repository.
The result? A total disaster. Managing commit hashes for each submodule was a nightmare. `detached HEAD` errors appeared constantly. If a developer forgot to push code to a sub-repo but updated the hash in the parent repo, the CI/CD pipeline would break immediately. That night, I spent 4 hours just cleaning up the mess of `Git pointers`.
The next morning, I decided to find another solution. That’s when I discovered repo (Git-repo). This is the ‘backbone’ tool Google uses to coordinate over 1,000 repositories in the Android project (AOSP).
Choosing the Right Path for Massive Microservices Projects
As a project grows from 10 to 100 microservices, how you manage your code determines your release speed. Let’s look at three common approaches:
1. Monorepo (All eggs in one basket)
- Pros: Easy code search, change multiple services in a single commit.
- Cons: Repos weighing dozens of GBs make cloning take all day. Permissions are extremely difficult, and CI/CD builds are slow without advanced tools like Bazel or Nx.
2. Git Submodule (Repo within a Repo)
- Pros: Built-in Git feature.
- Cons: Terrible UX. Tracking versions between parent and child repos is highly error-prone. With 50+ submodules, manual management becomes nearly impossible.
3. Repo Tool (Hybrid Solution)
- Pros: Keeps repos independent but centrally controlled via a Manifest file (XML). You can sync 200 repos with a single command.
- Cons: Requires installing an external Python script.
For teams of 20-50 people, repo is the perfect middle ground. It maintains the flexibility of Polyrepos while providing the centralized governance of a Monorepo.
The Core of Repo: Power from the Manifest File
repo doesn’t store code directly. It manages the repository list through a Manifest file. Think of it as a master map, specifying where each repo is located, which branch it uses, and where it’s stored locally.
A typical default.xml file is usually lightweight like this:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="origin" fetch=".." />
<default revision="main" remote="origin" sync-j="8" />
<!-- Microservices list -->
<project path="services/auth" name="my-org/auth-service" />
<project path="services/order" name="my-org/order-service" />
<project path="libs/common" name="my-org/shared-library" />
</manifest>Implementing Repo in 3 Steps
Step 1: Install the Tool
At its core, repo is a Python script wrapping Git. On Linux or macOS, you can install it in a heartbeat:
mkdir -p ~/.bin
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
chmod a+rx ~/.bin/repo
export PATH=$PATH:~/.binStep 2: Create a Manifest Repository
Create a separate repo (e.g., manifests.git) to hold the default.xml file. This will be the ‘single source of truth’ for your project structure.
Step 3: Initialize the Project
Instead of tiring yourself out with git clone for each one, developers just need to run:
# Connect to the manifest
repo init -u [email protected]:my-org/manifests.git -b main
# Download all 100+ repos to your machine
repo sync -j16The repo sync command will automatically clone or update the entire project. The -j16 parameter allows for 16 parallel threads, saving up to 70% of waiting time.
Life-Saving Commands for DevOps Engineers
When working with large systems, I frequently use these `pro tips` for quick handling:
Check system-wide status: Don’t cd into every folder. Use forall to scan everything at once:
repo forall -c 'git branch | grep "*"'Bulk create branches for a Release: Need to `create a release-v2.0 branch` for 80 repos simultaneously? Simple:
repo start release-v2.0 --allBattle-Tested Lessons from Reality
In a recent project with 85 repositories, repo helped us reduce configuration error tickets from 20 per week to nearly zero. However, keep in mind:
- Manifest is king: Any directory structure changes must be committed to the manifest before telling the team to sync code.
- Leverage multi-core power: Always use
-jwhen syncing. With high-speed office networks,-j16or-j32turns a ‘go get coffee’ wait into a few seconds. - Don’t use a sledgehammer to crack a nut: If your project only has 3-5 repos, stick to vanilla Git for simplicity.
repois only truly valuable when the number of repos exceeds human control.
After applying this process to an 8-person team, the ‘mismatched versions’ issues between services disappeared completely. Most importantly, no one had to stay up late just to check if a shared library was on the right version.
If you’re struggling with a fragmented pile of microservices, try spending an afternoon setting up repo. Believe me, your workflow will become much more professional.
