Git Multi-Pack-Index (MIDX): Consolidating Object Indexes for Optimal Read Performance in Large Repositories - ITFROMZERO - Share tobe shared! Git Multi-Pack-Index (MIDX) creates a shared index file across all pack files without merging them — significantly reducing the time spent on read operations in large repositories. This post shares real-world experience deploying MIDX in production, covering setup, git maintenance integration, and how to measure the performance gains.

Table of Contents

When Your Repository Becomes a Bottleneck

I first noticed this problem about 8 months ago, when our team migrated a large monorepo — over 15 years of commit history, nearly 2GB of pack files — to a new server. git status was taking 3–4 seconds, and git log --oneline -10 had noticeable lag. At first I thought it was a hardware issue, but after profiling, the root cause was somewhere else entirely: Git had to scan through too many individual pack files every time it looked up an object.

Git stores objects (commits, trees, blobs) in pack files — compressed binary files inside .git/objects/pack/. When a repository has just been cloned or recently had git gc run on it, there are usually only 1–2 pack files and everything stays fast. But after months of activity — continuous fetches, CI/CD running 24/7 — the number of pack files can climb into the dozens. Each pack file has its own index (.idx), and Git has to binary-search through each one to find an object. The more files there are, the greater the overhead.

The traditional solution is git gc --aggressive to consolidate everything into a single pack. But on a 2GB repo, that command runs for 20–30 minutes and blocks all operations during that time. Not viable for production.

How Git Multi-Pack-Index Solves the Problem

Multi-Pack-Index (MIDX) — introduced in Git 2.21 and stable since Git 2.25 — creates a single shared index file covering all existing pack files, without merging them together. This file lives at .git/objects/pack/multi-pack-index.

Instead of sequentially binary-searching through N .idx files, Git only needs a single lookup in the MIDX — which immediately tells it which pack file contains the object and at what offset. With 30 pack files, this is a significant improvement that requires no changes to the underlying data.

The most important point: MIDX does not modify existing pack files, does not risk data loss, and requires no downtime. Once created, it’s immediately in effect.

Enabling MIDX

Git 2.25+ ships with MIDX built in — nothing extra to install. Check your version first:

git --version
# git version 2.43.0 — OK, MIDX is supported

Enable MIDX for your repository:

# Enable globally (recommended for all repos)
git config --global core.multiPackIndex true

# Or only for the current repo
git config core.multiPackIndex true

Create the MIDX file for the first time:

cd /path/to/your/repo
git multi-pack-index write

# Confirm the file was created
ls -lh .git/objects/pack/multi-pack-index

On a large repo, the initial MIDX creation may take 10–30 seconds — but it doesn’t block anything; Git continues serving other operations normally in the meantime.

Detailed Configuration and Workflow Integration

Automatically Updating MIDX with git maintenance

MIDX is created once but doesn’t update itself. Every time a new pack file appears — after a fetch or gc — the existing MIDX no longer covers everything. You need to re-run git multi-pack-index write periodically.

Git 2.29+ includes git maintenance, which handles exactly this:

# Enable maintenance for the current repo
git maintenance start

# Check with a dry-run first
git maintenance run --task=incremental-repack --dry-run

git maintenance start registers a cron job (Linux/macOS) or Task Scheduler entry (Windows) to run maintenance periodically. The incremental-repack task will automatically update MIDX after each fetch without any manual intervention.

If you prefer more manual control, add this to your post-merge hook:

# .git/hooks/post-merge  (run chmod +x after creating)
#!/bin/bash
git multi-pack-index write 2>/dev/null &

Running it in the background (&) avoids slowing down the developer’s merge/fetch operations.

Combining with commit-graph

After enabling MIDX, I also enabled commit-graph — the two features complement each other well. MIDX optimizes object lookups, while commit-graph speeds up reachability queries like git log --graph and git merge-base:

git config --global fetch.writeCommitGraph true
git config --global core.multiPackIndex true

# Create both at the same time
git multi-pack-index write --bitmap
git commit-graph write --reachable --changed-paths

The --bitmap flag when writing MIDX generates an additional bitmap index — useful for server-side bare repositories. For local dev repos it’s not particularly necessary.

Configuration for Bare Repositories on a Server

If you self-host Gitea or GitLab, server-side configuration matters far more than local settings:

# On a bare repo (server)
git config pack.writeBitmaps true
git config repack.writeBitmaps true
git config core.multiPackIndex true

# Run maintenance — does not block client push/fetch
git maintenance run \
  --task=gc \
  --task=commit-graph \
  --task=incremental-repack

Schedule this to run at 2 AM to avoid peak hours:

# crontab -e
0 2 * * * cd /path/to/bare.git && git maintenance run --task=incremental-repack --task=commit-graph 2>>/var/log/git-maintenance.log

Verification and Monitoring

Verifying Integrity

# Verify MIDX has no errors (silent if OK, prints errors if problems found)
git multi-pack-index verify

# View detailed MIDX statistics
git multi-pack-index verify --object-dir=.git/objects

# Debug whether Git is loading the MIDX
GIT_TRACE2=1 git status 2>&1 | grep -i multi_pack

Measuring Real-World Improvement

Before enabling MIDX, I benchmarked with a simple approach:

# Drop the OS page cache before each measurement (Linux)
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'

# Measure time WITHOUT MIDX
time git rev-list --count HEAD
time git log --oneline -50 > /dev/null

# Create MIDX
git multi-pack-index write

# Drop cache and measure again
sudo bash -c 'echo 3 > /proc/sys/vm/drop_caches'
time git rev-list --count HEAD
time git log --oneline -50 > /dev/null

Results on our team’s repo (28 pack files, ~1.8GB): git rev-list --count HEAD dropped from 4.2s to 0.8s. git log --oneline -50 went from 2.1s down to 0.4s. With a team of 8, I rolled out this Git configuration across all developer machines and the CI/CD server — complaints about “slow git” dropped noticeably, especially after sprints with heavy branch merging.

When to Rebuild MIDX

A few situations that call for re-running git multi-pack-index write:

After any git fetch that creates a new pack file (if you’re not using the maintenance hook)
After git gc or git repack — these commands change the pack files
After a large rebase or merging a long-lived branch with many commits
When the number of loose objects spikes unexpectedly

# Check how many pack files currently exist
ls .git/objects/pack/*.pack | wc -l

# If > 10, it's worth rewriting MIDX
git multi-pack-index write

If the pack file count exceeds 50–100, consider running git repack -adf once to consolidate them — MIDX helps defer that point, but it won’t hold it off forever.

Practical Notes After 6 Months in Production

Git 2.25 is sufficient — Ubuntu 20.04 LTS ships with Git 2.25.1, which is enough for basic MIDX usage. For git maintenance you’ll need Git 2.29+.
MIDX only optimizes read operations — git add, git commit, and git push won’t be faster. The clearest gains are with: git log, git status, git fetch, and git merge.
The MIDX file is surprisingly small — a repo with 1.8GB of pack files produces a multi-pack-index file of only ~2MB. Negligible disk impact.
Safe to delete and roll back — deleting the MIDX file just makes Git slower; it doesn’t lose any data. You can test and roll back freely at any time.
Works on Windows too — Git for Windows 2.25+ has full support, no additional configuration required.

This isn’t a silver bullet, but it’s an optimization worth enabling immediately if your repo has grown past a few hundred MB and you’re starting to feel lag on everyday read commands.