Configuring Git Content Filters (Smudge and Clean): Automatically Transform Files on Checkout and Commit - ITFROMZERO - Share tobe shared! A hands-on guide to Git Content Filters using smudge and clean. Automate environment variable injection on checkout and strip secret keys on commit.

Table of Contents

1. A Familiar Headache: Half-Shared Configuration Files

Almost every developer has run into this: a configuration file like config.json, settings.php, or appsettings.json must be tracked in Git to share its schema across the team. However, each developer needs their own local database connection string or port.

Chaos quickly follows:

A new teammate accidentally pushes their personal DB_PASSWORD or API token to the shared GitLab repo.
Every time you run git pull, your config gets overwritten, breaking your local environment and wasting 15–30 minutes resolving meaningless conflicts.
A CI/CD script needs to stamp a Git commit hash into the code before deployment, but creating a dedicated commit just to bump a version clutters the commit log.

2. Why Default Git Fails to Solve This

Git tracks changes as snapshots. When you run git add, Git reads the exact contents of your working directory and creates a new blob object in the index.

The issue stems from two conflicting requirements:

On your local machine: The application requires real data to run, such as connecting to localhost:5432 or using a Stripe test key.
On the remote repo: You only want to store a template containing placeholders, such as __DB_HOST__ or __API_KEY__.

These two states clash on every single commit.

3. Three Common Workarounds and Their Bottlenecks

Before discovering Content Filters, most teams rely on these three workarounds:

Using `.env.example` and gitignoring `.env`

This is the standard best practice for environment variables. But if your project relies on complex XML or JSON files, or tracked code files that cannot simply be gitignored, this falls short. .gitignore only ignores entire files; it cannot filter line by line.

Using `git update-index --skip-worktree`

This command instructs Git to ignore local changes on an already tracked file. The catch? Whenever a teammate updates that file on the remote repo, running git pull will throw a conflict error or overwrite your local changes altogether.

Using Pre-commit Hooks

Pre-commit hooks prevent sensitive credentials from being committed. However, they only block the commit and raise an error. You still have to manually revert the secret and commit again.

4. The True Fix: Git Content Filters (Smudge & Clean)

Few developers realize that Git ships with built-in Content Filters configured via .gitattributes. Data flows automatically through two transformation pipelines:

Clean Filter: Triggered during git add (from working tree to index). This filter replaces real secrets with placeholders before writing them into commit history.
Smudge Filter: Triggered during git checkout or git switch (from repo to working tree). This filter takes placeholders and populates them with your machine’s real local secrets.

After implementing this workflow across our 8-person backend team, we completely eliminated accidental credential commits over the past 6 months.

Step 1: Declare the Filter in `.gitattributes`

Create a .gitattributes file in the repository root and commit it to Git:

# Assign the filter named "secret_filter" to config.json
config.json filter=secret_filter

Step 2: Configure Clean and Smudge Commands Locally

This configuration lives in each developer’s local .git/config and is never committed to the repo.

Suppose the config.json file in the repository contains a placeholder:

{
  "api_endpoint": "https://api.example.com",
  "api_key": "__LOCAL_SECRET_KEY__"
}

Run these two config commands in your terminal:

# 1. Smudge: On checkout, replace placeholder with local key
git config filter.secret_filter.smudge 'sed "s/__LOCAL_SECRET_KEY__/local-dev-token-999/g"'

# 2. Clean: When staging (git add), convert local key back to placeholder
git config filter.secret_filter.clean 'sed "s/local-dev-token-999/__LOCAL_SECRET_KEY__/g"'

Step 3: Automating via Shell Scripts and Environment Variables

Hardcoding tokens inside git config is not ideal. Instead, let a shell script pull directly from system environment variables.

Create the smudge script (.git-filters/smudge.sh):

#!/usr/bin/env bash
# Read from stdin, replace placeholder with MY_API_KEY environment variable
KEY="${MY_API_KEY:-sandbox_default_key}"
sed "s|__LOCAL_SECRET_KEY__|$KEY|g"

Create the clean script (.git-filters/clean.sh):

#!/usr/bin/env bash
# Clean data before committing
KEY="${MY_API_KEY:-sandbox_default_key}"
sed "s|$KEY|__LOCAL_SECRET_KEY__|g"

Grant execution permissions and register them in Git config:

chmod +x .git-filters/smudge.sh .git-filters/clean.sh
git config filter.secret_filter.smudge .git-filters/smudge.sh
git config filter.secret_filter.clean .git-filters/clean.sh

Testing the Setup

Verify that the filters are working as expected:

Delete the local file and check it out again: rm config.json && git checkout -- config.json. Open the file, and you will see __LOCAL_SECRET_KEY__ replaced by the environment variable value.
Edit other parts of config.json, then run git add config.json.
Run git diff --staged: Git displays only legitimate changes, while the secret string remains safely masked as __LOCAL_SECRET_KEY__.

Your local code retains valid runtime config, while the shared repository stays completely clean of sensitive secrets.