The Threat of Untrusted Docker Images in Production
While conducting security audits across more than ten systems recently, I noticed a familiar paradox. Engineering teams diligently patch OS vulnerabilities every week and tighten every firewall port. Yet, container images pulled into production servers are left virtually unchecked. Nobody verifies whether those image files are truly genuine and untampered with.
Docker tags are inherently mutable (easy to overwrite). This design flaw paves the way for serious real-world attack vectors:
- Leaked registry credentials: An attacker steals access tokens for Docker Hub or an internal Harbor instance, then overwrites the
:v1.0.0tag your team is preparing to deploy with a cryptominer like XMRig. - Man-in-the-Middle (MitM) attacks: Traffic pulling images over local networks or enterprise proxies gets intercepted, swapping out image manifests in transit.
- Pulling compromised base images: Developers mistakenly adopt an unvetted image packed with a pre-installed backdoor or unpatched, critical CVEs.
How can a production server tell the difference between an image built by your verified CI/CD pipeline and a malicious artifact slipped in along the way? This is the core challenge of Software Supply Chain Security.
Sigstore Cosign: A Lightweight, Pragmatic Code Signing Solution
In the past, Docker Content Trust (Notary v1) was the go-to solution. However, Notary required dedicated servers, complicated TUF metadata management, and significant operational overhead. Most DevOps teams quickly abandoned it due to its sheer complexity.
Cosign, developed under the Sigstore project and hosted by the Linux Foundation, takes a much cleaner approach. Cosign requires no extra databases. Instead, it leverages your existing Container Registry (Docker Hub, GHCR, AWS ECR, or Harbor) as the signature store. When signing, Cosign generates an OCI artifact containing the signature payload and pushes it straight to the registry, cryptographically bound to the original image’s sha256 digest.
There are currently two primary signing workflows:
- Key-based signing: Generates standard private/public key pairs. The private key is stored in your CI runner secrets for signing, while the public key sits on target production servers for verification. This model is exceptionally easy to set up for small to mid-sized teams.
- Keyless signing: Leverages OpenID Connect (OIDC) via Fulcio to issue short-lived certificates (valid for minutes) and logs them into the Rekor transparency log. This eliminates the headache and risk of managing long-lived private keys altogether.
Hands-On Guide: Signing and Verifying Docker Images Step by Step
Step 1: Install Cosign on Linux
Download the official binary directly from Sigstore:
# Download cosign v2.x binary
curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"
# Rename and grant execution permissions
sudo mv cosign-linux-amd64 /usr/local/bin/cosign
sudo chmod +x /usr/local/bin/cosign
# Verify installation
cosign versionStep 2: Generate a Cosign Key Pair
Run the key generation command:
cosign generate-key-pairYou will be prompted to enter a passphrase to encrypt your private key. When automating this within CI/CD pipelines, you can predefine the COSIGN_PASSWORD environment variable to prevent the script from stalling on interactive prompts.
The command outputs two files:
cosign.key: The private key used for signing. Keep this strictly confidential and store it only in GitHub Secrets or HashiCorp Vault.cosign.pub: The public key used for verification. This file is safe to distribute freely across your production nodes.
Step 3: Build and Push an Image to Your Registry
Important note: Cosign signs images only after they have been pushed to a remote registry. Log in and push a test image:
docker login
# Tag and push a test image
docker tag alpine:3.19 your-dockerhub-user/secure-app:1.0.0
docker push your-dockerhub-user/secure-app:1.0.0Step 4: Sign the Image
Run cosign sign with your private key:
cosign sign --key cosign.key your-dockerhub-user/secure-app:1.0.0Enter the passphrase configured in Step 2. Immediately after, Cosign uploads a new tag prefixed with sha256-<digest>.sig to your container registry alongside the image to store the signature.
Step 5: Verify the Signature in Production
On your production server (where only cosign.pub is stored), execute the verification command:
cosign verify --key cosign.pub your-dockerhub-user/secure-app:1.0.0If the signature matches and the image integrity is confirmed, Cosign returns the JSON payload containing the digest:
[
{
"critical": {
"identity": { "docker-reference": "index.docker.io/your-dockerhub-user/secure-app" },
"image": { "docker-manifest-digest": "sha256:d4e3..." },
"type": "cosign container image signature"
}
}
]Now simulate an attack scenario: build a different image and overwrite tag 1.0.0 without re-signing it. The verify command will immediately fail with an error:
Error: no matching signatures:
failed to verify signatureStep 6: Integrate Verification into Deployment Scripts
To prevent rogue containers from running, wrap the startup command in a script that checks the exit code:
#!/usr/bin/env bash
set -euo pipefail
IMAGE="your-dockerhub-user/secure-app:1.0.0"
PUBLIC_KEY="/opt/security/cosign.pub"
echo "[*] Checking integrity for: ${IMAGE}"
if cosign verify --key "${PUBLIC_KEY}" "${IMAGE}" > /dev/null 2>&1; then
echo "[+] Valid signature. Starting container..."
docker run -d --name production-app "${IMAGE}"
else
echo "[!] ERROR: Image is unsigned or tampered with. Aborting deployment!" >&2
exit 1
fiIf you run Kubernetes, you can replace shell scripts with Admission Controllers such as Kyverno or OPA Gatekeeper. They will automatically reject any Pod attempting to pull an image without a verified signature.
Final Thoughts
Hardening servers is a necessary baseline. Ensuring that only verified workloads land on those servers completes the puzzle. Setting up Cosign takes your team roughly 15 minutes, but it completely eliminates the risk of running unauthorized or compromised containers in production.
