What Happened at 2 AM
About six months ago, I got hit with a barrage of Fail2ban alerts — my server was being brute-forced over SSH at a rate I’d never seen before. I opened my laptop at 2 AM, remoted in, blocked the IPs, and combed through the logs line by line looking for any sign of a successful intrusion. Fortunately, nobody got in. But the feeling of sitting there auditing every file in the middle of the night… it’s a kind of discomfort that’s hard to describe. That night taught me a lesson I won’t forget: set up security from the start, don’t wait until something goes wrong.
Afterward, I sat down and thought: what if the attacker had gotten in with root? My private key sits at ~/.ssh/id_rsa — it has a passphrase, sure, but the file is still there on disk, copyable at any time. Once that key file is taken, the attacker can crack it offline with no time limit, with no need to ever connect to my server again.
The Core Problem: SSH Keys on Disk Are a Fatal Weakness
The standard way of storing SSH keys has a fundamental flaw: the private key is a file on disk. Even with 600 permissions and a strong passphrase, a root-level attacker can still:
- Copy
~/.ssh/id_rsaand crack it offline withhashcat— on a GPU like the RTX 4090, a weak passphrase can fall in minutes - Dump the memory of the
ssh-agentprocess while the key is loaded (the key lives in plaintext in RAM) - Hook into the OpenSSH shared library to intercept the key during signing
- Read the key via
/proc/<pid>/memwith root privileges
This isn’t about weak passphrases or wrong permissions. The fatal flaw is that the private key can leave the server. Files can be copied, memory can be dumped — no passphrase can patch that gap.
Solutions
Option 1: Hardware Security Keys (YubiKey, Nitrokey)
A YubiKey stores the private key inside its chip and makes extraction impossible. Each SSH session requires physically inserting the key. A YubiKey 5 Series runs about $50–60, with Nitrokey being a bit cheaper at ~$35–45. This works well for a personal laptop, but it’s impractical for a headless server running 24/7 — you can’t plug a YubiKey into a VPS sitting in a data center.
Option 2: SSH Certificate Authority with Short TTL
Use a CA to sign certificates with a short validity window (e.g., 8 hours). A compromised key is only usable within that window. The downside is the infrastructure overhead — HashiCorp Vault SSH Secrets Engine or a self-managed CA — and the key still exists as a file during that window.
Option 3: TPM 2.0 — Built-In Chip, No Extra Hardware Required
TPM (Trusted Platform Module) is a security chip built into most servers and laptops manufactured since 2016. The chip generates and stores key pairs entirely within the hardware — the private key never leaves the chip, even if an attacker has root, even if they dump all of RAM or the entire disk.
The Best Approach: SSH Keys with TPM 2.0 Using tpm2-pkcs11
Step 1: Check Whether Your Server Has TPM 2.0
# Check whether the TPM device exists
ls /dev/tpm*
# Or check the version
cat /sys/class/tpm/tpm0/tpm_version_major
# Output "2" = TPM 2.0
# Install required packages (Ubuntu/Debian)
sudo apt install tpm2-tools tpm2-abrmd tpm2-pkcs11 tpm2-pkcs11-tools
# Start the TPM resource manager daemon
sudo systemctl enable --now tpm2-abrmdIf you see /dev/tpm0 or /dev/tpmrm0, your server has a TPM. Most standard VPS instances don’t have TPM hardware, but some cloud providers now support vTPM — AWS Nitro, Google Cloud Confidential VMs, and Azure Gen 2 VMs.
Step 2: Initialize the PKCS#11 Token Store in TPM
The tpm2-pkcs11 library lets OpenSSH communicate with the TPM through PKCS#11 — a widely used standard interface for hardware security modules.
# Initialize the store
tpm2_ptool init
# Create a token with a label and two types of PIN:
# sopin = Security Officer PIN (used to reset the userpin)
# userpin = PIN used daily when SSHing
tpm2_ptool addtoken \
--pid=1 \
--label=ssh-keys \
--sopin=SuperAdminPin123 \
--userpin=MyUserPin456Step 3: Generate an SSH Key Inside the TPM
# Create an EC P-256 key (recommended — compact, fast, well-supported by TPM 2.0)
tpm2_ptool addkey \
--label=ssh-keys \
--userpin=MyUserPin456 \
--algorithm=ecc256
# Or RSA 2048 for compatibility with older systems
tpm2_ptool addkey \
--label=ssh-keys \
--userpin=MyUserPin456 \
--algorithm=rsa2048
# Verify the key was created inside the TPM
tpm2_ptool listobjects --label=ssh-keysStep 4: Export the Public Key and Add It to Your Server
# Export the public key from the TPM to copy to the target server
ssh-keygen -D /usr/lib/x86_64-linux-gnu/libtpm2_pkcs11.so
# Output looks like:
# ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAI...
# Pipe it directly to the remote server
ssh-keygen -D /usr/lib/x86_64-linux-gnu/libtpm2_pkcs11.so | \
ssh user@remote-server "cat >> ~/.ssh/authorized_keys"Configure ~/.ssh/config so SSH automatically uses the TPM key:
Host production-server
HostName your-server.com
User ubuntu
PKCS11Provider /usr/lib/x86_64-linux-gnu/libtpm2_pkcs11.soStep 5: Connect via SSH Using the TPM Key
# SSH will prompt for a PIN instead of a passphrase
ssh production-server
# Enter PIN for 'ssh-keys': ****
# Or specify the PKCS11 provider directly
ssh -I /usr/lib/x86_64-linux-gnu/libtpm2_pkcs11.so user@remote-serverVerification: Can the Key Really Not Be Copied?
I tested this myself to be sure. With root access, I tried every method to find the private key material:
# Search the entire filesystem for private key files
sudo find / -name "*.pem" -o -name "id_rsa" -o -name "id_ecdsa" 2>/dev/null
# → No private key files related to the TPM key
# List objects in the TPM
tpm2_ptool listobjects --label=ssh-keys
# Only metadata visible: CKA_ID, CKA_CLASS, CKA_LABEL...
# No raw private key material
# There is no export-private-key command in tpm2_ptool
# This isn't a permission issue — the TPM hardware simply
# does not provide this interface by designHere’s what actually happens: instead of loading the private key into RAM and signing there, OpenSSH sends a “please sign this data” request to the TPM chip. The TPM performs the signing operation internally and returns only the signature. The private key never leaves the chip in plaintext form — not even the kernel can read it.
Lessons Learned After 6 Months of Real-World Use
- Back up your PINs immediately: Store both the userpin and sopin in a password manager right after creating the token. If you lose both, the key is gone forever — there is no recovery path.
- Reinstalling the OS wipes the token store: Some distros clear the TPM on reinstall. You’ll need to regenerate the key pair and update
authorized_keyson every server. Document your setup process so you can redo it quickly. - Library path varies by distro: Ubuntu uses
/usr/lib/x86_64-linux-gnu/libtpm2_pkcs11.so, Arch Linux uses/usr/lib/libtpm2_pkcs11.so, Fedora uses/usr/lib64/pkcs11/libtpm2_pkcs11.so. Find it with:find /usr -name "libtpm2_pkcs11.so" - Performance impact is negligible: Signing through the TPM is about 50–100ms slower than a software key. In practice, SSH connection setup takes a tiny bit longer but you won’t notice it.
- Most VPS instances don’t have TPM: Standard VPS hosting generally lacks TPM hardware. In that case, consider SSH certificates with a short TTL, or a YubiKey if you’re SSHing from a personal machine.
Layering Your Security
TPM-backed keys close one specific attack vector: an attacker with root cannot extract the private key for lateral movement to other servers in your infrastructure. Here’s what else I run alongside it:
- SSH key stored in TPM — root cannot extract it
- Fail2ban blocking brute-force attempts by IP
- SSH running on a non-standard port to reduce log noise
AllowUsersin/etc/ssh/sshd_configto restrict which accounts can SSH inPasswordAuthenticationdisabled — key auth only
Since that 2 AM brute-force incident, I’ve never left a bare private key sitting on disk again. TPM 2.0 comes built into most modern hardware — set it up once, use it indefinitely, and even root can’t walk away with your key.
