The Problem: The Real-World Nightmare of Hardcoded Credentials
Ever opened a project’s .env file only to find root PostgreSQL passwords, AWS secret keys, or private TLS keys exposed in plain text? This is an all-too-familiar sight for sysadmins and DevOps engineers alike.
When you are managing just two or three servers, SSHing in to manually patch configuration files might get the job done. But once you scale up to 40 microservices with over 150 containers on Kubernetes, that approach falls apart completely. All it takes is one developer accidentally running git push with a config file to a public GitHub repository, and automated scraper bots will compromise those credentials in under 30 seconds.
After migrating our infrastructure from CentOS 7 to AlmaLinux 9, our team decided to close this vulnerability once and for all. We built a centralized secret management cluster on CentOS Stream 9. The most solid and widely adopted industry-standard solution today is HashiCorp Vault.
Core Operating Mechanics of HashiCorp Vault
Mastering these four core concepts before executing any commands will save you plenty of debugging headaches:
- Storage Backend: The location where encrypted data is stored. For clusters with fewer than 5 nodes, Vault’s built-in Integrated Storage (Raft) is the cleanest option. You don’t need to deploy and manage a separate, complex external Consul cluster.
- Unseal Keys & Shamir’s Secret Sharing: Upon startup, Vault is always in a sealed state. At this stage, data is completely unreadable. By default, Shamir’s algorithm splits the master key into 5 shards and requires a minimum threshold of 3 shards to reconstruct it. This ensures no single individual can unilaterally unlock the secret store.
- Secrets Engines: Plugins responsible for handling data. The most common is
kv(Key-Value), used for storing static API keys. For larger environments, you can leverage Dynamic Secrets to automatically generate temporary database credentials with an associated Time-To-Live (TTL). - Policies & Tokens: Vault enforces access control based on the principle of Least Privilege. Every incoming request to Vault must include a valid Token.
Hands-on: Deploying a Vault Server on CentOS Stream 9
The following steps were executed on a clean CentOS Stream 9 VPS (configured with 2 vCPUs and 4GB RAM).
Step 1: Install Vault from the HashiCorp Repository
CentOS Stream 9 uses DNF for package management. First, add the official repository to pull the latest releases:
# Install repository management tools
sudo dnf install -y dnf-plugins-core
# Add the official HashiCorp repository
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# Install Vault
sudo dnf install -y vault
Verify the binary installation:
vault --version
Step 2: Configure Raft Storage and the Web UI
Create the directory for Raft data and set proper permissions for the vault user:
sudo mkdir -p /opt/vault/data
sudo chown -R vault:vault /opt/vault/data
sudo chmod 750 /opt/vault/data
Overwrite the configuration file at /etc/vault.d/vault.hcl:
storage "raft" {
path = "/opt/vault/data"
node_id = "node1"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1 # Note: For production, you must configure valid TLS certificates and keys
}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "http://127.0.0.1:8201"
ui = true
disable_mlock = true
Enable and start the systemd service:
sudo systemctl enable --now vault
sudo systemctl status vault
Step 3: Open the Firewall Port
If firewalld is active on your server, allow traffic on port 8200 to access the API and Dashboard:
sudo firewall-cmd --permanent --add-port=8200/tcp
sudo firewall-cmd --reload
Step 4: Initialize the Cluster and Unseal Vault
Export the environment variable for the Vault CLI:
export VAULT_ADDR='http://127.0.0.1:8200'
Run the initialization command to generate the keys:
vault operator init
The output will display 5 Unseal Keys and 1 Initial Root Token:
Unseal Key 1: dE8jFn3q0...
Unseal Key 2: 9kLm2xP1a...
Unseal Key 3: xYz10uVb8...
Unseal Key 4: aB7cdE3f4...
Unseal Key 5: pQ4rtL9s2...
Initial Root Token: hvs.CAESIJ...
Important Warning: Store these 5 keys in a secure password manager immediately. Losing these keys means permanent, unrecoverable data loss.
Unseal the vault by running the unseal command three times, providing any three distinct keys:
vault operator unseal <Unseal_Key_1>
vault operator unseal <Unseal_Key_2>
vault operator unseal <Unseal_Key_3>
Once you see Sealed: false in the output, Vault is ready to accept data.
Step 5: Storing Secrets and JSON Handling Tips
Log in using the Root Token:
vault login <Initial_Root_Token>
Enable Key-Value engine version 2 (which supports secret versioning and rollbacks):
vault secrets enable -version=2 -path=secret kv
Write sample database credentials for an application:
vault kv put secret/my-app/database username="db_admin" password="K8#mP$9vL2@xQ!7z" port=5432
Retrieve and verify the data:
vault kv get secret/my-app/database
When provisioning system accounts, passwords should be strong and between 24 to 32 characters long. Instead of randomly typing characters, I regularly use the Password Generator on ToolCraft. It lets you customize length, include special characters, and calculate entropy in seconds.
Additionally, when applications query Vault via its REST API, responses often contain deeply nested JSON payloads. To quickly inspect payload structures while debugging with curl, I paste the data into the JSON Formatter & Validator. The best feature of ToolCraft is that all processing runs 100% client-side via JavaScript in your browser. Not a single byte is sent to a backend server, providing peace of mind when handling sensitive secrets.
Step 6: Creating Policies and Issuing Secure Tokens
Never grant Root Tokens to applications or CI/CD pipelines. Instead, create a scoped policy with read-only access to specific secrets. Create the app-policy.hcl file:
cat <<EOF > app-policy.hcl
path "secret/data/my-app/*" {
capabilities = ["read"]
}
EOF
Write the policy to Vault and generate a token with a 24-hour expiration:
vault policy write my-app-read app-policy.hcl
vault token create -policy="my-app-read" -ttl="24h"
This token is restricted to reading data under the secret/my-app/ path. After 24 hours, it will automatically expire without affecting any other services.
Conclusion
Deploying HashiCorp Vault completely eliminates the risky practice of scattering plain-text passwords across config files or Git repositories. Once you are comfortable with the Unseal lifecycle, the KV Engine, and Policies, you can explore Dynamic Secrets for MySQL/PostgreSQL or integrate Vault Agent to automatically inject secrets directly into Kubernetes Pods.

