Context: Why is effective secrets management essential?
Working with IT systems, I’ve observed a common but risky problem: managing sensitive information, also known as secrets. These secrets can be third-party API keys, database credentials, SSL certificates, access tokens, or even root server passwords. Often, developers tend to store them in configuration files (.env), environment variables, or, worse, hardcode them directly into the source code.
My server was once subjected to a brute-force SSH attack, and I had to handle an emergency in the middle of the night. Since then, I always prioritize setting up security from the start.
Security isn’t just about hardening systems or using strong passwords; it’s also about how we manage sensitive information like API keys, database credentials, or tokens. A small vulnerability in storing or transmitting secrets can lead to very serious consequences, potentially resulting in customer data breaches or even a complete loss of control over the entire system.
Manual secrets management is not only less secure but also poses challenges when scaling or making changes. For example, if you have dozens of services sharing a single database, updating the database password becomes a nightmare. This is why I sought professional solutions, and HashiCorp Vault stood out as a prominent choice.
To generate strong and unique passwords, I often use online tools like the Password Generator on ToolCraft.app. This tool runs entirely in the browser and doesn’t send data to the server, so I feel very secure using it to create passwords for multiple services. However, strong passwords alone are not enough. More importantly, it’s about how we store and use them securely and under control.
HashiCorp Vault is an open-source tool designed to securely manage the lifecycle of secrets. It provides centralized storage, encryption capabilities, policy-based access control, and, notably, the ability to generate dynamic secrets on demand. As a result, Vault helps minimize the risk of sensitive information exposure, simplifies management, and automates security tasks.
Installing HashiCorp Vault on Linux
In this section, I will guide you through installing HashiCorp Vault on a Linux server, specifically Ubuntu Server 22.04 LTS. This process is quite straightforward and very suitable for initial deployment.
1. Download and Install Vault
First, we need to download Vault. You can visit HashiCorp Vault’s official website to get the link for the latest version, or use wget as follows:
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vault
After installation, check the Vault version to confirm everything is ready:
vault --version
The displayed result will be similar to Vault v1.X.X ('X.X.X+ent' for enterprise).
2. Configure Vault Server
For Vault to function, we need to create a configuration file. I will set up a basic configuration with a filesystem storage backend, which is well-suited for development or testing environments. For production environments, you should consider using more robust backends like Consul, Raft, or PostgreSQL to ensure high availability.
Create necessary user, group, and directories
sudo useradd --system --home /etc/vault.d --shell /bin/false vault
sudo mkdir -p /etc/vault.d
sudo mkdir -p /var/lib/vault
sudo chown -R vault:vault /var/lib/vault
sudo chmod 755 /var/lib/vault
Create the configuration file /etc/vault.d/vault.hcl
Use the command sudo nano /etc/vault.d/vault.hcl then paste the following content:
storage "file" {
path = "/var/lib/vault"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "true" # Disable TLS for dev/test environments. TLS must be enabled for production!
}
# UI allows accessing Vault via a browser (optional)
ui = true
disable_mlock = true
Important note: tls_disable = "true" is for development environments only. In production environments, TLS MUST be enabled to encrypt all communication with Vault. I also enabled ui = true so that I can easily access Vault’s web interface later.
3. Create Systemd Service
To have Vault automatically start with the system and facilitate convenient management, I will create a Systemd service.
Create the file /etc/systemd/system/vault.service:
sudo nano /etc/systemd/system/vault.service
Next, paste the following content:
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep capabilities
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60s
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
Save and exit the file. Then, you need to reload Systemd, enable, and start the Vault service:
sudo systemctl daemon-reload
sudo systemctl enable vault
sudo systemctl start vault
To check the service status, use the command:
sudo systemctl status vault
You will see Vault displaying an active (running) status.
Detailed Vault Configuration
Once the Vault server has started, the next step is to initialize and configure it to begin storing and managing secrets.
1. Initialize Vault
When Vault first starts, it will be in a sealed state. This means Vault cannot access encrypted data. To unseal Vault, we need to perform initialization. This process will generate Unseal Keys and a Root Token.
Run the following command:
vault operator init
You will receive output similar to the following:
Unseal Key 1: Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unseal Key 2: Yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
Unseal Key 3: Zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Unseal Key 4: Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Unseal Key 5: Bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
Initial Root Token: hvs.Cccccccccccccccccccccccccccccccccccccccc
Vault initialized with 5 key shares and a key threshold of 3. Please
safely distribute the key shares you see above. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these key shares
to unseal it again.
Extremely Important: You must carefully store these Unseal Keys and Root Token in a secure location. The Root Token has full access to Vault and should only be used once for initial configuration. Afterward, revoke it and switch to using authentication methods with lower privileges. Unseal Keys are essential to unseal Vault each time it restarts or is sealed.
2. Unseal Vault
With a key_threshold of 3, I need to provide at least 3 of the 5 Unseal Keys received to unseal Vault.
vault operator unseal
# Paste Unseal Key 1 here
vault operator unseal
# Paste Unseal Key 2 here
vault operator unseal
# Paste Unseal Key 3 here
Once enough keys are provided, Vault will transition to an unsealed state.
3. Login
To interact with Vault, we need to log in. First, set the VAULT_ADDR environment variable:
export VAULT_ADDR='http://127.0.0.1:8200'
Now, log in using the Root Token:
vault login hvs.Cccccccccccccccccccccccccccccccccccccccc
You will see the message Success! You are now authenticated.
4. Enable and Configure Secret Engines (KV Secret Engine v2)
Secret Engines are Vault components for storing, creating, or encrypting secrets. The KV (Key-Value) Secret Engine is the most common type, allowing you to store secrets as key-value pairs.
Enable KV Secret Engine v2 (version 2 supports secret versioning):
vault secrets enable -version=2 kv
Write a secret:
vault kv put kv/my-app/config username=devuser password="supersecretpassword123"
Read secret:
vault kv get kv/my-app/config
The output will display the stored secrets. When working with JSON output from Vault, sometimes the JSON string is quite long and difficult to read. I often copy that output into the JSON Formatter & Validator on ToolCraft.app to reformat it, making it easier to analyze. It’s important to remember that sensitive information (secrets) must be handled carefully. Do not paste them into online tools if you are unsure about their security. For ToolCraft specifically, since all processing occurs in the browser, I feel more confident using it.
5. Policies
Policies in Vault define the permissions a token or AppRole can access. I will create a sample policy for a development application.
Create the file dev-policy.hcl:
path "kv/data/my-app/config" {
capabilities = ["read"]
}
path "kv/metadata/my-app/config" {
capabilities = ["list"]
}
This policy allows reading secrets at the path kv/my-app/config and listing its metadata.
Write the policy to Vault:
vault policy write dev dev-policy.hcl
6. Authentication Methods (Auth Methods – AppRole)
AppRole is a popular authentication method for applications. It allows applications to automatically retrieve tokens from Vault without hardcoding credentials.
Enable AppRole Auth Method:
vault auth enable approle
Create a role named dev-app and assign the newly created dev policy:
vault write auth/approle/role/dev-app token_policies="dev" token_ttl="1h" token_max_ttl="24h"
Read the role_id for the dev-app role (role_id is public and can be distributed to applications):
vault read auth/approle/role/dev-app/role-id
Create a secret_id for the dev-app role (secret_id is private, must be kept absolutely secure, and only granted to the application):
vault write -f auth/approle/role/dev-app/secret-id
Your application will use this role_id and secret_id to send requests to Vault. From there, it will receive a Vault Token with permissions according to the dev policy. This token will then be used to read the necessary secrets.
Example of Python code to retrieve a secret from Vault:
import os
import hvac # pip install hvac
VAULT_ADDR = os.environ.get('VAULT_ADDR', 'http://127.0.0.1:8200')
APPROLE_ROLE_ID = "your_role_id_here"
APPROLE_SECRET_ID = "your_secret_id_here"
client = hvac.Client(url=VAULT_ADDR)
try:
# Login using AppRole
login_response = client.auth.approle.login(role_id=APPROLE_ROLE_ID, secret_id=APPROLE_SECRET_ID)
client.token = login_response['auth']['client_token']
# Read secret
read_response = client.secrets.kv.v2.read_secret_version(path='my-app/config', mount_point='kv')
username = read_response['data']['data']['username']
password = read_response['data']['data']['password']
print(f"Username: {username}")
print(f"Password: {password}")
except hvac.exceptions.VaultError as e:
print(f"Error interacting with Vault: {e}")
except Exception as e:
print(f"Unknown error: {e}")
In practice, you would not hardcode role_id and secret_id directly into the code. Instead, provide them via environment variables or another secure mechanism during deployment.
Vault Monitoring and Auditing
After setting up Vault, monitoring and auditing are crucial steps. They ensure the system operates stably and securely.
1. Check Vault Status
You can check the overall status of Vault with the command:
vault status
This command will display information about Vault’s sealed status, high availability (if applicable), and address.
2. Audit Logging
Vault has the capability to log all requests and responses, providing you with a detailed record of who accessed what and when. This feature is indispensable for auditing purposes and enhancing security.
Enable file audit log:
vault audit enable file file_path=/var/log/vault_audit.log
Once enabled, all interactions with Vault will be logged to the file /var/log/vault_audit.log. You need to ensure this log file is protected and only authorized users have access.
3. Backup and Restore
Although I used a filesystem backend for this example, in a production environment, you MUST have a clear backup strategy for the backend storage (such as Consul, Raft, PostgreSQL, etc.). Most importantly, you must protect the Unseal Keys. Without the Unseal Keys, you will not be able to re-access the data when Vault restarts and returns to a sealed state.
4. Important Security Considerations
- Secure Unseal Keys: Absolutely do not store all Unseal Keys in the same location. Distribute them to truly trusted individuals.
- Use TLS: Always enable TLS for all communication with Vault in production environments. This helps protect data during transmission.
- Principle of Least Privilege: Grant users or applications only the necessary permissions, never more than what is required.
- Regular Auditing: Regularly review audit logs to promptly detect any unusual activities.
- Rotate Secrets and Tokens: Leverage Vault’s dynamic secret generation capabilities and periodically rotate tokens and secrets. This significantly reduces risk.
Conclusion
HashiCorp Vault is a flexible and effective solution for securely managing secrets. Investing time in learning and deploying Vault will bring long-term benefits to your system, giving you greater peace of mind regarding security. With this basic knowledge of installation and configuration, I hope you can embark on your journey to proactively and professionally protect sensitive information.
