How to Install and Configure Headscale: Self-Hosting a Tailscale Control Server for a Zero Trust VPN

Security tutorial - IT technology blog
Security tutorial - IT technology blog

Background & Why Self-Host Headscale?

Managing servers scattered across AWS, Hetzner, and a home lab setup can be a major headache. The common habit of exposing port 22 with an IP whitelist becomes highly inconvenient when you need to work from a coffee shop. Traditional OpenVPN setups also show significant drawbacks: all traffic gets routed through a central server, inadvertently turning the VPS into a bottleneck and causing latency to spike.

Tailscale solves this problem elegantly with a WireGuard-based Mesh VPN architecture. Two machines establish a direct P2P connection instead of tunneling through a central gateway. However, Tailscale’s free SaaS tier comes with constraints: a 3-user limit, reliance on a third-party control plane, and an inability to meet strict internal data governance requirements.

That is why Headscale was created. It is an open-source implementation of the Tailscale coordination server, giving you full control over the control plane on your own VPS. Your entire routing table, ACLs, and node lists remain strictly in your hands, while client devices continue using official Tailscale apps from the App Store, Google Play, or Linux repositories.

Installing Headscale on a Linux Server

A budget VPS with 1 vCPU and 1 GB of RAM (running Ubuntu 22.04/24.04 or Debian 12) is more than enough. In a homelab environment with under 100 nodes, the Headscale daemon typically consumes only 30 MB to 50 MB of RAM. You will need a subdomain (such as vpn.itfromzero.com) with an A record pointing to your server’s static IP.

Step 1: Download and Install the Binary

Visit the project’s GitHub releases page to grab the latest deb package for the amd64 architecture:

export HEADSCALE_VERSION="0.22.3"
wget --output-document=headscale.deb \
  https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_amd64.deb

sudo dpkg -i headscale.deb

This package automatically creates a system user named headscale, configures data directory permissions at /var/lib/headscale, and places an example configuration file at /etc/headscale/config.yaml.

Step 2: Set Up a Reverse Proxy with Nginx and Let’s Encrypt

While Headscale includes built-in ACME support for SSL certificates, placing Nginx in front of it remains the safest and most flexible approach. Nginx provides centralized certificate management, smooth WebSocket handling, and seamless routing if you host other web services on the same VPS.

sudo apt update && sudo apt install -y nginx certbot python3-certbot-nginx

Create a new virtual host file for Headscale:

sudo nano /etc/nginx/sites-available/headscale

Paste in the following configuration:

server {
    listen 80;
    server_name vpn.itfromzero.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name vpn.itfromzero.com;

    ssl_certificate /etc/letsencrypt/live/vpn.itfromzero.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/vpn.itfromzero.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $server_name;
        proxy_redirect http:// https://;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Open /etc/nginx/nginx.conf and ensure the WebSocket map block is declared inside the http {} block:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

Enable the virtual host, obtain an SSL certificate with Certbot, and reload the service:

sudo ln -s /etc/nginx/sites-available/headscale /etc/nginx/sites-enabled/
sudo certbot --nginx -d vpn.itfromzero.com
sudo nginx -t && sudo systemctl reload nginx

Detailed Headscale Configuration

Configuring config.yaml

Open the primary Headscale configuration file to adjust connection parameters:

sudo nano /etc/headscale/config.yaml

Review and modify the following core settings:

server_url: https://vpn.itfromzero.com:443
listen_addr: 127.0.0.1:8080
metrics_listen_addr: 127.0.0.1:9090

ip_prefixes:
  - 100.64.0.0/10
  - fd7a:115c:a1e0::/48

derp:
  server:
    enabled: false
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  paths: []
  auto_update_enabled: true
  update_frequency: 24h

database:
  type: sqlite3
  sqlite:
    path: /var/lib/headscale/db.sqlite

Enable and start the service via systemd:

sudo systemctl enable --now headscale
sudo systemctl status headscale

Creating an Administrative User

Headscale groups devices under Users (called Namespaces prior to version 0.23). Create the initial account for your internal team:

sudo headscale users create admin-team

Registering Clients with the Headscale Server

On the client machine (such as another Ubuntu server or a personal laptop), install the official Tailscale client:

curl -fsSL https://tailscale.com/install.sh | sh

# Connect to the self-hosted control server
sudo tailscale up --login-server https://vpn.itfromzero.com

The terminal will return an authentication link like this:

To authenticate, visit:
https://vpn.itfromzero.com/register/nodekey:xxxxxxxxxxxxxxxxxxxxxx

Copy the key string following the nodekey: prefix (or mkey: depending on the client version) and switch back to the Headscale server to approve it:

sudo headscale nodes register --user admin-team --key nodekey:xxxxxxxxxxxxxxxxxxxxxx

If you need automated bootstrapping via Ansible or Docker entrypoints, generate a single-use pre-authentication key with an expiration time:

# Create a single-use key that expires after 24 hours
sudo headscale preauthkeys create --user admin-team --expiration 24h

Clients can now join the network immediately with a single command:

sudo tailscale up --login-server https://vpn.itfromzero.com --authkey <YOUR_PREAUTH_KEY>

Testing & Monitoring the VPN Network

Verifying Connectivity and Mesh Performance

On the Headscale server, check the list of online devices:

sudo headscale nodes list

The output table displays the Node ID, Hostname, owning User, CGNAT IP address (within the 100.64.0.0/10 subnet), and Online status.

Next, test the connection quality directly from the client machine:

# List peers in the network alongside their assigned IP addresses
tailscale status

# Ping to verify direct routing
tailscale ping 100.64.0.2

A response line reading pong from ... via [public-ip]:port in 14ms (direct) confirms that the two nodes have successfully completed a P2P handshake over WireGuard. Bandwidth will hit the limits of your ISP line speed. Conversely, seeing via DERP(syd) indicates the connection is being relayed because both ends are behind strict NAT (Symmetric NAT), increasing latency to around 80–150ms.

Collecting Metrics with Prometheus

Headscale features a built-in Prometheus metrics endpoint on port 9090. Run a quick check using curl from localhost:

curl -s http://127.0.0.1:9090/metrics | grep -E '^headscale_'

The two most critical metrics to watch are headscale_nodes_online_count and headscale_sqlite_queries_total. For homelab deployments under 150 nodes, the default SQLite backend performs flawlessly. If you scale beyond 200 nodes with hundreds of concurrent route updates, simply switch database.type to postgres to eliminate database lock contention.

Share: