Posted inVirtualization

Self-Hosting a RustDesk Server: Master Remote Desktop with Blazing-Fast Speed and Absolute Security

Virtualization tutorial - IT technology blog
Virtualization tutorial - IT technology blog

Why You Should Stop Using TeamViewer and Self-Host RustDesk

If you’re managing a lab with dozens of virtual machines, you’re likely all too familiar with the “terrible lag” that comes with the free version of TeamViewer. Even worse are those moments when you’re in the middle of an urgent fix and get kicked out by AnyDesk due to suspected “commercial use.”

Currently, I run a Proxmox system with 12 VMs and containers for testing k8s and databases. Previously, every time I remoted into a Windows VM for configuration, I had to route through a third-party relay server. As a result, latency often hit 200-300ms, making mouse movements an absolute nightmare.

RustDesk is the perfect escape. It is an open-source remote control software that allows you to set up your own Self-hosted Server. When running your own server, data travels directly through your infrastructure. The speed becomes near-instant, depending only on your local network bandwidth instead of waiting for a response from a server located halfway across the world.

Hardware Requirements (Extremely Lightweight)

You only need a Linux VM (Ubuntu 22.04 LTS recommended) with a very modest configuration:

  • CPU: 1 Core.
  • RAM: 1GB (In reality, the hbbs and hbbr services only consume about 40-60MB of RAM).
  • Disk: 10GB SSD.
  • OS: Any distro that supports Docker.

Pro tip: If you want to control your home computer while at a coffee shop, rent a cheap local VPS to act as a relay server. Latency will drop from hundreds of ms to under 20ms.

Deploying RustDesk Server with Docker Compose

Using Docker is the cleanest way to deploy. It helps you isolate the environment and makes future upgrades a breeze.

Step 1: Install Docker

sudo apt update
sudo apt install docker.io docker-compose -y
sudo systemctl enable --now docker

Step 2: Configure the Docker Compose File

First, create a directory for your data for easier management:

mkdir rustdesk-server && cd rustdesk-server
nano docker-compose.yml

Paste the following configuration into the file:

version: '3'

services:
  hbbs:
    container_name: hbbs
    image: rustdesk/rustdesk-server:latest
    command: hbbs -r <YOUR_IP>:21117 -k _
    volumes:
      - ./data:/root
    network_mode: "host"
    depends_on:
      - hbbr
    restart: unless-stopped

  hbbr:
    container_name: hbbr
    image: rustdesk/rustdesk-server:latest
    command: hbbr -k _
    volumes:
      - ./data:/root
    network_mode: "host"
    restart: unless-stopped

Important note: Replace <YOUR_IP> with your Linux machine’s IP address. The -k _ parameter forces clients to use a Public Key to connect, blocking any uninvited guests.

Step 3: Activate the System

sudo docker-compose up -d

After launching, check the ./data folder. You will see a file named id_ed25519.pub. Use the following command to retrieve your security key string:

cat ./data/id_ed25519.pub

Client-Side Setup

Now, you need to connect both the controlling and the controlled machines to your newly built server.

  1. Open the RustDesk Client on your computer.
  2. Navigate to Settings > Display > ID/Relay Server.
  3. Enter the following information:
    • ID Server: Your Linux machine’s IP (e.g., 10.0.0.50).
    • Relay Server: Your Linux machine’s IP (e.g., 10.0.0.50).
    • Key: The string you just retrieved from the id_ed25519.pub file.

Real-world experience: For a more professional setup, I usually point a domain (e.g., remote.mycloud.com) to the server IP. This way, even if I change the infrastructure, I don’t have to manually reconfigure every client.

Security Optimization and Monitoring

To check if the server is running correctly, you can view the container’s real-time logs:

sudo docker logs -f hbbs

If you see “ID registered” in the logs, congratulations! You have successfully connected.

Firewall Configuration

If you are using a VPS, ensure the following ports are open so data isn’t blocked:

  • TCP: 21115 to 21119.
  • UDP: 21116.
sudo ufw allow 21115:21119/tcp
sudo ufw allow 21116/udp

Double Layer of Security

Never skip the -k parameter. On the open internet, port-scanning bots will find your server within minutes. Using a Key ensures only authorized devices can establish a connection.

Additionally, I often combine RustDesk with Tailscale. In this setup, the RustDesk Server only operates within a virtual private network. You must connect to the VPN before you can remote in. This is the ultimate security method for Proxmox systems or personal homelabs.

Conclusion

Self-hosting a RustDesk Server not only saves you money but also provides a significantly smoother experience. With Docker Compose, it takes less than 5 minutes to set up a professional remote control system. From now on, all your data and visuals are under your control, with no more reliance on third parties.

Share:
Tags: