Deploying Proxmox Mail Gateway as a Protective Layer for Your Mail Server: A Professional Guide to Virtualizing Spam and Malware Filtering

Mail server under spam attack — a real-world problem

If you’re running an internal mail server — whether Postfix, Exchange, or Zimbra — you’ve almost certainly stared at hundreds of junk emails every day and wondered: how do I stop this garbage before it ever reaches the user’s inbox?

That was exactly the problem I ran into when setting up a mail server for a small project. Postfix was working fine, but spam and phishing emails were getting through freely. The built-in antispam was weak, and paying for an external filtering service wasn’t in the budget. My solution: place Proxmox Mail Gateway (PMG) in front of the mail server to act as a gateway that filters all SMTP traffic.

I run a homelab with Proxmox VE managing 12 VMs and containers — it’s my playground for testing everything before pushing to production. I set up PMG as a separate VM that processes email before it reaches the main mail server. The result: spam dropped by over 95%, with zero additional cost.

What is Proxmox Mail Gateway and how does it work?

Proxmox Mail Gateway (PMG) is an open-source email security solution built on Debian, with well-known filtering tools baked right in:

• SpamAssassin — analyzes headers, content, and spam scoring
• ClamAV — scans for viruses and malware in attachments
• Postfix — MTA for receiving and relaying email
• Fetchmail + Razor/Pyzor — collaborative spam detection

The flow is straightforward:

1. Email from the internet hits your MX record (pointing to PMG)
2. PMG receives it, checks the spam score, scans for viruses, and applies rules
3. Clean email is relayed to the internal mail server
4. Dirty email is blocked, quarantined, or tagged

Your real mail server stays completely hidden behind PMG — it no longer needs to be exposed to the internet at all.

Hands-on: Step-by-step installation and configuration

Step 1: Prepare the environment

PMG needs a dedicated machine (or VM). Minimum specs for stable operation:

• CPU: 2 cores
• RAM: 2 GB (4 GB recommended for high traffic)
• Disk: 32 GB
• 2 IPs: 1 public (receives mail from the internet), 1 internal (relays to the mail server)

If you’re using Proxmox VE like I am, create a new VM, mount the PMG ISO, and install it as you normally would. The ISO is available on the Proxmox website — completely free.

Step 2: Install PMG

Boot from the ISO — the installer is pretty straightforward. The most important section is networking:

• Hostname: set a FQDN, e.g. mail-gw.yourdomain.com
• IP: public or internal IP depending on your topology
• Gateway: your default gateway

After installation, access the web UI at:

https://<IP_PMG>:8006

Log in with root and the password you set during installation.

Step 3: Configure Relay Domains

This is the most critical step — telling PMG which domains it should accept email for and where to relay them.

Go to Configuration → Mail Proxy → Relay Domains and add your domain:

# Domain to receive email for
yourdomain.com

# Relay target: IP and port of the internal mail server
# Example: 192.168.1.50:25

In the web UI, fill in the form:

• Domain: yourdomain.com
• Relay Host: 192.168.1.50
• Relay Port: 25

Step 4: Point Your MX Record to PMG

Once PMG is installed and the relay is configured, update your DNS records:

# DNS records to update
# MX record points to PMG (not the old mail server)
yourdomain.com.  MX  10  mail-gw.yourdomain.com.

# A record for the PMG hostname
mail-gw.yourdomain.com.  A  <IP_PUBLIC_PMG>

# SPF record (if using PMG for outbound mail)
yourdomain.com.  TXT  "v=spf1 ip4:<IP_PUBLIC_PMG> ~all"

Important: your internal mail server should have port 25 firewalled from the public internet — only allow connections from the PMG’s IP.

Step 5: Configure Spam Filter Rules

Go to Configuration → Mail Filter → Rules to create spam handling rules. PMG ships with sample rules, but I usually customize them:

# Rule 1: Block emails with a high spam score
# Condition: Spam Score >= 7
# Action: Block (reject with 5xx code)

# Rule 2: Tag suspicious emails
# Condition: Spam Score >= 4
# Action: Add Header X-Spam-Flag: YES

# Rule 3: Block dangerous attachments
# Condition: Attachment matches *.exe, *.bat, *.vbs, *.js
# Action: Block + Notify Admin

In the web UI, these rules are configured with drag-and-drop — fairly intuitive. Each rule has a priority; lower numbers are processed first.

Step 6: Enable ClamAV and Test

ClamAV comes pre-installed, but you need to make sure the database is up to date:

# SSH into PMG and manually update the virus database
freshclam

# Verify services are running
systemctl status clamav-daemon
systemctl status clamav-freshclam

# Test by sending the EICAR test virus through PMG
# EICAR string (harmless — for testing only):
echo 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' | \
  sendmail -v [email protected]

If ClamAV is working correctly, the email containing EICAR will be blocked and appear in the quarantine log.

Step 7: Monitor Logs and Quarantine

This is my favorite feature of PMG — the mail traffic monitoring interface is incredibly detailed:

• Mail Log: see every email that passed through, its spam score, and the action taken
• Quarantine: emails that were held — admins can release or delete them
• Statistics: spam/virus charts by day, week, and month

From the command line, you can also query the logs directly:

# View the recent mail queue
postqueue -p

# Stream PMG filter logs in real time
journalctl -u pmg-smtp-filter -f

# Inspect a specific queued message
postcat -vq <QUEUE_ID>

Step 8: Configure Whitelist and Blacklist

In practice, legitimate emails will occasionally be misidentified as spam (false positives). PMG lets you whitelist by sender, domain, or IP:

# Go to: Configuration → Mail Filter → Whitelist
# Add trusted domains:
github.com
google.com

# Or whitelist by sender IP (useful for trusted partners)
# Configuration → Mail Proxy → Networks

Conclusion

PMG solves a real-world problem in an elegant way: instead of cramming antispam functionality into your main mail server, you split it out into a dedicated layer. Your internal mail server doesn’t have to worry about spam or viruses — it just receives clean email from PMG and serves users.

What I appreciate most is the web UI — managing rules, viewing logs, releasing quarantined messages — everything is doable from a browser without memorizing complex commands. For a junior sysadmin just getting started with mail server management, that’s a huge advantage over manually configuring SpamAssassin.

If you’re already running Proxmox VE, it takes less than 30 minutes to spin up a PMG VM and start filtering spam right away. For moderate traffic (a few thousand emails per day), 2 CPUs and 2 GB of RAM is more than enough to run it comfortably.