Real-World Incident: When Flaky VM Networking Defeats Local Packet Captures
Last week, our 8-host ESXi cluster ran into a rather tricky issue. A VM hosting our payment gateway service would intermittently throw SSL Handshake Timeout errors when communicating with a banking gateway. It happened every 3 to 5 minutes per hour, dropping about 4% of total transactions. To make matters worse, manual curl and ping tests showed perfectly normal connectivity, returning HTTP 200 responses without a hitch.
The instinctive reaction for many engineers is to SSH into the VM and run tcpdump. In a production environment, however, you quickly run into significant roadblocks:
- The VM is a pre-packaged third-party appliance with no root access and no ability to install additional packages.
- PCI-DSS compliance strictly forbids installing packet-sniffing software directly on cardholder data servers to prevent data leaks and CPU spikes.
- Raw packets must be inspected at the virtual network adapter level before iptables or local OS firewalls can touch them.
With the VM off-limits, the only viable solution was to extract traffic directly from VMware’s virtual switch ports.
Under the Hood: Why Virtual Switches Block Cross-VM Packet Captures
A standard vSwitch operates at Layer 2. When VM A transmits a packet to VM B, the switch looks up its MAC address table and forwards the frame directly to the destination port. VM C, sitting on the same port group, is completely blind to this traffic stream unless it involves broadcast or multicast frames.
This architecture eliminates packet collisions and isolates traffic between tenants. Consequently, if you want VM C to capture traffic flowing between A and B, you have to intervene at the hypervisor switching layer.
3 Approaches to Packet Capture on VMware Infrastructure
Method 1: Enable Promiscuous Mode on the Port Group
The quickest workaround is switching Promiscuous Mode from Reject to Accept under the Port Group’s Security policy.
Pros: Takes less than a minute to configure. A VM running Wireshark attached to the same port group will immediately start receiving packets.
Cons: Severe security risks. Any VM within that port group can eavesdrop on neighboring traffic. Even worse, this approach fails completely if the target VM and sniffer VM reside on different ESXi hosts.
Method 2: Use pktcap-uw via ESXi Shell
VMware includes the native CLI utility pktcap-uw directly inside the ESXi kernel. You SSH into the ESXi host hosting the target VM and capture packets by switch port ID:
# List port IDs on the vSwitch
esxcfg-vswitch -l
# Capture packets on port 50331658 and write to datastore
pktcap-uw --switchport 50331658 --dir 2 -o /vmfs/volumes/datastore1/trace.pcap
Pros: Captures packets deep at the VMkernel layer, independent of switch configuration.
Cons: Requires enabling SSH on the ESXi host (usually strictly locked down under enterprise security baselines). Downloading the pcap file from the datastore to a local workstation is tedious and impractical for live troubleshooting during an active incident.
Method 3: Configure Port Mirroring on a vSphere Distributed Switch (vDS)
This feature works similarly to SPAN/ERSPAN on physical Cisco switches. The vDS automatically mirrors frames from the target VM’s vNIC and delivers them to the sniffer VM’s vNIC. It works seamlessly whether the two VMs reside on the same ESXi host or across different hosts in the cluster.
Step-by-Step Implementation: Configuring Distributed Port Mirroring on vDS
For enterprise environments, this is the safest and most robust approach: no exposed SSH ports on ESXi hosts, full compliance with internal security policies, and real-time packet inspection.
Step 1: Deploy a Dedicated Sniffer VM
Deploy a VM running Ubuntu 22.04 LTS or Windows Server, pre-installed with Wireshark and tshark. Attach two virtual network adapters (vNICs) to this VM:
- vNIC 1: Connected to your standard management network for SSH/RDP access.
- vNIC 2: Dedicated to capturing mirrored traffic (do NOT assign an IP address).
In vSphere Client, create a new Distributed Port Group named PG-Sniffer-Capture and assign it to vNIC 2. Under the Security settings of this port group, configure:
- Promiscuous mode: Accept
- MAC address changes: Accept
- Forged transmits: Accept
Step 2: Identify the Target VM’s Port ID
Open vSphere Client > Networking > Select your active vDS > Select the target VM’s Distributed Port Group > Navigate to the Ports tab. Locate the target VM by name and note down the value in the Port ID column (e.g., port 1024 for the payment-gw VM, and port 2048 for vNIC 2 of the sniffer VM).
Step 3: Create the Port Mirroring Session
- On the vDS configuration page, navigate to Configure > Port Mirroring.
- Click New to launch the session creation wizard.
- Select the session type: Distributed Port Mirroring (supports source and destination across different ESXi hosts).
- Name the session:
Mirror-Payment-GW-Trace, and toggle the status to Enabled. - Under Sources: Enter the target VM’s Port ID (
1024), and set traffic direction to Ingress / Egress to capture bidirectional traffic. - Under Destinations: Enter the capture vNIC’s Port ID on the sniffer VM (
2048). - Review the settings and click Finish.
If you need to automate this across multiple ports, you can use the following standard PowerCLI script:
# Connect to vCenter Server
Connect-VIServer -Server vcenter.corp.local -User [email protected] -Password "SecretP@ss!"
# Retrieve Distributed Switch details
$vds = Get-VDSwitch -Name "vDS-Core-Cluster"
$vdsView = Get-View $vds
# Initialize port mirroring session specifications
$configSpec = New-Object VMware.Vim.VMwareDVSConfigSpec
$session = New-Object VMware.Vim.VMwareDVSPortMirroringSessionConfigSpec
$session.key = "TraceSession01"
$session.name = "Live-Debug-Session"
$session.enabled = $true
$session.direction = "both"
$session.sourcePortKey = @("1024")
$session.destinationPortKey = @("2048")
$configSpec.portMirroringConfigSpec += $session
# Apply configuration directly to the vDS via API
$vdsView.ReconfigureDvs_Task($configSpec)
Step 4: Capture and Analyze Packets on the Sniffer VM
Log in to the sniffer VM. Assuming the capture vNIC is identified as ens224 on Linux, run a quick check to verify incoming packets:
# Inspect the first 10 incoming packets on the capture interface
sudo tshark -i ens224 -c 10
# Capture all port 443 traffic and write to a pcap file
sudo tcpdump -i ens224 -nn "tcp port 443" -w /tmp/payment_tls.pcap
Open payment_tls.pcap in Wireshark or inspect traffic directly through the GUI on interface ens224. Paste the following display filter into the search bar:
tcp.port == 443 && (tcp.flags.reset == 1 || tls.record.content_type == 21)This filter immediately isolates TCP RST packets (unexpected connection teardowns) and TLS Alerts (certificate or TLS version negotiation failures). In our real-world case, Wireshark caught a TLS Alert (Level: Fatal, Description: Handshake Failure) packet originating from the bank’s IP: they had upgraded their cipher suites without prior notice, causing connection requests from our payment gateway’s outdated OpenSSL library to be rejected.
Two Golden Rules When Running Port Mirroring
- Always disable the session immediately after troubleshooting: Mirrored traffic doubles switch throughput. For VMs pushing 2–5 Gbps, leaving a session running indefinitely will saturate the ESXi host’s physical uplinks and overwhelm disk I/O on the sniffer VM.
- Enable Sampling Rate under heavy load: When monitoring high-throughput database or cache clusters, configure a Sampling Rate (e.g., mirroring 1 out of every 5 packets) in the vDS session settings to protect the capture interface from dropping packets.
