Posted inVMware

Configuring NTP on VMware ESXi and vCenter: Permanently Fix SSO and SSL Certificate Errors

VMware tutorial - IT technology blog
VMware tutorial - IT technology blog

Background: The 7-Minute Clock Drift Nightmare

When running a production VMware cluster for banking or fintech, what keeps systems engineers up at night often isn’t a failed hard drive or a severed fiber link. It stems from something seemingly innocuous: system clock drift.

On Monday morning, none of the administrators could log in to vCenter. The vSphere Client interface stubbornly reported “Invalid credentials” despite typing the exact root password. Browsers collectively blocked access with the error “Signing certificate is not valid yet”. Scheduled backups via Veeam halted abruptly. All vMotion migrations between hosts failed across the board.

Upon reviewing the logs, the root cause emerged: the ESXi host hosting the vCenter Server Appliance (VCSA) virtual machine was running exactly 7 minutes and 12 seconds ahead of the Domain Controller. In VMware architecture, the Kerberos authentication mechanism and SAML Tokens issued by STS (Security Token Service) tolerate a maximum time skew of 300 seconds (5 minutes). Exceed this threshold, and all inter-service token exchanges are instantly rejected.

A virtualized infrastructure lacking accurate NTP synchronization triggers a cascading chain of failures:

  • vCenter SSO, Active Directory, and Identity Provider (IdP) authentication break down entirely.
  • SSL/TLS certificates are invalidated because the system assumes they are either not yet valid or have already expired.
  • Logs across ESXi hosts and vCenter drift in timestamps. When troubleshooting incidents or conducting security audits, correlating logs becomes nearly impossible.
  • vSphere HA and DRS make skewed failover and balancing decisions due to misinterpreting heartbeat timing between cluster nodes.

Pre-Deployment Prerequisites

Before executing commands, you must establish clear time sources and verify network connectivity.

1. Time Source Topology (NTP Topology)

  • Domain Environment: Point all ESXi hosts and vCenter directly to the PDC Emulator (the Domain Controller holding the PDC role). This PDC will, in turn, sync from external upstream sources.
  • Standalone Environment: Define at least 3 independent NTP servers to prevent split-brain scenarios. Prioritize internal Layer 3 Core Switches/Routers combined with reliable public pools like time.google.com, time.cloudflare.com, or pool.ntp.org.

2. Firewall Rules and Connectivity

NTP operates over UDP port 123. Verify Access Control Lists (ACLs) and firewalls between the VMware Management Network and the target NTP servers. Traffic must be fully bidirectional.

Quickly test UDP 123 reachability from an administrative workstation:

nc -zvu 192.168.10.1 123

Step-by-Step Configuration

1. Configuring NTP on VMware ESXi Hosts via SSH

While the ESXi Host Client GUI is straightforward, using the CLI is significantly faster and standardizes configuration when managing clusters with dozens of hosts.

Step 1: SSH into the ESXi host

ssh [email protected]

Step 2: Check current time and NTP configuration

# ESXi always displays system time in UTC
esxcli system time get

# View currently configured NTP servers
esxcli system ntp get

Step 3: Enable the firewall rule for NTP Client

esxcli network firewall ruleset set --ruleset-id=ntpClient --enabled=true
esxcli network firewall ruleset list | grep ntpClient

Step 4: Configure NTP servers and enable the service

# Add the server list (comma-separated)
esxcli system ntp set --servers="192.168.10.1,time.google.com,vn.pool.ntp.org"

# Set the startup policy to start with host and enable the service
esxcli system ntp set --enabled=true
/etc/init.d/ntpd restart

2. Configuring NTP on vCenter Server Appliance (VCSA)

On vCenter 7.x and 8.x, system time is not configured via the vSphere Client. It must be managed through the VAMI web interface or the Appliance Shell.

Method 1: Using the VAMI Web Interface (Recommended)

  1. Open your browser and navigate to port 5480: https://<vcenter-ip>:5480.
  2. Log in using the appliance root credentials.
  3. Select Time from the left navigation menu.
  4. In the Time synchronization pane, click Edit.
  5. Change the Mode from Host to NTP. Enter the IP addresses or FQDNs of your NTP servers: 192.168.10.1, time.google.com.
  6. Click Save.

Field Note: Never select Host mode (synchronization via VMware Tools from the underlying ESXi host) if both ESXi and VCSA independently point to external NTP servers. This dual synchronization creates an offset feedback loop, causing the vCenter clock to continually skip backward and forward.

Method 2: Configuring via Appliance Shell

SSH directly into the VCSA as root. At the Command> prompt, execute:

# Check the current mode
timesync.get

# Switch mode to NTP and specify the servers
timesync.set --mode NTP
timesync.set --servers "192.168.10.1,time.google.com"

# Verify the updated configuration
timesync.get

Post-Configuration Verification & Monitoring

Applying the configuration is only half the battle. Spend 3 minutes validating that the synchronization daemon is actively receiving valid upstream time data.

1. Measuring Skew on the ESXi Host

Run the peer inspection command on the ESXi host:

[root@esx-node01:~] ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.10.1    10.0.0.1         2 u   18   64  377    0.412   -0.035   0.021
+time.google.com 216.239.35.4     1 u   42   64  377   22.180    0.104   0.318
+vn.pool.ntp.org 133.243.3.8      2 u   11   64  377   14.650   -0.180   0.245

Key metrics to examine:

  • The * prefix: Indicates the primary reference NTP server currently utilized by the system.
  • The + prefix: Represents candidate peers that will automatically take over if the primary source becomes unreachable.
  • The reach = 377 column: An octal value confirming the last 8 consecutive poll attempts received valid responses. A value of 0 indicates that UDP port 123 traffic is blocked.
  • The offset column: The actual time offset in milliseconds (ms). Any value under 10ms is thoroughly optimal for production environments.

2. Verifying on vCenter

Access the VCSA Bash shell and inspect the system daemon status:

timedatectl status

Ensure both parameters confirm healthy operation: System clock synchronized: yes and NTP service: active.

3. Setting Up Proactive Alerts

Do not wait for a full-scale outage before diagnosing clock issues:

  • Create an Alarm Definition in vCenter monitoring the Host time out of sync event. Configure alerts to dispatch to Telegram or Email immediately when host skew exceeds 5 seconds.
  • If a physical host consistently drifts even when NTP services are operational, inspect the CMOS battery on the motherboard or flash the latest BIOS and server firmware updates from the vendor.
Share:
Tags: