A 3.5-Second Clock Drift Brought Down Our Payment Pipeline
About six months ago, our microservices cluster experienced an unforgettable on-call incident. Hundreds of payment transactions failed continuously because newly issued JWT tokens were instantly flagged as expired. When the on-call team opened Kibana to trace logs, events were jumping erratically across nodes, making it impossible to follow the execution flow.
The root cause was neither in the application code nor the database. The API server and worker nodes had drifted apart by exactly 3.5 seconds. For systems running Cassandra, CockroachDB, or Kubernetes etcd, a drift of just a few tens of milliseconds can disrupt consensus algorithms and corrupt data integrity.
Why Do Linux Servers Constantly Suffer from Clock Drift?
The motherboard’s real-time clock (RTC) relies on a quartz crystal oscillator. Its oscillation frequency fluctuates with server room temperature, voltage, and component aging. This clock drift causes physical servers to gain or lose roughly 1 to 2 seconds per day.
In virtualized environments like KVM, VMware, or AWS EC2, the issue is even more pronounced. When virtual CPUs suffer from resource contention (CPU steal) or when VMs migrate between hosts, the clock interrupts momentarily. Time offset can skyrocket to several minutes in just a few hours.
If every individual server syncs directly with public NTP servers over the Internet, unstable international transit leads to high network jitter. Furthermore, indiscriminately opening UDP port 123 to the outside world poses security risks, such as being exploited in NTP amplification attacks.
Reviewing Legacy Time Synchronization Methods
Before standardizing our infrastructure, I experimented with several approaches:
1. Running ntpdate via Cron Jobs
Many systems still schedule ntpdate via cron every 15 minutes. This approach is hazardous because ntpdate corrects time via step adjustments. Stepping the clock backward can easily freeze or corrupt timestamp-sensitive processes, such as MySQL replication binlogs or scheduled jobs.
2. Using the Traditional NTPd Daemon
ntpd has served the Linux ecosystem for decades. Its biggest drawback is slow synchronization convergence. When a server reboots or recovers from a network outage, ntpd can take 20 to 40 minutes to stabilize the clock. It also consumes more memory than modern alternatives.
3. Using the Default systemd-timesyncd
Built directly into Ubuntu and Debian, this tool is extremely lightweight. For a standalone workstation, it works great. However, systemd-timesyncd is strictly an SNTP client and cannot act as a local NTP server to serve time to other internal machines.
Production Solution: Setting Up an Internal Chrony NTP Server
After years of production operations, Chrony has proven to be the most reliable choice. This daemon synchronizes system time rapidly on boot, slews clock frequency smoothly without stepping time backward for running applications, and consumes less than 15MB of RAM.
Deployment Architecture
The standard topology consists of 1 to 2 Chrony Master servers placed in the DMZ or Management subnet. These servers fetch time directly from trusted upstream Stratum 1/2 sources. All internal database servers, backend nodes, and network switches/routers will point directly to this Master cluster.
Step 1: Install Chrony
On Debian and Ubuntu:
sudo apt update
sudo apt install chrony -yOn RHEL, AlmaLinux, Rocky Linux, or CentOS Stream:
sudo dnf install chrony -yStep 2: Configure Chrony as a Local NTP Server
The configuration file is located at /etc/chrony/chrony.conf (Ubuntu/Debian) or /etc/chrony.conf (RHEL/CentOS). Open the file with root privileges:
sudo nano /etc/chrony/chrony.confOptimized configuration for the Master server:
# Define upstream NTP servers close to your geographical region to minimize latency
server 0.asia.pool.ntp.org iburst
server 1.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
server time.google.com iburst
server time.cloudflare.com iburst
# Record clock drift and frequency offset to compensate on reboot
driftfile /var/lib/chrony/chrony.drift
# Allow stepping the clock if the offset exceeds 1 second during the first 3 updates
makestep 1.0 3
# Automatically sync the system clock to the hardware RTC
rtcsync
# Allow internal subnets to query time
allow 192.168.10.0/24
allow 10.10.0.0/16
# Serve time locally even when the Internet connection is lost
local stratum 10Subnetting Tip: When subnetting multiple server clusters, you can use toolcraft.app/en/tools/developer/ip-subnet-calculator to quickly compute accurate network and broadcast ranges, preventing misconfigured IP ranges in your allow directives.
Step 3: Configure Firewall and Enable the Service
NTP operates on UDP port 123. Open this port on the Master server’s firewall.
Using UFW (Ubuntu/Debian):
sudo ufw allow 123/udp
sudo ufw reloadUsing Firewalld (RHEL/CentOS/Rocky):
sudo firewall-cmd --add-service=ntp --permanent
sudo firewall-cmd --reloadEnable and start Chrony to run on system boot:
sudo systemctl enable --now chrony
sudo systemctl restart chronyStep 4: Verify Synchronization Status
Run the following command to list upstream time sources:
chronyc sources -vA source prefixed with ^* indicates that Chrony has selected it as the primary reference source:
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* time.cloudflare.com 3 6 377 25 -122us[ -150us] +/- 14ms
^+ 118.69.177.108 2 6 377 24 -450us[ -478us] +/- 22ms
^+ time.google.com 1 6 377 23 +85us[ +57us] +/- 18msCheck detailed system clock tracking parameters:
chronyc trackingFour key metrics to look out for:
- Reference time: The timestamp of the most recent successful synchronization.
- Stratum: The distance from the reference atomic clock (your Master server is typically Stratum 2 or 3).
- System time: The current offset from standard UTC. A healthy offset is under 0.0005 seconds (0.5ms).
- Root delay: Total network delay to the primary Stratum 1 source.
View active internal client connections:
chronyc clientsStep 5: Configure Clients to Sync with the Internal Master
On application nodes within your LAN, install Chrony, edit the configuration file, remove the public pools, and point exclusively to the Master server’s IP:
# Point directly to the internal Chrony Master IP
server 192.168.10.10 iburst
makestep 1.0 3
rtcsyncRestart the Chrony service on the client:
sudo systemctl restart chrony
chronyc sourcesKey Operational Best Practices
When deploying Chrony to production, keep these three essential technical guidelines in mind:
- Disable competing time services: Never run
systemd-timesyncdorntpdalongside Chrony. Completely disable them withsudo systemctl disable --now systemd-timesyncdto prevent race conditions over kernel clock control. - Control the makestep parameter: Only allow clock stepping during the first three updates at boot time. Stepping the clock backward under high application load can immediately corrupt distributed locks and database transactions.
- Configure at least 3 to 4 upstream sources: Chrony’s Marzullo-based algorithm requires at least three independent sources for reliable cross-validation. If one server advertises inaccurate time, Chrony automatically isolates and rejects the outlier.
