2:15 AM: Mass JWT Failures and the Elusive Clock Skew Culprit
The phone buzzed relentlessly with a flood of alerts from PagerDuty. The user authentication service began throwing 401 Unauthorized errors across the entire microservices cluster. Simultaneously, a PostgreSQL replica node triggered a split-brain warning.
Transaction logs revealed a bizarre anomaly: a request dispatched from the API gateway at 02:15:10 was logged as received by the backend at 02:14:48. Time had literally jumped backward by 22 seconds.
SSHing into the backend node showed ideal load metrics: CPU utilization was below 25%, and there was more than 12 GB of free RAM. Only after running the time check command did the issue surface:
timedatectl status
The root cause was instantly obvious. The clocks between the two API nodes had drifted apart by nearly 23 seconds. For JWT tokens carrying nbf (not before) or exp (expiration time) claims, a clock skew exceeding 5 seconds causes authentication libraries to reject incoming requests immediately, a failure pattern often observed when monitoring Keycloak with Prometheus and Grafana under severe drift conditions.
Why Did Chrony Suddenly Stop Synchronizing?
On modern Linux distributions (Ubuntu 20.04/22.04/24.04, RHEL 8/9, Rocky Linux), Chrony has completely superseded ntpd. For deeper insights into daemon architecture, see why Chrony is the modern standard over NTP on Linux. This daemon achieves fast clock convergence and provides smooth drift compensation. However, engineering teams often take it for granted after deployment, neglecting to set up proper health monitoring.
Inspecting Chrony’s detailed status on the failing node:
chronyc tracking
The output confirmed a complete loss of synchronization:
Reference ID : 00000000 () # No upstream server
Stratum : 0
Ref time (UTC) : Thu Jan 01 00:00:00 1970
System time : 22.841920853 seconds slow of NTP time # 22.8 seconds behind
Last offset : +0.000000000 seconds
RMS offset : 0.000000000 seconds
Frequency : 12.450 ppm slow
Residual freq : +0.000 ppm
Skew : 0.000 ppm
Root delay : 1.000000000 seconds
Root dispersion : 1.000000000 seconds
Update interval : 0.0 seconds
Leap status : Not synchronised
The three most common causes of this breakdown include:
- Blocked UDP Port 123: Network infrastructure teams tightening firewalls or updating security groups inadvertently drop outbound UDP 123 traffic. As a result, Chrony cannot reach the
pool.ntp.orgserver pool. - Service Crash Without Automatic Restart: The
chronyddaemon is killed by the OOM-killer or crashes unexpectedly. The default systemd service unit lacks automatic restart directives. - Virtual Machine Clock Drift: In virtualized environments (KVM, VMware, AWS EC2), CPU cycle contention makes virtual clocks drift rapidly. Without regular NTP discipline, the offset can compound by several seconds each day.
Troubleshooting: Quick Fix or Sustainable Solution?
Approach 1: Emergency Step Time Adjustment
When you need to restore production services immediately, you can force Chrony to step the clock rather than slewing it gradually:
# Restart the service and force a time step
sudo systemctl restart chrony || sudo systemctl restart chronyd
sudo chronyc makestep
This command snaps the offset back to safe limits within milliseconds. However, if UDP port 123 remains blocked or the daemon crashes again days later, the issue will inevitably recur.
Approach 2: Writing a Bash Script via Crontab
Some engineers opt for a quick workaround: schedule a cron job every 5 minutes to parse the offset and send alerts via a Slack or Telegram webhook:
#!/bin/bash
OFFSET=$(chronyc tracking | grep 'System time' | awk '{print $4}')
# Check if OFFSET exceeds 0.5s threshold and trigger an alert
While this patches the symptom, it quickly reveals major drawbacks. A cron job offers no time-series drift metrics, lacks failure alerts if the script itself fails, and is difficult to maintain consistently across hundreds of nodes.
Production-Grade Architecture: Monitoring with Prometheus Chrony Exporter
Instead of firefighting outages reactively, ingest clock offset, stratum, and synchronization status directly into a centralized Prometheus monitoring stack using chrony_exporter (open-sourced by SuperQ).
Step 1: Install Chrony Exporter on Linux Servers
Download the pre-compiled binary from the official GitHub releases:
ARCH="amd64"
VERSION="0.10.0"
curl -sLO "https://github.com/superq/chrony_exporter/releases/download/v${VERSION}/chrony_exporter-${VERSION}.linux-${ARCH}.tar.gz"
tar -xvf "chrony_exporter-${VERSION}.linux-${ARCH}.tar.gz"
sudo mv "chrony_exporter-${VERSION}.linux-${ARCH}/chrony_exporter" /usr/local/bin/
sudo chmod +x /usr/local/bin/chrony_exporter
rm -rf "chrony_exporter-${VERSION}"*
Step 2: Configure Systemd Service
To adhere to operational security best practices, run the exporter under a dedicated non-login system user:
sudo useradd -r -s /bin/false prometheus_exporter 2>/dev/null || true
sudo tee /etc/systemd/system/chrony_exporter.service << 'EOF'
[Unit]
Description=Prometheus Chrony Exporter
After=network.target chrony.service chronyd.service
Wants=chrony.service chronyd.service
[Service]
Type=simple
User=prometheus_exporter
Group=prometheus_exporter
ExecStart=/usr/local/bin/chrony_exporter --chrony.address=127.0.0.1:323
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now chrony_exporter
By default, Chrony exposes its management command socket over local UDP port 127.0.0.1:323. Verify your configuration in /etc/chrony/chrony.conf (or /etc/chrony.conf on CentOS/RHEL) to ensure cmdallow 127.0.0.1 or bindcmdaddress 127.0.0.1 is active.
Verify the local metrics endpoint:
curl -s http://localhost:9148/metrics | grep -E '^chrony_(last_offset|leap_status|stratum)'
Valid Prometheus metrics will look similar to this:
# HELP chrony_last_offset_seconds Last clock offset
# TYPE chrony_last_offset_seconds gauge
chrony_last_offset_seconds 0.000048120
# HELP chrony_leap_status Leap status (0: Normal, 1: 61s, 2: 59s, 3: Not synchronized)
# TYPE chrony_leap_status gauge
chrony_leap_status 0
# HELP chrony_stratum Chrony stratum
# TYPE chrony_stratum gauge
chrony_stratum 2
Step 3: Define Prometheus Scrape Job
Add the metrics target configuration to your prometheus.yml file:
scrape_configs:
- job_name: 'chrony_monitoring'
scrape_interval: 15s
static_configs:
- targets:
- '10.0.1.15:9148'
- '10.0.1.16:9148'
- '10.0.1.17:9148'
labels:
tier: 'backend'
env: 'production'
Reload Prometheus to apply the new configuration:
curl -X POST http://localhost:9090/-/reload
Step 4: Configure Early Warning Alert Rules
Do not wait until the clock drift hits 5 seconds before discovering an issue. Much like avoiding aggregate blind spots highlighted in Advanced PromQL: Don’t Let Average Numbers Mislead You, set up proactive threshold alerts in your alerting rules file (e.g., alerts/chrony_rules.yml):
groups:
- name: chrony_alerts
rules:
# Early warning when offset exceeds 100ms for 3 minutes
- alert: ChronyClockOffsetWarning
expr: abs(chrony_last_offset_seconds) > 0.1
for: 3m
labels:
severity: warning
annotations:
summary: "Server {{ $labels.instance }} clock drift detected"
description: "Clock offset is {{ $value | printf '%.4f' }}s, exceeding the 100ms safe threshold."
# Critical alert when offset exceeds 500ms
- alert: ChronyClockOffsetCritical
expr: abs(chrony_last_offset_seconds) > 0.5
for: 1m
labels:
severity: critical
annotations:
summary: "Critical clock skew on {{ $labels.instance }}"
description: "Clock offset reached {{ $value | printf '%.4f' }}s. High risk of JWT authentication failures and database split-brain."
# Alert when connection to all upstream NTP servers is lost
- alert: ChronyNotSynchronised
expr: chrony_leap_status == 3
for: 5m
labels:
severity: critical
annotations:
summary: "Chrony completely unsynchronized on {{ $labels.instance }}"
description: "Node has failed to synchronize with any upstream NTP server for the past 5 minutes."
# Alert when the exporter service is unresponsive
- alert: ChronyExporterDown
expr: up{job="chrony_monitoring"} == 0
for: 3m
labels:
severity: warning
annotations:
summary: "Chrony Exporter is down on {{ $labels.instance }}"
Conclusion
Time synchronization is one of the most overlooked components of IT infrastructure. Yet when it breaks down, the cascading failures can bring entire distributed systems to a halt, directly threatening distributed state consensus as seen when monitoring etcd clusters with Prometheus and Grafana. By actively tracking chrony_last_offset_seconds and chrony_leap_status, you can catch clock drift at the millisecond level and permanently eliminate late-night pages caused by out-of-sync servers.
