Managing Fedora CoreOS Auto-updates: Prevent Simultaneous Cluster Reboots – ITFROMZERO

Table of Contents

When Automation Backfires

Using Fedora as a dev machine is great because the packages are always up-to-date. However, when I brought Fedora CoreOS (FCOS) into production, its “fully automated” philosophy caused a rather ironic problem. By default, FCOS automatically downloads patches, installs them, and reboots to apply the new kernel.

Imagine you are running a 3-node Web Server cluster. At exactly 3 AM, all 3 nodes detect an update. They download it and simultaneously… reboot. The result is a total service outage for 3-5 minutes just because the system was too “diligent.” For sensitive systems like Etcd, losing 2/3 of nodes at once causes a loss of quorum and paralyzes the entire cluster.

Why Default Zincati Isn’t Enough

In the FCOS ecosystem, Zincati is the agent responsible for checking for new releases and coordinating rpm-ostree. The problem is that Zincati operates very independently. It doesn’t care whether neighboring nodes are up or down.

The “Atomic Update” mechanism forces the server to reboot to activate the new deployment. If you stick with the default configuration, you face three major risks:

Simultaneous cluster reboots causing service disruption (downtime).
No priority for standby nodes.
Risk of corrupting the state of distributed applications (like Kubernetes or Database clusters).

The Solution: Coordinating Reboots with FleetLock

To control the situation, we need a “permission” mechanism before rebooting. FleetLock acts as a gatekeeper. Before rebooting, Zincati must request a “lock.” If the FleetLock server approves, the node proceeds. Otherwise, it must wait its turn.

Step 1: Setting up the FleetLock Server

You need a small service to manage the lock state. The fastest way is to run fleetlock-server as a container. You can host it on a separate management machine or within the cluster itself if it has high availability.

# Run FleetLock server using Podman
podman run -d --name fleetlock-server \
  -p 8080:8080 \
  -v ./fleetlock.toml:/config/fleetlock.toml:Z \
  quay.io/coreos/fleetlock-server:latest

In the fleetlock.toml file, limit the number of nodes allowed to reboot at once. Typically, we only allow 1 node (slots = 1) to ensure safety:

[groups.production]
slots = 1

Step 2: Configuring Zincati on FCOS Nodes

We need to instruct Zincati on each node to stop rebooting arbitrarily. Instead, it must consult the FleetLock server. You should configure this via a Butane file before initializing the node.

Here is the configuration snippet I usually use to point to FleetLock:

variant: fcos
version: 1.4.0
storage:
  files:
    - path: /etc/zincati/config.d/50-fleetlock.toml
      contents:
        inline: |
          [updates]
          strategy = "fleetlock"
          [updates.fleetlock]
          base_url = "http://192.168.1.100:8080/"
          group = "production"

Important Notes:

strategy = "fleetlock": Tells Zincati to use the FleetLock protocol instead of rebooting freely.
base_url: The IP address of the FleetLock server you just set up.
group: The server group name to manage locks separately (e.g., staging, production).

Step 3: Verifying the Operation

After applying the configuration, you can check the agent status with the command:

systemctl status zincati

When an update is available, Zincati logs will show a “Waiting for lock” status. The first node to acquire the lock will reboot. Once it’s back online, the lock is released, and the next node begins the process. Your cluster will always have at least 2 active nodes (in a 3-node setup).

Real-world Best Practices

Through actual operation, I’ve drawn three key lessons to make the system more stable:

Set a Maintenance Window: Even with FleetLock, it’s best to limit updates to off-peak hours (e.g., 2 AM – 4 AM) using the updates.periodic.window configuration.
Monitor via Prometheus: Track the zincati_pushed_updates_total metric to identify nodes that are stuck or waiting too long for an update.
FleetLock Availability: If the FleetLock server goes down, Zincati will play it safe and… do nothing. This prevents simultaneous reboots but will delay security patches.

Combining Zincati and FleetLock turns Fedora CoreOS into a disciplined infrastructure. Instead of worrying every time a new patch arrives, I can now trust the system to self-heal without fear of being woken up at midnight by a service outage.