The Pain of Managing Dozens of Servers Manually
The sight of SSHing into each node in a cluster of 30-50 servers to type <a href="https://itfromzero.com/en/centos-vi-en/updating-centos-stream-9-when-to-reboot-using-needs-restarting-effectively.html">dnf update</a> or check for security vulnerabilities is a true nightmare. Six months ago, our system expanded rapidly, leading to a “mess” of versions: some machines running CentOS 7, some on Rocky Linux 8, and others on CentOS Stream 9. Controlling which packages were allowed into production at that time was nearly impossible.
After weighing Red Hat Satellite (high licensing costs) against Uyuni, I decided on Foreman combined with Katello. Foreman handles the provisioning and lifecycle management, while Katello is a powerful assistant for managing Repositories, Content Views, and Errata (patches).
Lessons learned from experience: Don’t view Foreman as just a simple installation tool. It serves as the “Single Source of Truth” for your entire Linux infrastructure. Here is the process I deployed on CentOS Stream 9.
System Requirements (Prioritize I/O Performance)
Foreman and Katello are resource-intensive because they run PostgreSQL, Candlepin, and Pulp simultaneously. Don’t try to run them on weak hardware unless you want the system to hang during repository synchronization. Recommended configuration based on my experience:
- CPU: Minimum 4 Cores (8 Cores recommended if managing over 100 clients).
- RAM: 12GB – 16GB (Below 8GB, Candlepin’s Java service is very prone to OOM – Out of Memory).
- Disk: At least 100GB for
/var/lib/pulp. If syncing multiple OS versions, prepare 300-500GB. - Network: Static IP and an accurate FQDN are mandatory.
Step 1: Setting up Hostname and Firewall
Foreman is extremely sensitive to server identification. If the hostname cannot be reverse-resolved, the installer will fail immediately. If you don’t have a dedicated DNS server, configure your /etc/hosts file carefully:
# Set FQDN
hostnamectl set-hostname foreman.itfromzero.local
# Declare internal IP
echo "192.168.1.100 foreman.itfromzero.local foreman" >> /etc/hosts
Next, update the system and open the necessary service ports. Katello requires quite a few ports to communicate with clients:
dnf update -y
firewall-cmd --permanent --add-service={http,https,dns,dhcp,dhcpv6,tftp,mcollective,puppetmaster,proxy-dhcp}
firewall-cmd --reload
Step 2: Configuring Repositories for Foreman & Katello
On CentOS Stream 9, we need to enable the official repositories from the Foreman and Puppet projects. Note that compatibility between Katello and Foreman versions is crucial.
# Install repo sources
dnf install -y https://yum.theforeman.org/releases/3.9/el9/x86_64/foreman-release.rpm
dnf install -y https://yum.theforeman.org/katello/4.11/katello/el9/x86_64/katello-repos-latest.rpm
dnf install -y https://yum.puppet.com/puppet7-release-el-9.noarch.rpm
# Enable appropriate Ruby and PostgreSQL modules
dnf module enable ruby:3.1 -y
dnf module enable postgresql:13 -y
Step 3: Executing the Foreman Installer
Instead of manually installing each component, Foreman provides a powerful script driven by Puppet. I use the katello scenario to enable full content management features.
dnf install -y foreman-installer-katello
foreman-installer --scenario katello \
--foreman-initial-admin-username admin \
--foreman-initial-admin-password YourSecurePassword \
--foreman-proxy-dns=false \
--foreman-proxy-dhcp=false
Note: I set the DNS and DHCP proxies to false because the system already has existing servers managing these services. The installation process usually takes 20 to 40 minutes depending on your disk speed.
Step 4: Content Management (Practical Experience)
Once you’ve accessed the Dashboard, don’t rush to add hosts immediately. Set up the Organization and Location first to organize your management structure logically from the start.
1. Smart Repository Synchronization
In the Content > Products section, when adding repositories like BaseOS or AppStream, you should select the “On Demand” download policy. This mode saves dozens of gigabytes of disk space because Foreman only downloads packages that the clients actually request.
2. Content Views – The Safety Barrier
This is Katello’s most valuable feature. Content Views allow you to create “snapshots” of your software.
- Dev Environment: Receives the latest patches daily.
- Prod Environment: Only receives patches that have been tested and stable for at least 2 weeks.
When a Zero-day vulnerability occurs, you just need to update the Content View and “Promote” it from Dev to Prod with just a few clicks.
Step 5: Connecting Clients and Monitoring Errata
To manage a new server, you don’t need a complicated installation. Use the global registration feature in the Hosts > Register Host menu.
# Run the registration script on the client machine
curl -X GET -k https://foreman.itfromzero.local/register_host_script > register.sh
chmod +x register.sh
./register.sh
After successful registration, all information regarding missing patches (Errata) will be displayed visually on the Web UI. You can see exactly which servers are affected by security vulnerabilities without having to SSH into each machine to check.
Summary After 6 Months of Operation
The system runs stably and has saved me about 80% of routine maintenance time. However, keep the following in mind:
- Clean up Tasks regularly: Katello generates many task logs. Go to Monitor > Tasks to delete old data and prevent the PostgreSQL database from bloating.
- Use LVM: Always place the
/var/lib/pulpdirectory on LVM so you can expand its capacity online as the number of repositories grows. - Backup is vital: Use the
foreman-maintain backup offlinecommand before every system upgrade.
Deploying Foreman + Katello may seem complex initially, but the value it brings in terms of centralized management and system security is well worth the effort.
