Installing Apache Kafka on CentOS Stream 9: KRaft Mode Deployment Without ZooKeeper - ITFROMZERO - Share tobe shared! System outages caused by synchronous transaction bottlenecks are a major headache during traffic spikes. Learn how to deploy Apache Kafka on CentOS Stream 9 using KRaft mode, eliminating ZooKeeper to save RAM and streamline operations.

Table of Contents

1. Real-World Scenario: When Infrastructure Collapses Under Transaction Bottlenecks

Imagine you run an online concert ticketing website. On normal days, the system handles only a few dozen scattered transactions. Everything runs smoothly.

Disaster strikes at 10 AM on the day exclusive VIP tickets go on sale. Roughly 20,000 users click ‘Checkout’ in the exact same second.

The fallout is instantaneous. Payment servers become overloaded, with CPU utilization spiking to 100%. Inventory services lag behind, leading to double-booked tickets. Worse, microservices communicate via synchronous HTTP calls (A calls B, B waits for C). A mere 3-second delay in any single service brings the entire pipeline grinding to a halt.

Frustrated users staring at loading spinners hit F5 repeatedly. Phantom requests surge fivefold. The primary database exhausts its connection pool and grinds to a complete standstill.

2. Root Cause Analysis: The Tight Coupling Trap and the ZooKeeper Burden

The incident stems from two fundamental architectural flaws:

Tight Coupling: Confirming an order requires both the inventory and email services to be online simultaneously. If the email service slows down, ticket purchases hang with it. This design completely strips the system of localized fault tolerance.
Misaligned Storage Tools: Relational databases (RDBMS) are built to maintain final state, not to withstand an avalanche of 20,000 to 30,000 incoming events per second.

Traditionally, the standard solution has been to deploy Apache Kafka to absorb this data stream. However, the legacy architecture introduced another operational headache: Apache ZooKeeper.

Anyone on on-call duty knows the pain of managing a ZooKeeper cluster alongside Kafka. ZooKeeper runs on its own JVM, demanding an extra 2 to 4 GB of RAM per node and requiring its own dedicated backup strategy. During network partitions, metadata synchronization between ZooKeeper and Kafka brokers often drifts, making cluster recovery excruciatingly slow.

3. Comparing Common Architectural Approaches

To address data ingestion bottlenecks, engineers typically weigh three options:

Vertical Database Scaling: Upgrading RAM from 32 GB to 128 GB and switching to Enterprise NVMe SSDs. This treats immediate symptoms, but infrastructure costs triple, and the bottleneck simply shifts to the next high-demand ticket sale.
Traditional Message Queues (RabbitMQ, Redis Pub/Sub): Decouples services effectively. However, when you need to retain event logs for 30 days for event replays or require throughput exceeding 100,000 messages/sec, RabbitMQ begins struggling with memory pressure.
Kafka in KRaft Mode: Retains Kafka’s proven capacity to handle millions of messages per second while completely eliminating the intermediate ZooKeeper layer.

4. Solution: Deploying Apache Kafka with KRaft on CentOS Stream 9

Starting with version 3.3, KRaft (Kafka Raft Metadata Mode) is officially production-ready. Kafka elects controllers and manages cluster metadata internally using the Raft consensus algorithm. Clusters start up in seconds instead of minutes, while effortlessly managing millions of partitions.

Step 1: Prepare the Environment and Install Java 17

Kafka runs on the JVM. CentOS Stream 9 provides robust, long-term support (LTS) for OpenJDK 17:

sudo dnf update -y
sudo dnf install -y java-17-openjdk-headless wget tar
java -version

Create a dedicated system user without interactive shell access for better security:

sudo useradd -r -d /opt/kafka -s /sbin/nologin kafka

Step 2: Download and Extract Apache Kafka

Download the Kafka binary distribution (version 3.7.0 compiled with Scala 2.13 is recommended):

cd /tmp
wget https://archive.apache.org/dist/kafka/3.7.0/kafka_2.13-3.7.0.tgz
sudo tar -xzf kafka_2.13-3.7.0.tgz -C /opt
sudo mv /opt/kafka_2.13-3.7.0 /opt/kafka
sudo chown -R kafka:kafka /opt/kafka

Step 3: Configure Kafka KRaft (Single-Node)

In KRaft mode, the primary configuration file resides at /opt/kafka/config/kraft/server.properties. Open it for editing:

sudo vi /opt/kafka/config/kraft/server.properties

Configure the basic parameters as follows (replace 192.168.1.50 with your server’s LAN/public IP):

# Configure node to act as both a broker and a metadata controller
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

# Set listener ports
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://localhost:9093
advertised.listeners=PLAINTEXT://192.168.1.50:9092
controller.listener.names=CONTROLLER

# Directory for message logs and retention period (7 days)
log.dirs=/var/lib/kafka/data
num.partitions=3
log.retention.hours=168

Create the log data directory and assign ownership to the kafka user:

sudo mkdir -p /var/lib/kafka/data
sudo chown -R kafka:kafka /var/lib/kafka

Step 4: Initialize the Storage Cluster ID

Before running Kafka for the first time, KRaft requires you to format the storage directory with a unique Cluster ID:

# Generate a unique cluster UUID
KAFKA_CLUSTER_ID=$(sudo -u kafka /opt/kafka/bin/kafka-storage.sh random-cluster-id)
echo "Your Cluster ID: $KAFKA_CLUSTER_ID"

# Format the log directory using the Cluster ID
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c /opt/kafka/config/kraft/server.properties

Step 5: Create a Systemd Service for Process Management

Create a service unit so Kafka automatically starts with CentOS Stream 9 on system boot:

sudo tee /etc/systemd/system/kafka.service << 'EOF'
[Unit]
Description=Apache Kafka Server (KRaft mode)
Documentation=https://kafka.apache.org/documentation/
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/jre-17-openjdk"
Environment="KAFKA_LOG_DIR=/opt/kafka/logs"
Environment="KAFKA_HEAP_OPTS=-Xmx1G -Xms1G"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

Reload the systemd daemon and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now kafka
sudo systemctl status kafka

Step 6: Open the Port in Firewalld

If your server has an active firewall, open port 9092 to allow client connections:

sudo firewall-cmd --permanent --add-port=9092/tcp
sudo firewall-cmd --reload

Step 7: Verify Message Publishing and Consumption

Verify that your messaging pipeline is working end-to-end using Kafka’s built-in CLI tools.

Create a test topic named orders-stream with 3 partitions:

/opt/kafka/bin/kafka-topics.sh --create --topic orders-stream --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092

Launch the console producer and publish two sample JSON transaction records:

/opt/kafka/bin/kafka-console-producer.sh --topic orders-stream --bootstrap-server localhost:9092
>{"order_id": 1001, "amount": 250000, "status": "PENDING"}
>{"order_id": 1002, "amount": 500000, "status": "PAID"}

Open another terminal session and start the console consumer to read messages from the beginning of the topic:

/opt/kafka/bin/kafka-console-consumer.sh --topic orders-stream --from-beginning --bootstrap-server localhost:9092

The consumer terminal instantly prints the two JSON messages. Data is now durably persisted to disk. Even if consumers fail or remain offline for hours, all messages stay safely stored on the Kafka broker, ready for replay at any moment.