An Incident at 2:15 AM
Prometheus alerts were blaring incessantly. Our Kafka cluster in the staging environment suffered a widespread outage. All workers handling order processing and sending notification emails ground to a halt. Over 45,000 messages were backed up in the queue.
Opening my laptop in a rush, I quickly checked the container status:
docker compose ps
docker compose logs --tail=100 zookeeperThe ZooKeeper container had crashed with the error code OOMKilled (Exit Code 137). ZooKeeper lost its quorum, causing metadata synchronization to break down. The entire cluster entered a split-brain state. The brokers were unable to elect partition leaders and rejected all incoming client connections. A sleepless night—all because a 3-node ZooKeeper cluster devoured all 4GB of RAM on our staging VPS.
The Burden Called ZooKeeper
Prior to Kafka 2.8 (and production-ready starting from 3.3+), Kafka strictly required ZooKeeper to store metadata, elect controllers, and manage topic configurations. This two-tier architecture exposed three major weaknesses:
- Resource waste: Running a minimal production cluster required 3 ZooKeeper nodes and 3 Kafka brokers. The ZooKeeper cluster alone consumed 2GB to 4GB of RAM just to maintain heartbeats.
- Bottlenecks during scaling: When the partition count exceeded 100,000, metadata became a major bottleneck. Recovery times after a broker restart could stretch from 2 to 10 minutes due to massive state synchronization from ZooKeeper.
- Operational complexity: You had to manage two separate distributed systems concurrently—two sets of network ports, two security configuration suites (SASL/TLS), and two disjointed logging sources.
Comparing Potential Solutions
Faced with infrastructure optimization challenges, our team evaluated 3 options:
1. Allocating more RAM and tuning the JVM for ZooKeeper
Increasing -Xmx from 1GB to 4GB per node and fine-tuning snapshot parameters. This approach only treats the symptoms. Infrastructure costs would surge while the clunky architecture remained unchanged.
2. Migrating to RabbitMQ or Redis Streams
This approach works well for smaller microservices. However, our system required long-term event log retention (30 days), message replay on failure, and a sustained throughput of 20,000 msg/s. RabbitMQ was not well-suited for these requirements.
3. Migrating to Kafka KRaft (Kafka Raft Metadata)
This is the optimal path forward. KRaft embeds the Raft consensus algorithm directly into the Kafka broker. Metadata is stored within an internal topic named @metadata. As a result, controller recovery time dropped from 30 seconds to under 500 milliseconds, while slashing memory consumption by 50%.
Deploying Kafka KRaft + Kafka UI Using Docker Compose
Here is a single-node Kafka KRaft setup (acting as both Broker and Controller) paired with the Kafka UI interface for testing locally or in staging environments.
Step 1: Prepare the docker-compose.yml file
Create the working directory:
mkdir -p ~/kafka-kraft-stack && cd ~/kafka-kraft-stack
nano docker-compose.ymlPaste the following configuration into the file:
services:
kafka:
image: apache/kafka:3.7.0
container_name: kafka-kraft
ports:
- "9092:9092"
environment:
KAFKA_NODE_ID: 1
KAFKA_CLUSTER_ID: "4L622nShTUiBenVKBpahAg"
KAFKA_PROCESS_ROLES: "broker,controller"
# Configure listeners for host machine (9092) and internal container traffic (29092, 9093)
KAFKA_LISTENERS: "PLAINTEXT://:9092,CONTROLLER://:9093,PLAINTEXT_INTERNAL://:29092"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://kafka:29092"
KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT"
KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
# Configure replication settings for a single-node cluster
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_LOG_DIRS: "/tmp/kraft-combined-logs"
volumes:
- kafka_data:/tmp/kraft-combined-logs
networks:
- kafka-net
kafka-ui:
image: provectuslabs/kafka-ui:latest
container_name: kafka-ui
ports:
- "8080:8080"
environment:
KAFKA_CLUSTERS_0_NAME: "kraft-local-cluster"
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092"
DYNAMIC_CONFIG_ENABLED: "true"
depends_on:
- kafka
networks:
- kafka-net
volumes:
kafka_data:
driver: local
networks:
kafka-net:
driver: bridgeStep 2: Launch the service cluster
Start the containers using Docker Compose v2:
docker compose up -dMonitor Kafka’s startup logs:
docker compose logs -f kafkaOnce you see the log entry [KafkaRaftServer id=1] Kafka Server started, the broker is ready to accept connections. Startup takes only about 3 to 5 seconds.
Step 3: Test Producing and Consuming
Create a sample topic named order-events with 3 partitions:
docker compose exec -it kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server localhost:9092 \
--create \
--topic order-events \
--partitions 3 \
--replication-factor 1Open a separate terminal tab to consume messages:
docker compose exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic order-events \
--from-beginningOpen a second terminal tab to produce messages:
docker compose exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server localhost:9092 \
--topic order-events
> {"order_id": 1001, "status": "PAID", "amount": 250000}
> {"order_id": 1002, "status": "PENDING", "amount": 120000}In the Consumer tab, the JSON payload will show up as soon as you press Enter.
Step 4: Visual Management with Kafka UI
Navigate to http://localhost:8080 in your browser. The Kafka UI dashboard enables you to:
- Inspect Controller health and Broker ID 1 status.
- Create, alter partitions, or modify retention policies directly.
- Inspect message payloads per partition and monitor Consumer Group Lag to identify bottlenecked workers.
Ever since eliminating ZooKeeper in favor of KRaft, our staging cluster has been running smoothly with only ~700MB of RAM—no more middle-of-the-night OOM alerts.
