Installing Fluent Bit on Ubuntu: Collecting System and Docker Container Logs, Forwarding to Elasticsearch - ITFROMZERO - Share tobe shared! A guide to installing Fluent Bit on Ubuntu 22.04 to collect logs from systemd and Docker containers, filter data, and forward to Elasticsearch. Includes a practical comparison of Logstash, Fluentd, Filebeat, and Fluent Bit to help you choose the right tool for your use case.

2:17 AM. Slack fires an alert: the API service is returning 500s non-stop. I SSH into the server, hop into each container running docker logs --tail 100 api_container, then journalctl -u nginx --since "5 minutes ago", then grep PostgreSQL’s log files. 15 minutes later, still no root cause because logs are scattered across 6 different places.

The next day I spent 2 hours setting up Fluent Bit. Since then — I’ve provisioned Ubuntu Server 22.04 on over 20 VPS instances, and installing Fluent Bit is one of the first things I always do — once it’s up, I breathe easy knowing that the next time something breaks at midnight, I won’t have to grep manually anymore.

Table of Contents

Comparing Log Collection Tools: Logstash, Fluentd, Filebeat, and Fluent Bit

Before jumping into installation, you need to pick the right tool. Choosing wrong means wasting time on a migration later.

Logstash — Powerful but RAM-hungry

As a core member of the Elastic Stack, it integrates with Elasticsearch nearly zero-config. Flexible pipeline DSL, massive plugin library. But it runs on the JVM — consuming 300–500MB RAM at idle, potentially climbing to 1–2GB under load. On a 2GB VPS already running your application, Logstash is an elephant in a small room.

Fluentd — Well-balanced, Strongest Ecosystem

Written in Ruby, considerably lighter than Logstash (~40–100MB RAM). Its real strength is breadth: Fluentd can ingest logs from dozens of sources, aggregate, transform, then route to multiple destinations simultaneously — and there’s a plugin for virtually anything you can think of. Fluentd and Fluent Bit are siblings from the same parent, both developed by Treasure Data. Many large-scale systems use Fluent Bit as an agent to collect logs from each node, forwarding to a central Fluentd aggregator for heavy processing.

Filebeat — Lightweight but Single-purpose

Written in Go, part of the Elastic Stack family like Logstash, with a lean footprint of ~30–50MB. The issue is that Filebeat was born to ship logs — read files, forward them, done. Want to parse and enrich complex logs? You have to push that work downstream to an Elasticsearch Ingest Pipeline.

Fluent Bit — Ultra-lightweight, Feature-complete

C-based. Memory footprint at idle is only around 450KB — that’s right, kilobytes not megabytes. 50+ input plugins, 40+ output plugins, capable of parse/filter/enrich/route before forwarding. For the systemd + Docker → Elasticsearch use case, no other tool does more while consuming less RAM.

Real-world Pros and Cons Analysis

Logstash: Extremely powerful, perfect Elastic Stack integration — but 300MB+ RAM rules it out for small VPS instances
Fluentd: Strongest plugin ecosystem, ideal as an aggregator node for centralized log processing from multiple servers
Filebeat: Lightweight and easy to install, but lacks filter capabilities so relies on Elasticsearch for processing
Fluent Bit: Ultra-lightweight, feature-complete for most use cases — ideal for running as an agent on every node in a fleet

Decision: Fluent Bit running on each server, collecting logs from systemd and Docker, forwarding directly to Elasticsearch. No intermediate aggregator needed when the fleet doesn’t exceed a few dozen nodes.

Installing Fluent Bit on Ubuntu 22.04

Fluent Bit has an official apt repository — installing this way means apt upgrade will update it automatically going forward, no extra steps to remember:

# Add GPG key and official repository
curl -fsSL https://packages.fluentbit.io/fluentbit.key | sudo gpg --dearmor -o /usr/share/keyrings/fluentbit-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/jammy jammy main" | \
  sudo tee /etc/apt/sources.list.d/fluent-bit.list

sudo apt-get update
sudo apt-get install -y fluent-bit

# Enable and start the service
sudo systemctl enable fluent-bit
sudo systemctl start fluent-bit
sudo systemctl status fluent-bit

The default config file is at /etc/fluent-bit/fluent-bit.conf. Back it up before editing — good habit:

sudo cp /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf.bak

Configuring Log Collection for System and Containers

Collecting Logs from systemd/journald

Fluent Bit’s systemd plugin reads directly from journald, bypassing files — critically, this means no log loss during log rotation. Add to the config file:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name            systemd
    Tag             system.*
    Systemd_Filter  _SYSTEMD_UNIT=nginx.service
    Systemd_Filter  _SYSTEMD_UNIT=postgresql.service
    Systemd_Filter  _SYSTEMD_UNIT=docker.service
    Read_From_Tail  On

To collect all system logs, remove the Systemd_Filter lines. I typically filter by service though, since Elasticsearch receives less data and queries run faster.

Collecting Logs from Docker Containers

The simplest approach is to let Fluent Bit read the log files Docker creates at /var/lib/docker/containers/:

[INPUT]
    Name              tail
    Tag               docker.*
    Path              /var/lib/docker/containers/*/*.log
    Parser            docker
    DB                /var/log/fluent-bit-docker.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

Fluent Bit runs under its own user and needs read access to the Docker directory. One command does it:

sudo usermod -aG docker fluent-bit
sudo systemctl restart fluent-bit

Filtering and Enriching Log Data

Raw logs from systemd tend to be messy — container IDs instead of service names, inconsistent timestamps across applications, heaps of metadata Elasticsearch doesn’t need. A few FILTER lines clean this up:

# Add hostname and environment to every log record
[FILTER]
    Name   record_modifier
    Match  *
    Record hostname ${HOSTNAME}
    Record env      production

# Keep only ERROR/WARN log levels from Docker containers
[FILTER]
    Name  grep
    Match docker.*
    Regex log (ERROR|WARN|WARNING|FATAL|CRITICAL)

# Parse JSON logs from Node.js or Python structlog applications
[FILTER]
    Name         parser
    Match        docker.*
    Key_Name     log
    Parser       json
    Reserve_Data True

Forwarding Logs to Elasticsearch

The es output plugin is built into Fluent Bit — no additional installation needed:

[OUTPUT]
    Name              es
    Match             *
    Host              elasticsearch.internal
    Port              9200
    Index             server-logs
    Type              _doc
    Logstash_Format   On
    Logstash_Prefix   server-logs
    Logstash_DateFormat %Y.%m.%d
    Retry_Limit       5

    # If Elasticsearch has authentication enabled
    HTTP_User         elastic
    HTTP_Passwd       your_password

    # If using HTTPS
    tls               On
    tls.verify        Off

With Logstash_Format On, Fluent Bit creates indices in the format server-logs-2026.11.14 — matching how Logstash does it, convenient for setting up ILM (Index Lifecycle Management) in Kibana later.

Testing and Verifying Results

Before restarting the service, verify the config syntax:

# Dry-run to check syntax
sudo fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run

# Restart if no errors found
sudo systemctl restart fluent-bit

# Monitor Fluent Bit's own logs
sudo journalctl -u fluent-bit -f

Check if logs are reaching Elasticsearch:

# Check if indices have been created
curl -X GET "http://elasticsearch.internal:9200/_cat/indices?v&index=server-logs*"

# View the first few records
curl -X GET "http://elasticsearch.internal:9200/server-logs-*/_search?pretty&size=5"

Once you see server-logs-YYYY.MM.DD indices appearing in Elasticsearch, the hard part is done. In Kibana, create an index pattern server-logs-*, open Discover — logs from all services stream in real-time.

Operational Tips

Disk buffering: If Elasticsearch goes down temporarily, Fluent Bit can buffer logs to disk instead of losing data. Add storage.type filesystem to the [SERVICE] section and specify storage.path
Backpressure: When Elasticsearch processes slower than log collection rate, Fluent Bit has automatic throttling. Set Mem_Buf_Limit in each INPUT to prevent OOM on smaller servers
Flush interval: For large fleets, set Flush 10 instead of the default 5 — cutting Elasticsearch requests in half without meaningfully impacting real-time delivery
Log rotation: The tail plugin handles log rotation automatically through the DB parameter — no additional logrotate configuration needed

The next time an alert fires at 2 AM, instead of manually grepping across 6 different places, I just open Kibana Discover, filter by time range and service name — the root cause usually surfaces within 2 minutes.