Vector: A Ultra-Lightweight Docker Logging Solution to Replace Logstash

Why You Should Stop Using Logstash or Fluentd

If you are running microservices, you’ve likely dealt with the headache of Logstash consuming up to 1GB of RAM just to forward a few lines of logs. Historically, the ELK stack (Elasticsearch – Logstash – Kibana) was the standard. However, Logstash is heavy because it runs on the JVM. Fluentd is better but still relies on Ruby, which can cause bottlenecks when data throughput spikes.

I once faced a tough situation: a VPS with 4GB of RAM was running four microservices smoothly until it “froze” because of Logstash. Just by installing it to ship logs, RAM usage immediately jumped to 90%. The system started lagging constantly. That’s when I found Vector and was truly impressed.

Vector is written in Rust, famous for its ability to process tens of thousands of events per second while consuming only about 20-30MB of RAM. This is 10-20 times lower than Logstash. Since switching to Vector on Docker Compose v2, my VPS maintenance costs have dropped significantly because I no longer need to upgrade RAM just to host a log agent.

Deploying Vector to Collect Docker Logs

We will set up a real-world model: Vector running as a container, “listening” for logs from the Docker Socket, and outputting them to the console. This entire process is based on Docker Compose v2 – the modern standard.

Step 1: Initialize the Configuration File

Vector controls everything via the vector.yaml file. Create the project directory and configuration file using the following commands:

mkdir vector-docker-logs && cd vector-docker-logs
touch vector.yaml

Here is a minimal configuration for Vector to start collecting data:

# vector.yaml
sources:
  docker_logs:
    type: "docker_logs"

sinks:
  out:
    type: "console"
    inputs:
      - "docker_logs"
    encoding:
      codec: "json"

Step 2: Set Up Docker Compose

The docker-compose.yml file will define Vector and an Nginx application to generate sample logs. Note the distroless image version for optimized security and size.

# docker-compose.yml
services:
  vector:
    image: timberio/vector:0.34.1-distroless-static
    container_name: vector
    volumes:
      - ./vector.yaml:/etc/vector/vector.yaml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "8686:8686"

  app-nginx:
    image: nginx:latest
    container_name: demo-nginx
    ports:
      - "8080:80"

Pro tip: Don’t forget to mount /var/run/docker.sock. Without this line, Vector will be completely “blind” because it cannot communicate with the Docker Daemon to retrieve data from other containers.

Professional Log Processing with VRL

Raw logs from Docker often contain redundant information. Vector’s true power lies in VRL (Vector Remap Language), which allows you to filter and transform data in real-time.

Filtering and Reformatting Data

Suppose you only want to keep logs from containers with names containing “demo” and add an environment label. Update your vector.yaml:

sources:
  docker_logs:
    type: "docker_logs"

transforms:
  # Only get logs from the demo-nginx container
  filter_nginx:
    type: "filter"
    inputs:
      - "docker_logs"
    condition: | 
      includes(string!(.container_name), "demo-nginx")

  # Add identification info
  parse_logs:
    type: "remap"
    inputs:
      - "filter_nginx"
    source: |
      .env = "production"
      .processed_at = now()
      .message = upcase(string!(.message))

sinks:
  stdout:
    type: "console"
    inputs:
      - "parse_logs"
    encoding:
      codec: "json"

VRL is extremely fast because it is compiled directly. Converting a message to uppercase (upcase) or adding a timestamp takes only a few microseconds, causing no delay to the pipeline.

Sending Logs to Loki or Elasticsearch

Once the logs are cleaned, you can push them to a centralized storage system like Grafana Loki:

sinks:
  loki_sink:
    type: "loki"
    inputs:
      - "parse_logs"
    endpoint: "http://loki:3100"
    labels:
      container_name: "{{ container_name }}"
      service: "my-app"

Real-World Performance Testing

Start the system with the command: docker compose up -d. Then, try accessing localhost:8080 to generate traffic for Nginx.

To see how Vector is processing, use the command: docker logs -f vector. You will see JSON log lines appearing immediately with the env and processed_at fields you configured.

A big plus is Vector’s internal dashboard. Add this code snippet to your configuration file:

api:
  enabled: true
  address: "0.0.0.0:8686"

When running vector top, you will see the throughput chart. Even when processing 5,000 – 10,000 log lines per second, Vector’s CPU usage usually stays below 5% on standard CPUs.

Conclusion

Switching to Vector is one of the best decisions for optimizing Docker infrastructure. It is not only lightweight but also extremely flexible thanks to the VRL language. If you are tired of maintaining Java/JVM for Logstash, try Vector today. Good luck with your deployment, and if you encounter any VRL syntax errors, feel free to leave a comment for support!