Installing and Configuring Vector: A High-Performance Log Pipeline Alternative to Logstash

Comparing Popular Modern Log Pipeline Solutions

Before migrating to Vector, our team struggled to collect logs from over 40 microservices. Whenever a flash sale hit, log volume spiked to 50GB per hour. At that point, our traditional shipper hit a bottleneck and drained all the server cluster’s resources.

The three most popular tools DevOps engineers typically consider are:

• Logstash (Elastic Stack): Runs on the JVM. It offers a massive plugin ecosystem and convenient grok parsing, but it is extremely heavy. A completely idle instance already consumes 1GB to 2GB of RAM right out of the box. Under heavy load, sudden garbage collection pauses cause backpressure across the queues.
• Fluent Bit: Written in C, making it remarkably lightweight and consuming only tens of megabytes of memory. This tool is ideal for shipping pod logs on Kubernetes nodes. However, complex string parsing and enrichment can be restrictive, and navigating multi-layered configuration files is prone to errors.
• Vector (Datadog open-source): Written in Rust, offering guaranteed memory safety. Production instances consume only about 40MB to 80MB of RAM. Most importantly, its built-in Vector Remap Language (VRL) enables deep data transformations without compromising throughput.

Pros & Cons Analysis: Logstash vs. Vector

The following real-world comparison table will help you evaluate options before re-architecting your pipeline:

When Should You Switch to Vector?

Our infrastructure runs on a 15-node Kubernetes cluster. Previously, whenever services threw widespread 500 errors, Logstash on the log-collecting nodes immediately suffered an OOM kill. CPU peaked at 100%. Alerting streams in Grafana lagged by up to 10 minutes, preventing the team from pinpointing the root cause.

You should consider switching to Vector right away in the following scenarios:

• You need to cut infrastructure costs: At 50,000 events/sec, Vector consumes roughly 1 CPU core and 90MB of RAM. Logstash requires up to 4 cores and 3GB of RAM.
• You want a lean agent: A single binary can tail Nginx log files, collect Docker container logs, and simultaneously fan out data to both ClickHouse for querying and S3 for cold storage.
• You require safe log validation: VRL enforces handling for null values and parse failures, eliminating the risk of pipeline crashes mid-stream.

Practical Implementation: Collecting and Parsing Nginx Logs to Elasticsearch

The following example reads Nginx access logs, parses the raw strings into clean JSON using VRL, and ships them directly into Elasticsearch.

Step 1: Install Vector on Ubuntu/Debian

Datadog maintains an official APT repository for Vector. This is the cleanest way to handle version upgrades and manage the service via systemd.

# Add Vector's official APT repository
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo -E bash

# Install the vector package
sudo apt-get update
sudo apt-get install -y vector

# Verify the binary
vector --version

Step 2: Vector’s Core 3-Component Architecture

Vector pipelines are configured via YAML or TOML, built around three core primitives:

1. Sources: Data intake points (log files, syslog, journald, sockets, stdin…).
2. Transforms: Parsing and sanitization layer (string parsing, type casting, IP enrichment, dropping redundant logs).
3. Sinks: Data destinations (Elasticsearch, Kafka, ClickHouse, VictoriaLogs, S3…).

Step 3: Build the Nginx Log Pipeline

Open /etc/vector/vector.yaml and paste the following configuration:

# 1. SOURCES: Read Nginx access log file
sources:
  nginx_access_logs:
    type: "file"
    include:
      - "/var/log/nginx/access.log"
    read_from: "beginning"

# 2. TRANSFORMS: Parse strings using VRL
transforms:
  parse_nginx:
    type: "remap"
    inputs:
      - "nginx_access_logs"
    source: |
      # Parse Combined Log Format using regex
      parsed, err = parse_regex(.message, r'^(?P<remote_addr>[\d\.]+) - (?P<remote_user>[^ ]*) \[(?P<time_local>[^\]]+)\] "(?P<request_method>[A-Z]+) (?P<request_uri>[^ ]+) (?P<server_protocol>[^"]+)" (?P<status>\d+) (?P<body_bytes_sent>\d+)')
      
      if err != null {
        .parse_status = "failed"
      } else {
        ., err = merge(., parsed)
        .status = to_int!(.status)
        .body_bytes_sent = to_int!(.body_bytes_sent)
        .parse_status = "success"
      }
      
      # Attach server tags for querying
      .host = get_hostname!()
      .service = "nginx-gateway"

# 3. SINKS: Ship logs to Elasticsearch and output to console for debugging
sinks:
  es_cluster:
    type: "elasticsearch"
    inputs:
      - "parse_nginx"
    endpoints:
      - "http://10.0.0.50:9200"
    mode: "bulk"
    bulk:
      index: "logs-nginx-%Y.%m.%d"
    auth:
      strategy: "basic"
      user: "elastic"
      password: "SecretPassword123"

  stdout_debug:
    type: "console"
    inputs:
      - "parse_nginx"
    encoding:
      codec: "json"

Step 4: Validate Syntax and Enable the Service

Vector includes a built-in config validation tool. Running this check prevents pipeline downtime caused by indentation errors or missing variables:

# Validate configuration file
vector validate --config /etc/vector/vector.yaml

# Enable and start the service
sudo systemctl enable --now vector

# Follow the processing logs
sudo journalctl -u vector -f

Once the configuration is validated, Vector begins ingesting access logs immediately. Fields like remote_addr, status, and request_uri are cleanly structured into JSON and shipped to Elasticsearch with sub-20ms latency.