Installing Pyrra: Managing SLOs and Alerts the Google SRE Way

The Paradox: Green Dashboards, Complaining Customers

Have you ever found yourself in a situation where your Prometheus dashboard shows perfectly stable CPU/RAM usage, yet Slack is blowing up because users are complaining about the application being slow? I once managed a system of 15 servers with full basic alerting. However, I realized I was measuring what the machines cared about instead of the actual user experience.

That’s when SLO (Service Level Objectives) became the key. Instead of just monitoring whether a server is up or down, we measure: “What percentage of requests were successful over the last 30 days?”. However, manually writing hundreds of lines of PromQL to calculate Error Budgets or Burn Rates is a nightmare. Pyrra was created to solve exactly this complexity.

4 Vital Concepts in System Administration

To use Pyrra effectively, you need to clearly distinguish between these 4 terms:

• SLI (Indicator): The actual measurement, such as the ratio of requests returning a 200 status code.
• SLO (Objective): The target you aim for. For example: 99.9% of requests must be successful. At this level, you only have 43.2 minutes of downtime per month.
• Error Budget: The allowed margin of error (0.1%). This is your “allowance” for maintenance or testing new features.
• Burn Rate: The rate at which you are “spending” your error budget. If the Burn Rate is 14.4, you will exhaust a month’s budget in just 2 days.

Pyrra helps you define SLOs using concise YAML files. It then automatically generates highly complex Prometheus Recording Rules for you.

Deploying Pyrra with Docker Compose

The fastest way to test it is using Docker Compose. Pyrra needs to connect directly to your Prometheus server.

version: '3.8'
services:
  pyrra:
    image: ghcr.io/pyrra-dev/pyrra:v0.7.0
    container_name: pyrra
    ports:
      - "9099:9099"
    command:
      - api
      - --prometheus-url=http://prometheus:9090
    restart: always

  pyrra-filesystem:
    image: ghcr.io/pyrra-dev/pyrra:v0.7.0
    container_name: pyrra-filesystem
    volumes:
      - ./slo-definitions:/etc/pyrra/slo
    command:
      - filesystem
      - --prometheus-url=http://prometheus:9090
      - --config-files=/etc/pyrra/slo/*.yaml
    restart: always

In this setup, the api component handles the UI. The filesystem component scans and loads SLO definition files from a local directory.

Defining Your First SLO

Suppose you have a service returning the http_requests_total metric. You want to set a 99% success rate SLO. Create the file slo-definitions/api-success-rate.yaml:

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: api-success-rate
  labels:
    service: backend-api
spec:
  description: The backend API success rate must be over 99%.
  target: "99"
  window: 28d
  indicator:
    ratio:
      errors:
        metric: http_requests_total{job="backend", code=~"5.."}
      total:
        metric: http_requests_total{job="backend"}

This structure is much more transparent than manually calculating rates over 28 days using PromQL.

Integration and Real-world Monitoring

Pyrra doesn’t store data; it only generates Rules. If you’re using Kubernetes, Pyrra automatically creates PrometheusRule objects via an Operator. For traditional servers, you can simply copy the rules from the Pyrra UI into your Prometheus configuration file.

When accessing http://localhost:9099, you will see key metrics:

• Objective: The 99% target.
• Availability: The actual performance the system is currently running at.
• Error Budget Remaining: The percentage of the error budget left before violating the commitment (SLA).

This mechanism helps me detect issues early. Just by looking at the Burn Rate, I know immediately if the system is “bleeding” even if it hasn’t completely failed yet.

Automated Alerting Based on Google Standards

Pyrra applies Google’s Multi-window Multi-burn-rate philosophy. Instead of alerting as soon as there is a single failed request, it only sends alerts based on the budget consumption rate:

• Critical Alert: You are burning your budget too fast; the system will go down in a few hours if not addressed.
• Warning Alert: The burn rate is slower but will jeopardize the SLO in the coming days.

This approach completely eliminates “flapping alerts” that cause fatigue for on-call engineers.

Conclusion

Using SLOs helps Dev and Ops teams speak the same language. Instead of arguing whether 80% CPU usage is important, look at the Error Budget. If the budget remains, deploy with confidence. If it’s running low, stop to optimize the system.

Pyrra is a powerful tool to help you start standardizing operations without needing to be a PromQL expert. If you are managing over 10 microservices, you should try installing it today.