The Headache of Instrumenting Each Application Individually
My monitoring setup with Prometheus + Grafana was tracking 15 servers, and that setup had already caught incidents before users reported them many times. But there was a blind spot I just couldn’t resolve: monitoring inside each application — latency per endpoint, error rate, the number of requests in flight.
Every time the team added a new service, the old process was: open the code, install the prometheus_client or OpenTelemetry SDK library, write middleware to measure latency, expose /metrics, then configure Prometheus to scrape it. Go services required one approach, Python another, Node.js yet another. For a small team, this consumed 2-3 hours per service — and was often pushed aside when deadlines were tight.
After 3 months of this, I realized that up to 40% of services had no metrics at all — all services written quickly that nobody had time to go back and instrument.
Why the Traditional Approach Is Hard to Maintain
The tools aren’t lacking. OpenTelemetry is very capable, and the Prometheus client library is feature-rich. But manual instrumentation creates technical debt in ways that are surprisingly hard to see:
- Developers have to learn the API of each SDK for each language
- Code becomes mixed between business logic and monitoring logic
- New services don’t naturally have metrics — someone has to remember to add them
- After refactoring, old metrics may no longer be accurate but still run, causing confusion
Commercial APMs like Datadog or New Relic have fairly good auto-instrumentation agents. But with 15 hosts, Datadog Pro works out to roughly $300-400/month — not counting custom metrics. Not suitable for a small team’s budget — open-source APM alternatives can help with cost, but they still require manual instrumentation.
Approaches I’ve Tried
Approach 1: Centralized Instrumentation at the API Gateway
Installing Nginx or Envoy as a gateway and measuring latency and request count there is fast, but you only see the outermost layer — you can’t tell which internal service is slow, and there are no end-to-end traces across multiple services.
Approach 2: OpenTelemetry Collector + Auto-Instrumentation Agent
Java has a very mature -javaagent. Python has opentelemetry-instrument. Node.js has @opentelemetry/auto-instrumentations-node (or the manual approach with prom-client). But Go doesn’t — Go compiles to a static binary, with no mechanism to inject an agent at runtime the way the JVM does.
I used this approach for Python and Node.js (with an OpenTelemetry Collector like Grafana Alloy to centralize data collection), but still had to modify each service’s entrypoint, and Go remained uncovered.
Approach 3 — What I’m Currently Using: Grafana Beyla with eBPF
Beyla is an open-source tool from Grafana Labs that uses eBPF (extended Berkeley Packet Filter) to hook into the Linux kernel. It automatically observes running applications — without needing to modify a single line of code. Beyla operates at the kernel level, intercepting HTTP/gRPC-related syscalls, then aggregates them into RED metrics (Rate, Errors, Duration) and traces.
After 6 months running on production, this is the approach I’ve found most effective for mixed-language environments.
Installing Grafana Beyla Step by Step
Requirements
- Linux kernel 5.8 or higher (check with:
uname -r) - Root access or
CAP_BPF,CAP_PERFMON,CAP_SYS_PTRACE - Prometheus running to receive metrics (or use Grafana Cloud)
Installing Beyla with Docker
The fastest way to get started is to run Beyla as a container alongside the application you want to monitor:
# Monitor application running on host port 8080
docker run --rm \
--pid=host \
--privileged \
-e BEYLA_OPEN_PORT=8080 \
-e BEYLA_PROMETHEUS_PORT=9400 \
-p 9400:9400 \
grafana/beyla:latestVisit http://localhost:9400/metrics — the http_server_request_duration_seconds metric will appear immediately, without needing to change anything in the application.
Installing Beyla as a Direct Binary
# Download the latest binary (check the releases page for the latest version)
wget https://github.com/grafana/beyla/releases/download/v1.8.0/beyla-linux-amd64
chmod +x beyla-linux-amd64
sudo mv beyla-linux-amd64 /usr/local/bin/beyla
# Run and auto-detect process listening on port 8080
sudo BEYLA_OPEN_PORT=8080 BEYLA_PROMETHEUS_PORT=9400 beylaYAML Configuration File for Production
Environment variables are only practical for testing. In production, a YAML configuration file is much easier to manage:
# /etc/beyla/config.yaml
open_port: 8080 # Port the application is listening on
service_name: my-api # Display name in Grafana
prometheus_export:
port: 9400
path: /metrics
# Optional: export traces to Grafana Tempo
otel_traces_export:
endpoint: http://tempo:4317
# Filter paths that don't need tracking (health checks, static files)
filters:
application:
- path_pattern: '/health'
method: GET# Run with config file
sudo beyla --config /etc/beyla/config.yamlIntegrating with Prometheus
Add a job to prometheus.yml:
scrape_configs:
- job_name: 'beyla-my-api'
static_configs:
- targets: ['localhost:9400']
relabel_configs:
- source_labels: [__address__]
target_label: instanceRunning Beyla as a systemd Service
To have Beyla start automatically with the server:
# /etc/systemd/system/beyla.service
[Unit]
Description=Grafana Beyla eBPF Auto-instrumentation
After=network.target
[Service]
ExecStart=/usr/local/bin/beyla --config /etc/beyla/config.yaml
Restart=on-failure
User=root
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now beyla
sudo systemctl status beylaHow Beyla Works with Each Language
The underlying mechanism differs by language, but the metrics output is consistent:
- Go: Beyla uses uprobes to hook into the net/http and gRPC functions in compiled binaries. This is something no other agent can do with Go.
- Python: Hooks into the WSGI/ASGI middleware layer (Django, Flask, and FastAPI all work).
- Node.js: Hooks into the V8 runtime and http/https modules.
The metrics produced follow the OpenTelemetry semantic conventions standard, so Grafana dashboards work right away without any customization.
Results After 6 Months of Real-World Use
What surprised me wasn’t that Beyla worked — it was that it worked well even on legacy services that nobody dared touch. I had a Go service written in 2021, no unit tests, nobody on the team fully understood it — and now it has complete latency histograms, error rates, and request throughput without touching a single line of code.
Metrics coverage across the team jumped from 60% to 100% in a single afternoon, simply by deploying Beyla to each host.
On overhead: I measured roughly 1-2% CPU and ~50MB RAM for a host processing ~500 req/s. Completely acceptable.
One thing to note: Beyla requires kernel 5.8+ and privileged access. If you’re running in a containerized environment with strict security policies (like GKE Autopilot), check compatibility beforehand.
Grafana Dashboard for Beyla
Grafana Labs provides ready-made dashboards on Grafana.com with IDs 19419 (RED Metrics) and 19420 (Service Map). To import into Grafana:
- Go to Grafana → Dashboards → Import
- Enter ID
19419→ Load - Select Prometheus datasource → Import
Once imported, the Service Map appears immediately — all services, traffic between them, and request flow are all displayed without any additional configuration. Request Rate, Error Rate, and P99 latency per endpoint are all available out of the box. No PromQL required — though recording rules and alerting rules can help you build production-grade alerts on top of these metrics.
