How to Install Falco for Runtime Security Monitoring and Anomaly Detection on Kubernetes - ITFROMZERO - Share tobe shared! Falco is a runtime security monitoring tool for Kubernetes that operates at the syscall layer to detect shell spawns in containers, sensitive file access, and anomalous behavior that Prometheus can't see. Covers installation with Helm, custom rule configuration, and routing alerts to Slack.

Table of Contents

A 2 AM Alert That Woke Me Up

The Kubernetes cluster was running fine. Prometheus + Grafana showed normal CPU and memory. No pods were crashing. But the payment system was processing unusually slowly — the kind of slow that normal metrics couldn’t explain.

After 40 minutes of digging through logs, I found the culprit: a container was running a curl process making outbound internet calls every 30 seconds. It wasn’t application code. It wasn’t any configured cron job. The image had been compromised at some point without anyone noticing.

Prometheus + Grafana was monitoring 15 servers at the time. Metrics were complete, dashboards looked clean. That setup is excellent for infrastructure metrics — but completely blind to what was happening inside containers at the syscall level. That’s when I decided to deploy Falco.

What Falco Is and Why It’s Different From What You’re Already Using

Falco was developed by Sysdig and is now a CNCF graduated project. What sets it apart isn’t the brand — it’s where it looks: instead of metrics or logs, it listens directly at the Linux kernel syscall layer. Everything a container does — opening files, creating processes, making network connections — passes through there.

If Prometheus + Grafana answers “how is the system performing”, Falco answers “who is doing what inside my containers”.

How It Works

There are two ways Falco can hook into the kernel:

Kernel module: loads a module directly into the kernel — higher performance but requires special privileges and depends on the kernel version.
eBPF probe: uses eBPF (extended Berkeley Packet Filter) — safer, requires no kernel module, and is the recommended choice for modern Kubernetes.

The mechanism is straightforward: read syscalls in real time, compare them against a set of rules defined in YAML, and fire an alert on a match. The default rules already cover the most common dangerous patterns: shell in a container, reading /etc/shadow, mounting sensitive paths, privilege escalation.

Installing Falco on Kubernetes with Helm

Prerequisites

Kubernetes cluster (1.19+)
Helm 3.x
kubectl with cluster-admin privileges

Step 1: Add the Helm Repository and Install

# Add the Falco repo
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Create a dedicated namespace
kubectl create namespace falco

# Install Falco with eBPF driver (recommended)
helm install falco falcosecurity/falco \
  --namespace falco \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true

The falcosidekick.enabled=true flag also installs Falcosidekick — a forwarder that routes alerts from Falco to Slack, Telegram, PagerDuty, Elasticsearch, and 50+ other outputs. Skipping it means the only way to receive alerts is to read logs manually.

Step 2: Verify Pods Are Running

kubectl get pods -n falco

# Expected output:
# NAME                                   READY   STATUS    RESTARTS
# falco-6xk9p                            2/2     Running   0
# falco-falcosidekick-7d8f9c-abc12       1/1     Running   0
# falco-falcosidekick-ui-5f6b8d-xyz34    1/1     Running   0

Falco runs as a DaemonSet — one pod per node, ensuring full cluster coverage.

Step 3: View Your First Alert

# View real-time logs from Falco
kubectl logs -n falco -l app.kubernetes.io/name=falco -f

# Try triggering an alert: exec into a running container
kubectl exec -it <pod-name> -- /bin/sh

# In Falco's logs, you'll immediately see:
# Notice A shell was spawned in a container with an attached terminal
# (user=root shell=sh parent=runc cmdline=sh container_id=abc123...)

Simply exec-ing into a container is enough to trigger an alert. “Terminal shell in container” is one of the most suspicious indicators — developers occasionally do this for debugging, but attackers do the same thing once they get in.

Customizing Rules for Your Environment

Falco’s default rules are fairly noisy — plenty of false positives if your team occasionally execs into containers for debugging. Tuning is necessary.

Creating Custom Rules

Create a file called custom-rules.yaml:

customRules:
  custom-rules.yaml: |-
    # Allow exec into containers in the 'dev' namespace without triggering an alert
    - rule: Terminal shell in container
      condition: >
        spawned_process
        and container
        and shell_procs
        and proc.tty != 0
        and not container.image.repository in (trusted_images)
        and not k8s.ns.name in (dev, staging)
      desc: A shell was spawned in a container
      output: >
        Shell spawned in production container
        (user=%user.name container=%container.name image=%container.image.repository
        pod=%k8s.pod.name ns=%k8s.ns.name)
      priority: WARNING
      tags: [shell, mitre_execution]

    # New rule: detect cryptocurrency miners
    - rule: Outbound connection to known crypto mining pool
      desc: Detect connection to cryptocurrency mining pools
      condition: >
        outbound
        and fd.sip.name in (mining_pool_domains)
      output: >
        Crypto mining connection detected
        (pod=%k8s.pod.name image=%container.image.repository
        dest=%fd.sip.name:%fd.sport)
      priority: CRITICAL
      tags: [network, cryptomining]

# Apply custom rules
helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  -f custom-rules.yaml

Verifying Rules Are Loaded

# Check that Falco loaded rules successfully with no syntax errors
kubectl logs -n falco -l app.kubernetes.io/name=falco | grep -E "(rule|error|warn)" | head -20

Connecting Falco to Slack or Telegram

Falcosidekick supports 50+ outputs. Configuring Slack is the fastest option:

helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl="https://hooks.slack.com/services/YOUR/WEBHOOK" \
  --set falcosidekick.config.slack.minimumpriority="warning"

From that point on, whenever there’s a shell in a container, someone accesses /proc/*/mem, or /etc/passwd gets written to — Slack will ping immediately. Alerts include full context: pod name, namespace, image, user, and command line.

Viewing the Falcosidekick UI

# Port-forward to view the dashboard
kubectl port-forward svc/falco-falcosidekick-ui 2802:2802 -n falco

# Open in browser: http://localhost:2802

I open this UI whenever I’m investigating an incident. Alert timeline, filter by priority (Critical/Warning/Notice), by namespace, by rule — one look and you can immediately see what’s happening and since when.

Rules You Should Enable From the Start

Hundreds of rules come bundled out of the box, but not all of them matter equally. Here are the ones I’ve found most valuable in production environments:

Write below etc: any process writing to /etc — extremely unusual inside a container.
Read sensitive file untrusted: reading /etc/shadow, /etc/sudoers, or SSH private keys.
Netcat Remote Code Execution in Container: netcat/ncat being used to create a reverse shell.
Contact K8S API Server From Container: a container making direct calls to the Kubernetes API — a possible sign of container escape.
Privilege Escalation Using Sudo: sudo inside a container is an obvious red flag.

Conclusion

After that 2 AM incident, I deployed Falco across the entire cluster the following week. Since then, it has caught two more incidents: one where a container had a reverse shell injected through a deserialization vulnerability, and another where a developer accidentally hardcoded credentials in an image that script kiddies had scanned out.

Falco doesn’t replace Prometheus or Grafana — it fills the gap that metric-based monitoring can’t see. Metrics tell you the system is slow; Falco tells you why — and sometimes the answer isn’t a performance issue, it’s a security incident.

If your cluster is running production workloads without runtime security monitoring, this is something you should have in place before an incident happens, not after.