Runtime Security for Kubernetes with Falco: Real-time Anomaly Detection

Runtime Security for Kubernetes with Falco: Real-time Anomaly Detection

Hello IT pros from itfromzero.com! When working with Kubernetes, you’ve probably noticed that Kubernetes security is an ongoing challenge. We need to pay attention to many aspects, from protecting the API server and managing RBAC to ensuring secure container images. However, one aspect is often overlooked until an incident occurs: **runtime security**.

Once, my team deployed a new application to Kubernetes. Everything seemed fine, but after a few days, I received an alert about a strange process running in one of the application’s pods. It turned out that a dependency in the image had a vulnerability, and an attacker exploited it to execute remote commands.

Fortunately, Falco quickly ‘caught’ this unusual behavior, allowing me to address it before greater damage occurred. Since then, I’ve always prioritized runtime monitoring, especially in dynamic K8s environments. In this article, I’ll share my practical experience with Falco so you can confidently protect your cluster.

Context & Why is Runtime Security for Kubernetes Essential?

Kubernetes offers tremendous flexibility and scalability, accompanied by its unique security challenges. Traditional solutions often focus on firewalls, network intrusion detection/prevention systems (IDS/IPS), or image scanning. In contrast, runtime security focuses on what is *actually happening* inside your containers and nodes *during operation*.

Picture this: You’ve carefully locked the doors (firewall) and checked the identities of those entering (image scanning), but once an intruder has gained access to the house (a compromised container), how do you know what they’re doing? That’s where runtime security comes into play.

Common runtime threats include:

• Container Escape: An attacker breaks out of a container and gains access to the host node.
• Malicious Command Execution: Running shells, modifying file systems, downloading malware.
• Sensitive Data Access: Reading configuration files, secrets, or application data.
• Unwanted Configuration Changes: Modifying critical system files or network configurations (e.g., changing `/etc/hosts` or `/etc/resolv.conf`).

Falco, an open-source CNCF project, is truly an ideal choice for addressing this challenge. It acts as a Runtime Threat Detection system. Falco operates by monitoring system-level events (system calls) and comparing them against a defined set of rules. If behavior violating these rules is detected, Falco immediately alerts you.

Installing Falco on Kubernetes

The best and recommended way to install Falco on Kubernetes is by using Helm. Falco will be deployed as a DaemonSet, ensuring it runs on every node in your cluster for comprehensive monitoring.

Step 1: Add the Falco Helm repository

First, you need to add the Falco repository to your Helm client:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Step 2: Create a Namespace (optional but recommended)

I usually create a dedicated namespace for security tools for easier management:

kubectl create namespace falco

Step 3: Install Falco using Helm

Now, you can install Falco into the namespace you just created. I typically use the latest version and let Falco automatically load the kernel module if needed:

helm install falco falcosecurity/falco \n  --namespace falco \n  --set falco.kind=DaemonSet \n  --set falco.loadKernelModule.enabled=true \n  --set falco.loadBPF.enabled=false

A brief explanation:

• falco.kind=DaemonSet: Ensures Falco runs on all nodes.
• falco.loadKernelModule.enabled=true: Falco will attempt to load the kernel module to monitor system calls. This is the most efficient method.
• falco.loadBPF.enabled=false: If the kernel module doesn’t work, Falco can use eBPF. I usually prioritize the kernel module, but you can enable eBPF if your environment supports it better or if you encounter issues with the kernel module.

Step 4: Verify installation status

After installation, check if Falco’s pods are running stably:

kubectl get pods -n falco

You should see pods with Running status. Next, view the logs of one of the Falco pods to ensure it started successfully and without errors:

kubectl logs -f <falco_pod_name> -n falco

Detailed Falco Configuration: The Power of Falco Rules

At the heart of Falco are its **Falco Rules**. These are the rules that define what behavior is normal and what is suspicious. Falco comes with a very good set of default rules, but to be truly effective, you need to customize them for your environment.

Structure of a Falco Rule

Each rule in Falco is defined in a YAML file and includes the following key components:

• rule: Unique name for the rule.
• desc: Description of the rule.
• condition: Boolean expression that determines when the rule is triggered (using Falco’s syntax).
• output: Message to be displayed when the rule is triggered.
• priority: Priority level of the alert (e.g., Critical, Warning, Notice).
• tags: Tags to categorize the rule (e.g., network, file, shell).

Example of a simple rule:

- rule: Detect Shell in Container
  desc: A shell was spawned in a container. This could be normal for debugging, but often indicates suspicious activity.
  condition: >
    spawned_process and container and proc.name in (bash, sh, csh, ksh, zsh)
    and not user.name in (root)
    and not proc.pname=containerd
  output: >
    Shell spawned in container (user=%user.name container=%container.name
    container_id=%container.id image=%container.image.repository
    command=%proc.cmdline parent=%proc.pname pid=%proc.pid)
  priority: WARNING
  tags: [shell, container, process]

Customizing Falco Rules

I typically avoid directly modifying the default rule files (falco_rules.yaml, falco_rules.local.yaml) provided by Falco. Instead, I create a separate falco_rules.custom.yaml file and mount it into the Falco pod. This approach makes it easier to upgrade Falco without losing my changes.

To add custom rules, you can update Falco’s Helm configuration. Create a values.yaml file as follows:

# values.yaml
falco:
  rules:
    customRules:
      my_custom_rules.yaml: |
        # Rule to detect sensitive file creation in /etc
        - rule: Write Sensitive File in etc
          desc: An attempt to write a sensitive file in /etc directory was made. This is highly suspicious.
          condition: >
            write and fd.name startswith "/etc/" and not fd.name in ("/etc/resolv.conf", "/etc/hostname", "/etc/mtab")
            and container
          output: >
            Sensitive file written in /etc (user=%user.name container=%container.name
            file=%fd.name command=%proc.cmdline)
          priority: CRITICAL
          tags: [filesystem, container, security]

        # Rule to detect unwanted 'kubectl exec' usage
        - rule: Kubectl Exec into Container
          desc: A kubectl exec command was detected which might indicate an interactive shell session in a container.
          condition: >
            spawned_process and proc.name=kubectl and proc.args contains "exec"
          output: >
            Kubectl exec detected (user=%user.name command=%proc.cmdline)
          priority: NOTICE
          tags: [kubernetes, shell]

        # Rule to detect sensitive file permission changes
        - rule: Change Sensitive File Permissions
          desc: File permissions changed on a sensitive file. Could indicate an attempt to elevate privileges.
          condition: >
            chmod and fd.name in ("/etc/passwd", "/etc/shadow", "/etc/sudoers")
            and container
          output: >
            Sensitive file permissions changed (user=%user.name container=%container.name
            file=%fd.name mode=%fd.mode command=%proc.cmdline)
          priority: CRITICAL
          tags: [filesystem, permissions, container]

Then, update the Falco installation using Helm:

helm upgrade falco falcosecurity/falco \n  --namespace falco \n  -f values.yaml

Falco also offers macros and lists to help you write more concise and manageable rules. For instance, you can define a list of shells for reuse.

Configuring Outputs

By default, Falco sends alerts to stdout (pod logs). You can configure Falco to send alerts to various destinations, making monitoring more convenient:

• File: Writes alerts to a log file.
• Syslog: Sends alerts to a syslog server.
• HTTP/HTTPS: Sends alerts as webhooks to services like Slack, PagerDuty, or a Security Information and Event Management (SIEM) system.
• gRPC: Provides an API for other applications to receive alerts.

To configure outputs, you also edit the values.yaml file during Helm chart installation or upgrade. For example, to send alerts to a Slack webhook:

# Add to values.yaml
falco:
  falco.jsonOutput: true
  falco.jsonOutputKey: output
  falco.outputs:
    stdout: true
    webhook:
      enabled: true
      url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
      insecure: false
      headers:
        Content-Type: "application/json"
      # customHeaders: [] # If custom headers are needed
      # template: "{"text":"Falco Alert: %output"}" # Customize Slack message format
      # Customize template for Falco to send direct JSON output, if Slack webhook supports it
      # or if a more complex format is required
      # template: |
      #   {
      #     "text": "Falco Alert: %output",
      #     "attachments": [
      #       {
      #         "color": "danger",
      #         "title": "%rule",
      #         "text": "%output",
      #         "fields": [
      #           {
      #             "title": "Priority",
      #             "value": "%priority",
      #             "short": true
      #           },
      #           {
      #             "title": "Container",
      #             "value": "%container.name",
      #             "short": true
      #           }
      #         ],
      #         "ts": "%evt.time.s"
      #       }
      #     ]
      #   }

After that, run helm upgrade to apply the new configuration.

Testing & Monitoring

Once installed, we need to verify if it’s working as expected, right? This is the part I find most interesting.

Testing Falco by Creating Events

I’ll create a rule-violating event to see how Falco reacts. Let’s try triggering the pre-provided Detect Shell in Container rule.

1. Find any pod: Choose a running pod in your cluster. For example, a simple nginx pod.

kubectl get pods

2. Execute a shell into that pod:

kubectl exec -it <your_pod_name> -- sh

If the pod doesn’t have sh, try bash, ash, or any available shell.

3. Check Falco’s logs: Open another terminal and view the logs of a Falco pod:

   kubectl logs -f <falco_pod_name> -n falco

   You will see an alert similar to this:

   ... Warning Shell spawned in container (user=root container=nginx-xxxx-xxxx container_id=xxxxxxxx image=nginx command=sh parent=runc pid=xxxx) ...

Try some other behaviors to test your custom rules:

• Inside the pod’s shell, try writing a file to /etc/:

echo "hello" > /etc/my_suspicious_file.txt

• Or try changing permissions of a sensitive file (if you have the rights, of course):

chmod 777 /etc/passwd

Each of these actions will trigger the corresponding rules, and Falco will log them.

Integration with Monitoring & SIEM Systems

Simply viewing Falco’s logs is not sufficient in a production environment. You need to integrate Falco with monitoring and event management systems to gain a comprehensive overview and enable rapid response:

• Prometheus & Grafana: Falco can export metrics for Prometheus to collect and display on Grafana. This helps you monitor the number and types of alerts over time.
• ELK Stack (Elasticsearch, Logstash, Kibana): Send Falco’s logs to Logstash, then store them in Elasticsearch and visualize them with Kibana. This is a very powerful way to analyze logs and search for threats.
• Commercial SIEM: Commercial SIEM solutions like Splunk and QRadar can also receive alerts from Falco via Syslog or webhooks.

Best Practices & Practical Tips

• Reduce False Positives: This is one of the biggest challenges. A false alert can cause you to overlook a genuine one. Fine-tune rules by adding and not conditions to exclude legitimate behaviors. For example, if a specific application process frequently writes to /tmp, add and not proc.name=my_app_process to the rule.
• Prioritize Alerts: Set appropriate priority levels for rules. CRITICAL alerts require immediate attention, while NOTICE alerts can be reviewed later.
• Create Application-Specific Rules: Each application has its own behavior. You should create specific rules for each application to detect particular anomalous behaviors.
• Periodically Review Rules: Kubernetes environments are constantly changing. Regularly review and update Falco’s rules to ensure they remain relevant to your current applications and infrastructure.
• Integrate with Automated Response: For extremely critical alerts, you might consider integrating Falco with automated response tools (e.g., Falco Sidekick can send events to a Lambda function to automatically kill a pod or scale down a deployment).

Conclusion

Falco is an extremely effective tool, almost a must-have in today’s Kubernetes security toolkit. It adds a crucial layer of runtime protection, helping you detect malicious activities early that other solutions might miss. Take the time to install, configure, and fine-tune Falco for your environment. Security is always an ongoing process, and Falco will be a powerful ally, providing greater peace of mind on that journey.

If you have any questions or want to share your experiences with Falco, don’t hesitate to leave a comment!