Your Kubernetes Cluster Runs Smoothly, but Is It Truly Secure?
Once a Kubernetes cluster is up, pods are running smoothly, and Ingress routes traffic properly, most engineers breathe a sigh of relief. I used to be the same way. But after a test server got hit by an automated SSH brute-force attack via an exposed port at 2 AM, I learned a crucial lesson: a working system is never guaranteed to be a secure system.
By default, Kubernetes prioritizes ease of use during initial setup. Configuration files for kube-apiserver, kubelet, or etcd often come with overly permissive file permissions (such as 644 or 777). If a web pod suffers an RCE vulnerability, an attacker could read service account tokens, exploit exposed API ports, and take full control of the entire cluster. To remediate these configuration vulnerabilities, we need an automated and reliable auditing tool.
What Are CIS Kubernetes Benchmark and Kube-bench?
To audit effectively, you first need to understand these two foundational resources:
- CIS Kubernetes Benchmark: A 200+ page guide published by the Center for Internet Security. It defines comprehensive security best practices and configuration standards for the Control Plane, etcd, Worker Nodes, and authentication mechanisms.
- Kube-bench: An open-source Go tool developed by Aqua Security. Kube-bench checks node configurations, file permissions, and process parameters against CIS standards, generating a detailed report in just 10–15 seconds.
Hands-on: Auditing Kubernetes Security with Kube-bench
Depending on your cluster architecture (self-managed with kubeadm or managed services like EKS and GKE), you can choose one of the two methods below.
Method 1: Running Kube-bench as a Kubernetes Job (Recommended)
This method allows you to quickly scan your cluster directly via kubectl without needing SSH access to each node.
Create the manifest file job-kube-bench.yaml:
cat <<EOF > job-kube-bench.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: kube-bench
spec:
template:
spec:
hostPID: true
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: kube-bench
image: aquasec/kube-bench:latest
command: ["kube-bench"]
volumeMounts:
- name: var-lib-etcd
mountPath: /var/lib/etcd
readOnly: true
- name: var-lib-kubelet
mountPath: /var/lib/kubelet
readOnly: true
- name: etc-systemd
mountPath: /etc/systemd
readOnly: true
- name: etc-kubernetes
mountPath: /etc/kubernetes
readOnly: true
- name: usr-bin
mountPath: /usr/local/mount-from-host/bin
readOnly: true
restartPolicy: Never
volumes:
- name: var-lib-etcd
hostPath:
path: /var/lib/etcd
- name: var-lib-kubelet
hostPath:
path: /var/lib/kubelet
- name: etc-systemd
hostPath:
path: /etc/systemd
- name: etc-kubernetes
hostPath:
path: /etc/kubernetes
- name: usr-bin
hostPath:
path: /usr/bin
EOFDeploy the Job and view the logs:
# Create the scan Job
kubectl apply -f job-kube-bench.yaml
# Wait for the Pod to complete
kubectl wait --for=condition=complete job/kube-bench --timeout=60s
# View the scan report
kubectl logs job/kube-benchMethod 2: Running the Binary Directly on Master / Worker Nodes
For bare-metal servers or self-managed VMs, running the standalone binary is the most straightforward way to audit each node individually.
# Download the latest release
curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.8.0/kube-bench_0.8.0_linux_amd64.tar.gz -o kube-bench.tar.gz
tar -xvf kube-bench.tar.gz
# Audit the Control Plane node
sudo ./kube-bench run --targets master
# Audit the Worker node
sudo ./kube-bench run --targets nodeUnderstanding Results and Remediating FAIL Warnings
Kube-bench categorizes scan results into four statuses:
[PASS]: Complies with CIS benchmarks.[FAIL]: Critical security issue that requires immediate attention.[WARN]: Warning that should be reviewed based on your specific use case.[INFO]: Informational item for reference.
Beneath every FAIL item, Kube-bench provides a == Remediation == block with specific remediation steps. Here are two of the most common issues I frequently encounter on newly deployed K8s clusters:
1. Incorrect API Server Configuration File Permissions (Section 1.1.1)
By default, manifest files containing initialization secrets and tokens may be readable by standard OS users. Restrict permissions to 600:
sudo chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml
sudo chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml2. Kubelet Anonymous Authentication Enabled (Section 4.2.1)
If anonymous-auth is enabled, anyone with access to the node’s internal network can send requests directly to Kubelet on port 10250. Edit /var/lib/kubelet/config.yaml:
authentication:
anonymous:
enabled: false
webhook:
enabled: trueRestart the service to apply changes:
sudo systemctl restart kubeletReal-World Tips for Auditing Kubernetes Clusters
After using Kube-bench extensively across staging and production environments, here are a few key takeaways:
- Don’t blindly aim for 100% PASS: Some rules recommending disabling anonymous requests or blocking hostPath might break your CNI (like Calico, Cilium) or Storage CSI plugins. Always test thoroughly in staging before applying fixes in production.
- Set up automated CronJobs and alerts: Schedule a weekly CronJob scan (e.g., Sunday at 3 AM). Forward error logs directly to your DevOps team’s Slack or Telegram channel so you are instantly notified if file permissions are accidentally altered.
- Use the matching benchmark version: Always verify your current K8s version and pass the appropriate flag, such as
--benchmark cis-1.8. Running mismatched versions will produce inaccurate results.
Infrastructure security is an ongoing process. Integrating Kube-bench into your regular pipeline alongside tools like Trivy Operator and Kyverno policy automation will give you peace of mind knowing your Kubernetes cluster isn’t exposed to basic misconfigurations.
