Real-Time K8s Security: Blocking Malicious Processes with eBPF & Tetragon

In the early days of Kubernetes, security was largely a perimeter game. We focused on ingress controllers, network policies, and scanning container images before they reached the registry. While these remain essential, they share a common flaw: they are static or reactive. If an attacker bypasses your perimeter or exploits a zero-day vulnerability in a running application, your static defenses become irrelevant.

This is where runtime security enters the picture. However, traditional runtime security often relies on heavy agents or intrusive sidecars that introduce significant performance overhead. Enter eBPF (Extended Berkeley Packet Filter) and Tetragon. By shifting security logic into the Linux kernel, we can achieve deep observability and—more importantly—real-time enforcement with negligible latency.

The Shift from Observability to Enforcement

Most security tools in the Kubernetes ecosystem are built for observability. They tell you that something bad happened. Tools like Falco, for example, are excellent at streaming alerts when a sensitive file is accessed or a shell is opened in a pod. But by the time you receive the alert and your automated response kicks in, the attacker might have already exfiltrated data or established persistence.

Tetragon, a component of the Cilium project, changes the paradigm by focusing on enforcement. Because it operates via eBPF directly within the kernel's execution path, it can intervene at the moment a system call (syscall) is made. It doesn't just watch the process; it can stop it in its tracks before the malicious action completes.

Why eBPF is the Right Tool for the Job

To understand why Tetragon is effective, we have to look at the underlying technology. eBPF allows us to run sandboxed programs within the Linux kernel without changing kernel source code or loading traditional modules.

Historically, if you wanted to monitor process execution, you might use ptrace or auditd. These methods are notoriously slow because they require frequent context switching between user space and kernel space. For a high-traffic Kubernetes node, this overhead can degrade application performance by 20% or more.

eBPF programs run directly in the kernel context. They are triggered by specific events (kprobes, tracepoints, or LSM hooks). When a process attempts to execute a binary (the execve syscall), an eBPF program can inspect the arguments, environment variables, and the identity of the process. If the action violates a security policy, the kernel can terminate the process immediately.

Getting Started with Tetragon

Tetragon is typically deployed as a DaemonSet in your Kubernetes cluster. It listens to the Linux kernel and maps kernel-level events (like process IDs and namespaces) back to Kubernetes-level metadata (like Pod names and Labels).

Installation

The easiest way to install Tetragon is via Helm:

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system

Once installed, the Tetragon agent runs on every node, monitoring the kernel for activity defined in your TracingPolicy custom resources.

Implementing Your First TracingPolicy

The core of Tetragon is the TracingPolicy. This is a Kubernetes Custom Resource Definition (CRD) where you define what to watch and what action to take.

Let's look at a common security requirement: preventing anyone from running a shell (sh or bash) inside a production container. While kubectl exec is useful for debugging, it is a primary vector for manual exploitation.

Example: Blocking Shell Access

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "block-shell-execution"
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string" # The path to the binary
      selectors:
        - matchArgs:
            - index: 0
              operator: "In"
              values:
                - "/bin/sh"
                - "/bin/bash"
          matchActions:
            - action: Sigkill

In this policy:

1. We hook into the sys_execve syscall.
2. We inspect the first argument (index: 0), which represents the binary path.
3. We use a selector to check if that path is /bin/sh or /bin/bash.
4. If it matches, the Sigkill action is triggered, terminating the process before it can start.

Moving Beyond Simple Path Matching

Attackers are clever. If you block /bin/sh, they might upload a renamed binary or use a different execution path. Tetragon allows for more sophisticated filtering based on process lineage and Kubernetes metadata.

Protecting Sensitive Files

Suppose you have a pod that handles TLS certificates stored at /etc/certs. No process other than your main application should ever touch these files. You can write a policy that monitors the fd_install or do_sys_open kernel functions to prevent unauthorized reads.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "protect-certs"
spec:
  kprobes:
    - call: "security_file_open"
      syscall: false
      args:
        - index: 0
          type: "file" # The file being opened
      selectors:
        - matchPIDs:
            - operator: "NotIn"
              values: [1] # Allow the main init process
          matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/etc/certs"
          matchActions:
            - action: Sigkill

This policy uses a Linux Security Module (LSM) hook (security_file_open). It checks if the process opening the file is NOT the main container process (PID 1). If a sidecar or an injected process tries to open anything in /etc/certs, it is killed.

Real-World Scenario: Blocking Reverse Shells

A common post-exploitation step is the "reverse shell," where the compromised container initiates an outbound connection to an attacker-controlled server.

By combining process execution monitoring with network event monitoring, Tetragon can identify a process that was spawned from a web server (like Nginx) and is now trying to open a network socket—a classic sign of a reverse shell.

Unlike a standard firewall that only sees the IP and port, Tetragon sees the intent. It knows which process is opening the connection and can block the socket creation at the kernel level if that process isn't on an allow-list.

Operational Considerations

Implementing runtime enforcement isn't without its challenges. As a senior engineer, you must consider the operational impact of "blocking" mode.

1. The Danger of False Positives

If your policy is too broad, you might accidentally kill a critical system process or a legitimate maintenance script. Always start with action: Post (which just logs the event) before moving to action: Sigkill. Monitor the logs for a few days to ensure your selectors are precise.

2. Performance at Scale

While eBPF is efficient, complex policies with deep inspection can still consume CPU cycles. Use selectors effectively to limit the scope of your probes. For example, instead of monitoring every execve on the node, use Kubernetes namespace selectors to only monitor sensitive production workloads.

3. Integration with CI/CD

Security policies should be treated as code. Your TracingPolicy manifests should live in the same repository as your application code and be deployed via your standard GitOps pipeline (e.g., ArgoCD or Flux). This ensures that as your application evolves, your security posture evolves with it.

Tetragon vs. Pod Security Admissions (PSA)

You might ask: "Can't I just use Pod Security Admissions or OPA Gatekeeper?"

PSAs and Admission Controllers are "Gatekeepers." They check the configuration of a Pod before it is created. They can prevent a Pod from running as root or mounting a host path. However, they cannot see what happens inside the container once it is running.

Tetragon is the "Bodyguard." It doesn't care how the Pod was configured; it cares about what the processes are actually doing in real-time. You need both for a defense-in-depth strategy.

Conclusion and Actionable Steps

Runtime security is no longer an optional luxury for Kubernetes clusters. As the complexity of our supply chains increases, we must assume that some layer of our defense will eventually be breached.

eBPF-based enforcement via Tetragon provides a powerful, low-overhead way to stop attacks in progress. By shifting from passive alerting to active blocking, you significantly reduce the window of opportunity for an attacker.

To get started today:

1. Audit your critical workloads: Identify the top three malicious actions you want to prevent (e.g., shell access, unauthorized network calls, or sensitive file access).
2. Deploy Tetragon in Audit Mode: Use the TracingPolicy to log these actions without blocking them.
3. Refine your selectors: Use Kubernetes labels and namespace filters to minimize false positives.
4. Enforce: Once you have a high degree of confidence, switch the action to Sigkill for your most sensitive production environments.

By moving security logic into the kernel, you aren't just watching for trouble—you're preventing it.