Real-Time Runtime Security: Blocking Malicious Processes with eBPF and Tetragon

The Shift from Detection to Prevention

For years, the standard approach to Kubernetes security has followed a predictable pattern: scan your images in the CI pipeline, audit your configurations with admission controllers, and set up alerts for suspicious behavior in production. While these are essential layers of a defense-in-depth strategy, they share a common weakness: they are often reactive.

By the time a traditional security tool alerts you that a shell has been opened in a production container, the attacker has likely already moved laterally or exfiltrated data. The challenge in modern cloud-native environments is moving from detection—telling you something bad happened—to prevention—stopping the bad thing before it executes.

This is where eBPF (extended Berkeley Packet Filter) and Tetragon come into play. By operating at the kernel level, we can achieve real-time enforcement that is decoupled from the application code and carries minimal performance overhead.

Why Traditional Runtime Security Falls Short

To understand why we need eBPF-based tools like Tetragon, we have to look at the limitations of existing methods:

1. LD_PRELOAD and User-Space Hooks: These can be easily bypassed by statically linked binaries or by an attacker who gains enough privilege to unset environment variables.
2. Syscall Auditing (Auditd/Seccomp): While powerful, standard Linux auditing can generate massive amounts of logs, leading to "alert fatigue." Seccomp is excellent for reducing attack surface but is notoriously difficult to manage at scale across diverse microservices; one missed syscall can crash your entire application.
3. Context Gap: Many security tools look at events in isolation. They see a write() syscall, but they don't necessarily know the full process lineage, the Kubernetes metadata (pod name, namespace), or the network state associated with that specific execution.

The eBPF Advantage

eBPF has revolutionized observability and security because it allows us to run sandboxed programs inside the Linux kernel without changing kernel source code or loading kernel modules.

When a process tries to execute a file, open a socket, or modify a sensitive configuration, it must interact with the kernel via system calls. eBPF programs can hook into these calls (using kprobes or tracepoints) and even into the Linux Security Module (LSM) hooks. Because the eBPF program runs within the kernel execution path, it can make a decision to allow or deny the action before the kernel actually completes the request. This is the difference between an alarm going off after a door is kicked in and a lock that refuses to turn for the wrong key.

Introducing Tetragon

Tetragon is the real-time security observability and runtime enforcement component of the Cilium project. While Cilium focuses on the network layer, Tetragon focuses on the process and file system layer.

What sets Tetragon apart is its ability to perform kernel-level enforcement. It doesn't just watch; it acts. Using a Custom Resource Definition (CRD) called a TracingPolicy, you can define exactly which behaviors are permitted and which should be blocked. If a process violates a policy, Tetragon can send a SIGKILL to that process immediately, effectively neutralizing the threat in microseconds.

Implementing Tetragon in Kubernetes

To get started, you typically install Tetragon via Helm. It runs as a DaemonSet, ensuring every node in your cluster is protected.

helm install tetragon cilium/tetragon -n kube-system

Once installed, Tetragon begins monitoring process execution. However, its true power is unlocked through TracingPolicy. Let's look at three practical scenarios where Tetragon provides superior protection.

Scenario 1: Blocking Unauthorized Binaries

Imagine you have a production Nginx pod. There is no legitimate reason for an engineer or an automated script to run nmap, netcat, or apt-get inside that container. In a standard setup, an attacker who gains access might download these tools to probe your internal network.

With Tetragon, we can create a policy that specifically targets these binaries. Here is a simplified example of a TracingPolicy that monitors and blocks unauthorized execution:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "block-unauthorized-tools"
spec:
  kprobes:
    - call: "tcp_connect"
      syscall: false
      args:
        - index: 0
          type: "sock"
      selectors:
        - matchBinaries:
            - operator: "In"
              values:
                - "/usr/bin/nmap"
                - "/usr/bin/nc"
          matchActions:
            - action: Sigkill

In this policy, if any process matching those binary paths attempts to initiate a TCP connection, the kernel will immediately issue a Sigkill. The process dies before the first packet even leaves the pod.

Scenario 2: Preventing Sensitive File Access

Attackers often target /etc/shadow, Kubernetes service account tokens, or custom application secrets. While Kubernetes RBAC handles API access, it doesn't prevent a compromised process from reading files on the local disk if the container user has permissions.

We can use Tetragon to monitor the fd_install or do_sys_open functions in the kernel to prevent access to specific paths.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "protect-sensitive-files"
spec:
  kprobes:
    - call: "do_sys_openat2"
      syscall: false
      args:
        - index: 1
          type: "string" # This is the filename
      selectors:
        - matchArgs:
            - index: 1
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/var/run/secrets/kubernetes.io/serviceaccount"
          matchActions:
            - action: Sigkill

Scenario 3: Restricting Process Lineage

One of the most common attack vectors is "living off the land"—using legitimate tools for illegitimate purposes. For example, curl is often present in images for health checks. However, curl should only be called by your startup scripts or specific cron jobs, not by the www-data user after the server has been running for three days.

Tetragon allows you to filter based on the parent process. You can allow curl if its parent is a specific initialization script, but block it if its parent is the shell (sh or bash).

Real-World Considerations: Performance and Stability

As a senior engineer, your first concern with kernel-level hooking should be performance. "Will this slow down my high-throughput API?"

The beauty of eBPF is that the filtering happens in the kernel context. Unlike older technologies that context-switch to user-space for every event (like some FUSE implementations or older versions of Falco), Tetragon only sends data to user-space when a policy match occurs. If a process is behaving normally and doesn't trigger a policy, the overhead is nearly imperceptible—typically less than 1-2% CPU utilization.

However, there are risks. A poorly written eBPF program could cause issues, but Tetragon uses the kernel's built-in eBPF verifier. This verifier ensures that the eBPF code is safe, cannot loop infinitely, and cannot access unauthorized memory regions. This provides a layer of safety that traditional kernel modules lack.

Operationalizing Tetragon: The "Audit First" Strategy

You should never jump straight to Sigkill in a production environment. Blocking a legitimate process is a fast way to cause a self-inflicted Denial of Service (DoS).

1. Observability Mode: Deploy your TracingPolicy without the Sigkill action. Tetragon will log the events to its stdout (which you can collect via FluentBit or Vector).
2. Analysis: Review the logs to identify false positives. Perhaps your monitoring agent uses cat on a file you thought was off-limits.
3. Refinement: Adjust your matchBinaries or matchArgs to exclude legitimate internal traffic.
4. Enforcement: Once you have a clean log for 48-72 hours, update the policy to include action: Sigkill.

Integrating with the Security Stack

Tetragon doesn't replace your existing tools; it completes them.

• Image Scanning: Still necessary to find vulnerabilities in libraries (CVEs).
• Admission Controllers: Still necessary to ensure pods don't run as root.
• Tetragon: Provides the final, unbreakable line of defense that stops execution when the previous layers are bypassed.

The logs generated by Tetragon are JSON-formatted and include rich Kubernetes metadata. This makes them incredibly valuable for SOC (Security Operations Center) teams. Instead of a vague "Process executed" log, they get:

"Pod 'payment-processor-59f', Namespace 'prod', Node 'worker-3' attempted to execute '/usr/bin/apt-get' and was terminated by Sigkill."

Conclusion

Implementing real-time runtime security with eBPF and Tetragon represents a significant leap forward in how we protect Kubernetes workloads. By moving enforcement into the kernel, we gain the ability to stop attacks in progress with minimal performance impact and high precision.

To get started today:

1. Audit your most sensitive workloads: Identify which pods handle PII or financial data.
2. Deploy Tetragon in a dev cluster: Experiment with the default observability logs to see the sheer amount of data eBPF provides.
3. Draft your first TracingPolicy: Start by monitoring access to /etc/ or the execution of shell binaries.
4. Adopt a 'Deny by Default' mindset: Gradually work toward a state where only the exact binaries required for your application's lifecycle are permitted to run.

Security is no longer just about building higher walls; it's about having a kernel that knows exactly who is allowed to hold the keys.