Continuous Profiling for Go: Using eBPF and Pyroscope to Solve CPU Hotspots

Metrics, logs, and traces have long been the 'three pillars' of observability. They tell us when a service is slow, which request failed, and where the latency occurred. However, for Go developers running high-throughput microservices, a critical question often remains unanswered: What exactly is the CPU doing during those latency spikes?

Traditional profiling in Go using net/http/pprof is powerful but often reactive. By the time you manually trigger a profile, the transient spike is gone. Continuous profiling solves this by recording performance data 24/7. When combined with eBPF (Extended Berkeley Packet Filter) and OpenTelemetry, we can achieve deep visibility into our Go applications with near-zero overhead and zero code changes.

The Shift to Continuous Profiling

In a standard Go environment, profiling usually involves exposing a /debug/pprof endpoint and using go tool pprof to analyze a 30-second snippet of CPU activity. While effective for debugging local issues, it fails in production for three reasons:

1. Selection Bias: You only profile when you notice a problem, missing the intermittent regressions that slowly erode performance.
2. Overhead Concerns: While pprof is efficient, manual invocation requires coordination and can occasionally impact performance if not managed.
3. Lack of Context: A standalone profile doesn't show how CPU usage has evolved over several deployments.

Continuous profiling transforms this into a stream of data. By integrating with OpenTelemetry—the industry standard for observability—profiling data becomes part of a unified telemetry pipeline.

Why eBPF is a Game Changer for Profiling

Historically, profiling required some level of instrumentation. eBPF changes the paradigm. It allows us to run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules.

For Go microservices, eBPF-based profiling offers several advantages:

• Zero Instrumentation: You don't need to import a library or recompile your binary. The profiler sits outside the application, sampling the stack traces of running processes.
• System-Wide Visibility: eBPF sees everything. It captures not just your Go code, but also Cgo calls, kernel syscalls, and garbage collection (GC) cycles that might be invisible to user-space profilers.
• Low Overhead: Modern eBPF profilers use frequency-based sampling (e.g., 100Hz). This typically results in less than a 1% CPU overhead, making it safe for production use.

The Architecture: OpenTelemetry, eBPF, and Pyroscope

To implement this, we use a three-tier architecture:

1. The Agent (eBPF): A profiler (like the Pyroscope agent or the OpenTelemetry eBPF profiler) runs on the host. It samples the stack traces of our Go processes.
2. The Collector (OpenTelemetry): The OTel Collector receives the profiling data. It can enrich the data with metadata (service name, pod ID, deployment version).
3. The Backend (Pyroscope): Now part of Grafana Phlare, Pyroscope acts as the storage and visualization layer, allowing us to query flame graphs across time.

Practical Implementation

Let’s look at how to set up continuous profiling for a Go microservice. We will simulate a common bottleneck: inefficient JSON processing in a hot loop.

1. The Vulnerable Code

Consider a service that processes large JSON payloads. A common mistake is repeatedly unmarshaling data into an interface or a map instead of a structured struct, or failing to reuse buffers.

func processData(raw []byte) {
    // High CPU overhead due to reflection-heavy unmarshaling
    var result map[string]interface{}
    for i := 0; i < 1000; i++ {
        json.Unmarshal(raw, &result)
    }
}

2. Deploying the Pyroscope Agent with eBPF

Instead of modifying the Go code, we deploy the Pyroscope agent as a sidecar or a DaemonSet in Kubernetes. The agent uses eBPF to discover the Go process and start sampling.

In a Docker Compose or local environment, you can start the agent and point it to your backend:

services:
  pyroscope:
    image: grafana/pyroscope:latest
    ports:
  - "4040:4040"

  agent:
    image: grafana/pyroscope:latest
    command: ["agent", "--server-address=http://pyroscope:4040"]
    privileged: true # Required for eBPF
    volumes:
      - /sys/kernel/debug:/sys/kernel/debug

3. Integrating with OpenTelemetry

OpenTelemetry is currently standardizing the profiling signal. By using the OTel Collector, you can route profiling data to multiple backends or add resource attributes that link profiles to specific traces.

receivers:
  pyroscope:
    endpoint: 0.0.0.0:4040

exporters:
  otlphttp/pyroscope:
    endpoint: "http://pyroscope-server:4040/otlp"

service:
  pipelines:
    profiles:
      receivers: [pyroscope]
      exporters: [otlphttp/pyroscope]

Identifying CPU Hotspots in Go

Once the data flows into Pyroscope, you are presented with a Flame Graph. For a senior engineer, reading a flame graph is like reading an X-ray. Here’s what to look for in a Go microservice:

Garbage Collection (GC) Pressure

If you see runtime.gcDrain or runtime.mallocgc occupying a large horizontal slice of your flame graph, your CPU is spending too much time managing memory rather than executing business logic.

• The Fix: Use sync.Pool to reuse objects or reduce the frequency of short-lived allocations in hot loops.

Reflection and Encoding

Go’s encoding/json relies heavily on reflection. In the flame graph, this appears as large blocks of reflect.Value.Interface or reflect.Typeof.

• The Fix: Switch to a code-generation-based JSON library like easyjson or ffjson, or move to Protobuf for internal service communication.

Inefficient String Concatenation

If you see runtime.concatstrings, it means you are likely using the + operator in a loop.

• The Fix: Use strings.Builder to pre-allocate memory and reduce allocations.

Regex Compilation

Seeing regexp.(*Regexp).doExecute is fine, but seeing regexp.Compile frequently is a red flag.

• The Express Fix: Ensure regexp.MustCompile is called once in init() or a global variable, never inside a request handler.

The Power of Differential Profiling

One of the most powerful features of continuous profiling is the Diff View. After a new deployment, you can compare the CPU profile of the new version against the previous one.

If the total CPU usage increased by 10%, the Diff View will highlight the specific functions responsible for that delta in red. This allows you to catch performance regressions before they trigger alerts or increase your cloud bill.

Cost and Performance Considerations

While eBPF is low-overhead, storing continuous profiles isn't free. Pyroscope uses specialized compression and downsampling to manage storage costs. For a large-scale microservice architecture:

1. Sampling Frequency: Start with 100Hz. If the overhead is too high (unlikely for Go), drop to 49Hz.
2. Retention: You rarely need minute-by-minute granularity from six months ago. Set aggressive retention policies for raw data (e.g., 7 days) and use aggregated data for long-term trends.
3. Tagging: Use OTel attributes to tag profiles with service.version and deployment.environment. This makes the data actionable.

Conclusion

Continuous profiling with eBPF and OpenTelemetry bridges the gap between knowing that a service is slow and knowing why. By moving away from ad-hoc pprof sessions and toward a continuous, system-wide view, Go developers can proactively identify CPU hotspots that were previously invisible.

Actionable Steps:

1. Audit your hot paths: Identify services with high CPU utilization or high cloud costs.
2. Deploy a PoC: Set up a Pyroscope instance and the eBPF agent in a staging environment.
3. Analyze GC and JSON: Look for runtime.gcDrain and reflect in your flame graphs—these are the low-hanging fruit of Go performance optimization.
4. Integrate with CI/CD: Use differential profiling to compare performance across releases.

By treating performance as a continuous signal rather than a one-off debugging task, you build more resilient, cost-effective, and faster microservices.