Go GOMAXPROCS in Containers: The CPU Detection Problem

I once spent a day hunting latency that was just GOMAXPROCS in a container. “Our Go service is throttled 70% of the time despite low CPU usage.” I was staring at Grafana dashboards showing our API gateway hovering at 30% CPU utilization, yet the p99 latency was through the roof, and our users were complaining about intermittent slowdowns. The metrics didn’t make sense—how could a service be slow when it was barely using its allocated resources?

The culprit turned out to be one of the most common Go-in-containers mistakes: GOMAXPROCS misconfiguration. Go’s runtime was detecting all 64 CPUs on the host machine, completely ignoring our container’s 2 CPU limit. This mismatch created a cascade of problems that took us days to diagnose.

This problem is particularly insidious because it doesn’t show up in typical metrics. CPU usage looks fine. Memory is fine. Everything seems fine—except for those inexplicable latency spikes that happen “randomly” throughout the day.

Tested on: Go 1.21, Kubernetes 1.28, 2 CPU limit container on 64-core host

Understanding Go’s Scheduler

Before diving into the fix, let’s understand why Go behaves this way. Go has its own scheduler that manages goroutines—lightweight threads that are the backbone of Go’s concurrency model. The Go scheduler is a sophisticated piece of engineering that multiplexes potentially millions of goroutines onto a smaller number of OS threads.

The GOMAXPROCS Variable

GOMAXPROCS controls how many OS threads can execute user-level Go code simultaneously. It’s essentially the parallelism limit for your Go program. When your program starts, Go queries the system to determine how many CPUs are available and sets GOMAXPROCS accordingly.

// What Go does by default
runtime.GOMAXPROCS(0)  // Returns host CPU count

// On a 64-core host with 2 CPU container limit:
// GOMAXPROCS = 64  (wrong!)
// Go creates 64 OS threads for goroutine scheduling
// But only 2 CPUs worth of quota available

The problem is that Go’s CPU detection uses Linux’s sysconf(_SC_NPROCESSORS_ONLN) or reads from /proc/cpuinfo. Both of these return the host’s CPU count, not the container’s cgroup-limited quota. Go was designed before containers became ubiquitous, and this detection mechanism made perfect sense for bare metal and VM deployments.

The Container Mismatch

Containers don’t actually limit which CPUs your process can see—they limit how much CPU time your process gets. Your Go service can “see” all 64 CPUs on the host. It just can’t use more than 2 CPUs worth of time per scheduling period. This is fundamentally different from a VM where you’d actually have a 2-CPU virtual machine.

Why This Causes Throttling

The Linux Completely Fair Scheduler (CFS) enforces CPU limits through a quota mechanism. With a 2 CPU limit in Kubernetes, your container gets 200ms of CPU time every 100ms period (2 × 100ms = 200ms quota).

Host: 64 CPUs
Container limit: 2 CPUs (200ms quota per 100ms period)

Go runtime with GOMAXPROCS=64:
┌─────────────────────────────────────────────────────────────┐
│ 64 OS threads competing for 2 CPUs worth of time           │
│                                                              │
│ Thread 1:  [run].....[run].....[run].....                   │
│ Thread 2:  .....[run].....[run].....[run]                   │
│ Thread 3:  ..[run].....[run].....[run]...                   │
│ ...                                                          │
│ Thread 64: [run].....[run].....[run].....                   │
│                                                              │
│ CFS sees: 64 threads × time slices = exceeds 200ms quota   │
│ Result: THROTTLED even at "low" CPU usage                   │
└─────────────────────────────────────────────────────────────┘

Symptoms:
- High CFS throttling (nr_throttled in cpu.stat)
- Latency spikes during throttle periods
- CPU usage appears low in metrics

Here’s where things get counterintuitive. When CFS throttles your container, it stops ALL threads in that container until the next period begins. This creates coordinated latency spikes—every request being processed when throttling kicks in experiences a delay.

With 64 threads, each one takes a small time slice. Even if most are idle, the scheduler still context-switches between them, consuming CPU quota just for the overhead of thread management. You’re burning CPU time not on actual work, but on managing threads that shouldn’t exist.

The Context Switching Tax

Context switches aren’t free. Each switch requires saving and restoring register state, flushing TLB entries, and potentially invalidating CPU caches. With 64 threads competing for 2 CPUs, you get constant context switches. I’ve seen services where the context switching overhead consumed 20-30% of their total CPU budget.

The irony is that the extra threads don’t help you go faster—they actively slow you down. A 2 CPU service with GOMAXPROCS=2 will outperform the same service with GOMAXPROCS=64 by a significant margin.

Measuring the Impact

Before you fix anything, measure the current state. These commands help you understand the severity of the problem:

# Check current GOMAXPROCS in running container
curl localhost:6060/debug/pprof/goroutine?debug=2 | head -20

# Check CFS throttling (cgroup v2)
cat /sys/fs/cgroup/cpu.stat
# nr_throttled should be near 0, not thousands

# Check CFS throttling (cgroup v1)
cat /sys/fs/cgroup/cpu/cpu.stat

# Check context switches
cat /proc/self/status | grep ctxt
# voluntary_ctxt_switches: should be reasonable
# nonvoluntary_ctxt_switches: high = problem

# Check actual GOMAXPROCS from inside the container
go run -e 'runtime.GOMAXPROCS(0)'

If you see thousands or millions of nr_throttled events and high nonvoluntary_ctxt_switches, you’ve found your problem.

Solutions

There are several approaches to fix this, ranging from dead simple to slightly more involved.

1. Use automaxprocs (Uber’s Library)

This is the recommended solution for most teams. Uber’s automaxprocs library automatically detects container CPU limits and sets GOMAXPROCS appropriately. It reads from cgroup files to determine the actual CPU quota.

import _ "go.uber.org/automaxprocs"

func main() {
    // automaxprocs automatically sets GOMAXPROCS
    // based on container CPU quota
    // 2 CPU limit → GOMAXPROCS=2
}

# go.mod
require go.uber.org/automaxprocs v1.5.3

The beauty of automaxprocs is that it’s a blank import. You add one line to your main package, and it handles everything during init. It also logs the detected value, so you can verify it’s working:

maxprocs: Updating GOMAXPROCS=2: determined from CPU quota

Automaxprocs handles both cgroup v1 and v2, deals with fractional CPU limits (rounding appropriately), and falls back to the host CPU count if it can’t detect a limit (useful for local development).

2. Set via Environment Variable

If you prefer not to add a dependency, Go respects the GOMAXPROCS environment variable. You can set it explicitly in your Kubernetes deployment:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            limits:
              cpu: "2"
          env:
            - name: GOMAXPROCS
              value: "2"

The downside is that you need to remember to keep the environment variable in sync with your CPU limit. Change one without the other, and you’re back to problems. This approach works but requires discipline.

3. Downward API for Dynamic Setting

A more robust approach uses Kubernetes Downward API to inject the CPU limit as an environment variable, which your code then reads:

# deployment.yaml - dynamic GOMAXPROCS
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            limits:
              cpu: "2"
          env:
            - name: CPU_LIMIT
              valueFrom:
                resourceFieldRef:
                  resource: limits.cpu

package main

import (
    "os"
    "runtime"
    "strconv"
)

func init() {
    if cpuLimit := os.Getenv("CPU_LIMIT"); cpuLimit != "" {
        if n, err := strconv.Atoi(cpuLimit); err == nil {
            runtime.GOMAXPROCS(n)
        }
    }
}

This keeps your deployment DRY—the CPU limit is defined once, and GOMAXPROCS follows automatically. However, it still requires code changes and doesn’t handle fractional limits well.

4. Programmatic Detection

For the most control, you can read cgroup files directly. This is essentially what automaxprocs does internally:

package main

import (
    "os"
    "runtime"
    "strconv"
    "strings"
)

func setCPULimit() {
    // Read cgroup v2 CPU max
    data, err := os.ReadFile("/sys/fs/cgroup/cpu.max")
    if err != nil {
        return // Fall back to default
    }

    parts := strings.Fields(string(data))
    if len(parts) >= 2 && parts[0] != "max" {
        quota, _ := strconv.Atoi(parts[0])
        period, _ := strconv.Atoi(parts[1])
        if period > 0 {
            cpus := quota / period
            if cpus > 0 {
                runtime.GOMAXPROCS(cpus)
            }
        }
    }
}

This approach gives you full control and zero dependencies, but you need to handle both cgroup v1 and v2, edge cases, and testing. Unless you have specific requirements, automaxprocs is easier.

Benchmarks

I ran benchmarks on a real Kubernetes cluster to quantify the impact. The results were dramatic:

Scenario: HTTP server handling 1000 req/s
Container: 2 CPU limit on 64-core host

GOMAXPROCS=64 (default):
  p50 latency: 45ms
  p99 latency: 890ms
  Throttle rate: 72%
  Context switches/s: 45,000

GOMAXPROCS=2 (correct):
  p50 latency: 12ms
  p99 latency: 35ms
  Throttle rate: 3%
  Context switches/s: 2,100

Improvement: 4x better p50, 25x better p99

The p99 improvement is the most striking. Those 890ms tail latencies were entirely artificial—caused by throttling pauses, not by actual work taking that long. With correct GOMAXPROCS, the tail latencies dropped to 35ms, close to what you’d expect for the actual request processing time.

The context switch reduction from 45,000/s to 2,100/s also has downstream effects. Less cache thrashing means better CPU efficiency. Less scheduler overhead means more time for actual work.

Monitoring

Set up monitoring to catch this problem before it affects users. The go_gomaxprocs metric is exposed by the default Prometheus client library:

# Prometheus alert for GOMAXPROCS mismatch
- alert: GoMaxProcsNotSet
  expr: |
    go_gomaxprocs >
    (container_spec_cpu_quota / container_spec_cpu_period) * 2
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "GOMAXPROCS ({{ $value }}) exceeds CPU limit"

# Alert on high throttling
- alert: ContainerCPUThrottling
  expr: |
    rate(container_cpu_cfs_throttled_periods_total[5m]) /
    rate(container_cpu_cfs_periods_total[5m]) > 0.25
  for: 10m
  annotations:
    summary: "Container {{ $labels.container }} throttled >25%"

The first alert catches services where GOMAXPROCS exceeds the CPU limit. The multiplier of 2 provides some slack—GOMAXPROCS=4 on a 2 CPU limit is suboptimal but not catastrophic.

The second alert catches throttling regardless of cause. Even with correct GOMAXPROCS, if your service is genuinely CPU-bound and exceeds its limit, you’ll see throttling. That’s different from the pathological throttling caused by GOMAXPROCS misconfiguration.

Dashboard Panels

Add these to your Grafana dashboard:

# GOMAXPROCS vs CPU limit ratio
go_gomaxprocs / on(pod)
(container_spec_cpu_quota / container_spec_cpu_period)

# Throttle percentage
rate(container_cpu_cfs_throttled_periods_total[5m]) /
rate(container_cpu_cfs_periods_total[5m]) * 100

# Context switches per second
rate(container_context_switches_total[5m])

Checklist

## Go Container CPU Configuration

### Setup
- [ ] Add `go.uber.org/automaxprocs` to all Go services
- [ ] Or set GOMAXPROCS env var equal to CPU limit
- [ ] Verify with /debug/pprof endpoint

### Monitoring
- [ ] Export go_gomaxprocs metric
- [ ] Alert on GOMAXPROCS > CPU limit
- [ ] Track CFS throttling rate

### Verification
- [ ] Check nr_throttled in cpu.stat
- [ ] Compare context switch rates before/after
- [ ] Load test to verify latency improvement

Conclusion

Go’s container CPU detection problem is one of those issues that seems obscure until you hit it—and then it explains mysterious performance problems you’ve been chasing for weeks. The fix is simple: add go.uber.org/automaxprocs to your services.

Key takeaways:

1. Go detects host CPUs, not container limits—this is by design, not a bug
2. GOMAXPROCS=64 on a 2 CPU container creates massive overhead from context switching and thread management
3. Use automaxprocs for automatic, correct detection with zero configuration
4. Monitor throttling to catch this and similar issues early
5. The fix takes 5 minutes and can improve p99 latency by 25x

Check your Go services today. Run curl localhost:6060/debug/pprof/goroutine?debug=2 | head -5 and look at the GOMAXPROCS value. If it’s higher than your CPU limit, you have work to do.

Related Articles

• Python GIL and K8s CPU Limits - Similar issue in Python
• K8s CPU Throttling Autopsy - Deep dive on CFS throttling