Kubernetes CPU Throttling Pitva: Prečo p99 Latencia Exploduje pri 40% CPU Usage

CPU limity vyzerali bezpecne, kym sa throttling neukazal v latency grafoch. “CPU usage je 40%, ale p99 latencia skočila z 50ms na 800ms.” Toto je klasický symptóm CPU throttlingu v Kubernetes, ktorý sa nezobrazuje v štandardných dashboardoch.

Problém: Kubernetes CPU limity používajú Linux CFS (Completely Fair Scheduler), ktorý throttluje aj keď “priemerné” CPU vyzerá OK.

Testované na: Kubernetes 1.28, Java 21, Go 1.22, Prometheus + Grafana

Ako CFS Throttling Funguje

CPU Limity v Kubernetes

resources:
  requests:
    cpu: "500m"   # Garantovaných 0.5 CPU
  limits:
    cpu: "1000m"  # Maximum 1 CPU

CFS Quota Mechanizmus

CFS Period: 100ms (default)
CPU Limit 1000m = 100ms CPU time per 100ms period

Timeline:
|----100ms period----|----100ms period----|
|▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓|                    |
 ↑ 80ms CPU burst    ↑ 20ms čakanie       ↑ Ďalší request čaká!

Problém: Bursts

Scenár: Web request handler
- Väčšina času: idle (0% CPU)
- Pri requeste: 100% CPU na 80ms
- CFS quota: 100ms per 100ms period

Request 1: 80ms CPU → OK (zostáva 20ms quota)
Request 2: 80ms CPU → THROTTLED! (chýba 60ms quota)
         → Request 2 čaká 60ms na ďalší period

Výsledok: Request 2 má +60ms latency (throttling)

Diagnostika: CFS Metriky

Prometheus Query

# Throttled sekundy za minútu
sum(rate(container_cpu_cfs_throttled_seconds_total{
    namespace="production",
    pod=~"api-.*"
}[5m])) by (pod)

Korelácia s Latenciou

# Throttling vs P99 latency
# Panel 1: Throttled periods
sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod)
/
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

# Panel 2: Request latency P99
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Reprodukovateľný Lab

Java Service

// ThrottlingDemo.java
@RestController
public class ThrottlingDemo {

    @GetMapping("/cpu-burst")
    public String cpuBurst() {
        // Simuluj CPU-intensive prácu
        long start = System.nanoTime();
        double result = 0;
        for (int i = 0; i < 10_000_000; i++) {
            result += Math.sin(i) * Math.cos(i);
        }
        long elapsed = (System.nanoTime() - start) / 1_000_000;
        return "Computed in " + elapsed + "ms, result: " + result;
    }
}

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: throttling-demo
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: app
        image: throttling-demo:latest
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "500m"      # Tight limit!
            memory: "512Mi"
        ports:
        - containerPort: 8080

Load Test

# Pošli 10 concurrent requests
hey -n 1000 -c 10 http://throttling-demo:8080/cpu-burst

Výsledky

# S CPU limit 500m:
Summary:
  Requests/sec: 12.3
  Latency distribution:
    50%: 120ms
    90%: 450ms
    99%: 890ms  ← Throttling!

# Bez CPU limitu (len request):
Summary:
  Requests/sec: 45.2
  Latency distribution:
    50%: 45ms
    90%: 52ms
    99%: 68ms   ← 13× lepšie!

Riešenia

1. Odstráň CPU Limity (Kontroverzné!)

resources:
  requests:
    cpu: "500m"    # Ponechaj request pre scheduling
  # limits:        # ŽIADNY CPU limit!
  #   cpu: "1000m"

Prečo to funguje:

• Bez limitu nie je CFS quota
• Bursts môžu využiť voľné CPU na node
• Request garantuje minimálne CPU

Riziká:

• “Noisy neighbor” - jeden pod môže vyžrať CPU
• Menej predvídateľné správanie
• Potrebuješ dobrý monitoring

2. Nastav Vyšší Limit

resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "2000m"  # 4× request = priestor pre bursts

Rule of thumb: limit = 2-4 × request pre bursty workloads.

3. Burstable vs Guaranteed QoS

# Guaranteed QoS (request == limit)
resources:
  requests:
    cpu: "1000m"
  limits:
    cpu: "1000m"  # Rovnaké = Guaranteed

# Burstable QoS (request < limit)
resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "2000m"  # Vyššie = Burstable

4. Java: GOMAXPROCS Equivalent

// JVM automaticky detekuje CPU limit v kontajneri
// Ale môže nastaviť príliš málo GC threads

// Dockerfile alebo deployment:
// -XX:ActiveProcessorCount=4
// Explicitne nastav počet CPU pre JVM

java -XX:ActiveProcessorCount=4 \
     -XX:ParallelGCThreads=4 \
     -jar app.jar

5. Go: GOMAXPROCS

// automaxprocs automaticky nastaví podľa CFS quota
import _ "go.uber.org/automaxprocs"

func main() {
    // GOMAXPROCS automaticky nastavené podľa CPU limit
    // Nie podľa počtu CPU na node
}

Monitoring Dashboard

Grafana Panel: Throttling Overview

# Throttled percentage
100 * (
  sum(rate(container_cpu_cfs_throttled_periods_total{namespace="$namespace"}[5m])) by (pod)
  /
  sum(rate(container_cpu_cfs_periods_total{namespace="$namespace"}[5m])) by (pod)
)

Alert Rule

# prometheus_rules.yml
groups:
- name: cpu_throttling
  rules:
  - alert: HighCPUThrottling
    expr: |
      sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod, namespace)
      /
      sum(rate(container_cpu_cfs_periods_total[5m])) by (pod, namespace)
      > 0.25
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High CPU throttling on {{ $labels.pod }}"
      description: "Pod {{ $labels.pod }} is being throttled >25% of the time"

Benchmark: Limit vs No Limit

Gotchas

1. GC Spikes

Java GC (G1) potrebuje CPU burst
- Minor GC: 10-50ms CPU spike
- S tight limitom: GC trvá dlhšie kvôli throttling
- → Dlhšie GC pauzy → Horšia latencia

2. JIT Compilation

JVM JIT compiler beží v backgrounde
- Potrebuje CPU pre kompiláciu
- S throttlingom: pomalšia kompilácia
- → Dlhšie warm-up → Horšia performance na štarte

3. Sidecar Kontajnery

# Istio sidecar tiež spotrebúva CPU limit!
containers:
- name: app
  resources:
    limits:
      cpu: "1000m"
- name: istio-proxy  # Automaticky pridaný
  resources:
    limits:
      cpu: "100m"   # Default, môže byť málo!

Checklist

## CPU Throttling Diagnostika

### Identifikácia
- [ ] Pridaj metriku container_cpu_cfs_throttled_seconds_total do dashboardu
- [ ] Koreluj throttling s p99 latenciou
- [ ] Skontroluj či GC spikes korelujú s throttlingom

### Riešenie
- [ ] Zvýš CPU limit na 2-4× request
- [ ] Alebo odstráň limit úplne (s monitoringom)
- [ ] Nastav explicitne GOMAXPROCS / ActiveProcessorCount

### Monitoring
- [ ] Alert na throttled_periods > 25%
- [ ] Dashboard s korelációu throttling vs latency
- [ ] Sleduj trends po zmene limitov

Záver

CPU throttling je skrytý zabijak latencie v Kubernetes:

1. 40% average CPU neznamená “OK” - bursts sú throttlované
2. Odstránenie CPU limitov môže paradoxne zlepšiť stabilitu
3. Monitoruj cfs_throttled_seconds_total - nie len CPU usage
4. Java/Go potrebujú explicit CPU count nastavenie

Pri bursty workloadoch (web servery, API) je tight CPU limit anti-pattern.

Súvisiace články

• K8s PostgreSQL Connection Storm - Connection management
• Go GOMAXPROCS v Kontajneroch - Go tuning