JVM Native Memory v Kubernetes: Prečo Pod Dostane OOMKilled s 50% Heap

Stale sme ladili heap a stale sme dostavali OOMKilled. “Pod OOMKilled pri 2GB memory limite, ale heap mal len 1GB a 50% využitie.” Kam zmizlo ďalších 1GB?

JVM používa viac ako len heap. Native memory zahŕňa Metaspace, thread stacky, NIO buffery, JIT kompilovaný kód a viac. V kontajneroch toto často spôsobuje OOMKilled.

Testované na: Java 21, Kubernetes 1.28, Spring Boot 3.2, container limit 2GB

Anatómia JVM Memory

Total Memory = Heap + Non-Heap

Container Memory (2GB limit)
├── Heap (-Xmx)                    ~1GB
│   ├── Young Generation
│   ├── Old Generation
│   └── [Kontrolované -Xmx]
│
├── Metaspace                      ~100-300MB
│   ├── Class metadata
│   ├── Method metadata
│   └── [Kontrolované -XX:MaxMetaspaceSize]
│
├── Thread Stacky                  ~200-500MB
│   ├── Každý thread: ~1MB default
│   ├── 200 threads = 200MB
│   └── [Kontrolované -Xss]
│
├── Code Cache                     ~50-240MB
│   ├── JIT kompilovaný kód
│   └── [Kontrolované -XX:ReservedCodeCacheSize]
│
├── Direct Buffers (NIO)           ~variabilné
│   ├── Off-heap pre I/O
│   └── [Kontrolované -XX:MaxDirectMemorySize]
│
├── Native Libraries               ~variabilné
│   ├── JNI alokácie
│   └── Native kód
│
└── Ostatné                        ~50-100MB
    ├── GC štruktúry
    ├── Symbol tables
    └── Interné štruktúry

Diagnostika Problému

Povoľ Native Memory Tracking

# Spusti JVM s NMT
java -XX:NativeMemoryTracking=summary \
     -jar app.jar

# Skontroluj rozklad memory
jcmd <pid> VM.native_memory summary

# Alebo detailný rozklad
jcmd <pid> VM.native_memory detail

NMT Výstup Príklad

Native Memory Tracking:

Total: reserved=3145728KB, committed=2097152KB  ← Committed = skutočne použité

-                 Java Heap (reserved=1048576KB, committed=1048576KB)
                            (mmap: reserved=1048576KB, committed=1048576KB)

-                     Class (reserved=312456KB, committed=289345KB)
                            (classes #25678)
                            (  instance classes #24567, array classes #1111)

-                    Thread (reserved=215678KB, committed=215678KB)
                            (thread #210)
                            (stack: reserved=214567KB, committed=214567KB)

-                      Code (reserved=253456KB, committed=178934KB)
                            (mmap: reserved=253456KB, committed=178934KB)

-                        GC (reserved=89567KB, committed=89567KB)

-                  Internal (reserved=56789KB, committed=56789KB)

-                    Symbol (reserved=23456KB, committed=23456KB)

Problém: Matematika Nesedí

Container limit:    2GB
Heap (-Xmx):        1GB
Očakávaný overhead: ~500MB
Celkom očakávané:   ~1.5GB

Skutočné využitie:
  Heap:             1GB
  Metaspace:        280MB (25k tried)
  Threads:          210MB (210 threadov)
  Code Cache:       180MB
  Direct Buffers:   256MB (Netty)
  Ostatné:          150MB
  Celkom:           2.08GB → OOMKilled!

Riešenia

1. Vypočítaj Container Memory Správne

# Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
spec:
  containers:
  - name: app
    resources:
      limits:
        memory: "2Gi"
    env:
    - name: JAVA_OPTS
      value: >-
        -Xmx1g
        -Xms1g
        -XX:MaxMetaspaceSize=256m
        -XX:MaxDirectMemorySize=256m
        -Xss512k
        -XX:ReservedCodeCacheSize=128m

Výpočet Memory Rozpočtu

Container Limit: 2GB

Alokácie:
  Heap (-Xmx):                    1000MB
  Metaspace:                       256MB
  Thread stacky (200 × 512KB):     100MB
  Code Cache:                      128MB
  Direct Memory:                   256MB
  GC + Internal:                   150MB
  Safety buffer:                   110MB
                                 -------
  Celkom:                         2000MB ✓

2. Použi Container-Aware JVM Nastavenia

# Java 17+ - automatická detekcia kontajnera
env:
- name: JAVA_OPTS
  value: >-
    -XX:MaxRAMPercentage=75.0
    -XX:InitialRAMPercentage=75.0
    -XX:MaxMetaspaceSize=256m
    -Xss512k

# Toto nastaví Xmx na 75% container limitu automaticky
# Nechá 25% pre non-heap

3. Limituj Počet Threadov

// Spring Boot - limituj Tomcat thready
// application.yml
server:
  tomcat:
    threads:
      max: 100  # Znížené z 200 default
      min-spare: 10

// Alebo pre async workery
spring:
  task:
    execution:
      pool:
        max-size: 50

4. Kontroluj Direct Memory

// Skontroluj použitie direct memory
import java.lang.management.ManagementFactory;
import java.lang.management.BufferPoolMXBean;

List<BufferPoolMXBean> pools = ManagementFactory.getPlatformMXBeans(BufferPoolMXBean.class);
for (BufferPoolMXBean pool : pools) {
    System.out.println(pool.getName() + ": " + pool.getMemoryUsed() / 1024 / 1024 + "MB");
}
// Output: direct: 256MB

# Limituj direct memory
-XX:MaxDirectMemorySize=256m

# Netty špecifické
-Dio.netty.maxDirectMemory=0  # Použi Xmx pre limit

Monitoring Native Memory

Prometheus Metriky

// Micrometer metriky pre non-heap
@Bean
MeterBinder jvmNativeMemory() {
    return registry -> {
        Gauge.builder("jvm.memory.native.used", () -> {
            // Z NMT ak je povolené
            // Alebo odhad z /proc/self/status
            return parseVmRSS();
        }).register(registry);
    };
}

private long parseVmRSS() {
    try {
        String status = Files.readString(Path.of("/proc/self/status"));
        // Parsuj VmRSS riadok
        return extractVmRSS(status);
    } catch (IOException e) {
        return 0;
    }
}

Container Memory vs JVM Memory

# Container memory (z cAdvisor)
container_memory_usage_bytes{pod="myapp-xyz"}

# JVM heap použitý
jvm_memory_used_bytes{area="heap"}

# Non-heap = container - heap
container_memory_usage_bytes - on(pod) jvm_memory_used_bytes{area="heap"}

Grafana Dashboard

{
  "panels": [
    {
      "title": "Memory Breakdown",
      "targets": [
        {"expr": "jvm_memory_used_bytes{area='heap'}", "legendFormat": "Heap"},
        {"expr": "jvm_memory_used_bytes{area='nonheap'}", "legendFormat": "Non-Heap (čiastočné)"},
        {"expr": "container_memory_usage_bytes", "legendFormat": "Container Celkom"}
      ]
    }
  ]
}

Bežné Memory Leaky

1. Class Loader Leaky (Metaspace)

// Problém: Dynamická generácia tried
// Groovy skripty, JAXB, reflection proxies

// Detekcia
jcmd <pid> VM.native_memory detail | grep -A5 "Class"

// Fix: Limituj generáciu tried, reuse classloaderov
-XX:MaxMetaspaceSize=256m  // Tvrdý limit

2. Thread Leaky

// Problém: Thready nikdy neukončené
ExecutorService executor = Executors.newCachedThreadPool();
// Thready rastú neobmedzene

// Detekcia
jcmd <pid> Thread.print | grep -c "java.lang.Thread"

// Fix: Použi bounded pools
ExecutorService executor = Executors.newFixedThreadPool(50);

3. Direct Buffer Leaky

// Problém: NIO buffery neuvoľnené
ByteBuffer buffer = ByteBuffer.allocateDirect(1024 * 1024);
// Nie explicitne uvoľnené, čaká na GC

// Detekcia
jcmd <pid> VM.native_memory summary | grep -A2 "Internal"

// Fix: Manuálne cleanup alebo System.gc() hint
((DirectBuffer) buffer).cleaner().clean();

4. JNI Memory Leaky

// Problém: Native kód alokuje, nikdy neuvoľňuje
// Bežné s image processing knižnicami

// Detekcia
// NMT ukazuje "Other" rastúce

// Fix: Skontroluj dokumentáciu native knižnice
// Použi try-with-resources pre native resources

Produkčná Konfigurácia

Odporúčané JVM Flagy

# Pre 2GB kontajner
java \
  # Heap: 50% kontajnera
  -Xmx1g \
  -Xms1g \
  \
  # Metaspace: ohraničené
  -XX:MaxMetaspaceSize=256m \
  \
  # Threads: menšie stacky
  -Xss512k \
  \
  # Code Cache: ohraničené
  -XX:ReservedCodeCacheSize=128m \
  \
  # Direct Memory: ohraničené
  -XX:MaxDirectMemorySize=256m \
  \
  # Container awareness
  -XX:+UseContainerSupport \
  \
  # NMT pre debugging (mierny overhead)
  -XX:NativeMemoryTracking=summary \
  \
  # GC logging
  -Xlog:gc*:file=/logs/gc.log:time \
  \
  -jar app.jar

Alert Pravidlá

groups:
- name: jvm_memory
  rules:
  - alert: ContainerMemoryNearLimit
    expr: |
      container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85
    for: 5m
    annotations:
      summary: "Container {{ $labels.pod }} na >85% memory"

  - alert: NonHeapMemoryHigh
    expr: |
      (container_memory_usage_bytes - jvm_memory_used_bytes{area="heap"})
      / container_spec_memory_limit_bytes > 0.4
    for: 10m
    annotations:
      summary: "Non-heap memory >40% container limitu"

Checklist

## JVM Container Memory Sizing

### Výpočet
- [ ] Urči container memory limit
- [ ] Nastav heap na 50-60% limitu
- [ ] Rozpočet Metaspace (skontroluj počet tried)
- [ ] Rozpočet threads (počet × stack size)
- [ ] Rozpočet direct memory (Netty, NIO)
- [ ] Nechaj 10-15% safety buffer

### JVM Flagy
- [ ] Nastav -Xmx a -Xms explicitne
- [ ] Nastav -XX:MaxMetaspaceSize
- [ ] Nastav -Xss (zníž z 1MB default)
- [ ] Nastav -XX:MaxDirectMemorySize
- [ ] Povoľ -XX:+UseContainerSupport

### Monitoring
- [ ] Sleduj container_memory_usage_bytes
- [ ] Sleduj jvm_memory_used_bytes (heap + non-heap)
- [ ] Povoľ NMT v staging pre debugging
- [ ] Alert na >85% container memory

### Testovanie
- [ ] Load test s produkčnou záťažou
- [ ] Monitoruj memory 24+ hodín
- [ ] Over že nie je OOMKilled pod stresom

Záver

JVM v kontajneroch potrebuje explicitný memory management:

1. Heap nie je celková memory - Non-heap môže byť 500MB+
2. Rozpočet všetkých komponentov - Metaspace, threads, direct buffers
3. Nastav limity explicitne - Nespoliehaj sa na defaults
4. Monitoruj container memory - Nie len JVM heap

Container limit - 25% buffer = Maximálny bezpečný -Xmx.

Súvisiace články

• Go GOMAXPROCS v Kontajneroch - Container runtime tuning
• K8s CPU Throttling Pitva - Resource limity