JVM Metaspace OOM in Kubernetes: Why MaxMetaspaceSize Alone Won't Save You

The only OOM that kept killing us was metaspace, not heap. “Set -XX:MaxMetaspaceSize=256m but pod still OOMKilled.” The cause: Metaspace is off-heap memory that grows with loaded classes, your container limit doesn’t have enough headroom, and the JVM isn’t unloading unused classes.

Environment: JVM 11+, Kubernetes with memory limits, Spring Boot/microservices, dynamic class loading (reflection, proxies, Groovy)

The Problem

The Mysterious OOM

Pod lifecycle:

T+0:00   Pod starts, memory usage: 400MB
         Heap: 256MB, Metaspace: 80MB, Other: 64MB

T+1:00   Traffic increases, more code paths executed
         Metaspace: 120MB (loading more classes)

T+4:00   Metaspace: 200MB (proxies, reflection, lambdas)

T+8:00   Metaspace: 280MB → MaxMetaspaceSize hit!
         java.lang.OutOfMemoryError: Metaspace

         Or worse: Container limit hit first
         OOMKilled (exit code 137) - no Java error!

The Container Math Problem

# Your Kubernetes config
resources:
  limits:
    memory: "512Mi"
  requests:
    memory: "512Mi"

# Your JVM flags
JAVA_OPTS: "-Xmx256m -XX:MaxMetaspaceSize=256m"

# What you think:
# Heap (256MB) + Metaspace (256MB) = 512MB ✓

# What actually happens:
# Heap: 256MB
# Metaspace: up to 256MB
# Code cache: 48MB (default)
# Thread stacks: 1MB × N threads
# Direct buffers: varies
# JNI/native: varies
# GC overhead: varies
# Total: 256 + 256 + 48 + 50 + X = 610MB+ → OOMKilled!

Root Cause

What Goes Into Metaspace

Metaspace contents:

┌─────────────────────────────────────────────────────────────┐
│ Metaspace                                                   │
│ ├─ Class metadata (Klass structures)                       │
│ ├─ Method metadata (bytecode, JIT compiled code refs)      │
│ ├─ Constant pools                                           │
│ ├─ Annotations                                              │
│ ├─ Method counters and profiling data                       │
│ └─ Static variables (moved from PermGen in Java 8)         │
│                                                             │
│ What makes it GROW:                                         │
│ - Spring proxies (CGLib/ByteBuddy)                         │
│ - Hibernate entity proxies                                  │
│ - Lambda expressions (each is a class!)                     │
│ - Reflection-heavy frameworks                               │
│ - Groovy/scripting engines                                  │
│ - Dynamic class generation                                  │
│ - ClassLoader leaks                                         │
└─────────────────────────────────────────────────────────────┘

Why Classes Don’t Unload

// Classes can only be unloaded when their ClassLoader is unreachable
// This rarely happens in application servers

// Spring Boot: Single application ClassLoader
// → Classes loaded once, never unloaded

// Common ClassLoader leak pattern:
public class CacheWithClassRef {
    // This cache holds Class references
    private static Map<String, Class<?>> classCache = new HashMap<>();

    // Classes in cache can NEVER be unloaded
    // Even if you remove entries, if anything references the Class...
}

// ThreadLocal holding class instance:
private static ThreadLocal<MyService> service = new ThreadLocal<>();
// If thread pool reuses threads, classes stay loaded

Diagnosis

Check Metaspace Usage

# Get Metaspace metrics from JVM
kubectl exec pod-name -- jcmd 1 VM.metaspace

# Output:
# Total: reserved=280MB, committed=260MB, used=245MB
# Class: reserved=256MB, committed=240MB, used=230MB
# ├─ Classloader: 1234 classes, 180MB used
# ├─ Classloader: 567 classes, 45MB used
# └─ ...

# Count loaded classes
kubectl exec pod-name -- jcmd 1 VM.classloaders

# Or via JMX
kubectl exec pod-name -- jcmd 1 GC.class_stats

# List all loaded classes with sizes
kubectl exec pod-name -- jcmd 1 VM.class_hierarchy

Monitor Class Loading Over Time

// Add JVM flags for class loading visibility
-XX:+TraceClassLoading
-XX:+TraceClassUnloading
-Xlog:class+load=info
-Xlog:class+unload=info

// In logs you'll see:
// [class,load] com.example.Proxy$123 source: __JVM_DefineClass__
// Many dynamically generated classes = Metaspace growth

Find ClassLoader Leaks

# Heap dump and analyze ClassLoaders
kubectl exec pod-name -- jcmd 1 GC.heap_dump /tmp/heap.hprof
kubectl cp pod-name:/tmp/heap.hprof ./heap.hprof

# In Eclipse MAT or VisualVM:
# 1. Find all ClassLoader instances
# 2. Check retained size of each
# 3. Look for duplicate classes loaded by different loaders

The Fix

Option 1: Right-Size Container Memory

# Account for ALL JVM memory regions
resources:
  limits:
    memory: "768Mi"  # Give more headroom
  requests:
    memory: "768Mi"

# JVM flags with explicit limits
env:
  - name: JAVA_OPTS
    value: >-
      -Xmx256m
      -XX:MaxMetaspaceSize=128m
      -XX:ReservedCodeCacheSize=64m
      -XX:MaxDirectMemorySize=64m
      -XX:+UseContainerSupport
      -XX:MaxRAMPercentage=50.0

# Memory budget:
# Heap: 256MB (Xmx)
# Metaspace: 128MB (MaxMetaspaceSize)
# Code cache: 64MB (ReservedCodeCacheSize)
# Direct memory: 64MB (MaxDirectMemorySize)
# Thread stacks: ~100MB (100 threads × 1MB)
# JVM overhead: ~50MB
# Total: ~662MB with buffer to 768MB

Option 2: Enable Class Unloading

# JVM flags to encourage class unloading
-XX:+CMSClassUnloadingEnabled  # For CMS (deprecated)
-XX:+ClassUnloading            # For G1/ZGC (default on)
-XX:+ClassUnloadingWithConcurrentMark  # For G1

# More aggressive Metaspace GC
-XX:MinMetaspaceFreeRatio=10   # Default 40
-XX:MaxMetaspaceFreeRatio=30   # Default 70

# Force Metaspace to shrink after unloading
-XX:+ShrinkHeapInSteps

Option 3: Reduce Dynamic Class Generation

// Spring: Prefer interfaces over classes for proxying
@Service
public class MyServiceImpl implements MyService {
    // Interface proxy (JDK dynamic proxy) vs
    // Class proxy (CGLib - generates more classes)
}

// Spring Boot config
spring.aop.proxy-target-class=false  # Use JDK proxies

// Hibernate: Reduce proxy generation
spring.jpa.properties.hibernate.bytecode.use_reflection_optimizer=false
spring.jpa.properties.hibernate.bytecode.provider=none

// Lambda optimization - reuse instead of inline
// BAD: Each lambda is a new class
list.forEach(item -> process(item));

// BETTER: Method reference (may share class)
list.forEach(this::process);

// BEST for hot paths: Reuse functional interface instance
private final Consumer<Item> processor = this::process;
list.forEach(processor);

Option 4: ClassLoader Isolation

// For plugin systems or scripting engines
// Use separate ClassLoaders that can be garbage collected

public class PluginLoader {
    public void loadAndExecute(String pluginJar) {
        // URLClassLoader can be closed and GC'd
        try (URLClassLoader loader = new URLClassLoader(
                new URL[]{new URL("file:" + pluginJar)},
                null  // No parent - isolates classes
        )) {
            Class<?> plugin = loader.loadClass("com.plugin.Main");
            plugin.getMethod("run").invoke(plugin.getDeclaredConstructor().newInstance());
        }
        // After try block, loader is closed
        // Classes can be unloaded on next GC
    }
}

Option 5: Native Memory Tracking

# Enable NMT to see all memory regions
-XX:NativeMemoryTracking=summary  # or detail

# Get memory breakdown
kubectl exec pod-name -- jcmd 1 VM.native_memory summary

# Output shows:
# Total: reserved=1500MB, committed=800MB
# - Java Heap: 256MB
# - Class (Metaspace): 180MB
# - Thread: 120MB (120 threads)
# - Code: 50MB
# - GC: 30MB
# - Internal: 20MB
# - Symbol: 15MB
# ...

Monitoring

groups:
  - name: jvm-metaspace
    rules:
      - alert: MetaspaceHigh
        expr: |
          jvm_memory_used_bytes{area="nonheap", id="Metaspace"} /
          jvm_memory_max_bytes{area="nonheap", id="Metaspace"} > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Metaspace usage above 85%"

      - alert: ClassLoadingRate
        expr: |
          rate(jvm_classes_loaded_classes_total[5m]) > 100
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "High class loading rate - possible leak"

      - alert: ContainerOOMRisk
        expr: |
          container_memory_working_set_bytes /
          container_spec_memory_limit_bytes > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Container near memory limit - OOMKill risk"

Checklist

## JVM Metaspace OOM Kubernetes

### Diagnosis
- [ ] Check Metaspace usage: jcmd 1 VM.metaspace
- [ ] Count loaded classes: jcmd 1 VM.classloaders
- [ ] Enable NMT: -XX:NativeMemoryTracking=summary
- [ ] Look for class loading trends over time

### Container Sizing
- [ ] Calculate total JVM memory (heap + metaspace + codecache + threads + overhead)
- [ ] Set container limit 20-30% above JVM total
- [ ] Explicitly set MaxMetaspaceSize
- [ ] Explicitly set ReservedCodeCacheSize

### Reducing Metaspace
- [ ] Prefer JDK proxies over CGLib
- [ ] Reuse lambda instances in hot paths
- [ ] Check for ClassLoader leaks in heap dump
- [ ] Close/dispose ClassLoaders when done

Conclusion

The lesson: MaxMetaspaceSize limits Metaspace but doesn’t prevent container OOM. You need to account for ALL JVM memory regions in your container limit, and actively manage class loading.

Key principles:

1. Container limit > Heap + Metaspace + CodeCache + Threads + Overhead
2. Classes rarely unload - once loaded, they stay
3. Dynamic proxies and lambdas create classes - each one costs Metaspace
4. Use NMT to understand actual memory usage

Related Articles

• Java Native Memory OOMKilled - Off-heap memory issues
• Kubernetes OOM Killer Memory Limits - Container memory management