Tail-Based Sampling in OpenTelemetry: Sizing, Memory Crashes and Cost Model

I learned tail sampling after a very real telemetry bill. “Tail sampling is great”—I heard this at a conference. The promise was compelling: instead of randomly dropping 80% of traces at the start (head sampling), you wait until the trace is complete, then intelligently decide what to keep. Keep all errors. Keep all slow traces. Sample the boring successful ones. It sounded like the perfect solution to our observability costs.

A month later, our OTel Collector was OOMkilling every 30 minutes. We had set decision_wait: 60s because some of our distributed traces took a while to complete. What we didn’t calculate was the memory implications: at 1,000 traces per second, a 60-second wait means 60,000 traces in memory simultaneously, each containing multiple spans with attributes. We were trying to hold gigabytes of trace data in a container with 512MB of RAM.

The problem with tail sampling documentation is that it explains HOW to enable the feature without explaining the sizing implications. Nowhere does it say: “If you set decision_wait to X and you have Y traces per second, you need Z gigabytes of memory.” Nowhere does it explain that when num_traces is exceeded, the collector starts dropping the OLDEST traces—which might include the error traces you wanted to keep.

This guide is the practical sizing knowledge I wish I’d had before deploying tail sampling in production.

Tested on: OTel Collector 0.96+, Kubernetes, Jaeger backend. Production on systems with 10k spans/s.

Why Tail Sampling

Head Sampling (Traditional)

Request → Sample decision → Trace
                 ↓
         80% dropped (random)
         20% kept

Problem: You drop 80% of traces BEFORE knowing if they’re interesting.

Tail Sampling

Request → Collect ALL spans → Wait for completion → Decision
                                                        ↓
                                            Keep: errors, slow, interesting
                                            Drop: fast, successful

Benefit: You see 100% of error traces, 100% of slow traces, sample healthy ones.

Basic Configuration

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      # Always keep errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep slow traces
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 500

      # Sample healthy traces
      - name: probabilistic-sample
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Sizing: Key Parameters

decision_wait

How long to wait for all spans before making a decision.

decision_wait: 10s  # Wait max 10s for complete trace

Trade-off:

• Too short → you don’t see all spans (distributed traces)
• Too long → high memory consumption

Recommendation: max_latency_P99 + 2-3s buffer

num_traces

Maximum number of traces in memory at once.

num_traces: 100000

Calculation:

num_traces = expected_new_traces_per_sec × decision_wait × safety_factor

Example:
1000 traces/s × 10s × 2 = 20,000 traces

What if exceeded? Collector starts dropping OLDEST traces (including error traces!).

Memory Estimation

memory_per_trace ≈ 10-50KB (depends on span count)
total_memory = num_traces × memory_per_trace

Example:
100,000 traces × 20KB = 2GB RAM for tail sampling buffer

Memory Sizing Formula

Required Memory (GB) =
  (traces_per_second × decision_wait_seconds × avg_spans_per_trace × bytes_per_span)
  / 1_000_000_000

Where:
- bytes_per_span ≈ 500-2000 (depends on attributes)
- safety_factor = 1.5-2x

Example

Input:
- 1000 traces/s
- decision_wait: 15s
- 10 spans/trace
- 1KB/span

Calculation:
1000 × 15 × 10 × 1000 = 150,000,000 bytes = 150MB

With safety factor 2x: 300MB for sampling buffer
+ base collector overhead: ~200MB
= Minimum 500MB, recommend 1GB

Production Deployment

Kubernetes Resources

# collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  template:
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector-contrib:0.96.0
        resources:
          requests:
            memory: 1Gi
            cpu: 500m
          limits:
            memory: 2Gi
            cpu: 1000m
        env:
        - name: GOMEMLIMIT
          value: "1800MiB"  # 90% of limit

Memory Limiter Processor

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1800
    spike_limit_mib: 400

Complete Pipeline

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1800
    spike_limit_mib: 400

  batch:
    timeout: 1s
    send_batch_size: 1024

  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 500
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: sample-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

exporters:
  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]

Cost Model

Traces Kept vs Dropped

Input: 1,000,000 spans/day

Without sampling:
- Storage: 1M × 1KB = 1GB/day
- 30 days retention: 30GB
- Cost @ $0.10/GB: $3/day = $90/month

With tail sampling (10% + all errors):
- Errors (1%): 10,000 spans
- Slow (5%): 50,000 spans
- Sampled (10% of rest): 94,000 spans
- Total: ~154,000 spans (15.4%)

- Storage: 154KB × 1KB = 154MB/day
- 30 days: 4.6GB
- Cost: $0.46/day = $14/month

Savings: 84%

Break-even Analysis

Tail Sampling Costs:
- Collector resources: ~$50/month (1 replica with 2GB RAM)
- Complexity: Engineering time

Savings:
- Storage: $76/month
- Query performance: Faster (less data)

ROI: Positive if >500k spans/day

Monitoring Tail Sampling

Prometheus Metrics

# Add prometheus exporter
exporters:
  prometheus:
    endpoint: 0.0.0.0:8888

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888

Key Metrics

# Sampling decisions
rate(otelcol_processor_tail_sampling_sampling_decision_latency_count[5m])

# Traces dropped due to num_traces limit
rate(otelcol_processor_tail_sampling_sampling_traces_dropped[5m])

# Memory usage
process_resident_memory_bytes{job="otel-collector"}

# Queue depth (if using batching)
otelcol_exporter_queue_size

Alerts

groups:
  - name: otel-collector
    rules:
      - alert: TailSamplingDropping
        expr: rate(otelcol_processor_tail_sampling_sampling_traces_dropped[5m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Tail sampling is dropping traces"
          description: "Increase num_traces or reduce decision_wait"

      - alert: CollectorHighMemory
        expr: process_resident_memory_bytes{job="otel-collector"} / 1e9 > 1.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OTel Collector memory > 1.5GB"

Common Pitfalls

1. Too Long decision_wait

# BAD: 60s wait = huge RAM consumption
decision_wait: 60s

# GOOD: 10-15s for most use cases
decision_wait: 10s

2. Too Small num_traces

# BAD: Drops traces during spike
num_traces: 1000

# GOOD: 2x expected load
num_traces: 50000

3. Composite Policy vs Multiple Policies

# BAD: Each policy evaluates independently
policies:
  - name: errors
    type: status_code
    status_codes: [ERROR]
  - name: slow
    type: latency
    threshold_ms: 500

# GOOD: Composite for AND/OR logic
policies:
  - name: composite-policy
    type: composite
    composite:
      max_total_spans_per_second: 1000
      policy_order: [errors, slow-errors, sample]
      composite_sub_policy:
        - name: errors
          type: status_code
          status_codes: [ERROR]
        - name: slow-errors
          type: and
          and:
            - name: slow
              type: latency
              latency:
                threshold_ms: 500
            - name: probabilistic
              type: probabilistic
              probabilistic:
                sampling_percentage: 50

Conclusion

Tail sampling is a powerful tool, but requires proper sizing. Key points:

1. decision_wait = P99 latency + buffer (10-15s typically)
2. num_traces = traces/s × decision_wait × 2
3. Memory = num_traces × 20KB (minimum estimate)
4. Monitoring = watch dropped traces and memory
5. Cost = ROI positive from 500k+ spans/day

FAQ

What if I have very long traces (minutes)?

Increase decision_wait, but prepare for higher memory. Alternative: split trace into smaller segments.

Can I scale horizontally?

Not directly. Tail sampling needs all spans of one trace on one collector. Use load balancing with trace ID affinity.

What if collector crashes?

You lose in-flight traces. Use persistent queue (file storage) for recovery.

Related Articles

• K8s Connection Storm - Monitoring pod rollouts
• CI/CD for Monorepo - Integrating OTel into pipeline