Envoy Outlier Detection Brownouts: When the Mesh Ejects Healthy Pods

Your app looks healthy. Pods are Ready. CPU is fine. Yet clients see bursts of 503s, and the errors don’t correlate with app logs.

If you’re running Istio (or any Envoy-based mesh), one common culprit is outlier detection: Envoy decides an endpoint is “bad” and ejects it from load balancing. With small endpoint pools, aggressive retry policies, or cross-zone constraints, that “self-healing” can become a brownout generator.

Tested on: Kubernetes 1.29–1.31, Istio 1.21–1.23, Envoy 1.29–1.31 (sidecar), HTTP/2 + gRPC workloads.

Incident narrative (anonymized)

A payments API ran with 6 pods behind an Istio Gateway. A new DestinationRule was applied as part of “hardening”:

• outlier detection set to eject on a single 5xx
• retries enabled (3 attempts)
• small connection pool limits

During a routine deploy, a subset of pods had 1–2 second cold-start stalls (JIT + cache warmup). Envoy interpreted the transient errors as “unhealthy”, ejected endpoints, and retries amplified load on the remaining pods. The system oscillated:

• eject → fewer endpoints → more load → more 5xx → more eject

Blast radius: 30–50% of traffic for ~12 minutes until rollback.

Constraint: we could not disable the mesh (mTLS + routing rules were required).

Timeline

• T-0: 503 rate spikes at the gateway; app dashboards mostly green.
• T+5m: Istio access logs show response flags UF and UH increasing.
• T+12m: Envoy stats show outlier_detection.ejections_active > 0 for the upstream cluster.
• T+20m: istioctl proxy-config cluster reveals aggressive outlier settings on the service cluster.
• T+30m: Quick mitigation: relax outlier thresholds + reduce retries.
• T+45m: Ejections stop oscillating; 503 rate returns to baseline.

Mechanism: how outlier detection creates brownouts

Outlier detection is per-proxy and local

Each Envoy sidecar (or gateway) keeps local error stats for upstream endpoints. When an endpoint crosses thresholds (e.g. consecutive 5xx), Envoy ejects it for a period (baseEjectionTime). Ejected endpoints are removed from load balancing decisions.

With many clients (many sidecars), you can get synchronized ejections even if the “real” issue is transient.

Small pools and locality constraints amplify ejections

If you have:

• only a few pods
• strict zone affinity / locality load balancing
• “prefer local zone” policies

Then ejecting even 1–2 pods can push the cluster into panic mode behavior or simply overload remaining pods. Retries then make the overload worse.

Retries turn a blip into a storm

A retry policy is a load multiplier:

• 1 request + 3 retries = up to 4 upstream attempts
• if upstream is already overloaded, retries increase error rate and trigger more ejections

This often produces oscillation: recover → retry storm → eject → partial recover → repeat.

Runbook: proving outlier detection is the root cause

What to check first

1. Look for Envoy response flags in access logs

At gateway or sidecar logs, check x-envoy-response-flags (or the equivalent field in your log format).

Common flags in these incidents:

• UF (upstream connection failure)
• UH (no healthy upstream)
• UO (upstream overflow)
• URX (upstream retry limit exceeded)
• UC (upstream connection termination)

Example snippet:

... "response_code":503, "response_flags":"UH", "upstream_cluster":"outbound|8080||payments.default.svc.cluster.local" ...

2. Confirm ejections via Envoy admin stats

Pick one affected proxy (gateway is easiest) and query stats:

kubectl -n istio-system exec deploy/istio-ingressgateway -c istio-proxy -- \
  curl -s localhost:15000/stats | grep -E 'outlier|eject|healthy|panic' | head

You want to find stats that indicate:

• active ejections
• total ejections
• panic mode / degraded health

How to confirm the hypothesis

A. Inspect xDS config (Istio view of Envoy config)

Find a pod sending traffic (or the gateway) and inspect cluster config:

istioctl proxy-config cluster -n default deploy/payments-api --fqdn payments.default.svc.cluster.local

Look for:

• outlier detection settings
• connection pool settings
• circuit breakers / max requests
• locality / load balancing policies

B. Identify which policy applied (DestinationRule / VirtualService)

Search the namespace for DestinationRules that target the host:

kubectl -n default get destinationrule -o yaml | grep -n "payments.default.svc.cluster.local" || true

Then open the matching DestinationRule and look at trafficPolicy.outlierDetection.

C. Correlate ejection time windows with errors

If baseEjectionTime is 30s and your error bursts show a ~30s periodicity, that’s a strong signal you’re in an ejection loop.

Safe mitigations

1. Relax outlier detection for small pools

• Increase consecutive5xx
• Increase interval
• Reduce baseEjectionTime
• Cap maxEjectionPercent below 50–60%

2. Reduce or temporarily disable retries

• Treat retries as a budget, not a default
• Prefer fast-fail with good timeouts over “retry until storm”

3. Scale endpoints before enabling ejection

• If you have 2–3 pods, ejection is extremely risky.

4. Fix readiness / startup behavior

• Don’t route traffic to pods while they are cold-starting.
• Ensure graceful shutdown (avoid transient 5xx during rollouts).

Risky mitigations

1. Disabling outlier detection everywhere

• You may hide a real unhealthy endpoint problem and shift failure modes elsewhere.

2. Increasing retries to “reduce errors”

• Often reduces errors briefly, then amplifies the next wave.

3. Aggressively shrinking connection pools

• Can trigger UO / overflow behavior under bursty traffic.

What we changed (concrete diffs)

We changed a DestinationRule that was far too aggressive.

Before:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments-api
spec:
  host: payments.default.svc.cluster.local
  trafficPolicy:
    outlierDetection:
      consecutive5xx: 1
      interval: 1s
      baseEjectionTime: 5m
      maxEjectionPercent: 100

After (safer defaults for a 6–10 pod service):

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments-api
spec:
  host: payments.default.svc.cluster.local
  trafficPolicy:
    outlierDetection:
      consecutive5xx: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

We also reduced retries in the VirtualService during rollouts:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments-api
spec:
  hosts:
  - payments.default.svc.cluster.local
  http:
  - retries:
      attempts: 1
      perTryTimeout: 200ms
      retryOn: "5xx,connect-failure,refused-stream"

How to verify (measurable)

1. Ejections stop oscillating

• Envoy stats show ejections_active stays at 0 most of the time (except real failures).

2. 503 flags change

• UH should disappear (or be extremely rare).
• If you still see errors, they should correlate with actual app saturation or real upstream failures.

3. Retry budget holds

• Watch for URX (retry limit exceeded) and overall upstream attempt rate.

4. Rollouts don’t trigger brownouts

• Use canary deploy, observe ejections and 503s.

Prevention / guardrails

Reliability contracts

• No outlier detection on services with < N endpoints (choose N like 5–10 based on your traffic).
• Retry budget contract: retries must be < X% of total upstream attempts.
• Outlier ejection budget: alert when ejections exceed a small threshold per minute.

Alerts worth adding

• Non-zero ejections active for critical clusters for > 5 minutes
• Spike in UH or UO flags at the gateway
• Increase in upstream attempt rate vs request rate (retry amplification)

Related reading

• Circuit Breaker Anti-Patterns: When Protection Causes Outages
• Circuit Breaker vs Rate Limiter vs Bulkhead: When to Use Which Pattern
• gRPC Deadline Propagation: Preventing Cascading Failures
• gRPC Keepalive Mismatch: Transport Closing After Idle
• Kubernetes Graceful Shutdown as a Contract: Zero 502s During Rollouts (HTTP + gRPC)
• HTTP Keep-Alive Connection Reset: Why Your Requests Fail with ‘Connection Reset by Peer’
• Kubernetes Cross-Zone Traffic: The Hidden Cost Eating Your Cloud Bill