Envoy/Istio 503 UF/UO/UT: When the Mesh, Not the App, Is Your Outage

The symptom pattern usually shows up as “the app is flaky” until you look at the right logs:

• clients see bursts of HTTP 503 (sometimes 504)
• pods are Ready, CPU is not crazy
• the same request works on retry
• the response often has server: envoy and x-envoy-response-flags

The key mental shift: in a service mesh, Envoy is a first-class production component. It has its own connection pools, queue limits, retries, and outlier ejection. If those are mis-sized (or retries amplify load), Envoy can become the bottleneck even when your app is fine.

Tested on: Kubernetes 1.29–1.31, Istio 1.21–1.24, Envoy 1.28–1.30, HTTP/1.1 + HTTP/2, sidecars + ingress gateways.

Incident narrative (anonymized)

We had a service that was stable for months. Then we shipped a “small” Istio policy change:

• a DestinationRule with conservative connection pool limits (copy-pasted)
• retries left enabled with a generous per-try timeout
• outlier detection turned on with aggressive ejection

At peak traffic:

• Envoy started returning 503s
• app metrics looked normal (it wasn’t receiving many requests anymore)
• retries amplified the load on the mesh layer
• the outage looked “random” because it depended on per-pod proxy load

Constraint: we couldn’t rip out the mesh during an incident. We needed a safe rollback/patch that reduces mesh pressure immediately.

Timeline

• T-0: 503s spike; p99 latency rises.
• T+10m: I confirm responses include Envoy flags (x-envoy-response-flags).
• T+20m: Envoy admin stats show pending overflow and upstream connection overflow increasing.
• T+30m: I find the DestinationRule with too-low connection pool limits and retries that amplify.
• T+40m: Mitigation: reduce retries + increase connection pool budget for that one service.
• T+60m: 503s stop; p99 stabilizes.
• T+1d: We add “mesh contracts” in CI: forbid tiny pool limits on critical services and cap retries.

Mechanism: what the Envoy flags actually mean

Envoy gives you clues in access logs and headers. These flags matter because they map directly to one class of root causes.

A quick decoder (most common in incidents):

You’ll often see more than one flag in a cascade. My workflow is: treat flags as a hypothesis generator, then validate with proxy stats.

Runbook: diagnose and fix mesh-generated 503s

What to check first

1) Prove the 503 is coming from Envoy

From a client or debug pod, check response headers:

curl -s -D- http://<service>.<ns>.svc.cluster.local/health -o /dev/null | sed -n '1,20p'

Look for:

• server: envoy
• x-envoy-response-flags: ...

If you have access logs, extract the flags (format varies). In Istio, you can often see them via proxy logs.

2) Pull Envoy stats from the sidecar

The fastest “is Envoy overloaded?” answer is in admin stats on port 15000.

kubectl -n <ns> exec -it <pod> -c istio-proxy -- sh -lc \
  "curl -s http://127.0.0.1:15000/stats | egrep 'upstream_rq_pending_overflow|upstream_cx_overflow|upstream_rq_timeout|no_healthy_upstream' | head -n 50"

If upstream_rq_pending_overflow or upstream_cx_overflow climbs during the incident, you’re not debugging the app — you’re debugging the proxy budget.

3) Find the relevant Istio config (DestinationRule / VirtualService)

This is where most root causes live.

kubectl -n <ns> get destinationrule,virtualservice
kubectl -n <ns> get destinationrule <dr> -o yaml
kubectl -n <ns> get virtualservice <vs> -o yaml

If you use istioctl, it’s even faster to see the effective config:

istioctl proxy-config clusters <pod> -n <ns> | head -n 40
istioctl proxy-config routes <pod> -n <ns> | head -n 40

How to confirm the hypothesis

Pick one dominating flag, then confirm:

• UO → look for overflow stats + tiny connection pool limits in DestinationRule
• UT → check per-try timeout vs backend p99 + queue depth
• UF → check upstream health and TLS/mTLS compatibility
• NR → check routes/hosts

A very common confirmation loop is:

• reduce retries → overflow disappears → 503s drop
• then resize pools properly (instead of living with “retries off” forever)

Safe mitigations (during incident)

1) Reduce retry amplification

Retries are multiplicative. If one client request becomes 3 upstream attempts, you just tripled load on the proxy + backend.

A safe emergency change is to cap retries and keep per-try timeout small:

retries:
  attempts: 1
  perTryTimeout: 1s
  retryOn: gateway-error,connect-failure,refused-stream
timeout: 3s

This reduces retry storms while preserving some resiliency.

2) Increase connection pool budget for the hot service

If you see overflow counters, increase the limits for that one upstream cluster.

3) Relax outlier detection temporarily

If outlier detection is ejecting endpoints aggressively, you can create a feedback loop:

• a few 5xx → endpoints ejected → fewer endpoints → more overload → more 5xx

During an incident, reduce ejection sensitivity or disable temporarily for that service.

Risky mitigations

• Setting “infinite” pool limits everywhere
  • you can shift the bottleneck to the node/kernel and create larger outages
• Disabling mTLS to “fix UF”
  • you might create a security incident and still not fix the root cause
• Removing sidecars wholesale
  • huge blast radius, messy rollback

What we changed (concrete)

1) We sized connection pools like a budget

The mistake was a tiny pool limit copy-paste. We replaced it with a budget derived from:

• expected concurrency
• upstream latency
• headroom for bursts

Example DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments
spec:
  host: payments.<ns>.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 500
      http:
        http1MaxPendingRequests: 2000
        maxRequestsPerConnection: 1000

2) We capped retries and made timeouts intentional

We tightened VirtualService retries/timeouts so we don’t create retry storms:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments
spec:
  hosts:
  - payments.<ns>.svc.cluster.local
  http:
  - timeout: 3s
    retries:
      attempts: 1
      perTryTimeout: 1s
      retryOn: gateway-error,connect-failure,refused-stream
    route:
    - destination:
        host: payments.<ns>.svc.cluster.local

3) We added a “mesh contract” CI check

We lint Istio configs:

• forbid tiny maxConnections for critical services without a justification label
• cap retries and require explicit timeouts
• disallow aggressive outlier detection defaults in shared templates

This prevented the same outage class from reappearing.

How to verify (measurable)

1) Envoy overflow counters stop increasing

Re-check sidecar stats:

kubectl -n <ns> exec -it <pod> -c istio-proxy -- sh -lc \
  "curl -s http://127.0.0.1:15000/stats | egrep 'upstream_rq_pending_overflow|upstream_cx_overflow' | head -n 20"

Expected: counters stop climbing quickly after config is applied.

2) x-envoy-response-flags disappear (or change to real backend failures)

If errors remain, they should be backend errors, not proxy overflow.

3) p99 stabilizes without retry hiding

Watch p99 plus request rate. If p99 improves but traffic drops, you only moved the bottleneck.

Prevention / guardrails

• Budgets
  • per-service connection pool budgets and retry budgets
• Invariants
  • “critical service must survive 1 endpoint loss without UO”
• Alerts
  • overflow counters, no_healthy_upstream, proxy queue depth
• Change safety
  • any template change in DestinationRule/VirtualService is a production change (treat it like code)

Related reading

• gRPC Keepalive Mismatch: Transport Closing After Idle
• gRPC Deadline Propagation: Preventing Cascading Failures
• Circuit Breaker Anti-Patterns: When Protection Causes Outages
• Circuit Breaker vs Rate Limiter vs Bulkhead: When to Use Which Pattern
• Kubernetes Graceful Shutdown as a Contract: Zero 502s During Rollouts (HTTP + gRPC)
• PMTU Blackholes: When Only Large Responses Hang
• Adaptive Concurrency Limits: Stop Guessing Thread Pool Sizes