Circuit Breaker Anti-Patterns: When Protection Causes Outages

We once made an outage worse by adding a circuit breaker in the wrong place. Circuit breakers are supposed to prevent outages. They can also create them—sometimes with impressive efficiency—when the configuration is slightly wrong.

I’ve seen a payment incident where one buggy endpoint (getBalance) failed constantly, tripped a shared circuit breaker, and suddenly charge() and refund() were dead too. The circuit breaker did exactly what we told it to do… which was the problem.

In this post I’ll walk through five circuit breaker anti-patterns I keep seeing in production—and the fixes that make circuit breakers boring again: shared breakers, hypersensitive thresholds, half-open stampedes, missing fallbacks, and testing gaps.

Tested with: Resilience4j 2.1, Hystrix (legacy), Go kit circuit breaker

Anti-Pattern 1: Shared Circuit Breaker

The most common anti-pattern is using a single circuit breaker for an entire service. It seems efficient—one breaker, one configuration, less code. But it creates a failure coupling that didn’t exist before. If any endpoint becomes unhealthy, all endpoints become unavailable.

The Problem

// WRONG: Single breaker for all endpoints
@Bean
public CircuitBreaker paymentCircuitBreaker() {
    return CircuitBreaker.of("payment-service", config);
}

// Used for all payment operations
paymentClient.charge(amount);      // Uses same breaker
paymentClient.refund(txId);        // Uses same breaker
paymentClient.getBalance(userId);  // Uses same breaker

// What happens:
// 1. getBalance endpoint has bug, fails 100%
// 2. Shared breaker opens
// 3. charge() and refund() stop working too!
// 4. All payment operations fail due to healthy endpoints being blocked

The Fix

// CORRECT: Separate breakers per operation/endpoint
@Bean
public CircuitBreakerRegistry circuitBreakerRegistry() {
    return CircuitBreakerRegistry.of(defaultConfig);
}

public void charge(BigDecimal amount) {
    CircuitBreaker breaker = registry.circuitBreaker("payment-charge");
    breaker.executeSupplier(() -> paymentClient.charge(amount));
}

public void refund(String txId) {
    CircuitBreaker breaker = registry.circuitBreaker("payment-refund");
    breaker.executeSupplier(() -> paymentClient.refund(txId));
}

// Now: getBalance failures don't affect charge/refund

Anti-Pattern 2: Threshold Too Low

This anti-pattern comes from good intentions—you want to protect your system from failures, so you make the circuit breaker very sensitive. But in distributed systems, transient failures are normal. Network blips happen. Services restart. A single timeout doesn’t mean the service is down. Setting thresholds too low means your circuit breaker opens on normal operational noise, creating artificial outages.

The Problem

// WRONG: Opens on any failure
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(1)        // 1% failure opens breaker
    .slidingWindowSize(10)          // Only 10 requests in window
    .build();

// Reality:
// 1 failure in 10 requests = 10% failure rate
// 10 requests at 1000 RPS = 10ms of data
// Single network blip → breaker opens → 30s outage

The Fix

// CORRECT: Reasonable thresholds
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)             // 50% failures to open
    .slowCallRateThreshold(80)            // 80% slow calls to open
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .slidingWindowType(SlidingWindowType.TIME_BASED)
    .slidingWindowSize(30)                // 30 second window
    .minimumNumberOfCalls(20)             // Need 20+ calls to evaluate
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .permittedNumberOfCallsInHalfOpenState(5)
    .build();

// Now: Needs significant failure pattern to trigger
// Quick recovery with half-open testing

Anti-Pattern 3: No Fallback Strategy

A circuit breaker without a fallback is like a fire alarm without an evacuation plan. Yes, you’ve detected the problem. Now what? If your only response to an open circuit is to throw an error to the user, you’ve just converted a degraded dependency into a complete failure of your service. The whole point of the circuit breaker is to allow graceful degradation, but degradation requires you to define what “degraded” looks like.

The Problem

// WRONG: Breaker opens, request fails
public Order getOrder(String orderId) {
    return circuitBreaker.executeSupplier(
        () -> orderService.getOrder(orderId)
    );
    // When breaker is open: CallNotPermittedException thrown
    // User sees: 500 Internal Server Error
}

The Fix

// CORRECT: Meaningful fallbacks
public Order getOrder(String orderId) {
    return circuitBreaker.executeSupplier(
        () -> orderService.getOrder(orderId),
        throwable -> getFallbackOrder(orderId, throwable)
    );
}

private Order getFallbackOrder(String orderId, Throwable t) {
    if (t instanceof CallNotPermittedException) {
        // Breaker is open - return cached data
        return orderCache.get(orderId);
    }
    if (t instanceof TimeoutException) {
        // Timeout - return partial data
        return Order.builder()
            .id(orderId)
            .status(OrderStatus.UNKNOWN)
            .message("Order details temporarily unavailable")
            .build();
    }
    throw new ServiceUnavailableException("Order service down", t);
}

Anti-Pattern 4: Testing in Half-Open is Too Aggressive

The half-open state is when your circuit breaker tests whether the service has recovered. The intuition is “let’s send some requests and see if they work.” But if you send too many test requests, you can prevent the service from recovering. A service that’s struggling under load doesn’t need 100 test requests—it needs a gentle probe. This is especially dangerous with services that have cold-start behavior or connection pool limits.

The Problem

// WRONG: Half-open immediately sends traffic
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .waitDurationInOpenState(Duration.ofSeconds(60))
    .permittedNumberOfCallsInHalfOpenState(100)  // 100 requests!
    .build();

// What happens:
// 1. Breaker opens due to failures
// 2. After 60s, sends 100 requests to test
// 3. If service is still recovering, 100 users impacted
// 4. Breaker reopens immediately
// 5. Repeat - service never recovers due to load

The Fix

// CORRECT: Gradual recovery
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .waitDurationInOpenState(Duration.ofSeconds(30))    // Shorter wait
    .permittedNumberOfCallsInHalfOpenState(3)           // Just 3 probes
    .build();

// Or implement exponential backoff
public class AdaptiveCircuitBreaker {
    private int consecutiveFailures = 0;
    
    public Duration getWaitDuration() {
        // 10s, 20s, 40s, 80s, max 5min
        long seconds = Math.min(10 * Math.pow(2, consecutiveFailures), 300);
        return Duration.ofSeconds((long) seconds);
    }
}

Anti-Pattern 5: Ignoring Timeout Configuration

This is perhaps the most insidious anti-pattern because it involves the interaction between two different systems. Your HTTP client has a timeout. Your circuit breaker has a “slow call” threshold. If these aren’t coordinated, the circuit breaker can’t do its job. A request that hangs for 30 seconds isn’t counted as “slow” by the circuit breaker until it completes—but by then, you’ve already exhausted your thread pool and connection pool. The circuit breaker never gets a chance to help because the damage happens before the call completes.

The Problem

// WRONG: Default timeouts that don't match SLA
@Bean
public WebClient paymentClient() {
    return WebClient.builder()
        .baseUrl("http://payment-service")
        // No timeout configured - defaults to infinite!
        .build();
}

CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .slowCallDurationThreshold(Duration.ofSeconds(5))
    .build();

// What happens:
// 1. Payment service hangs for 30 seconds
// 2. Request not counted as "slow" because it hasn't completed
// 3. Thread pool exhausts before breaker can react
// 4. Connection pool exhausts
// 5. Cascading failure across all services

The Fix

// CORRECT: Layered timeout strategy
@Bean
public WebClient paymentClient() {
    HttpClient httpClient = HttpClient.create()
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)  // 2s connect
        .responseTimeout(Duration.ofSeconds(5));              // 5s response
    
    return WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(httpClient))
        .build();
}

// Circuit breaker with matching config
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .slowCallDurationThreshold(Duration.ofSeconds(3))  // Lower than HTTP timeout
    .slowCallRateThreshold(50)
    .build();

// Add timeout decorator as well
TimeLimiterConfig timeLimiterConfig = TimeLimiterConfig.custom()
    .timeoutDuration(Duration.ofSeconds(4))  // Between slow threshold and HTTP timeout
    .build();

Complete Configuration Example

@Configuration
public class ResilienceConfig {
    
    @Bean
    public CircuitBreakerConfig defaultCircuitBreakerConfig() {
        return CircuitBreakerConfig.custom()
            // When to open
            .failureRateThreshold(50)
            .slowCallRateThreshold(50)
            .slowCallDurationThreshold(Duration.ofSeconds(2))
            
            // Measurement
            .slidingWindowType(SlidingWindowType.TIME_BASED)
            .slidingWindowSize(30)  // 30 seconds
            .minimumNumberOfCalls(10)
            
            // Recovery
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .permittedNumberOfCallsInHalfOpenState(3)
            .automaticTransitionFromOpenToHalfOpenEnabled(true)
            
            // What counts as failure
            .recordExceptions(IOException.class, TimeoutException.class)
            .ignoreExceptions(BusinessException.class)
            .build();
    }
    
    @Bean
    public TimeLimiterConfig defaultTimeLimiterConfig() {
        return TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofSeconds(3))
            .cancelRunningFuture(true)
            .build();
    }
}

Monitoring

# Prometheus alerts for circuit breaker issues
- alert: CircuitBreakerOpen
  expr: |
    resilience4j_circuitbreaker_state{state="open"} == 1
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "Circuit breaker {{ $labels.name }} is open"

- alert: CircuitBreakerHighFailureRate
  expr: |
    resilience4j_circuitbreaker_failure_rate > 25
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Circuit breaker {{ $labels.name }} has >25% failure rate"

Checklist

## Circuit Breaker Configuration

### Design
- [ ] Separate breakers per operation/endpoint
- [ ] Define fallback for each breaker
- [ ] Document expected failure modes

### Thresholds
- [ ] Set failure rate threshold 25-50%
- [ ] Configure slow call threshold matching SLO
- [ ] Use minimum call count before evaluation
- [ ] Time-based sliding window for variable load

### Recovery
- [ ] Wait duration matches service recovery time
- [ ] Half-open permits 3-5 test calls max
- [ ] Consider exponential backoff for flaky services

### Timeouts
- [ ] HTTP timeout < circuit breaker slow threshold
- [ ] Time limiter timeout between them
- [ ] Connection pool has lower timeout than all

### Testing
- [ ] Load test with injected failures
- [ ] Verify fallback behavior
- [ ] Test recovery from open state

Conclusion

The common thread through all these anti-patterns is that circuit breakers require thoughtful configuration specific to your system. The defaults are rarely right. The tutorials usually show simplified examples. And the consequences of misconfiguration only appear during incidents—exactly when you need your resilience patterns to work correctly.

The fundamental insight is that circuit breakers are about isolating failures, not just detecting them. Each configuration decision should be made with isolation in mind. Separate breakers per endpoint isolate failures between endpoints. Reasonable thresholds isolate real failures from transient noise. Fallbacks isolate the user experience from backend problems. Gentle half-open testing isolates recovery from additional load. Layered timeouts isolate slow calls from resource exhaustion.

Key principles for circuit breaker configuration:

1. Use separate breakers per endpoint - failures in one operation shouldn’t affect others
2. Set realistic thresholds - not too sensitive, require statistically significant failure rates
3. Always have fallbacks - cache, defaults, graceful degradation with user-friendly messages
4. Test half-open carefully - gradual recovery with minimal probe traffic
5. Layer timeouts correctly - HTTP timeout < TimeLimiter < SlowCall threshold

Most importantly, test your circuit breakers with real failures before production. Use chaos engineering tools to simulate outages and verify that your breakers open and close as expected, that fallbacks work correctly, and that recovery is smooth. A circuit breaker you’ve never seen trip is a circuit breaker you can’t trust.

Related Articles

• gRPC Deadline Propagation - Timeout handling
• Database Connection Pool Exhaustion - Resource exhaustion