Circuit Breaker Anti-Patterns: When Protection Causes Outages
Circuit breakers prevent cascading failures but wrong config makes them worse. I show 5 anti-patterns: shared breakers, wrong thresholds, instant open, no fallback, and testing gaps.
#distributed-systems
8 posts
Elasticsearch Hot Shard Problem: When One Node Does All the Work
5 data nodes but one is at 100% CPU. Uneven routing keys create hot shards. I show how to detect skew and fix it with routing strategies.
gRPC Deadline Propagation: Preventing Cascading Failures
Frontend gives up after 5s but backend keeps working for 30s. Without deadline propagation, you waste resources on doomed requests. I show how to implement it in Go.
Cache Stampede Prevention: Probabilistic Early Expiration (X-Fetch)
100 requests hit expired cache simultaneously. All 100 query the database. I implement the X-Fetch algorithm that refreshes cache before expiration without locks.
Transactional Outbox: Solving the Dual Write Problem Without 2PC
Practical Outbox pattern implementation in Node.js/TypeScript with PostgreSQL LISTEN/NOTIFY. Race-condition case study and production-ready solution.
Gossip Protocol Ghost Nodes: IP Reuse Haunting Your Cluster
New node joins cluster but gets shunned. Old node's IP is still in gossip protocol's failure detection blacklist. The zombie membership record lives on.
Double Charges From Idempotency Keys: The Replica Lag Trap
Perfect idempotency logic, but customers still get charged twice. The cause: checking idempotency keys against a read replica that's seconds behind the primary during traffic spikes.
Split-Brain From a Clock Step Backwards: Wall Time in Lease-Based Systems
Two nodes both believe they hold the leader lease. The cause: a small NTP time step backwards combined with code that mixes wall-clock time with duration-based timeouts.