Your home’s electrical panel has circuit breakers. Plug in too many appliances, the breaker trips, cutting power to prevent fires. You can’t use those outlets until you flip it back on. Annoying, but it saves your house.
Circuit breaker pattern in distributed systems: automatically cut connections to failing services to prevent cascade failures.
Without Circuit Breakers
Overload Scenario
Hosting a party:
- Kitchen: Microwave, toaster, coffee maker running
- Living room: TV, sound system, holiday lights
- Garage: Power tools, refrigerator, freezer
Without breakers:
- Wires heat up from excess current
- Insulation starts melting
- Sparks fly
- House fire
The desire to power everything destroys everything.
Daisy Chain
One bad appliance affecting all:
- Faulty toaster shorts out
- All outlets on circuit affected
- Other appliances malfunction
- Entire electrical system compromised
One problem becomes everyone’s problem.
Circuit Breaker States
Closed (Normal)
Requests flow to service. Monitor success/failure rate. Everything operating normally.
Open (Tripped)
Failure rate exceeds threshold. Stop sending requests. Return immediate error. Protect both caller and service.
Half-Open (Testing)
After cooldown period, send test request. Success? Close breaker. Failure? Stay open.
This diagram requires JavaScript.
Enable JavaScript in your browser to use this feature.
Failure Scenarios
Database Overload
Without breaker:
- Query takes 30 seconds
- Web threads wait
- More requests pile up
- All threads blocked
- Entire website down
With breaker:
- First few queries slow
- Breaker detects pattern
- Opens circuit
- Fast-fail responses
- Website degraded but alive
Third-Party API
API having issues:
- 9:00 AM: API responding slowly
- 9:05: Timeouts increasing
- 9:10: Circuit breaker opens
- 9:10-9:20: Fast failures, no waiting
- 9:30: Test request, working! Close circuit
Protected your system during their outage.
Breaker Types
Count-Based
Trips after N failures:
- Set threshold: 5 failures
- Count consecutive failures
- 5th failure: Open circuit
Simple but effective.
Percentage-Based
Trips on failure rate:
- Monitor last 100 requests
- If >50% fail: Open circuit
Adapts to load. More sophisticated.
Time-Based
Trips on sustained problems:
- “5 failures in 1 minute”
- Prevents hair-trigger trips
- Handles brief spikes
Common Problems
Flapping
Constantly opening/closing:
- System on edge of failure
- Breaker trips
- Recovers briefly
- Trips again
Solutions: Longer cooldown periods, gradual recovery, root cause analysis.
Stuck-Open
Never recovers:
- Service actually healthy
- Breaker won’t close
- Permanent degradation
Solutions: Maximum open duration, forced retry logic, monitoring alerts.
Decision Rules
Start conservative: high failure tolerance, long cooldown periods, monitor actual patterns, tune based on data.
Test your breakers: chaos engineering, failure injection, load testing, recovery verification.
Circuit breakers turn catastrophic failures into manageable degradations. Some functionality is lost when a breaker opens, but total system failure is worse.