Circuit Breaker (Polly, at the gateway)

The problem: retrying a corpse makes things worse

When a downstream service is down or crawling, the instinct — retry — is exactly wrong. Every retry adds load to a service that already can't cope, ties up a caller thread waiting for a timeout, and the waiting threads pile up until the caller falls over too. One sick service takes out its callers; a cascading failure. The fix is to stop calling a service that clearly can't answer, give it room to recover, and fail fast in the meantime.

Where a circuit breaker belongs (and where it doesn't)

A circuit breaker guards a synchronous call to a dependency. In this system the saga is asynchronous — services talk by passing events over RabbitMQ, and a slow account-service just means the message waits in a queue, with retry + DLQ already handling transient failures. There is no thread blocked on a response to protect.

The synchronous hops are at the edge: the gateway proxying HTTP to account-service and transfer-service. That is where a breaker earns its keep, so that is where we put it. (Async resilience is a different toolkit — timeout propagation and load shedding.)

The pattern

A breaker is a state machine:

• Closed — calls flow; failures are counted.
• Open — too many failures, so calls are rejected immediately (no downstream call) for a cool-off window.
• Half-open — after the window, one trial call is allowed; success closes the breaker, failure re-opens it.

gateway/src/Gateway/Resilience/DownstreamResilience.cs builds a Polly v8 pipeline per YARP cluster: an outer circuit breaker wrapping an inner timeout, so a timeout counts as a failure for the breaker. It is wired into YARP's proxy pipeline in Program.cs (MapReverseProxy(p => p.UseDownstreamResilience())).

• State is per cluster. A failing account-service trips only the account-service breaker; transfer-service traffic is untouched.
• What counts as a failure? YARP doesn't throw when a downstream returns 5xx — it proxies the response. So after the forward we inspect IForwarderErrorFeature (connection refused / timeout) and the status code, and surface a 5xx or a forwarder error as an exception the breaker records.
• Open → 503. When the circuit is open, the pipeline throws BrokenCircuitException before the forward runs; we translate that into a clean 503 and never touch downstream. A timeout becomes 504.

Polly v8 is ratio-based, not count-based

The classic breaker is "N consecutive failures → open". Polly v8 is a sliding window: it opens when, within SamplingDuration, throughput ≥ MinimumThroughput and the failure ratio ≥ FailureRatio. Our defaults (MinimumThroughput: 5, FailureRatio: 0.5, SamplingSeconds: 10, BreakSeconds: 30) approximate "about 5 failures trips it; stay open 30s", but the ratio model is more robust under mixed traffic — five failures buried in a thousand successes shouldn't trip the breaker, and the ratio says so.

Trade-offs

• An open breaker is a deliberate outage. You are returning 503 to clients who might have succeeded, betting that protecting the downstream is worth more than the odd request that would have squeaked through. Tune the thresholds to that bet.
• Timeout ≠ cancellation. Our timeout returns 504 to the client promptly, but the in-flight YARP forward isn't hard-cancelled (wiring Polly's token into the forward is extra plumbing). The client is freed; the orphaned forward completes and is discarded. Documented, not hidden.
• Half-open is a single probe. Recovery hinges on one trial call; if that one unluckily fails, you wait another full window. Fine here; high-traffic systems sometimes allow a few probes.
• Breakers don't help async. Putting one around a message publish would mostly fight the outbox/retry machinery. We deliberately scope breakers to the sync edge.

Interview Q&A

Q: What problem does a circuit breaker actually solve? Cascading failure. It stops a caller from hammering (and being dragged down by) a dependency that's already failing, and lets the dependency recover instead of being kept on its knees by retries.

Q: Circuit breaker vs. retry — don't they conflict? They're layered. Retry handles a transient blip (one dropped packet); the breaker handles a sustained outage (the service is down). Retry-with-backoff inside a breaker is common; retrying a service whose breaker is open is pointless, which is the whole point.

Q: Where would you NOT put a circuit breaker here, and why? Around the async saga hops. They're not synchronous calls — a slow consumer just means a deeper queue, which retry + DLQ + backpressure already handle. A breaker there adds complexity without protecting a blocked thread.

Q: Count-based vs. ratio-based breakers? Count ("5 in a row") is simple but brittle — it trips on a burst even when the service is mostly healthy, and misses a 40%-failure service that never gets 5 in a row. Polly v8's ratio-over-a-window handles both honestly.

Q: How does the breaker know a downstream failed if YARP returns the 5xx itself? We inspect the forwarder error feature and the response status after the proxy step and raise an exception the breaker counts — YARP swallowing the 5xx into a normal response would otherwise hide the failure from Polly.

Try to break it

1. Trip it. Point a cluster at a downstream that returns 500. After ~5 failures the next call should come back 503 instantly and the downstream call count must not increase — prove the breaker actually short-circuits (the test asserts exactly this).
2. Recovery. Heal the downstream and wait out BreakSeconds; the next call half-opens, succeeds, and closes the breaker. Heal it but keep failing the very first probe and watch it re-open for another window.
3. Isolation. Trip account-service's breaker and confirm transfer-service still proxies normally — per-cluster state, not a global switch.
4. The timeout caveat. Make the downstream sleep past TimeoutSeconds; the client gets 504 quickly, but check the downstream logs — the orphaned forward still ran to completion. That's the cancellation gap to close if it matters.