Bulkhead Isolation & Load Shedding

The Problem: One bad apple spoils the bunch

When a microservice begins to slow down (e.g., due to a database lock, high CPU, or network latency), the immediate effect is that HTTP requests back up. In a naive API Gateway, these slow requests consume threads and connection pool resources. If the gateway exhausts its connection pool waiting for account-service, it will be unable to serve requests for transfer-service—even if transfer-service is perfectly healthy. One failing downstream brings down the entire edge.

Furthermore, queuing unbounded amounts of work in front of a struggling service only makes the problem worse (the "thundering herd" effect).

The Pattern

• Bulkhead Isolation: Name derived from ship hulls. If one compartment floods, the ship shouldn't sink. In software, this means allocating strict concurrency limits per downstream service. If account-service is allowed 100 concurrent requests, the 101st request is immediately rejected, preserving gateway resources for other services.
• Load Shedding: When the bulkhead is full, it is better to return a fast 503 Service Unavailable than to hold the connection and timeout later. Rejecting work early sheds load from the struggling system, giving it breathing room to recover.

Where it lives in our code

• The Gateway (Edge): In gateway/src/Gateway/Resilience/DownstreamResilience.cs, we use Polly v8's AddConcurrencyLimiter.
  • We configure it with a PermitLimit (e.g., 100) and a QueueLimit (e.g., 50).
  • It wraps the AddCircuitBreaker and AddTimeout in a pipeline (AddResilienceHandler).
  • When the RateLimiterRejectedException is thrown, our custom middleware catches it and returns a 503 Service Unavailable indicating "Downstream is overloaded".
• The Message Broker (Async): In account-service's Program.cs, we set e.PrefetchCount = 10 and e.ConcurrentMessageLimit = 5.
  • The PrefetchCount acts as a network-level backpressure mechanism. The consumer won't pull more than 10 messages into memory from RabbitMQ.
  • The ConcurrentMessageLimit acts as an async bulkhead, ensuring the consumer only uses 5 worker threads, preventing thread starvation in the host process.

Trade-offs

• Rejection vs. Wait: Load shedding forces the client to handle the failure. The client must either retry (with backoff) or fail gracefully. It trades individual request success for overall system survival.
• Tuning Limits: Setting the concurrency limit too low causes unnecessary rejections during normal bursts. Setting it too high defeats the purpose of the bulkhead. These limits must be tuned based on real-world load testing.

Interview Q&A

Q: What is the difference between a Rate Limiter and a Bulkhead (Concurrency Limiter)? A Rate Limiter restricts requests over time (e.g., 100 req/sec per user) to prevent abuse. A Bulkhead restricts concurrent in-flight requests (e.g., max 50 active threads) to protect internal capacity and isolate failures.

Q: Why put the Bulkhead before the Circuit Breaker in the pipeline? The pipeline executes from the outside in. If the Bulkhead is on the outside, it can reject requests immediately without ever interacting with the Circuit Breaker's state machine, keeping overhead extremely low.

Try to break it

See chaos-drills.md for the Slow DB Injection drill, which proves the Bulkhead protects the gateway.