PortfolioTech Notebook
Menu
patterns

chaos-drills

Chaos Drills (Formalized Fault Injection)

The Problem: "It works on my machine"

Microservices fail in creative ways. A node crashes mid-saga, a database connection pool runs dry, or a message broker gets partitioned. If you only test the happy path, you are relying on hope. Resilience is not something you "turn on"; it is something you must prove by deliberately breaking things.

The Pattern

Formalize failure scenarios as Chaos Drills. These are reproducible steps that inject faults into the running system to prove that the resilience mechanisms (circuit breakers, outbox relays, DLQs, sagas) work exactly as designed.

Our Drills

You can execute these drills locally against the Docker Compose stack.

1. Kill -9 Mid-Saga (The Outbox/Inbox Test)

Goal: Prove that crashing a service after a database commit but before publishing the event does not lose data or duplicate side effects.

Drill:

  1. Put a breakpoint or a Task.Delay(10000) right after the Marten SaveChangesAsync() in account-service but before the OutboxRelay publishes.
  2. Send a POST /transfers request.
  3. Hard kill the account-service container (docker kill micro-service-async-account-service-1).
  4. Verification:
    • The transfer is stuck in PENDING.
    • Restart the service (docker start micro-service-async-account-service-1).
    • The OutboxRelay should wake up, find the unpublished event, and publish it.
    • The saga should resume and finish COMPLETED.
    • The consumer inbox prevents any duplicate processing if the event had been partially published.

2. Broker Partition (The DLQ/Retry Test)

Goal: Prove that temporary broker unavailability doesn't drop messages, and persistent failures correctly route to the dead-letter queue (parking lot).

Drill:

  1. Disconnect the account-service from the rabbitmq container using Docker network commands: docker network disconnect micro-service-async_default micro-service-async-rabbitmq-1
  2. Send a POST /transfers.
  3. The transfer-service outbox will fail to publish and will keep retrying.
  4. Reconnect the broker: docker network connect micro-service-async_default micro-service-async-rabbitmq-1
  5. Verification:
    • The outbox publisher reconnects and successfully publishes.
    • The saga completes.
  6. Poison Message Drill: Modify a consumer to throw an unhandled exception. Send a message. Verify it retries 3 times and is then moved to the banking.dead-letters queue, triggering the ParkingLotNonZero Prometheus alert.

3. Slow DB Injection (The Circuit Breaker / Bulkhead Test)

Goal: Prove that a slow downstream service trips the circuit breaker and triggers bulkhead load shedding, protecting the gateway from thread exhaustion.

Drill:

  1. Inject a 15-second Task.Delay in the account-service's GET /accounts endpoint.
  2. Fire 200 concurrent requests to the gateway's /api/accounts endpoint.
  3. Verification:
    • The gateway's Polly concurrency limiter (bulkhead) immediately rejects requests over the PermitLimit with a 503 Service Unavailable ("Downstream is overloaded").
    • The requests that get through eventually hit the timeout and trip the Polly circuit breaker.
    • Subsequent requests fail fast with a 503 ("circuit open") without waiting.
    • The transfer-service endpoints (/api/transfers) remain completely unaffected and responsive, proving cluster isolation.