PortfolioTech Notebook
Menu
patterns

timeout-propagation

Timeout Propagation (deadline in the envelope)

The problem: a timeout on one hop doesn't help the saga

A per-call timeout (the gateway's circuit breaker has one) protects that call. But our saga is a chain of asynchronous hops over RabbitMQ, and "the transfer is taking too long" is a property of the whole chain, not any one step. Two failure modes fall out of that:

  1. Zombie work. A message sits in a queue (or the parking lot) for minutes, everyone gives up, the transfer is marked failed — and then the message is finally delivered and account-service debits the account anyway. Money moves for a transfer that no longer exists.
  2. Stuck sagas. A consumer is down, so the next event never arrives. The transfer sits in PENDING or DEBITED forever; the money (if already debited) is stranded.

The fix is one idea applied in two places: an absolute deadline that travels with the saga, enforced both by the worker (don't act late) and by the initiator (give up and clean up).

The pattern

When transfer-service accepts a transfer it computes an absolute deadline = now + budget and:

  • stamps it into the event envelope (metadata.deadline, ISO-8601) on the initial Transfer.Requested, and it rides every subsequent event because the saga handler copies metadata forward (transfers.service.ts, account-events.handler.ts);
  • stores it on the transfer so it can be swept locally.

Two enforcers act on that deadline:

  • account-service refuses late work. TransferRequestedConsumer / CreditRequestedConsumer check SagaSession.DeadlineExceeded(metadata) before moving money; past the deadline they emit DebitFailed / CreditFailed ("deadline exceeded") instead of debiting/crediting. Compensation deliberately ignores the deadline — you can't "time out" a refund.
  • transfer-service sweeps stuck sagas. TransferTimeoutSweeper (a scheduled job) finds non-terminal transfers past their deadline and, in a transaction: PENDING → FAILED (the debit was never confirmed) and DEBITED → COMPENSATING + a CompensationRequested (refund the source). Every transition is re-checked inside the transaction, so it can't race the saga handler into a double compensation — the state machine assertion makes the loser a no-op.

Why an absolute deadline, not a per-hop timeout

A relative timeout ("each step gets 10s") resets at every hop, so a saga that limps through five slow-but-not-timed-out steps can still take a minute. An absolute deadline is a single budget for the whole flow: wherever a step runs, it asks one question — "is it already too late?" — and every participant answers it the same way against the same instant. That only works because the deadline is data that travels, not config baked into each service.

Trade-offs

  • Clock skew. Comparing "now" against a deadline minted on another host trusts loosely-synced clocks. Fine for a 30s budget; a 50ms budget would need NTP discipline or relative-time math. Documented, not ignored.
  • The two-generals reality. When the sweeper fires, it doesn't know whether account-service is about to act. The deadline check on the account side is what makes the sweep safe: account-service won't debit after the deadline, so a swept PENDING → FAILED can't be contradicted by a late debit. The two enforcers are a pair; neither is safe alone.
  • A deadline is a guess. Too short and healthy-but-slow transfers get killed; too long and stuck money sits longer. It's a tunable (TRANSFER_DEADLINE_MS), and in production it'd be an SLO-derived number, not a constant.
  • Compensation has no deadline. Refunds must always run, so the symmetry is intentionally broken there.

Interview Q&A

Q: Why not just rely on per-call timeouts? They bound a single hop, not the end-to-end saga. A timeout that resets every hop can't express "this whole transfer must finish within 30s". An absolute deadline carried in the envelope can, and every participant enforces the same one.

Q: What stops a timed-out transfer from being debited late? The deadline travels on the event, and account-service checks it before moving money. Past the deadline it rejects the debit, so the sweeper's "fail it" decision can never be contradicted by a late success.

Q: How do you avoid double-compensating when the sweeper and the saga both fire? Both run inside a MongoDB transaction and re-read the transfer's state; the legal transition table allows DEBITED → COMPENSATING exactly once, so whoever loses the race hits an illegal transition and no-ops. No second refund is enqueued.

Q: Isn't comparing timestamps across services fragile? It trusts clock sync, which is acceptable at a 30s budget and risky at millisecond budgets. That's the honest limitation; tighter budgets want relative-deadline math or strong time sync.

Q: Why does compensation ignore the deadline? Because a refund must always complete — "the deadline passed" is never a reason to leave a customer's money debited. Only the forward steps (debit, credit) are deadline-gated.

Try to break it

  1. Late debit. Publish a Transfer.Requested whose metadata.deadline is in the past. account-service must emit DebitFailed and leave the balance untouched — no debit ever happens (covered by a test).
  2. Stuck PENDING. Create a transfer, never deliver a Debited event, expire its deadline, run the sweeper: it must land in FAILED with a deadline reason.
  3. Stuck DEBITED. Drive a transfer to DEBITED, expire its deadline, sweep: it must go COMPENSATING and publish a CompensationRequested so the source is refunded.
  4. Double-compensation race. Fire the sweeper and a real CreditFailed at the same DEBITED transfer. Exactly one CompensationRequested should result — prove the state-machine assertion (not luck) is what enforces it.