Saga (Choreography) & Compensating Events

The problem: a transaction across two databases

A money transfer touches the Account service (PostgreSQL) and the Transfer service (MongoDB). Each service owns its data — there is no shared database (see ddd-bounded-contexts.md) and no distributed transaction. Yet "debit Alice, credit Bob" must end in a consistent state: either both happened, or — observably — neither.

The pattern

A saga is a sequence of local transactions, each publishing an event that triggers the next step. If a step fails, the saga runs compensating actions — new forward transactions that semantically undo the completed steps. Nothing is ever "rolled back" across services; mistakes are corrected, the way real banks issue refunds rather than rewriting history.

Ours is choreography: no central coordinator. Each service reacts to events and knows only its own step:

Transfer svc                    Account svc
────────────                    ───────────
Transfer.Requested ───────────► debit source
PENDING                            │
         ◄── Account.Debited ──────┤ (or DebitFailed ──► FAILED, saga over)
DEBITED
Transfer.CreditRequested ─────► credit destination
         ◄── Account.Credited ─────┤ (or CreditFailed...)
COMPLETED ✓

— compensation branch —
         ◄── Account.CreditFailed ─┤  money is in limbo: debited, not credited!
COMPENSATING
Transfer.CompensationRequested ► refund source (DebitReversed)
COMPENSATED ✓ (failed, but money restored)

Where it lives in our code

• State machine (the saga's brain): transfer-service/src/transfers/transfer-state.ts — legal transitions only; terminal states are frozen.
• Reactions: transfer-service/src/transfers/account-events.handler.ts — each account event advances the state and stages the next event in the same Mongo transaction (outbox).
• Steps + compensation: account-service/src/AccountService/Consumers/ — TransferRequestedConsumer (debit), CreditRequestedConsumer (credit), CompensationRequestedConsumer (refund).
• Pivot point: the debit is the pivot transaction. Before it, failure just means FAILED. After it, failure requires compensation — there is real money in limbo between Debited and Credited.

Business failures vs technical failures — the key design line

• Business failure (insufficient funds, unknown account): expected. The consumer publishes a failure event (DebitFailed) — it does not throw. Retrying "insufficient funds" can never help.
• Technical failure (DB down, deadlock, concurrency conflict): unexpected. The consumer throws; retry + DLQ machinery handles it (retry-dlq.md).

Mixing these up is the classic saga bug: a thrown InsufficientFundsException would retry three times, land in the DLQ, and stall the saga — instead of cleanly failing the transfer.

Choreography vs orchestration

	Choreography (this project)	Orchestration (Phase 2+ option)
Coordinator	none — events drive everything	a saga orchestrator tells each service what to do
Coupling	services only know event contracts	orchestrator knows every step
Visibility	flow is implicit, spread across services	flow is explicit in one place
Sweet spot	few steps, few services	long flows, many branches, timeouts

With 2 services and 3 steps, choreography is the right size. At 5+ steps the "where is the flow defined?" question starts to hurt, and an orchestrator (MassTransit State Machine, Temporal) pays off.

Interview Q&A

Q: Why not a distributed transaction instead of all this? 2PC across PostgreSQL, MongoDB and RabbitMQ isn't supported, and even where it is, it couples availability (all participants must be up to commit) and holds locks across the network. Sagas trade isolation for availability: intermediate states are visible (Alice's balance dips before Bob's rises), and that's a business conversation, not just a technical one.

Q: What happens if the compensation itself fails? Compensations must be designed to eventually succeed — ours is a credit, which has no business reason to fail. Technical failures retry; after max attempts the message parks in the DLQ and a human intervenes. That's why CompensationRequestedConsumer throws on a missing account instead of swallowing it: losing a refund is unacceptable.

Q: Where's the saga's state? I thought choreography meant stateless. Choreography means no coordinator, not no state. Each transfer document IS the saga instance — its state field plus stateHistory is the saga log. Without it you couldn't answer "did this transfer finish?"

Q: How do you prevent a duplicated or late event from corrupting a finished saga? Two layers: messageId dedupe (inbox) catches literal redeliveries; the state-machine transition table rejects semantically stale events (Debited arriving on a COMPLETED transfer throws IllegalTransition and is acked away). Terminal states have no outgoing transitions, so a finished saga is immutable.

Q: Can the user see Alice debited but Bob not yet credited? Yes — that's eventual consistency, and it's exposed honestly: the API returns 202 and the transfer state (DEBITED) tells the truth. Hiding the intermediate state would require locks we deliberately gave up.

Try to break it

1. Run ./scripts/e2e.sh scenario 3 manually and watch stateHistory: PENDING → DEBITED → COMPENSATING → COMPENSATED — the saga's story, queryable forever.
2. docker compose stop account-service, POST a transfer, wait, docker compose start account-service — the saga resumes and completes. Durable messages + retries make the choreography crash-proof.
3. Transfer to an account with a different currency (open one with "currency":"EUR"): the credit fails after the debit — compensation kicks in.