Transactional Outbox

Phase 3 update: account-service no longer uses a transactional outbox. Once it became event-sourced, the append was the only write, so the event store itself became the log — see event-sourcing ("the event store IS the outbox"). transfer-service still uses its hand-built Mongo outbox, described below.

The problem: dual writes

A service often needs to do two things atomically:

1. Commit a state change to its database
2. Publish an event to the message broker

These are two different systems — there is no transaction that spans both. Whatever order you pick, a crash between the two steps corrupts the system:

• DB first, then publish → crash after commit = money moved but nobody was told. The saga stalls forever.
• Publish first, then DB → crash after publish = the world reacts to a change that never happened.

This is the dual-write problem, and it is the single most common correctness bug in event-driven systems.

The pattern

Write the event into an outbox table in the same database, inside the same transaction as the business change. A separate process (relay) reads pending rows and publishes them to the broker, marking them sent.

┌────────────── one DB transaction ──────────────┐
│  UPDATE accounts SET balance = ...             │
│  INSERT INTO ledger_entries ...                │
│  INSERT INTO outbox_messages (envelope) ...    │
└────────────────────────────────────────────────┘
                     │ later, asynchronously
                     ▼
        relay reads pending → publish to RabbitMQ → mark sent

Now there is exactly one source of truth (the DB commit). Either both the state change and the event exist, or neither does.

Our two implementations (compare them!)

	Account service (.NET)	Transfer service (NestJS)
Outbox storage	OutboxMessage table (PostgreSQL), generated by MassTransit	outbox_messages collection (MongoDB), hand-built
Write path	context.Publish(...) inside consumer is intercepted and stored	OutboxService.enqueue(session, ...) called explicitly in the Mongo transaction
Relay	MassTransit's UseBusOutbox background delivery service	OutboxPublisher.drain() polling every 500 ms
Code	account-service/src/AccountService/Program.cs (AddEntityFrameworkOutbox), consumers in Consumers/	transfer-service/src/messaging/outbox.service.ts, outbox.publisher.ts

The hand-built one exists so you can explain exactly what the library one does under the hood.

Delivery semantics

The relay gives at-least-once delivery: if it crashes after publishing but before marking the row sent, the row is re-published on restart — with the same messageId. That is why every consumer deduplicates (see idempotent-consumer.md). At-least-once + dedupe = effectively-once processing. Exactly-once delivery does not exist in distributed systems — say that confidently in interviews.

Trade-offs

• Latency: events leave on the relay's schedule (≤ ~1 s here), not instantly.
• Ordering: our relay publishes oldest-first from a single instance. Scaling relays horizontally breaks ordering unless you partition.
• Polling vs CDC: polling is simple but adds DB load; Change Data Capture (Debezium tailing the WAL/oplog) is the heavy-duty alternative — planned in the roadmap.

Interview Q&A

Q: Why not just publish the event after committing the transaction? It almost always works. "Almost always" is the problem: a crash or network failure between commit and publish silently loses the event, and these bugs are unreproducible in testing. The outbox makes the event part of the committed state, turning a correctness problem into a latency trade-off.

Q: Why not use a distributed transaction (2PC) across the DB and the broker? 2PC needs every participant to support XA, holds locks during the protocol (kills throughput), and the coordinator is a single point of failure. RabbitMQ doesn't support it anyway. The industry abandoned 2PC across heterogeneous systems in favor of outbox + idempotency.

Q: Your outbox relay published the event but crashed before marking it sent. What happens? The event is published again on restart with the same messageId; consumers dedupe it. This is precisely why the outbox pattern and the idempotent-consumer pattern are inseparable — you never implement just one.

Q: How does this scale? The outbox table becomes a bottleneck. Mark-and-sweep cleanup of sent rows (MassTransit does this automatically), partition the relay by aggregate id to keep per-aggregate ordering, or graduate to CDC so the relay reads the replication log instead of querying the table.

Try to break it

1. docker compose stop rabbitmq, then POST a transfer. Watch outbox_messages grow (docker compose exec mongo mongosh transfers --eval 'db.outbox_messages.find({status:"pending"}).toArray()'). Start RabbitMQ again — the backlog drains, the saga completes. Nothing was lost.
2. Look at the same thing on the .NET side: stop RabbitMQ, POST /accounts... nothing breaks, because account opening doesn't publish. Then run a transfer and stop RabbitMQ between the debit and the credit. The OutboxMessage table holds the pending event until the broker returns.