Idempotent Consumer (Inbox)

Phase 3 update: account-service's inbox is now a Marten document (SagaSession: pre-check + a unique-PK backstop, written in the same transaction as the event append) rather than the MassTransit EF inbox. The principle below is unchanged — dedupe on messageId in the same transaction as the state change. See event-sourcing.

The problem: at-least-once delivery

RabbitMQ (like Kafka, SQS, etc.) guarantees at-least-once delivery: a message is redelivered if the consumer dies before acking, if the connection drops after processing but before the ack arrives, or if our own outbox relay re-publishes after a crash. "At least once" means duplicates are normal operation, not an edge case.

For us a duplicate is not cosmetic: processing Transfer.Requested twice debits Alice twice. In a banking system, idempotency is the difference between a correct ledger and money invented or destroyed.

The pattern

Record every processed messageId in the same transaction as the side effects. A duplicate delivery then fails the uniqueness check before doing anything, and is acknowledged without effect.

BEGIN TRANSACTION
  INSERT INTO processed_messages (messageId)   ← unique index; duplicate aborts here
  ... business state change ...
  ... outbox insert (next event) ...
COMMIT

Atomicity is the crux: if the dedupe record were written in a separate step, a crash between business change and dedupe record would let the redelivery process the message again.

Our two implementations

	Account service (.NET)	Transfer service (NestJS)
Mechanism	MassTransit inbox (UseEntityFrameworkOutbox on the endpoint): InboxState table keyed (MessageId, ConsumerId)	hand-built processed_messages collection, unique index on messageId
Duplicate handling	the inbox middleware skips delivery to the consumer entirely	E11000 duplicate-key error caught → log + ack (account-events.handler.ts)
Second line of defence	unique ledger index (TransferId, Type) in AccountsDbContext — even a bug can't double-debit one transfer	state machine rejects illegal transitions (saga-choreography.md)

Where does the messageId come from? The publisher mirrors the envelope's messageId into the AMQP message-id property (outbox.publisher.ts; the test fixture does the same), and republications reuse it — dedupe only works because the id is stable across retries.

Natural idempotency — the alternative

Some operations are idempotent by construction: "set status to SHIPPED", "upsert this row". No inbox needed. But "subtract 2500 from balance" is NOT naturally idempotent — applied twice, it's wrong twice. Rule of thumb: absolute writes are naturally idempotent, relative writes need an inbox. You can sometimes convert one into the other (e.g. store per-transfer ledger entries with a unique key — which we also do).

Interview Q&A

Q: Why doesn't the broker just deliver exactly once? Exactly-once delivery requires the broker and consumer to commit atomically — same dual-write problem as the outbox, between different parties. What systems actually offer (Kafka's "exactly-once semantics" included) is at-least-once delivery plus transactional dedupe on the consumer side — exactly what we built by hand.

Q: Couldn't you dedupe with a Redis SET of seen messageIds? Only if you enjoy distributed-systems horror stories: Redis and your database don't commit atomically, so a crash between "mark seen in Redis" and "commit business change" (in either order) loses or duplicates work. The dedupe record must live in the SAME datastore, in the SAME transaction, as the side effects.

Q: How long do you keep dedupe records? As long as a duplicate can plausibly arrive: retries span minutes, so days are plenty. MassTransit's inbox cleans up automatically (configurable DuplicateDetectionWindow, ours 30 min); the Mongo collection would get a TTL index in production.

Q: The consumer processed the message and crashed before acking. The inbox skips the redelivery — but then who acks it? The redelivery is still consumed — the inbox/handler recognises the duplicate, does nothing, and acks. Dedupe doesn't mean "ignore the message", it means "ack without side effects".

Try to break it

1. Re-run the duplicate test and watch it from the inside: cd transfer-service && npm run test:integration -- -t idempotency. Two identical deliveries, one CreditRequested.
2. Publish a duplicate by hand from the RabbitMQ management UI (http://localhost:15672, exchange banking.events, routing key transfers.transfer.requested, paste an envelope you saw in the logs — same messageId). Balance moves once.
3. Delete the unique index from processed_messages and repeat #2. Watch the double-debit happen. Put the index back and appreciate it.