PortfolioTech Notebook
Menu
patterns

event-sourcing

Event Sourcing (Marten)

The problem: the balance is a side effect, not a fact

In Phase 1 the account was a row: balance = 7500. Every debit overwrote that number and, as a courtesy, appended a ledger_entries row so we could reconstruct how we got there. But the row was the truth and the ledger was a copy — so the two could drift, "why is this balance what it is?" meant trusting a log nobody enforced, and a bug that wrote the wrong balance left no way to recompute the right one.

Event sourcing inverts that. The events are the truth; the balance is a fold of them and is never stored authoritatively. You can throw the balance away and recompute it, because the only writes that ever happened are the events.

The pattern

  • An account is a stream of events (AccountOpened, AccountDebited, AccountCredited, DebitReversed, plus DebitRejected / CreditRejected).
  • State is rebuilt by folding: Apply(AccountDebited e) => BalanceMinor -= e.AmountMinor. The current balance is events.Aggregate(seed, Apply).
  • Replaying the whole stream on every read is wasteful, so Marten keeps an inline snapshot: a materialised Account document updated inside the same transaction as the append. Reads are O(1); the stream is still the source of truth the snapshot is derived from.
  • Concurrent writers are handled with optimistic concurrency: appends carry an expected version and the event table has UNIQUE(stream_id, version). Two debits that both load version 4 and both try to write version 5 — one wins, the other gets a ConcurrencyException and the message is retried. This is the event-sourced equivalent of the Phase 1 xmin row-version check.

Where it lives in our code

  • account-service/src/AccountService/Domain/AccountEvents.cs — the events.
  • Domain/Account.cs — the aggregate. Two halves kept apart on purpose:
    • Decisions (Open/Debit/Credit/ReverseDebit) look at current state and a command and return the event(s) to append. They never mutate — that is what keeps "balance never goes negative" enforceable in exactly one place.
    • Apply folds an already-decided event into state. By the time an event exists, it happened; Apply cannot reject anything.
  • Program.cs — AddMarten(...) with Projections.Snapshot<Account>(Inline) and per-event metadata (correlationId + headers) enabled.
  • Consumers/*.cs — each saga step FetchForWriting<Account>(id) (which captures the expected version), asks the aggregate for the event, AppendOne, and saves.

The key insight: the event store IS the outbox

Phase 1 needed a transactional outbox because state and events lived in two places and had to be written atomically. With event sourcing there is only one write — the append — so the dual-write problem disappears. The Marten event store is already an ordered, durable log, so we publish from it instead of from a second table:

Infrastructure/OutboxRelay.cs is a BackgroundService that walks Marten's global event Sequence from a persisted RelayCheckpoint and publishes each event's integration counterpart to RabbitMQ. It is safe without a distributed transaction:

  • the append is the only write — nothing to reconcile;
  • delivery is at-least-once (publish, then advance the checkpoint — a crash in between re-publishes on restart);
  • re-publishing is harmless because the integration messageId is the Marten event Id — stable across replays — and every consumer deduplicates on messageId. At-least-once + idempotent = effectively once.

Failures are events too (DebitRejected on a transfer-keyed stream), so DebitFailed/CreditFailed are relayed with the same guarantee as successes — there is no second, less-reliable publish path for the unhappy case.

Idempotent consumption uses a Marten inbox document (SagaSession): the incoming messageId is Inserted in the same transaction as the append, so a redelivery rolls the whole thing back.

Trade-offs

  • Event sourcing is not an audit log. An audit log is written alongside state for humans to read; an event stream is the state and the system replays it. You can delete the snapshot; you cannot delete the events.
  • Schema evolution is forever. You can never "migrate" old events — they are immutable history. New shapes are handled with upcasting / versioned events (see event contracts & versioning).
  • Denormalised post-balance in the event. Each balance event also stores BalanceAfterMinor. That is a convenience for the relay and the ledger view; the fold still derives the balance from the running total, so the stored value is a checksum, not the source of truth. A "pure" delta-only stream avoids the redundancy at the cost of a re-fold on every projection.
  • Single-writer relay. One account-service instance runs the relay; a second would double-publish (still correct, just wasteful). Production hardening is leader election via a Postgres advisory lock.

Interview Q&A

Q: Event sourcing vs. an append-only audit/ledger table? The ledger in Phase 1 was a copy kept beside the authoritative balance row — it could drift and nothing replayed it. Here the events are the only writes and the balance is recomputed from them; there is no other source to drift from.

Q: How do you get the current balance without replaying millions of events? A snapshot/projection. Marten updates an inline Account document in the same transaction as the append, so reads are O(1). The snapshot is disposable — delete it and it rebuilds from the stream.

Q: Two debits hit the same account at once — what stops a lost update? Optimistic concurrency on append: UNIQUE(stream_id, version). Both load version N, both try to write N+1, one wins, the loser gets ConcurrencyException and the message is retried against the new state.

Q: With event sourcing, do you still need a transactional outbox? No — that is the elegant part. The store is already the log. A checkpointed relay publishes from it; because the broker messageId is the stable event Id, at-least-once delivery is idempotent end-to-end.

Q: A failed debit changes no state — how does it still reach the saga reliably? It is recorded as a DebitRejected event (on a transfer-keyed stream) and relayed like any other. Modelling rejections as events means there is one delivery path, not a reliable one for successes and a fire-and-forget one for failures.

Q: How do you fix a bug that produced wrong balances? Fix the projection/Apply logic and rebuild the snapshot (and read models) from the event log. The events are correct; only the derived view was wrong.

Try to break it

  1. Replay determinism. Wipe the Account snapshot table and let it rebuild from the stream. The balances must come back identical. If a fold is non-deterministic (e.g. reads DateTimeOffset.UtcNow), replay diverges — that is why decisions, not Apply, own all non-determinism.
  2. The concurrency window. Fire two simultaneous debits at one account. Exactly one AccountDebited should land per balance; the other must retry, not silently overwrite. Confirm with UNIQUE(stream_id, version) in mt_events.
  3. At-least-once, not at-most-once. Kill account-service after the relay publishes but before it writes the checkpoint. On restart the event is published again — and transfer-service's inbox must drop it because the messageId (= event Id) repeats. Change the relay to mint a fresh messageId and watch the duplicate slip through.
  4. Snapshot identity. The aggregate's Id/state need public setters or Marten loads them as defaults (we hit exactly this: a private set left Id empty on load). Make BalanceMinor private-set again and watch every read return 0.