PortfolioTech Notebook
Menu
patterns

metrics-slos-grafana

Metrics, SLOs & Grafana

The problem: is the saga healthy right now?

Traces explain one request; logs explain one process. Neither answers "across all traffic in the last 5 minutes, what fraction of transfers completed, and is the saga slower than our promise?" That's an aggregate question, and the tool for aggregates over time is metrics.

The pattern

  • A metric is a numeric time series with labels: a counter (monotonic, e.g. transfers_terminal_total{state="COMPLETED"}), a gauge (up/down, e.g. queue depth), or a histogram (bucketed distribution, e.g. saga duration → P50/P99).
  • An SLI (Service Level Indicator) is a metric that measures user-facing quality: saga success rate, saga latency.
  • An SLO (Service Level Objective) is a target for an SLI: "99% of transfers complete within 5 seconds." The SLO is the line; the SLI is where you actually are.

Services emit metrics via the OTel SDK → OTLP → collector → Prometheus (pull-based: Prometheus scrapes the collector's Prometheus exporter and RabbitMQ's plugin). Grafana queries Prometheus with PromQL and renders the SLO dashboard.

  services ──OTLP──► collector ──Prometheus exporter:8889──┐
  rabbitmq_prometheus plugin :15692 ──────────────────────┤
                                                  Prometheus (scrape, store)
                                                           │ PromQL
                                                        Grafana (SLO dashboard)

Where it lives in our code

  • Runtime metrics, for free: the same AddOpenTelemetry().WithMetrics(...) (.NET) and NodeSDK (transfer-service) that produce traces also export HTTP server/client durations, Kestrel connections, EFCore, and MassTransit meters. Verified in Prometheus: http_server_duration_*, kestrel_*, aspnetcore_rate_limiting_* for all three services.
  • The custom SLI — the saga's own health: transfer-service/src/observability/saga-metrics.ts defines
    • transfers_terminal_total{state} — a counter incremented when a saga reaches a terminal state (COMPLETED / FAILED / COMPENSATED),
    • transfer_saga_duration_seconds — a histogram of wall-clock from request to terminal state, with buckets straddling the 5s SLO. Both are recorded in account-events.handler.ts via recordTerminal(state, transfer.createdAt) when isTerminal(transfer.state). The OTel metrics API is a no-op unless the SDK started, so this adds nothing in tests.
  • Queue depths (gauges): observability/rabbitmq/enabled_plugins turns on rabbitmq_prometheus (:15692); observability/prometheus.yml scrapes it. This is where the parking-lot depth (rabbitmq_queue_messages{queue="banking.dead-letters"}) comes from — see retry-dlq.md.
  • The dashboard: observability/grafana/dashboards/saga-dashboard.json, auto-provisioned. Five panels: throughput by terminal state, saga P99 vs the 5s SLO line, error rate (1 - COMPLETED/total), queue depths, and parking-lot depth (zero = healthy; anything else is alert-worthy).

Why the saga SLI is measured at the terminal transition

A transfer's success isn't known when the 202 returns — the money hasn't moved yet. It's known only when the saga reaches COMPLETED/FAILED/COMPENSATED, which happens in the async consumer, possibly seconds later and in a different service. So the SLI is recorded there, and the duration is measured from transfer.createdAt (request time) to now (terminal time) — the real user-perceived latency, spanning the whole choreography, not just the HTTP call.

Trade-offs: push vs. pull, and cardinality

  • Pull (Prometheus scrapes) vs push: pull gives Prometheus control over rate and a free liveness signal (a target that stops responding is down). Our services push OTLP to the collector, and Prometheus pulls from the collector — best of both: apps don't manage scrape endpoints, Prometheus still owns ingestion.
  • Label cardinality is the footgun. state has 3 values — safe. Labelling by transferId or userId would mint a new time series per id and blow up Prometheus memory. SLI labels must be low-cardinality dimensions you'd actually group by.

Interview Q&A

Q: SLI vs. SLO vs. SLA? SLI = the measured number (99.2% completed). SLO = the internal target (99% in 5s). SLA = the contractual promise to customers with consequences if missed (usually looser than the SLO, so you have headroom). You instrument the SLI, alert on the SLO, and report against the SLA.

Q: Why a histogram for latency instead of an average? Averages hide tails. A 200ms average can still mean 1% of users wait 8s. A histogram lets you compute percentiles (histogram_quantile(0.99, ...)), and P99 is what users feel. SLOs are written on percentiles, never on means.

Q: Counter vs. gauge vs. histogram — pick for "transfers that failed." A counter (transfers_terminal_total{state="FAILED"}): it only ever goes up, and you take its rate() for "failures per second." A gauge would be wrong (failures don't decrease); a histogram is for distributions, not counts.

Q: How do you compute an error-rate SLI from a counter? 1 - rate(transfers_terminal_total{state="COMPLETED"}[5m]) / rate(transfers_terminal_total[5m]). Rates over a window, divide success by total, subtract from one. The dashboard's error-rate panel is exactly this (with clamp_min to avoid divide-by-zero when idle).

Q: Push or pull metrics — which and why? Prometheus is pull by design: it scrapes targets, which doubles as health-checking and centralises rate control. Push (via the OTel collector or Pushgateway) suits short-lived jobs that die before a scrape. We combine them: OTLP push to the collector, Prometheus pull from the collector.

Q: What makes a good SLO dashboard? It answers "are we meeting our promise?" at a glance: the SLI plotted against the SLO line (P99 vs 5s), an error budget / error rate, and the leading indicators that predict a breach (queue depth climbing, parking-lot non-zero). Ours puts the parking-lot panel in red on any non-zero value because a parked message means a saga is silently stuck.

Try to break it

  1. Run ./scripts/e2e.sh, open Grafana (http://localhost:3300, anonymous admin) → Banking / Saga SLOs. Watch transfers_terminal_total split into COMPLETED / FAILED / COMPENSATED as the three scenarios run.
  2. Fire a burst of transfers and watch P99 vs the 5s SLO line move; the red threshold marks a breach. (We measured P99 ≈ 2.5s under light load — comfortably under SLO.)
  3. Make the parking lot light up: stop account-service, POST several transfers so messages exhaust retries, and watch rabbitmq_queue_messages{queue="banking.dead-letters"} climb the parking-lot panel into the red — the alert-worthy signal that the saga is stuck. Start account-service and replay to clear it.
  4. Query Prometheus directly: sum by (state) (transfers_terminal_total) — the raw SLI behind panel #1.