Distributed Transactions and Sagas

One-liner: Across services you either lock everyone with 2PC or chain local transactions with compensations (saga); the outbox pattern makes the events reliable.

Say this#

SAY THIS

2PC (two-phase commit) — coordinator asks all participants to PREPARE, then COMMIT/ABORT; blocking protocol.
Coordinator / transaction manager — the 2PC orchestrator; its failure blocks participants holding locks.
XA — the standard 2PC API for databases and message brokers.
3PC / Paxos commit — non-blocking variants; rarely used in practice.
Saga — sequence of local transactions, each with a compensating transaction to undo it.
Choreography — services react to each other's events; no central brain.
Orchestration — a saga orchestrator (Temporal, AWS Step Functions, Camunda) tells each service what to do.
Compensating transaction — semantic undo (refund), not a rollback.
Pivot transaction — the step after which the saga cannot be aborted, only compensated forward.
Transactional outbox — write the event into an outbox table in the same DB transaction; a relay publishes it.
CDC (change data capture) — Debezium reads the DB log to publish outbox rows to Kafka.
Idempotent consumers — required because relays deliver at-least-once.
Dual-write problem — writing DB then broker without a transaction loses or duplicates events.

How it works#

2PC:

TEXT

coordinator -> PREPARE -> [A] [B] [C]   each writes to log, holds locks, votes YES/NOcoordinator -> COMMIT  -> [A] [B] [C]   if all YES; else ABORTcoordinator -> PREPARE -> [A] [B] [C]   each writes to log, holds locks, votes YES/NO
coordinator -> COMMIT  -> [A] [B] [C]   if all YES; else ABORT

Saga (orchestration, e.g. order flow):

Orchestrator: ReserveInventory → ChargePayment → CreateShipment.
Each step is a local ACID transaction in its own service.
If ChargePayment fails: run ReleaseInventory (compensation) in reverse order.
Orchestrator state persisted so it survives crashes (Temporal workflow history).

Outbox:

TEXT

BEGIN; INSERT order; INSERT outbox(event); COMMIT;relay/CDC -> Kafka topic -> consumers (idempotent, dedupe on event_id)BEGIN; INSERT order; INSERT outbox(event); COMMIT;
relay/CDC -> Kafka topic -> consumers (idempotent, dedupe on event_id)

Trade-offs#

Choice	Pro	Con
2PC	Atomic across resources, simple mental model	Blocking on coordinator failure, locks held across network, latency, poor availability
Saga choreography	Loose coupling, no central component	Hard to see the flow; cyclic event dependencies; debugging
Saga orchestration	Explicit flow, easy to reason about and retry	Orchestrator is a component to run HA; some coupling
Outbox + CDC	Exactly-once-ish publication, no dual write	Extra table, relay to operate, at-least-once to consumers
Just use one DB	Real ACID	Doesn't fit service-per-DB boundaries

Numbers#

2PC adds at least 2 round trips plus 2 fsyncs per participant before commit; cross-region this is 100+ ms.
Sagas are eventually consistent: the intermediate state is visible for the saga's duration (ms to hours).
Spanner/CockroachDB use 2PC over Paxos/Raft groups internally; the lock hold time is bounded by consensus latency, not a flaky coordinator.

Say this#

How it works#

Trade-offs#

Numbers#

In the interview#