Distributed transactions are a promise that every backend engineer eventually learns to distrust. Two-Phase Commit (2PC) and the Saga Pattern are the two most cited solutions, yet both are built on assumptions that break under production pressure. The real skill is knowing which failure mode you can live with—and when to abandon the transaction model entirely.

The Two-Phase Commit Promise You Shouldn't Believe

Two-Phase Commit looks elegant on paper. A coordinator asks each participant to prepare, then issues a commit or abort. In theory, it guarantees atomicity across nodes. In practice, it introduces synchronous locking that kills throughput. Every participant holds resources—database rows, file handles, network connections—while waiting for the coordinator's final decision. Under load, these locks cascade into timeouts and deadlocks.

The coordinator itself becomes a single point of failure. If it crashes after sending prepare but before commit, participants are stuck in an indeterminate state, holding locks until a recovery process intervenes. This blocking protocol is especially dangerous under network partitions. A split-brain scenario leaves some participants prepared and others uncertain, requiring manual intervention or a timeout-based heuristic that breaks atomicity.

Many production teams disable 2PC entirely after initial trials. Uber's Chief Product Officer Sachin Kansal, in a recent TechCrunch interview, discussed how the company avoids 2PC for ride matching—even though a ride is a multi-step process involving driver dispatch, payment authorization, and route calculation. Instead, Uber relies on idempotent APIs and eventually consistent state reconciliation. The message is clear: if a company handling millions of real-time transactions avoids 2PC, the pattern's promise is not what it seems.

Georgia Tech researchers have documented that 2PC's timeout rate can exceed 5% in clusters with even modest latency variance (around 100ms). The protocol assumes a reliable, low-latency network that rarely exists outside controlled data centers. For any system spanning regions or cloud providers, 2PC becomes a liability.

Consider a concrete scenario: a retail checkout system that uses 2PC to reserve inventory, authorize payment, and create a shipping label. If the inventory service is slow due to a spike in traffic, the entire checkout stalls. Customers see a spinning wheel, and some abandon the purchase. The coordinator holds a lock on the payment authorization, preventing other transactions from using that payment method. After a timeout, the coordinator aborts, but the inventory reservation may have already expired, leading to overselling. This race condition is not uncommon in 2PC deployments.

Saga Pattern: The More Honest Alternative That Still Lies

The Saga Pattern emerged as a response to 2PC's rigidity. Instead of locks, it uses a sequence of local transactions, each with a compensating action to undo previous steps if something fails. No global lock means higher concurrency. But the trade-off is that atomicity is no longer guaranteed at any point—only eventual consistency is promised, and that promise often surprises teams in production.

Compensating transactions are notoriously hard to test. A cancellation action for a hotel booking must undo a reservation that may have already been confirmed to the guest. The compensation logic often duplicates the forward logic, creating two code paths that must remain in sync. Partial failures cascade unpredictably: if step three fails, compensations for steps two and one run, but what if step two's compensation also fails? The system enters a state where some work is undone and some is not, with no coordinator to reconcile.

LFortran's compiler pipeline, as documented in a recent Hacker News article, uses a saga-like sequence of compilation steps with rollback points. But the team noted that debugging required distributed tracing—a non-trivial infrastructure investment. YouTrackDB, JetBrains' object-oriented graph database, explicitly avoids sagas for graph transactions, opting instead for local graph traversals that never leave a single node. The pattern's complexity often outweighs its benefits for all but the most carefully designed systems.

Eventual consistency surprises in practice because the time window between operations is unbounded. A saga that updates a user's profile and then sends a notification might have the profile update succeed but the notification fail. The compensation, reverting the profile, could race with a user viewing their changes. The result is a user-visible inconsistency that no automated test caught.

Another example: an e-commerce order saga that charges a gift card, then applies a coupon, then ships the item. If the shipment fails, the compensation must reverse the coupon and refund the gift card. But gift card refunds often have different rules than original charges—some systems only allow refunds to the same card, and only once. If the compensation runs twice due to a retry, the gift card could be refunded twice, causing a loss. Testing this scenario requires mocking the gift card provider's exact behavior, which is rarely available in staging environments.

Where Two-Phase Commit Actually Works (and Where It Breaks)

2PC has a narrow sweet spot: small clusters with stable, low-latency networks. Financial ledgers that batch transactions within a single data center can tolerate blocking because the coordinator is rarely far from participants. Some banking systems still use 2PC for settlement operations where the cost of inconsistency outweighs the cost of locking. But even there, the trend is toward idempotent retries and eventually consistent ledgers.

The breaking point is latency variance. Georgia Tech's research on campus systems found that 2PC's timeout rate jumps sharply when round-trip times exceed 10ms. In a multi-region deployment, where inter-datacenter latency can vary from 1ms to 200ms, the protocol spends most of its time waiting. Big Tech companies like Microsoft and Amazon have largely shifted to idempotent operations with at-least-once delivery for their internal services, as documented in various architecture retrospectives.

Microsoft's Windows Search, which recently stripped ads from its search menu as reported by The Verge, also simplified its underlying data pipeline. Earlier versions used a distributed transaction coordinator for indexing across multiple machines, but the team found that blocking on index updates caused noticeable search latency. They eventually moved to a local state model where each machine indexes independently and syncs via a log.

The lesson is that 2PC works when you can control the network and limit participants. It fails when you need geographic distribution or high availability. The blocking nature means a single slow participant drags down the entire transaction, making it unsuitable for any system that serves user-facing requests.

One counter-argument: some modern databases like Spanner use a variant of 2PC with TrueTime to achieve external consistency across regions. However, this requires a globally synchronized clock and a sophisticated infrastructure that most teams cannot replicate. For ordinary PostgreSQL or MySQL deployments, 2PC remains a risky choice.

Saga in Production: The Compensating Action Trap

Compensating actions sound straightforward: if step three fails, run undo for step two and step one. But undo logic often duplicates forward logic. A payment capture's compensation is a refund; a refund's compensation is... nothing, because refunds are final in most payment systems. The asymmetry means that some compensations are not true inverses, leaving partial state behind.

Partial failures cascade unpredictably. Consider a saga that books a flight, a hotel, and a car rental. If the car rental fails, the saga compensates the hotel and flight. But what if the hotel compensation fails because the hotel system is down? The saga is now inconsistent: the flight is canceled, but the hotel remains booked. The system must decide whether to retry compensation, escalate to a manual process, or accept the inconsistency. Each choice has operational cost.

Debugging these failures requires distributed tracing. Without a unified trace across services, engineers cannot determine which step failed and which compensations ran. Microsoft's Windows Search team, in their earlier distributed indexing system, found that saga debugging consumed disproportionate engineering time. They abandoned the pattern for local state replication, where each node independently manages its data and conflicts are resolved via timestamps.

Testing compensations costs roughly 2x development time, according to anecdotal reports from several microservices teams. Each compensation path must be tested in isolation, in combination with other compensations, and under network failures. The combinatorial explosion makes thorough testing impractical, which is why many production sagas have untested failure paths.

Consider a travel booking saga with five steps: flight, hotel, car, insurance, and payment. If the payment fails, compensations must run for the previous four steps. But each compensation may depend on the state of the forward action—for example, the hotel compensation may need to know whether the booking was refundable. If the forward action was partially successful (e.g., a non-refundable deposit was taken), the compensation may only be able to recover part of the cost. This partial recovery is often a surprise to business stakeholders who expected full rollback.

The Real Cost of Both: Latency and Observability

2PC adds 50–200ms per transaction due to the two-round-trip protocol. In a system handling thousands of transactions per second, that overhead compounds into significant latency budget consumption. Sagas avoid the synchronous overhead but introduce compensating action latency that can be even higher if compensations require human approval or external system calls.

Both patterns require distributed tracing infrastructure to debug failures. A Hacker News discussion on distributed transactions flagged the observability gap as the most underappreciated cost. Teams invest in the transaction logic but neglect the monitoring that tells them when it breaks. Without tracing, a saga that silently leaves inconsistent state can go undetected for hours.

Sagas also need a saga log—a durable store that records each step's outcome. This log becomes a critical piece of infrastructure that must be highly available and consistent. Enzyme autodiff, as used in the LFortran compiler pipeline, avoids distributed state entirely by keeping all gradient computation local. The analogy holds: if you can keep your transaction within a single node, you sidestep the entire distributed transaction problem.

The cost of observability is often greater than the cost of the transaction protocol itself. Teams that adopt 2PC or sagas without investing in tracing and logging end up with opaque systems that fail silently. The real expense is not the protocol—it's the operational maturity required to run it.

To quantify: a typical 2PC deployment might require a dedicated coordinator service with its own database for logging, plus a monitoring dashboard for lock duration and timeout rates. Setting this up takes weeks and ongoing maintenance. A saga deployment requires a saga log (often a separate database or Kafka topic), distributed tracing (e.g., Jaeger or Zipkin), and a retry/compensation framework. The total operational overhead can easily exceed the development cost of the transaction logic itself.

When to Pick Neither: Practical Alternatives

Many production systems avoid distributed transactions altogether. Idempotent operations with at-least-once delivery handle most failures without coordination. If an operation is idempotent, retrying it after a failure is safe. This pattern works well for payment captures, email sends, and state updates where duplicates are acceptable or detectable.

The outbox pattern is another alternative. Instead of coordinating across services, each service writes its intent to a local outbox table, then a separate process reads and delivers messages reliably. This ensures that the local transaction and the message sending are atomic, without a distributed coordinator. The pattern is simpler than sagas and avoids compensating actions because failures are handled by retries.

Event sourcing with replayable logs goes further. Every state change is recorded as an event in an append-only log. Services rebuild their state by replaying events, so there is no need for distributed transactions—consistency is achieved through log ordering. This pattern is used by many financial systems and was notably adopted by Uber's financial services team, as mentioned in the TechCrunch interview, where idempotent APIs and event logs replaced earlier transaction attempts.

Robotaxi coordination, as discussed in the same interview, uses local consensus algorithms rather than sagas. Each vehicle makes decisions based on local sensor data and a shared map, with conflicts resolved through a lightweight consensus protocol that does not require global transactions. The lesson is that domain-specific solutions often outperform generic transaction patterns.

Another alternative is the use of CRDTs (Conflict-free Replicated Data Types). For data that does not require strict ordering, such as user profile fields that are updated independently, CRDTs allow each node to update locally and merge automatically. This eliminates the need for any transaction coordination. Georgia Tech's campus systems use CRDTs for student records, where each department can update its own fields without locking.

For systems that must maintain strong consistency, consider whether the data can be colocated. If all the data needed for a transaction can be stored in a single database instance, use that database's native transactions. This is often possible by denormalizing or aggregating data. The distributed transaction should be a last resort, not a first choice.

What Practitioners Actually Do: Patterns That Survive

Idempotency keys for all write operations are the most common recommendation from experienced distributed systems engineers. Every write request includes a unique key; if the server receives the same key twice, it returns the previous result without applying the change again. This eliminates the need for transactions in many cases.

Circuit breakers around saga steps prevent cascading failures. If a step repeatedly fails, the circuit breaker trips and the saga can abort before wasting resources on retries. This pattern is well-documented in the microservices community and is implemented in frameworks like Hystrix and Resilience4j.

Temporal workflows provide a runtime for long-running sagas. The Temporal system manages state, retries, and compensations, allowing developers to write saga logic as straightforward code without managing the infrastructure. This approach reduces the operational burden but still requires the team to understand the failure semantics of compensating actions.

Georgia Tech's campus systems, which manage student records and course registration, use CRDTs (Conflict-free Replicated Data Types) to avoid transactions entirely. Each node can update its local copy, and conflicts are resolved automatically by the data type's merge logic. This pattern works well for data that does not require strict ordering.

Ultimately, the most common advice from practitioners is to prefer single-node transactions over distributed ones. If your data can live on one database, use that database's native transaction support. It is simpler, faster, and more reliable than any distributed alternative. The distributed transaction is a last resort, not a first choice.

In summary, both 2PC and sagas have their place, but their failure modes are often underestimated. The key is to understand the trade-offs: 2PC offers strong consistency at the cost of availability and latency, while sagas offer availability at the cost of eventual consistency and complex compensation logic. For most modern web applications, the best approach is to avoid distributed transactions altogether by using idempotent APIs, the outbox pattern, or event sourcing. When you must use a distributed transaction, invest heavily in observability and testing, and be prepared for the operational cost.