diff --git a/docs/knowledge/architecture/consistency-in-distributed-systems.md b/docs/knowledge/architecture/consistency-in-distributed-systems.md new file mode 100644 index 0000000..6675613 --- /dev/null +++ b/docs/knowledge/architecture/consistency-in-distributed-systems.md @@ -0,0 +1,208 @@ +--- +tags: + +- architecture +- distributed-systems +- consistency +- replication +- system-design + +--- + +# Understanding Consistency in Distributed Systems + +Consistency is one of those words that means something precise in distributed +systems and something vague in everyday engineering conversation. This article +builds the intuition from scratch, then turns it into a practical checklist +you can apply when designing (or debugging) a real system. + +## What Consistency Actually Means + +In a single-machine system with one copy of the data, there's no ambiguity: +you write a value, you read it back, you get what you wrote. Consistency +becomes a *problem* the moment there is more than one copy of the same data — +which is nearly always, once you introduce replication for availability or +performance. + +**Consistency** is a guarantee about what a reader is allowed to see relative +to writes that have happened, when there are multiple copies (replicas) of +the data that don't all update at exactly the same instant. + +## Strong Consistency vs. Eventual Consistency + +- **Strong consistency** — every read reflects the latest committed write, no + matter which replica serves it. To get this, replicas typically have to + coordinate on every write (e.g. wait for a quorum of acknowledgments) before + the write is considered done, or route all reads through a single + source of truth. This costs latency and, under network partitions, + availability. +- **Eventual consistency** — a write is accepted quickly, and replicas + propagate it to each other in the background. Readers may see stale data + for some window of time, but *if no new writes occur*, all replicas + eventually converge on the same value. This buys low latency and high + availability, at the cost of temporary staleness. + +Neither is "better" in the abstract — they're different trade-offs, and the +right one depends on what happens if a reader sees stale data. + +## Why Replication Introduces Stale Views + +The instant you copy data to more than one place, you've created the +possibility that those copies disagree, at least momentarily: + +- **Network latency** — propagating a write to another replica (possibly in + another region) takes non-zero time. A reader hitting that replica before + the update arrives sees the old value. +- **Replication lag** — asynchronous replication (common for read replicas) + means the primary can be arbitrarily far ahead of a replica under load, + widening the staleness window from milliseconds to, in bad cases, minutes. +- **Network partitions** — when replicas can't talk to each other at all, + they have to choose: keep serving reads/writes with what they have locally + (favoring availability, at the cost of consistency), or refuse to serve + until they can safely coordinate (favoring consistency, at the cost of + availability). This is the practical shape of the trade-off popularized by + the CAP theorem. + +## The Trade-off: Correctness, Availability, Latency, UX + +Every consistency choice is really a trade between four things: + +- **Correctness** — can the system tolerate a reader seeing a stale or + wrong value? +- **Availability** — can the system keep responding when parts of it can't + communicate? +- **Latency** — how long does a write or read have to wait for + coordination? +- **User experience** — does staleness show up to the user as something + confusing or harmful, or is it invisible/acceptable? + +Strong consistency buys correctness by spending latency and, in a partition, +availability. Eventual consistency buys latency and availability by spending +short-term correctness. There is no configuration that gives you all four at +once — the job is picking which one you can afford to spend, per piece of +data. + +## Deciding the Required Consistency Level + +The right question is never "what consistency model does this system use?" +in the abstract — it's "what's the business impact if this specific piece of +data is stale?" Work backward from that impact: + +1. What breaks if the reader sees a stale value? (Nothing? A bad but + recoverable UX? Money lost? A safety issue?) +1. How stale could it realistically get, given the replication mechanism in + use? +1. Is the staleness self-correcting (the next read gets the fresh value) or + does it need active reconciliation (e.g. a double-booked resource)? + +If the answer to (1) is "irrecoverable harm," lean strong. If it's +"mildly annoying, corrects itself in under a second," eventual is usually the +right call — paying for strong consistency there is spending latency and +availability on a problem that doesn't exist. + +## Real-World Examples + +- **Payments and financial transactions** → strong consistency. Two + concurrent reads of an account balance must agree, and a transfer must not + be visible on one side of the ledger before the other. The cost of a stale + balance (double-spending, incorrect authorization) far outweighs the extra + latency of coordinating the write. +- **Product/menu availability** → some staleness is acceptable. Showing an + item as available for a few seconds after it sold out is a minor UX + hiccup, not a correctness failure — *as long as* there's a validation step + at the critical moment (checkout, order confirmation) that re-checks the + authoritative state before committing. +- **Driver location** → eventual consistency is the natural fit. The only + thing that matters is "where was the driver most recently seen," and a + location that's a second or two old is functionally identical to the + current one for a map UI. Optimizing for low latency (frequent, cheap + updates) matters far more than perfect real-time accuracy here. + +## Graceful Degradation + +When a replica can't get a fresh answer — because of lag, a partition, or a +timeout — the fallback doesn't have to be an error page. Showing the **last +known good state**, clearly if you want (e.g. "last updated 30s ago"), keeps +the product usable instead of failing the whole experience over one stale +field. This is often a better user experience than strict consistency would +have given you anyway, because "slightly stale" beats "completely broken." + +## Different Guarantees for Different Parts of the Same System + +A single system is not one consistency model. It's normal — and usually +correct — for the payment ledger, the inventory count, and the driver-location +feed inside the *same* application to each use a different consistency +guarantee, because each has a different answer to "what's the business +impact of staleness here?" Treating consistency as a system-wide setting +instead of a per-data-type decision is a common source of both over- +engineering (paying strong-consistency costs everywhere) and under- +engineering (using eventual consistency somewhere that can't tolerate it). + +## Common Misconceptions + +- **"Stronger consistency is always better."** It's always *safer* against + staleness, but never free — it costs latency and, during partitions, + availability. Applying it where staleness genuinely doesn't matter is + paying a real cost for zero benefit. +- **"Eventual consistency means the data is unreliable."** It means there's + a bounded (usually short) window where reads may lag writes, not that the + system is flaky. Most eventually-consistent systems converge in well under + a second under normal conditions. +- **"Consistency is a single global property of the architecture."** As + above — it's a per-data-type decision, not a single knob for the whole + system. + +## Architectural Mental Model + +Use this checklist when evaluating a system, or explaining a consistency +choice in a design discussion: + +1. What data or operations must always be correct? +1. What data can be stale? +1. How much staleness is acceptable (milliseconds, seconds, minutes)? +1. What happens when replicas or services cannot communicate? +1. What are the latency, availability, and operational costs of the chosen + consistency model? +1. What business risk are we accepting with the trade-off? + +A useful reasoning pattern for articulating the decision, in an interview or +a design doc alike: + +!!! tip "Reasoning Pattern" + **"I chose X because Y, and the cost/trade-off is Z."** Example: *"I chose + eventual consistency for driver location because the business only needs + the latest known position and low latency matters more than + millisecond-perfect accuracy; the trade-off is that a rider might briefly + see a slightly outdated pin, which self-corrects on the next update."* + +## Summary + +- Consistency only becomes a question once there's more than one copy of the + data. +- Strong consistency trades latency/availability for guaranteed freshness; + eventual consistency trades short-term freshness for latency/availability. +- The right choice is driven by the business impact of stale data, not by a + system-wide default. +- Different parts of the same system can and should use different + consistency guarantees. +- Graceful degradation (serving last-known-good data) is often better UX + than failing outright when a fresh read isn't available. +- Stronger consistency isn't automatically better — it's a cost you should + only pay where staleness is genuinely unacceptable. + +## Related Articles + +- [Event-Driven Architecture: Kafka vs. SQS vs. RabbitMQ](event-driven-architecture.md) + — how asynchronous propagation between services relates to replication lag + and eventual consistency. +- [Kafka Consumers Behind a FastAPI API on Kubernetes](kafka-consumers-fastapi-kubernetes.md) + — a concrete case of consumer-side state lagging behind the source of + truth. + +## Future Follow-ups + +Potential deeper dives building on this foundation: the CAP theorem in more +formal detail, distributed transactions (2PC/Saga), quorum-based systems +(Raft/Paxos-backed stores), conflict resolution strategies (last-write-wins, +CRDTs, vector clocks), and consistency models beyond the strong/eventual +split (causal consistency, read-your-writes, monotonic reads). diff --git a/docs/knowledge/architecture/index.md b/docs/knowledge/architecture/index.md index 57c7d04..7aca38e 100644 --- a/docs/knowledge/architecture/index.md +++ b/docs/knowledge/architecture/index.md @@ -25,6 +25,9 @@ within a service, and how services communicate with each other. - [Dependency Inversion via Interfaces and Abstract Classes](dependency-inversion-abstract-classes.md) — the SOLID principle behind DI, and `abc.ABC` as the nominal-typing way to express the abstraction it requires. +- [Understanding Consistency in Distributed Systems](consistency-in-distributed-systems.md) + — strong vs. eventual consistency, why replication introduces staleness, + and a checklist for choosing the right guarantee per piece of data. ## Patterns