Add article on consistency in distributed systems

docs/knowledge/architecture/consistency-in-distributed-systems.md

@@ -0,0 +1,208 @@
+---
+tags:
+
+- architecture
+- distributed-systems
+- consistency
+- replication
+- system-design
+
+---
+
+# Understanding Consistency in Distributed Systems
+
+Consistency is one of those words that means something precise in distributed
+systems and something vague in everyday engineering conversation. This article
+builds the intuition from scratch, then turns it into a practical checklist
+you can apply when designing (or debugging) a real system.
+
+## What Consistency Actually Means
+
+In a single-machine system with one copy of the data, there's no ambiguity:
+you write a value, you read it back, you get what you wrote. Consistency
+becomes a *problem* the moment there is more than one copy of the same data —
+which is nearly always, once you introduce replication for availability or
+performance.
+
+**Consistency** is a guarantee about what a reader is allowed to see relative
+to writes that have happened, when there are multiple copies (replicas) of
+the data that don't all update at exactly the same instant.
+
+## Strong Consistency vs. Eventual Consistency
+
+- **Strong consistency** — every read reflects the latest committed write, no
+  matter which replica serves it. To get this, replicas typically have to
+  coordinate on every write (e.g. wait for a quorum of acknowledgments) before
+  the write is considered done, or route all reads through a single
+  source of truth. This costs latency and, under network partitions,
+  availability.
+- **Eventual consistency** — a write is accepted quickly, and replicas
+  propagate it to each other in the background. Readers may see stale data
+  for some window of time, but *if no new writes occur*, all replicas
+  eventually converge on the same value. This buys low latency and high
+  availability, at the cost of temporary staleness.
+
+Neither is "better" in the abstract — they're different trade-offs, and the
+right one depends on what happens if a reader sees stale data.
+
+## Why Replication Introduces Stale Views
+
+The instant you copy data to more than one place, you've created the
+possibility that those copies disagree, at least momentarily:
+
+- **Network latency** — propagating a write to another replica (possibly in
+  another region) takes non-zero time. A reader hitting that replica before
+  the update arrives sees the old value.
+- **Replication lag** — asynchronous replication (common for read replicas)
+  means the primary can be arbitrarily far ahead of a replica under load,
+  widening the staleness window from milliseconds to, in bad cases, minutes.
+- **Network partitions** — when replicas can't talk to each other at all,
+  they have to choose: keep serving reads/writes with what they have locally
+  (favoring availability, at the cost of consistency), or refuse to serve
+  until they can safely coordinate (favoring consistency, at the cost of
+  availability). This is the practical shape of the trade-off popularized by
+  the CAP theorem.
+
+## The Trade-off: Correctness, Availability, Latency, UX
+
+Every consistency choice is really a trade between four things:
+
+- **Correctness** — can the system tolerate a reader seeing a stale or
+  wrong value?
+- **Availability** — can the system keep responding when parts of it can't
+  communicate?
+- **Latency** — how long does a write or read have to wait for
+  coordination?
+- **User experience** — does staleness show up to the user as something
+  confusing or harmful, or is it invisible/acceptable?
+
+Strong consistency buys correctness by spending latency and, in a partition,
+availability. Eventual consistency buys latency and availability by spending
+short-term correctness. There is no configuration that gives you all four at
+once — the job is picking which one you can afford to spend, per piece of
+data.
+
+## Deciding the Required Consistency Level
+
+The right question is never "what consistency model does this system use?"
+in the abstract — it's "what's the business impact if this specific piece of
+data is stale?" Work backward from that impact:
+
+1. What breaks if the reader sees a stale value? (Nothing? A bad but
+   recoverable UX? Money lost? A safety issue?)
+1. How stale could it realistically get, given the replication mechanism in
+   use?
+1. Is the staleness self-correcting (the next read gets the fresh value) or
+   does it need active reconciliation (e.g. a double-booked resource)?
+
+If the answer to (1) is "irrecoverable harm," lean strong. If it's
+"mildly annoying, corrects itself in under a second," eventual is usually the
+right call — paying for strong consistency there is spending latency and
+availability on a problem that doesn't exist.
+
+## Real-World Examples
+
+- **Payments and financial transactions** → strong consistency. Two
+  concurrent reads of an account balance must agree, and a transfer must not
+  be visible on one side of the ledger before the other. The cost of a stale
+  balance (double-spending, incorrect authorization) far outweighs the extra
+  latency of coordinating the write.
+- **Product/menu availability** → some staleness is acceptable. Showing an
+  item as available for a few seconds after it sold out is a minor UX
+  hiccup, not a correctness failure — *as long as* there's a validation step
+  at the critical moment (checkout, order confirmation) that re-checks the
+  authoritative state before committing.
+- **Driver location** → eventual consistency is the natural fit. The only
+  thing that matters is "where was the driver most recently seen," and a
+  location that's a second or two old is functionally identical to the
+  current one for a map UI. Optimizing for low latency (frequent, cheap
+  updates) matters far more than perfect real-time accuracy here.
+
+## Graceful Degradation
+
+When a replica can't get a fresh answer — because of lag, a partition, or a
+timeout — the fallback doesn't have to be an error page. Showing the **last
+known good state**, clearly if you want (e.g. "last updated 30s ago"), keeps
+the product usable instead of failing the whole experience over one stale
+field. This is often a better user experience than strict consistency would
+have given you anyway, because "slightly stale" beats "completely broken."
+
+## Different Guarantees for Different Parts of the Same System
+
+A single system is not one consistency model. It's normal — and usually
+correct — for the payment ledger, the inventory count, and the driver-location
+feed inside the *same* application to each use a different consistency
+guarantee, because each has a different answer to "what's the business
+impact of staleness here?" Treating consistency as a system-wide setting
+instead of a per-data-type decision is a common source of both over-
+engineering (paying strong-consistency costs everywhere) and under-
+engineering (using eventual consistency somewhere that can't tolerate it).
+
+## Common Misconceptions
+
+- **"Stronger consistency is always better."** It's always *safer* against
+  staleness, but never free — it costs latency and, during partitions,
+  availability. Applying it where staleness genuinely doesn't matter is
+  paying a real cost for zero benefit.
+- **"Eventual consistency means the data is unreliable."** It means there's
+  a bounded (usually short) window where reads may lag writes, not that the
+  system is flaky. Most eventually-consistent systems converge in well under
+  a second under normal conditions.
+- **"Consistency is a single global property of the architecture."** As
+  above — it's a per-data-type decision, not a single knob for the whole
+  system.
+
+## Architectural Mental Model
+
+Use this checklist when evaluating a system, or explaining a consistency
+choice in a design discussion:
+
+1. What data or operations must always be correct?
+1. What data can be stale?
+1. How much staleness is acceptable (milliseconds, seconds, minutes)?
+1. What happens when replicas or services cannot communicate?
+1. What are the latency, availability, and operational costs of the chosen
+   consistency model?
+1. What business risk are we accepting with the trade-off?
+
+A useful reasoning pattern for articulating the decision, in an interview or
+a design doc alike:
+
+!!! tip "Reasoning Pattern"
+    **"I chose X because Y, and the cost/trade-off is Z."** Example: *"I chose
+    eventual consistency for driver location because the business only needs
+    the latest known position and low latency matters more than
+    millisecond-perfect accuracy; the trade-off is that a rider might briefly
+    see a slightly outdated pin, which self-corrects on the next update."*
+
+## Summary
+
+- Consistency only becomes a question once there's more than one copy of the
+  data.
+- Strong consistency trades latency/availability for guaranteed freshness;
+  eventual consistency trades short-term freshness for latency/availability.
+- The right choice is driven by the business impact of stale data, not by a
+  system-wide default.
+- Different parts of the same system can and should use different
+  consistency guarantees.
+- Graceful degradation (serving last-known-good data) is often better UX
+  than failing outright when a fresh read isn't available.
+- Stronger consistency isn't automatically better — it's a cost you should
+  only pay where staleness is genuinely unacceptable.
+
+## Related Articles
+
+- [Event-Driven Architecture: Kafka vs. SQS vs. RabbitMQ](event-driven-architecture.md)
+  — how asynchronous propagation between services relates to replication lag
+  and eventual consistency.
+- [Kafka Consumers Behind a FastAPI API on Kubernetes](kafka-consumers-fastapi-kubernetes.md)
+  — a concrete case of consumer-side state lagging behind the source of
+  truth.
+
+## Future Follow-ups
+
+Potential deeper dives building on this foundation: the CAP theorem in more
+formal detail, distributed transactions (2PC/Saga), quorum-based systems
+(Raft/Paxos-backed stores), conflict resolution strategies (last-write-wins,
+CRDTs, vector clocks), and consistency models beyond the strong/eventual
+split (causal consistency, read-your-writes, monotonic reads).

docs/knowledge/architecture/index.md

@@ -25,6 +25,9 @@ within a service, and how services communicate with each other.
 - [Dependency Inversion via Interfaces and Abstract Classes](dependency-inversion-abstract-classes.md)
   — the SOLID principle behind DI, and `abc.ABC` as the nominal-typing way
   to express the abstraction it requires.
+- [Understanding Consistency in Distributed Systems](consistency-in-distributed-systems.md)
+  — strong vs. eventual consistency, why replication introduces staleness,
+  and a checklist for choosing the right guarantee per piece of data.
 
 ## Patterns