diff --git a/docs/knowledge/python/closures-lexical-scope.md b/docs/knowledge/python/closures-lexical-scope.md new file mode 100644 index 0000000..0a0f092 --- /dev/null +++ b/docs/knowledge/python/closures-lexical-scope.md @@ -0,0 +1,109 @@ +--- +tags: + +- python +- closures +- scope +- functions + +--- + +# Closures and Lexical Scope in Python + +Python resolves names by **lexical (static) scope**: where a function is *written* +determines which variables it can see — not where it is *called* from. A closure is what +you get when an inner function keeps using variables from the enclosing function after +that function has returned. + +## The LEGB Rule + +When Python looks up a name inside a function it searches, in order: + +1. **L**ocal — the current function. +1. **E**nclosing — any enclosing function scopes, innermost first. +1. **G**lobal — the module level. +1. **B**uilt-in — `len`, `print`, etc. + +Only functions, classes, and modules create scopes. `if`/`for`/`while`/`with` blocks do +**not**, so a variable assigned in a loop body is visible after the loop. + +## Closures + +```python +def make_counter(): + count = 0 + + def increment(): + nonlocal count + count += 1 + return count + + return increment + + +c = make_counter() +c(), c(), c() # 1, 2, 3 +``` + +`make_counter` has returned, yet `count` survives: `increment` holds a reference to it +through a **cell**. You can inspect this: + +```python +c.__closure__[0].cell_contents # 3 +c.__code__.co_freevars # ('count',) +``` + +Each call to `make_counter()` creates a new, independent cell — two counters do not share +state. + +## `nonlocal` and `global` + +Assignment makes a name **local to the whole function** at compile time. So this fails: + +```python +def outer(): + x = 0 + def inner(): + x += 1 # UnboundLocalError: x is treated as local + inner() +``` + +`nonlocal x` says "this name lives in the nearest enclosing function scope"; `global x` +says "module scope". Reading a free variable needs no declaration — only rebinding does. +Mutating an object (`items.append(...)`) is not rebinding, so it needs no `nonlocal`. + +## The Classic Gotcha: Late Binding + +Closures capture the **variable**, not its value at definition time: + +```python +funcs = [lambda: i for i in range(3)] +[f() for f in funcs] # [2, 2, 2] -- all see the final i +``` + +Fixes: bind the value as a default argument, or use `functools.partial`: + +```python +funcs = [lambda i=i: i for i in range(3)] # [0, 1, 2] +``` + +## Where Closures Are Used + +- **Function factories** and configuration ("make me a validator with this threshold"). +- **Callbacks** that need a bit of context without a full class. +- **[Decorators](decorators.md)** — the wrapper closes over the decorated function and any + decorator arguments. +- Lightweight private state; a class is usually clearer once there is more than one + piece of state or more than one operation. + +## Key Takeaways + +- Scope is decided by where code is written (LEGB), not by the call site. +- Closures capture variables via cells; rebinding them requires `nonlocal`. +- Late binding bites in loops — capture with a default argument. +- Only functions/classes/modules create scope; blocks don't. + +## Related Articles + +- [Python Tips & Tricks](python-tips.md) — a short intro to scopes and closures. +- [Decorators](decorators.md) diff --git a/docs/knowledge/python/context-managers.md b/docs/knowledge/python/context-managers.md new file mode 100644 index 0000000..48d4cdf --- /dev/null +++ b/docs/knowledge/python/context-managers.md @@ -0,0 +1,139 @@ +--- +tags: + +- python +- context-managers +- resources +- contextlib + +--- + +# Context Managers in Python + +A context manager guarantees that setup and teardown code run as a pair, no matter how +the block in between exits — normally, via `return`, or via an exception. It is Python's +answer to "always release this resource", and the `with` statement is its syntax. + +## What `with` Actually Does + +```python +with open("data.txt") as f: + data = f.read() +``` + +is roughly equivalent to: + +```python +mgr = open("data.txt") +f = type(mgr).__enter__(mgr) +try: + data = f.read() +except BaseException as exc: + if not type(mgr).__exit__(mgr, type(exc), exc, exc.__traceback__): + raise +else: + type(mgr).__exit__(mgr, None, None, None) +``` + +Two methods make up the protocol: + +- `__enter__(self)` runs on entry; its return value is bound to the name after `as`. +- `__exit__(self, exc_type, exc, tb)` runs on exit, always. If the block raised, the + exception details are passed in; if it did not, all three are `None`. **Returning a + truthy value from `__exit__` suppresses the exception**; returning `None`/`False` lets + it propagate. + +## Class-Based Context Managers + +```python +import time + + +class Timer: + def __enter__(self) -> "Timer": + self.start = time.perf_counter() + return self + + def __exit__(self, exc_type, exc, tb) -> None: + self.elapsed = time.perf_counter() - self.start + # returning None -> exceptions still propagate + + +with Timer() as t: + sum(range(10_000_000)) +print(t.elapsed) +``` + +## Generator-Based: `contextlib.contextmanager` + +For most cases a generator is shorter. Everything before `yield` is `__enter__`, the +yielded value is the `as` target, and everything after is `__exit__`: + +```python +from contextlib import contextmanager + + +@contextmanager +def transaction(conn): + try: + yield conn + conn.commit() + except Exception: + conn.rollback() + raise + finally: + conn.close() +``` + +The gotcha: **an exception in the `with` body is re-raised at the `yield`**. If you want +cleanup to happen on failure you need `try`/`finally` (or `except`) around the `yield` — +without it, code after `yield` is skipped when the body raises. + +## Useful `contextlib` Tools + +- `suppress(*exceptions)` — swallow specific exceptions: + `with suppress(FileNotFoundError): os.remove(path)`. +- `closing(obj)` — call `obj.close()` on exit for objects that have `close()` but are + not context managers themselves. +- `redirect_stdout(target)` — temporarily redirect `print` output. +- `ExitStack` — manage a *dynamic* number of context managers, or register arbitrary + cleanup callbacks: + +```python +from contextlib import ExitStack + +with ExitStack() as stack: + files = [stack.enter_context(open(p)) for p in paths] + # all files are closed on exit, in reverse order, even if one open() failed +``` + +- `nullcontext()` — a no-op stand-in, handy for "optionally lock/trace": + `with (lock if needs_lock else nullcontext()):`. + +## Multiple Managers and Async + +Multiple managers can share one `with`; they are entered left to right and exited in +reverse: + +```python +with open("in.txt") as src, open("out.txt", "w") as dst: + dst.write(src.read()) +``` + +For async resources (DB connections, HTTP sessions) the protocol is `__aenter__` / +`__aexit__`, used with `async with`; `contextlib.asynccontextmanager` is the generator +equivalent. See [asyncio concurrency](asyncio-concurrency.md). + +## Key Takeaways + +- `with` = guaranteed `__exit__`, including on exceptions and early `return`. +- `__exit__` returning truthy **swallows** the exception — do it deliberately. +- Prefer `@contextmanager` for simple cases; always wrap `yield` in `try`/`finally` when + cleanup must run on error. +- Reach for `ExitStack` when the number of resources is only known at runtime. + +## Related Articles + +- [Decorators](decorators.md) — `@contextmanager` is itself a decorator. +- [Metaprogramming & Dunder Methods](metaprogramming-dunder-methods.md) — the data model + protocol `__enter__`/`__exit__` belong to. diff --git a/docs/knowledge/python/decorators.md b/docs/knowledge/python/decorators.md new file mode 100644 index 0000000..3dd1cc4 --- /dev/null +++ b/docs/knowledge/python/decorators.md @@ -0,0 +1,122 @@ +--- +tags: + +- python +- decorators +- functools +- closures + +--- + +# Decorators in Python + +A decorator is a callable that takes a function (or class) and returns a replacement, +usually a wrapper that adds behavior. The `@` syntax is sugar: + +```python +@decorator +def f(): ... + +# is exactly +def f(): ... +f = decorator(f) +``` + +It happens once, at definition time. Decorators rely on +[closures](closures-lexical-scope.md) to remember the original function. + +## A Well-Behaved Decorator + +```python +import functools +import time + + +def timed(func): + @functools.wraps(func) + def wrapper(*args, **kwargs): + start = time.perf_counter() + try: + return func(*args, **kwargs) + finally: + print(f"{func.__name__} took {time.perf_counter() - start:.3f}s") + + return wrapper +``` + +Always use **`functools.wraps`**: it copies `__name__`, `__doc__`, `__module__`, and sets +`__wrapped__`, so debugging, introspection, and `help()` still point at the original +function instead of "wrapper". Forward `*args, **kwargs` so any signature works, and +return the result. + +## Decorators With Arguments + +`@retry(times=3)` means `retry(times=3)` is called first and must *return* the real +decorator — so you need three nested levels: + +```python +def retry(times: int = 3, exceptions=(Exception,)): + def decorator(func): + @functools.wraps(func) + def wrapper(*args, **kwargs): + for attempt in range(1, times + 1): + try: + return func(*args, **kwargs) + except exceptions: + if attempt == times: + raise + return wrapper + return decorator + + +@retry(times=5, exceptions=(ConnectionError,)) +def fetch(): ... +``` + +## Stacking and Order + +Decorators apply bottom-up (closest to the function first) but the resulting wrappers run +top-down: + +```python +@a +@b +def f(): ... # f = a(b(f)); calling f() enters a's wrapper first +``` + +Order matters: `@login_required` above `@cache` means auth is checked before the cache; +reverse them and cached results skip the check. + +## Class-Based and Class Decorators + +A class with `__call__` can be a decorator, which is convenient when it needs state +(call counts, a cache). A decorator can also receive a *class* and return it modified +(e.g. `@dataclass`) — often a lighter alternative to a metaclass; see +[Metaprogramming & Dunder Methods](metaprogramming-dunder-methods.md). + +## Decorating Methods and Async Functions + +- A plain function wrapper works on methods because `self` arrives in `*args`. +- For `async def`, the wrapper must itself be `async def` and `await` the call — + otherwise you return an un-awaited coroutine and skip timing/retry logic entirely. + +## Built-in Decorators Worth Knowing + +- `@property`, `@staticmethod`, `@classmethod` +- `@functools.lru_cache` / `@functools.cache` — memoization (arguments must be hashable; + on methods it also keeps `self` alive) +- `@functools.singledispatch` — dispatch on argument type +- `@contextlib.contextmanager` — see [context managers](context-managers.md) +- `@dataclasses.dataclass` + +## Key Takeaways + +- `@dec` is `f = dec(f)`, run at definition time. +- Use `functools.wraps`, forward `*args, **kwargs`, return the result. +- Decorator with arguments = a factory returning the decorator (three levels). +- Stacking applies bottom-up, executes top-down. + +## Related Articles + +- [Closures and Lexical Scope](closures-lexical-scope.md) +- [Context Managers](context-managers.md) diff --git a/docs/knowledge/python/garbage-collection-cycles.md b/docs/knowledge/python/garbage-collection-cycles.md new file mode 100644 index 0000000..a70a1a2 --- /dev/null +++ b/docs/knowledge/python/garbage-collection-cycles.md @@ -0,0 +1,119 @@ +--- +tags: + +- python +- garbage-collection +- memory +- cpython + +--- + +# Cyclic Garbage Collector in Python + +CPython manages memory with two cooperating mechanisms: **reference counting**, which +handles almost everything immediately, and a **cyclic garbage collector**, which exists +for one case reference counting cannot solve — reference cycles. + +## Reference Counting + +Every object carries a count of references to it. When it drops to zero the object is +freed on the spot (and `__del__` runs). This is deterministic: leaving a function scope +frees its locals right away. + +```python +import sys +x = [] +sys.getrefcount(x) # 2: `x` plus the temporary argument +``` + +## Where It Fails: Cycles + +```python +class Node: + def __init__(self): + self.other = None + +a, b = Node(), Node() +a.other = b +b.other = a +del a, b # counts are still 1 each -> never reach zero +``` + +Each object keeps the other alive although nothing outside can reach them. Self-references +(`lst.append(lst)`) and closures/callbacks that point back at their owner create the same +problem. + +## How the Cycle Collector Works + +The `gc` module's collector only tracks **container objects** (lists, dicts, instances, +etc. — things that can hold references). Atomic types like `int` and `str` can't form +cycles and aren't tracked. It finds garbage roughly like this: + +1. For each tracked object, copy its refcount. +1. Subtract references that come *from other tracked objects* (internal references). +1. Anything left with a positive count is referenced from outside the set → reachable. +1. Everything reachable from those is kept; the rest is unreachable cyclic garbage and is + freed. + +### Generations + +Most objects die young, so collecting everything each time would waste work. Objects are +grouped into three **generations**: + +- Gen 0 — new objects; collected most often. +- Gen 1 / Gen 2 — survivors get promoted; collected progressively less often. + +A collection of a generation is triggered when the count of allocations minus +deallocations exceeds that generation's threshold. (Newer CPython versions have adjusted +the exact scheme, so treat the numbers as tunable, not fixed.) + +```python +import gc + +gc.get_threshold() # e.g. (700, 10, 10) +gc.collect() # force a full collection, returns number unreachable found +gc.disable() # only do this if you know you create no cycles +gc.get_stats() +``` + +## Finalizers and `weakref` + +Since Python 3.4 (PEP 442), objects with `__del__` in cycles *can* be collected, but +finalizer timing is still unpredictable — don't rely on `__del__` for releasing resources; +use [context managers](context-managers.md). + +Break cycles by design with **weak references**, which don't increase the refcount: + +```python +import weakref + +class Child: + def __init__(self, parent): + self.parent = weakref.ref(parent) # call self.parent() to dereference +``` + +`weakref.WeakValueDictionary` and `WeakSet` are handy for caches and registries that +shouldn't keep objects alive. + +## Practical Notes + +- Memory "leaks" in Python are usually lingering references (globals, caches, closures, + stored exceptions/tracebacks), not GC failures. `gc.get_referrers`, `tracemalloc`, and + `objgraph` help track them down. +- Large latency-sensitive services sometimes `gc.freeze()` after startup or tune + thresholds to avoid long pauses, but measure first. +- Freed memory isn't always returned to the OS; the allocator (pymalloc) keeps arenas. +- Other implementations differ: PyPy uses tracing GC and no refcounts, so don't depend on + immediate destruction. + +## Key Takeaways + +- Refcounting frees most objects instantly; the cyclic GC is a backstop for cycles. +- Only container objects are tracked; collection is generational. +- Avoid cycles with `weakref`; don't use `__del__` for resource cleanup. + +## Related Articles + +- [GIL, Threading, Multiprocessing and the Memory Model](gil-threading-multiprocessing.md) +- [Closures and Lexical Scope](closures-lexical-scope.md) — closures are a common source + of hidden references. diff --git a/docs/knowledge/python/gil-threading-multiprocessing.md b/docs/knowledge/python/gil-threading-multiprocessing.md new file mode 100644 index 0000000..db6c997 --- /dev/null +++ b/docs/knowledge/python/gil-threading-multiprocessing.md @@ -0,0 +1,120 @@ +--- +tags: + +- python +- gil +- threading +- multiprocessing +- concurrency +- memory-model + +--- + +# GIL, Threading, Multiprocessing and the Memory Model + +This is the deeper companion to the short section in +[Python Tips & Tricks](python-tips.md#concurrency-and-parallelism). It covers *why* the +GIL exists, what it does and doesn't protect, and how memory is shared (or not) under +threads versus processes. + +## Why the GIL Exists + +CPython's memory management uses non-atomic reference counting. If two threads changed a +refcount simultaneously the count could be corrupted, leading to leaks or use-after-free. +The **Global Interpreter Lock** is one mutex that only lets one thread execute Python +bytecode at a time, making refcounts and interpreter internals safe without fine-grained +locks — and keeping single-threaded code and C extensions simple and fast. + +Trade-off: no CPU parallelism across threads in one process. + +## When the GIL Is Released + +- Blocking I/O: sockets, file reads, `time.sleep`. +- Many C extensions during heavy work (NumPy, hashing, compression, `re` in some cases). +- The interpreter forces a switch periodically (`sys.getswitchinterval()`, 5 ms by + default) so a CPU-bound thread can't starve others forever. + +So threads help for **I/O-bound** work and for C code that releases the lock, but not for +pure-Python CPU-bound loops. + +## The GIL Does Not Make Your Code Thread-Safe + +A single bytecode is atomic, but a statement usually isn't: + +```python +counter = 0 + +def work(): + global counter + for _ in range(100_000): + counter += 1 # load, add, store -> a switch can happen between them +``` + +Run in several threads, the final `counter` can fall short. Use `threading.Lock`, or +better, `queue.Queue` and immutable data. Things like `list.append` or `dict[key] = v` are +atomic in CPython, but relying on that is fragile — compound "check then act" sequences +still race. + +## Threads vs. Processes + +| | `threading` | `multiprocessing` | +| --- | --- | --- | +| Memory | Shared address space | Separate memory per process | +| GIL | One shared | One per process | +| CPU-bound speedup | No (pure Python) | Yes | +| Startup / overhead | Cheap | Heavy (fork/spawn) | +| Data exchange | Direct object access | Pickled through pipes/queues or shared memory | +| Failure isolation | A crash can take down all | Process dies alone | + +```python +from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor + +with ThreadPoolExecutor(max_workers=16) as pool: # I/O-bound + pages = list(pool.map(fetch, urls)) + +with ProcessPoolExecutor() as pool: # CPU-bound + results = list(pool.map(crunch, chunks)) +``` + +`concurrent.futures` gives both a single interface; prefer it over raw `Thread`/`Process` +unless you need fine control. + +## Multiprocessing Memory Model + +- Each child has its own copy of memory. Arguments and return values are **pickled**, so + they must be picklable and large payloads are expensive. +- Start method matters: `fork` (Linux default in older versions) copies the parent + copy-on-write — fast, but unsafe with threads or held locks; `spawn` (macOS/Windows + default) starts a fresh interpreter and re-imports your module, so guard entry points + with `if __name__ == "__main__":`. +- To share data without copying, use `multiprocessing.shared_memory`, `Value`/`Array`, or + a manager; otherwise pass messages. +- Note that reference counting writes to objects, so "copy-on-write" pages get copied + anyway as children touch shared objects. + +## Choosing a Model + +- Waiting on network/disk, moderate concurrency → threads. +- Massive number of concurrent connections → [asyncio](asyncio-concurrency.md). +- Heavy pure-Python computation → processes (or push work into NumPy/C/Rust). +- Mixed → asyncio or threads for I/O, `run_in_executor` with a process pool for CPU. + +## The GIL's Future + +Python 3.13 introduced an experimental **free-threaded** build (PEP 703) that can run +without the GIL, and later releases continue to develop it. It requires extension +compatibility and has some single-thread overhead, so the default builds still have the +GIL. Even without it, shared mutable state still needs locks. + +## Key Takeaways + +- The GIL protects the interpreter's internals, not your data. +- Threads for I/O, processes for CPU; asyncio for many connections. +- Processes don't share memory: pay for pickling or use explicit shared memory. +- Always protect compound operations on shared state with locks or queues. + +## Related Articles + +- [Python Tips & Tricks](python-tips.md) +- [Designing Concurrent Tasks with asyncio](asyncio-concurrency.md) +- [Cyclic Garbage Collector](garbage-collection-cycles.md) diff --git a/docs/knowledge/python/index.md b/docs/knowledge/python/index.md index e293b3d..d695c0c 100644 --- a/docs/knowledge/python/index.md +++ b/docs/knowledge/python/index.md @@ -28,6 +28,18 @@ Python day to day. model, `TaskGroup` vs. `gather`, cancellation, and the synchronization primitives. - [Logging & Tracing in Python](logging-and-tracing.md) — the `logging` module done right, `contextvars` for async-safe request context, and distributed tracing basics. +- [Context Managers](context-managers.md) — the `with` protocol, `@contextmanager`, + and `ExitStack`. +- [Closures and Lexical Scope](closures-lexical-scope.md) — LEGB, cells, `nonlocal`, and + late-binding gotchas. +- [Decorators](decorators.md) — `functools.wraps`, decorators with arguments, stacking + order, and async. +- [Multiple Inheritance & MRO](multiple-inheritance-mro.md) — C3 linearization, what + `super()` really does, and mixins. +- [Cyclic Garbage Collector](garbage-collection-cycles.md) — reference counting, cycles, + generations, and `weakref`. +- [GIL, Threading, Multiprocessing & Memory Model](gil-threading-multiprocessing.md) — + why the GIL exists, thread safety, and sharing memory across processes. ## Contributing diff --git a/docs/knowledge/python/multiple-inheritance-mro.md b/docs/knowledge/python/multiple-inheritance-mro.md new file mode 100644 index 0000000..b0d1eaa --- /dev/null +++ b/docs/knowledge/python/multiple-inheritance-mro.md @@ -0,0 +1,116 @@ +--- +tags: + +- python +- oop +- inheritance +- mro + +--- + +# Multiple Inheritance and MRO + +Python lets a class inherit from several parents. That raises the question: when a method +exists in more than one ancestor, which wins, and what does `super()` mean? The answer is +the **Method Resolution Order (MRO)** — a single linear ordering of a class's ancestors. + +## The Diamond Problem + +```python +class A: + def hello(self): print("A") + +class B(A): + def hello(self): print("B"); super().hello() + +class C(A): + def hello(self): print("C"); super().hello() + +class D(B, C): + def hello(self): print("D"); super().hello() + +D().hello() # D, B, C, A +``` + +`A` is reached only once, and `C` runs between `B` and `A`, even though `B` inherits +directly from `A`. Inspect the order with: + +```python +D.__mro__ # (D, B, C, A, object) +D.mro() +``` + +## C3 Linearization + +Python computes the MRO with the **C3 algorithm**, which guarantees: + +1. A class comes before its parents. +1. The order of bases in the `class` statement is preserved (`D(B, C)` → B before C). +1. The ordering is monotonic — a subclass never reorders what its parents established. + +Formally, `L[D] = D + merge(L[B], L[C], [B, C])`. If no consistent order exists Python +raises `TypeError: Cannot create a consistent method resolution order` +(e.g. `class X(A, B)` where `B` subclasses `A`: base order contradicts the hierarchy). + +## What `super()` Really Does + +`super()` does **not** mean "my parent". It means "the next class after me in the MRO of +`type(self)`". In the example above, `B`'s `super().hello()` calls `C.hello`, not +`A.hello`, because `self` is a `D`. + +Consequences: + +- Every class in a cooperative hierarchy must call `super()` so the chain isn't cut off. +- Signatures must be compatible. The idiom is to accept and forward `**kwargs`: + +```python +class Base: + def __init__(self, **kwargs): + super().__init__(**kwargs) + +class Named(Base): + def __init__(self, name, **kwargs): + self.name = name + super().__init__(**kwargs) +``` + +- Calling `Parent.__init__(self)` explicitly bypasses the MRO and can run a shared base + twice or skip siblings — avoid it in multi-inheritance code. + +## Mixins: The Practical Use + +Most healthy multiple inheritance is the **mixin**: a small class with one focused +behavior, no state of its own, not meant to be instantiated alone. + +```python +class JSONMixin: + def to_json(self): + import json + return json.dumps(self.__dict__) + +class User(JSONMixin, Base): ... +``` + +Put mixins **before** the main base so their overrides win. Name them `*Mixin` so intent +is clear. + +## Guidelines + +- Prefer composition; use inheritance for "is-a" and mixins for orthogonal capabilities. +- Keep hierarchies shallow, and avoid diamonds unless they're cooperative by design. +- When debugging surprise behavior, print `Cls.__mro__` first. +- `isinstance`/`issubclass` follow the same ancestry; ABCs and `Protocol` are alternatives + when you only need an interface (see + [Dependency Injection with Protocols](../architecture/patterns/dependency-injection-protocols.md)). + +## Key Takeaways + +- MRO is a C3-computed linear order; read it from `__mro__`. +- `super()` = next in the MRO of the instance's type, not the literal parent. +- Cooperative multiple inheritance needs every class to call `super()` and forward + `**kwargs`. + +## Related Articles + +- [Metaprogramming & Dunder Methods](metaprogramming-dunder-methods.md) — `__init_subclass__` + and metaclasses, which also participate in class construction.