From 0e49a36db23b00ef7056ab4e95f9348f87df1b00 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:02:22 +0000 Subject: [PATCH 1/8] Add ADR-054: lean indexer processes and a polite background summarizer Co-Authored-By: Claude [ADR-054] --- docs/adr/ADR-054-lean-and-polite-processes.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/adr/ADR-054-lean-and-polite-processes.md diff --git a/docs/adr/ADR-054-lean-and-polite-processes.md b/docs/adr/ADR-054-lean-and-polite-processes.md new file mode 100644 index 0000000..fcf5149 --- /dev/null +++ b/docs/adr/ADR-054-lean-and-polite-processes.md @@ -0,0 +1,74 @@ +# ADR-054: Lean Indexer Processes and a Polite Background Summarizer + +**Status:** proposed +**Date:** 2026-10-20 +**Branch:** `feature/adr-054-process-footprint` +**Reviewer:** @edb +**Backlog:** [B-062](../backlog.md#b-062) (OpenBLAS thread buffers), [B-013](../backlog.md#b-013) (the summarizer competes with foreground work) +**Depends on:** none +**Depended on by:** none yet + +## Context + +**B-062.** On @edb's machine every indexer process commits about 3.6 GB of private memory before it +loads a model, and 2.1 GB of that is `import faiss`. The faiss-cpu wheel bundles OpenBLAS, which +allocates a buffer for every thread it might use (14 there) when the library loads. `core.py` pins +faiss's OpenMP pool to one thread, but only after the import, so the buffers are already committed. +Measured by @edb with `GetProcessMemoryInfo().PrivateUsage` on Windows, Python 3.14, faiss-cpu 1.14.3: + +| `OPENBLAS_NUM_THREADS` | `import faiss` | `import core` | +|---|---|---| +| unset | 2.10 GB | 3.58 GB | +| 4 | 0.54 GB | 1.69 GB | +| 1 | 0.07 GB | 1.12 GB | + +Three MCP servers and a model host held about 17 GB of commit between them, and Windows grew +`pagefile.sys` to 58 GB to back it. + +**B-013.** With `[summarization].enabled = true`, every reindex generates a summary per new chunk with +a 1.5B model. That is the indexer's dominant cost, and it runs at normal priority beside whatever the +developer is doing. Summarization runs in one of two places: + +- **the model host** (ADR-028), the default on a GPU (`[model_host].enabled = "auto"`). One scheduler + thread runs both summaries and every query embed, one at a time, and yields summaries at a batch + boundary when an embed arrives; +- **the isolated summarizer subprocess** (`IsolatedChunkSummarizer`, a one-worker + `ProcessPoolExecutor`), used when the host is off (the default on CPU) and as the fallback when no + host can be used (ADR-048). + +## Decision + +1. **`OPENBLAS_NUM_THREADS` defaults to 1 before the first numpy or faiss import, in every process.** + A new leaf module, `src/blas_threads.py`, does `os.environ.setdefault("OPENBLAS_NUM_THREADS", "1")` + and nothing else. It is the first import of `core.py` and of each entry module that could pull in + numpy or faiss before `core`: `MCPServer.py`, `incremental_indexer.py`, `hybrid_retriever.py` and + `model_host.py`. `setdefault` keeps an explicit value. Child processes (the model host, the + summarizer worker) inherit the variable through the environment. +2. **The isolated summarizer worker lowers its own priority when it starts**: `os.nice(10)` on POSIX, + `SetPriorityClass(BELOW_NORMAL_PRIORITY_CLASS)` on Windows, before it imports torch. Not + configurable: nothing interactive ever waits on that process. +3. **The model host keeps normal priority.** Its one scheduler thread also embeds every search query. + See the alternatives below and the open questions. + +## Consequences + +To be completed as the build lands. + +## Alternatives Considered + +| Option | Why rejected | +|--------|-------------| +| Set the variable in `core.py` only | Too late wherever numpy or faiss is imported before `core`, as `incremental_indexer.py` and `hybrid_retriever.py` do. | +| `faiss.omp_set_num_threads` / `threadpoolctl` after import | The buffers are committed when the library loads; nothing after the import gets them back. | +| Lower the whole model host (`BELOW_NORMAL_PRIORITY_CLASS` in `model_client._spawn`) | The same thread embeds search queries, which a user waits on. Under foreground CPU load a query would wait too. | +| Lower only the summarizing thread in the host | See the Implementation Log. | + +## Implementation Log + +- [ ] B-062: leaf module, imported first in `core` and the entry modules +- [ ] B-062: subprocess test that the variable is set before numpy/faiss on every entry point +- [ ] B-062: Linux memory and FAISS-speed measurements, and a `tools/` probe for Windows +- [ ] B-013: lower the isolated summarizer worker; tests for both OS branches +- [ ] Stretch: lazy `sentence_transformers`/`transformers` imports in `core` + +**Notes:** From 5dcc26c8ac94de53a434406bc5e470ed5a194077 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:31:13 +0000 Subject: [PATCH 2/8] Set OPENBLAS_NUM_THREADS before numpy or faiss loads in every process OpenBLAS sizes its per-thread buffers when the library loads, so the faiss.omp_set_num_threads(1) call in core came too late (B-062). A new leaf module, blas_threads, defaults the variable to 1 with setdefault, and core and every entry module import it first. A subprocess test checks the variable's value at the moment numpy and faiss are first looked up, for each entry point. Co-Authored-By: Claude [ADR-054 src/MCPServer.py src/blas_threads.py src/core.py src/hybrid_retriever.py src/incremental_indexer.py src/model_host.py] --- src/MCPServer.py | 1 + src/blas_threads.py | 17 +++++ src/core.py | 1 + src/hybrid_retriever.py | 1 + src/incremental_indexer.py | 1 + src/model_host.py | 1 + tests/test_footprint_blas_threads.py | 94 ++++++++++++++++++++++++++++ 7 files changed, 116 insertions(+) create mode 100644 src/blas_threads.py create mode 100644 tests/test_footprint_blas_threads.py diff --git a/src/MCPServer.py b/src/MCPServer.py index b0aeb0b..1734a23 100644 --- a/src/MCPServer.py +++ b/src/MCPServer.py @@ -1,3 +1,4 @@ +import blas_threads # noqa: F401 — B-062: before anything imports numpy or faiss import contextvars import functools import os diff --git a/src/blas_threads.py b/src/blas_threads.py new file mode 100644 index 0000000..0bc623e --- /dev/null +++ b/src/blas_threads.py @@ -0,0 +1,17 @@ +"""blas_threads.py — one OpenBLAS thread per process unless the user says otherwise (ADR-054, B-062). + +OpenBLAS allocates a buffer for every thread it might use when the library loads, +not when it first runs. The faiss-cpu wheel bundles OpenBLAS, so ``import faiss`` +alone committed 2.10 GB on a 14-thread Windows machine, against 0.07 GB at one +thread. ``faiss.omp_set_num_threads`` runs after the import and cannot get it back. + +Import this module before anything that imports numpy or faiss: it is the first +import of ``core`` and of every entry module. ``setdefault`` keeps an explicit +``OPENBLAS_NUM_THREADS``. Child processes (the model host, the summarizer worker) +inherit the variable through the environment. + +A leaf: ``os`` only. Importing anything else here could pull numpy in first. +""" +import os + +os.environ.setdefault("OPENBLAS_NUM_THREADS", "1") diff --git a/src/core.py b/src/core.py index 314fc6a..c17c569 100644 --- a/src/core.py +++ b/src/core.py @@ -1,3 +1,4 @@ +import blas_threads # noqa: F401 — B-062: OPENBLAS_NUM_THREADS before faiss and numpy load import os import sqlite3 import faiss diff --git a/src/hybrid_retriever.py b/src/hybrid_retriever.py index 085f721..98edf36 100644 --- a/src/hybrid_retriever.py +++ b/src/hybrid_retriever.py @@ -38,6 +38,7 @@ """ from __future__ import annotations +import blas_threads # noqa: F401 — B-062: before anything imports numpy or faiss import logging import os import re diff --git a/src/incremental_indexer.py b/src/incremental_indexer.py index 314d676..608cc06 100644 --- a/src/incremental_indexer.py +++ b/src/incremental_indexer.py @@ -63,6 +63,7 @@ """ from __future__ import annotations +import blas_threads # noqa: F401 — B-062: before anything imports numpy or faiss import argparse import os import subprocess diff --git a/src/model_host.py b/src/model_host.py index 0ada7f0..8f2baab 100644 --- a/src/model_host.py +++ b/src/model_host.py @@ -34,6 +34,7 @@ """ from __future__ import annotations +import blas_threads # noqa: F401 — B-062: before torch or core pulls in numpy import base64 import heapq import hmac diff --git a/tests/test_footprint_blas_threads.py b/tests/test_footprint_blas_threads.py new file mode 100644 index 0000000..fe759fe --- /dev/null +++ b/tests/test_footprint_blas_threads.py @@ -0,0 +1,94 @@ +"""ADR-054 (B-062): OPENBLAS_NUM_THREADS is set before numpy or faiss loads, in every entry point. + +Each case imports one module in a fresh interpreter with a finder on sys.meta_path +that records the variable's value at the moment numpy, faiss or torch is first +looked up, which is when OpenBLAS would allocate its per-thread buffers. +""" +import json +import os +import subprocess +import sys + +import pytest + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_SRC = os.path.join(_ROOT, "src") + +_PROBE = r""" +import json, os, sys +seen = {} + +class Recorder: + def find_spec(self, name, path=None, target=None): + if name in ("numpy", "faiss", "torch") and name not in seen: + seen[name] = os.environ.get("OPENBLAS_NUM_THREADS") + return None + +sys.meta_path.insert(0, Recorder()) +sys.path.insert(0, sys.argv[1]) +for statement in sys.argv[2:]: + exec(statement) +print(json.dumps({"seen": seen, "after": os.environ.get("OPENBLAS_NUM_THREADS")})) +""" + + +def _probe(*statements: str, env_value: str | None = None) -> dict: + env = dict(os.environ) + env.pop("OPENBLAS_NUM_THREADS", None) + env["CODE_INDEXER_DEVICE"] = "cpu" + if env_value is not None: + env["OPENBLAS_NUM_THREADS"] = env_value + proc = subprocess.run([sys.executable, "-c", _PROBE, _SRC, *statements], env=env, + capture_output=True, text=True, encoding="utf-8", timeout=300) + assert proc.returncode == 0, proc.stderr[-3000:] + return json.loads(proc.stdout.strip().splitlines()[-1]) + + +# Every module a process can start from: the two console scripts, the model host +# (started as a subprocess), the retriever the eval tools import first, and core. +ENTRY_MODULES = ["core", "MCPServer", "incremental_indexer", "hybrid_retriever", "model_host"] + + +@pytest.mark.parametrize("module", ENTRY_MODULES) +def test_variable_is_set_before_numpy_and_faiss_load(module): + out = _probe(f"import {module}") + assert out["after"] == "1" + for lib, value in out["seen"].items(): + assert value == "1", f"{lib} loaded with OPENBLAS_NUM_THREADS={value!r} via {module}" + + +@pytest.mark.parametrize("module", ["core", "MCPServer", "incremental_indexer", "hybrid_retriever"]) +def test_entry_modules_do_load_numpy_and_faiss(module): + """Keeps the test above honest: these modules really do pull both libraries in.""" + out = _probe(f"import {module}") + assert {"numpy", "faiss"} <= set(out["seen"]) + + +def test_model_host_sets_it_before_its_lazy_imports(): + out = _probe("import model_host", "import faiss, numpy") + assert out["seen"] == {"faiss": "1", "numpy": "1"} + + +def test_tui_launches_through_mcpserver_with_it_set(): + out = _probe("import tui.backend", "import MCPServer") + assert out["seen"].get("numpy") == "1" + + +@pytest.mark.parametrize("module", ["core", "MCPServer"]) +def test_an_explicit_value_is_kept(module): + out = _probe(f"import {module}", env_value="4") + assert out["after"] == "4" + assert set(out["seen"].values()) == {"4"} + + +def test_the_probe_catches_numpy_imported_first(): + """Negative control: importing numpy before core is the bug this guards against.""" + out = _probe("import numpy", "import core") + assert out["seen"]["numpy"] is None + assert out["after"] == "1" + + +def test_blas_threads_is_a_leaf(): + """It must not import anything that could load numpy before it sets the variable.""" + out = _probe("import blas_threads", "import sys; assert 'numpy' not in sys.modules") + assert out["after"] == "1" and out["seen"] == {} From 1c1d48bd6c1498dcdda1819231930116549eafa9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:31:13 +0000 Subject: [PATCH 3/8] Run the isolated summarizer worker at below-normal priority The worker's initializer now drops its CPU priority before it loads torch: os.nice(10) on POSIX, SetPriorityClass(BELOW_NORMAL) on Windows (B-013). The model host keeps normal priority because its one thread also embeds search queries. Co-Authored-By: Claude [ADR-054 src/summarizer.py] --- src/summarizer.py | 40 +++++- tests/test_footprint_summarizer_priority.py | 134 ++++++++++++++++++++ 2 files changed, 173 insertions(+), 1 deletion(-) create mode 100644 tests/test_footprint_summarizer_priority.py diff --git a/src/summarizer.py b/src/summarizer.py index f0f6aec..99b12f1 100644 --- a/src/summarizer.py +++ b/src/summarizer.py @@ -57,6 +57,8 @@ import atexit import logging +import os +import sys import time from dataclasses import asdict, dataclass, fields from typing import Callable, Sequence @@ -292,6 +294,40 @@ def _worker_init(model_id: str, device: str, dtype_str: str) -> None: print(" [Summarizer] Ready.", flush=True) +# ADR-054 (B-013): how far the isolated worker drops its CPU priority. Nothing waits +# on it interactively, so it should yield to the developer's foreground work rather +# than finish first. Below-normal on Windows; nice 10 (what `nice` uses) on POSIX. +_WORKER_NICE = 10 +_BELOW_NORMAL_PRIORITY_CLASS = 0x00004000 + + +def _lower_priority() -> None: + """Drop this process to below-normal CPU priority. Never raises. + + Runs in the worker before torch loads, so the threads torch starts inherit it: + on Linux a nice value belongs to the thread that set it and the threads it + starts afterwards. The model host never calls this. Its one scheduler thread + also embeds search queries, which someone is waiting on (ADR-054 §3). + """ + try: + if sys.platform == "win32": + import ctypes + kernel32 = ctypes.windll.kernel32 + if not kernel32.SetPriorityClass(kernel32.GetCurrentProcess(), + _BELOW_NORMAL_PRIORITY_CLASS): + raise OSError(f"SetPriorityClass failed (error {kernel32.GetLastError()})") + else: + os.nice(_WORKER_NICE) + except Exception as exc: # noqa: BLE001 — a worker at normal priority still works + logger.warning("summarizer worker could not lower its priority: %s", exc) + + +def _isolated_worker_init(model_id: str, device: str, dtype_str: str) -> None: + """IsolatedChunkSummarizer's initializer: lower the worker's priority, then load.""" + _lower_priority() + _worker_init(model_id, device, dtype_str) + + def _messages(code: str) -> list[dict]: return [ {"role": "system", "content": _SYSTEM_PROMPT}, @@ -592,6 +628,8 @@ class IsolatedChunkSummarizer: • An OOM or segfault in the worker raises BrokenProcessPool (a Python exception) in the caller — caught here, summarization is disabled gracefully, and indexing continues for all remaining files. + • The worker runs at below-normal CPU priority (ADR-054), so a reindex's + summaries yield to the developer's foreground work. dtype defaults to "float16" to halve model RAM on CPU (~3 GB vs ~6 GB for 1.5B float32). transformers supports float16 inference on CPU. @@ -640,7 +678,7 @@ def _ensure_executor(self) -> None: from concurrent.futures import ProcessPoolExecutor self._executor = ProcessPoolExecutor( max_workers=1, - initializer=_worker_init, + initializer=_isolated_worker_init, initargs=(self._model_id, self._device, self._dtype), ) diff --git a/tests/test_footprint_summarizer_priority.py b/tests/test_footprint_summarizer_priority.py new file mode 100644 index 0000000..223796e --- /dev/null +++ b/tests/test_footprint_summarizer_priority.py @@ -0,0 +1,134 @@ +"""ADR-054 (B-013): the isolated summarizer worker drops to below-normal priority; the host does not.""" +import concurrent.futures +import os +import subprocess +import sys +import types + +import pytest + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src")) + +import model_client as mc # noqa: E402 +import summarizer as sm # noqa: E402 + +BELOW_NORMAL = 0x00004000 + + +class _FakeKernel32: + def __init__(self, ok=True): + self.ok = ok + self.calls = [] + + def GetCurrentProcess(self): # noqa: N802 — the Win32 name + return -1 + + def SetPriorityClass(self, handle, cls): # noqa: N802 + self.calls.append((handle, cls)) + return 1 if self.ok else 0 + + def GetLastError(self): # noqa: N802 + return 5 + + +@pytest.fixture +def posix(monkeypatch): + calls = [] + monkeypatch.setattr(sys, "platform", "linux") + monkeypatch.setattr(os, "nice", lambda inc: calls.append(inc) or inc) + return calls + + +@pytest.fixture +def windows(monkeypatch): + import ctypes + kernel32 = _FakeKernel32() + monkeypatch.setattr(sys, "platform", "win32") + monkeypatch.setattr(ctypes, "windll", types.SimpleNamespace(kernel32=kernel32), raising=False) + monkeypatch.setattr(os, "nice", lambda inc: pytest.fail("os.nice on Windows"), raising=False) + return kernel32 + + +def test_posix_worker_renices_itself(posix): + sm._lower_priority() + assert posix == [sm._WORKER_NICE] and sm._WORKER_NICE > 0 + + +def test_windows_worker_asks_for_below_normal(windows): + sm._lower_priority() + assert windows.calls == [(-1, BELOW_NORMAL)] + + +def test_windows_failure_is_logged_not_raised(windows, caplog): + windows.ok = False + sm._lower_priority() + assert "could not lower its priority" in caplog.text + + +def test_posix_failure_is_logged_not_raised(monkeypatch, caplog): + monkeypatch.setattr(sys, "platform", "linux") + + def refuse(_inc): + raise PermissionError("not allowed") + monkeypatch.setattr(os, "nice", refuse) + sm._lower_priority() + assert "could not lower its priority" in caplog.text + + +def test_worker_lowers_priority_before_loading_the_model(monkeypatch): + order = [] + monkeypatch.setattr(sm, "_lower_priority", lambda: order.append("lower")) + monkeypatch.setattr(sm, "_worker_init", lambda *a: order.append(("load", a))) + sm._isolated_worker_init("m", "cpu", "float16") + assert order == ["lower", ("load", ("m", "cpu", "float16"))] + + +def test_spawn_path_uses_the_lowering_initializer(monkeypatch): + seen = {} + + class FakePool: + def __init__(self, **kwargs): + seen.update(kwargs) + + def shutdown(self, **_kw): + pass + monkeypatch.setattr(concurrent.futures, "ProcessPoolExecutor", FakePool) + s = sm.IsolatedChunkSummarizer(model_id="m", device="cpu") + s._ensure_executor() + assert seen["initializer"] is sm._isolated_worker_init + assert seen["initargs"] == ("m", "cpu", "float16") + s._shutdown() + + +@pytest.mark.skipif(sys.platform == "win32", reason="os.nice is POSIX") +def test_a_real_worker_runs_niced(): + parent = os.nice(0) + with concurrent.futures.ProcessPoolExecutor(max_workers=1, initializer=sm._lower_priority) as pool: + child = pool.submit(os.nice, 0).result(timeout=60) + assert child == min(parent + sm._WORKER_NICE, 19) + + +# ── the model host keeps normal priority (ADR-054 §3) ────────────────────── + +@pytest.mark.parametrize("platform", ["win32", "linux"]) +def test_model_host_spawn_keeps_normal_priority(platform, monkeypatch, tmp_path): + monkeypatch.setenv("CODE_INDEXER_HOST_DIR", str(tmp_path)) + monkeypatch.setattr(sys, "platform", platform) + for name, value in (("DETACHED_PROCESS", 0x8), ("CREATE_NEW_PROCESS_GROUP", 0x200)): + monkeypatch.setattr(subprocess, name, value, raising=False) + spawned = {} + monkeypatch.setattr(subprocess, "Popen", lambda args, **kw: spawned.update(kw)) + mc._spawn() + spawned["stdout"].close() + assert not spawned.get("creationflags", 0) & BELOW_NORMAL + assert "preexec_fn" not in spawned + + +def test_model_host_summarizer_load_does_not_lower_priority(monkeypatch): + import model_host + monkeypatch.setenv("CODE_INDEXER_DEVICE", "cpu") + monkeypatch.setattr(sm, "_lower_priority", lambda: pytest.fail("the host lowered itself")) + loaded = [] + monkeypatch.setattr(sm, "_worker_init", lambda *a: loaded.append(a)) + model_host.TorchBackend().load(model_host.SUMMARIZER) + assert loaded and loaded[0][1] == "cpu" From 49bfaecc21b7201177e3cd49b1f7b69bc3c4251b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:31:13 +0000 Subject: [PATCH 4/8] Add a probe for per-import memory and flat-search timing Prints PrivateUsage on Windows or VmSize/VmRSS on Linux after each import step, per OPENBLAS_NUM_THREADS setting, and times flat FAISS search with --search, so B-062 can be re-measured with one command. Co-Authored-By: Claude [ADR-054] --- tools/footprint_probe.py | 193 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 tools/footprint_probe.py diff --git a/tools/footprint_probe.py b/tools/footprint_probe.py new file mode 100644 index 0000000..6cdbd47 --- /dev/null +++ b/tools/footprint_probe.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python +"""footprint_probe.py — memory after each import step, per OPENBLAS_NUM_THREADS (ADR-054, B-062). + +Re-measures B-062's table with one command. Each setting runs in a fresh child +process, which imports the same modules in the order ``MCPServer.py`` meets them +and prints its memory after each step: + + Windows PrivateUsage (commit charge) via GetProcessMemoryInfo — what B-062 measured + Linux VmSize and VmRSS from /proc/self/status. Linux overcommits, so neither + is commit charge: VmSize counts reserved address space, VmRSS only + the pages touched. Neither will match a Windows number. + +``--search`` instead times flat FAISS search per setting, the check B-062 asked +for before shipping: a synthetic 50,000 x 1536 float32 IndexFlatIP, 100 queries, +top-50, both as one batch (which goes through BLAS) and one query at a time (what +a search does: one query per tier). + +Usage: + python tools/footprint_probe.py # settings: unset, 4, 1 + python tools/footprint_probe.py --threads unset 1 + python tools/footprint_probe.py --steps faiss core + python tools/footprint_probe.py --search --threads unset 1 + +The ``unset`` child imports numpy/faiss directly, before ``core`` would set the +default, so it shows what the old order cost. Nothing here loads a model. +""" +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_SRC = os.path.join(_ROOT, "src") + +# The order MCPServer.py imports them in (B-062's table), plus numpy on its own. +STEPS = { + "numpy": ["numpy"], + "faiss": ["faiss"], + "torch": ["torch"], + "transformers": ["transformers", "sentence_transformers"], + "core": ["core"], +} + + +def memory() -> dict: + """This process's memory in GB: PrivateUsage on Windows, VmSize/VmRSS on Linux.""" + if sys.platform == "win32": + import ctypes + from ctypes import wintypes + + class Counters(ctypes.Structure): # PROCESS_MEMORY_COUNTERS_EX + _fields_ = [("cb", wintypes.DWORD), ("PageFaultCount", wintypes.DWORD)] + [ + (name, ctypes.c_size_t) for name in ( + "PeakWorkingSetSize", "WorkingSetSize", "QuotaPeakPagedPoolUsage", + "QuotaPagedPoolUsage", "QuotaPeakNonPagedPoolUsage", + "QuotaNonPagedPoolUsage", "PagefileUsage", "PeakPagefileUsage", + "PrivateUsage")] + + counters = Counters() + counters.cb = ctypes.sizeof(counters) + psapi = ctypes.WinDLL("psapi") + kernel32 = ctypes.WinDLL("kernel32") + kernel32.GetCurrentProcess.restype = wintypes.HANDLE + psapi.GetProcessMemoryInfo.argtypes = [wintypes.HANDLE, ctypes.POINTER(Counters), wintypes.DWORD] + if not psapi.GetProcessMemoryInfo(kernel32.GetCurrentProcess(), ctypes.byref(counters), + counters.cb): + raise OSError("GetProcessMemoryInfo failed") + return {"PrivateUsage": counters.PrivateUsage / 2**30, + "WorkingSet": counters.WorkingSetSize / 2**30} + out = {} + with open("/proc/self/status", encoding="utf-8") as fh: + for line in fh: + key, _, value = line.partition(":") + if key in ("VmSize", "VmRSS"): + out[key] = int(value.split()[0]) / 2**20 # kB -> GB + return out + + +def _child(steps: list[str]) -> None: + """Import each step in turn and print one JSON line per step.""" + sys.path.insert(0, _SRC) + print(json.dumps({"step": "bare python", **memory()}), flush=True) + for step in steps: + try: + for module in STEPS[step]: + __import__(module) + except ImportError as exc: + print(json.dumps({"step": step, "error": f"not installed ({exc.name})"}), flush=True) + continue + print(json.dumps({"step": f"import {step}", **memory(), + "torch_loaded": "torch" in sys.modules}), flush=True) + try: # not a dependency; sentence-transformers brings it in through scikit-learn + from threadpoolctl import threadpool_info + except ImportError: + return + # Which BLAS each library loaded and the threads it took. An OpenMP-threaded OpenBLAS + # (faiss-cpu's on Linux) follows OMP_NUM_THREADS and ignores OPENBLAS_NUM_THREADS. + for pool in threadpool_info(): + print(json.dumps({"blas": os.path.basename(pool["filepath"]), "api": pool["internal_api"], + "threads": pool["num_threads"], + "layer": pool.get("threading_layer") or "-"}), flush=True) + + +def _search_child(n: int, dim: int, queries: int, k: int, repeats: int) -> None: + """Time flat search the way core sets faiss up (OMP pinned to 1); print one JSON line.""" + import time + import numpy as np + import faiss + faiss.omp_set_num_threads(1) + rng = np.random.default_rng(0) + xb = rng.standard_normal((n, dim), dtype=np.float32) + xq = rng.standard_normal((queries, dim), dtype=np.float32) + index = faiss.IndexIDMap(faiss.IndexFlatIP(dim)) + index.add_with_ids(xb, np.arange(n, dtype=np.int64)) + del xb + index.search(xq[:2], k) # warm up + + def best(fn) -> float: + times = [] + for _ in range(repeats): + started = time.perf_counter() + fn() + times.append(time.perf_counter() - started) + return min(times) + + batch = best(lambda: index.search(xq, k)) + single = best(lambda: [index.search(xq[i:i + 1], k) for i in range(queries)]) + print(json.dumps({"batch_s": batch, "one_at_a_time_s": single, + "env": os.environ.get("OPENBLAS_NUM_THREADS")}), flush=True) + + +def _run(setting: str, steps: list[str], search: bool = False) -> list[dict]: + env = dict(os.environ) + env.pop("OPENBLAS_NUM_THREADS", None) + if setting != "unset": + env["OPENBLAS_NUM_THREADS"] = setting + child = ["--search-child"] if search else ["--child", *steps] + proc = subprocess.run([sys.executable, os.path.abspath(__file__), *child], + env=env, capture_output=True, text=True, encoding="utf-8") + if proc.returncode != 0: + sys.exit(f"probe child for OPENBLAS_NUM_THREADS={setting} failed:\n{proc.stderr}") + return [json.loads(line) for line in proc.stdout.splitlines() if line.startswith("{")] + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("--threads", nargs="+", default=["unset", "4", "1"], + help='OPENBLAS_NUM_THREADS values to try; "unset" removes it') + ap.add_argument("--steps", nargs="+", default=list(STEPS), choices=list(STEPS)) + ap.add_argument("--search", action="store_true", help="time flat FAISS search instead") + ap.add_argument("--repeats", type=int, default=5, help="--search: best of this many runs") + ap.add_argument("--child", nargs="*", help=argparse.SUPPRESS) + ap.add_argument("--search-child", action="store_true", help=argparse.SUPPRESS) + args = ap.parse_args() + if args.child is not None: + _child(args.child) + return + if args.search_child: + _search_child(50_000, 1536, 100, 50, args.repeats) + return + if args.search: + print(f"python {sys.version.split()[0]} on {sys.platform}, {os.cpu_count()} CPUs; " + "flat IP search, 50,000 x 1536 float32, 100 queries, top-50, best of " + f"{args.repeats}") + for setting in args.threads: + row = _run(setting, [], search=True)[0] + print(f" OPENBLAS_NUM_THREADS={setting:<6} batch of 100 {row['batch_s'] * 1000:8.1f} ms " + f"one at a time {row['one_at_a_time_s'] * 1000:8.1f} ms") + return + + print(f"python {sys.version.split()[0]} on {sys.platform}; memory in GB, cumulative per step") + if sys.platform != "win32": + print("Linux overcommits: VmSize is reserved address space and VmRSS touched pages. " + "Neither is Windows commit charge.") + for setting in args.threads: + print(f"\nOPENBLAS_NUM_THREADS={setting}") + for row in _run(setting, args.steps): + if "blas" in row: + print(f" {row['api']:<8} {row['threads']:>3} threads ({row['layer']}) {row['blas']}") + continue + if "error" in row: + print(f" {row['step']:<22} {row['error']}") + continue + cols = " ".join(f"{k} {v:6.2f}" for k, v in row.items() + if k not in ("step", "torch_loaded")) + print(f" {row['step']:<22} {cols}") + + +if __name__ == "__main__": + main() From bcdae0ed62152247440d3581c71c192ccb7c9ed5 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 20:31:13 +0000 Subject: [PATCH 5/8] Record ADR-054's decisions, measurements and open questions Co-Authored-By: Claude [ADR-054] --- docs/adr/ADR-054-lean-and-polite-processes.md | 180 +++++++++++++++--- 1 file changed, 157 insertions(+), 23 deletions(-) diff --git a/docs/adr/ADR-054-lean-and-polite-processes.md b/docs/adr/ADR-054-lean-and-polite-processes.md index fcf5149..19b635f 100644 --- a/docs/adr/ADR-054-lean-and-polite-processes.md +++ b/docs/adr/ADR-054-lean-and-polite-processes.md @@ -12,9 +12,10 @@ **B-062.** On @edb's machine every indexer process commits about 3.6 GB of private memory before it loads a model, and 2.1 GB of that is `import faiss`. The faiss-cpu wheel bundles OpenBLAS, which -allocates a buffer for every thread it might use (14 there) when the library loads. `core.py` pins +allocates a buffer for each thread it might use (14 there) when the library loads. `core.py` pins faiss's OpenMP pool to one thread, but only after the import, so the buffers are already committed. -Measured by @edb with `GetProcessMemoryInfo().PrivateUsage` on Windows, Python 3.14, faiss-cpu 1.14.3: +@edb measured this with `GetProcessMemoryInfo().PrivateUsage` on Windows, Python 3.14 and +faiss-cpu 1.14.3: | `OPENBLAS_NUM_THREADS` | `import faiss` | `import core` | |---|---|---| @@ -30,29 +31,74 @@ a 1.5B model. That is the indexer's dominant cost, and it runs at normal priorit developer is doing. Summarization runs in one of two places: - **the model host** (ADR-028), the default on a GPU (`[model_host].enabled = "auto"`). One scheduler - thread runs both summaries and every query embed, one at a time, and yields summaries at a batch - boundary when an embed arrives; + thread runs both summaries and every embed, query embeds included, one at a time. It yields + summaries at a batch boundary when an embed arrives. - **the isolated summarizer subprocess** (`IsolatedChunkSummarizer`, a one-worker - `ProcessPoolExecutor`), used when the host is off (the default on CPU) and as the fallback when no - host can be used (ADR-048). + `ProcessPoolExecutor`). It is used when the host is off, which is the default on CPU, and as the + fallback when no host can be used (ADR-048). ## Decision 1. **`OPENBLAS_NUM_THREADS` defaults to 1 before the first numpy or faiss import, in every process.** A new leaf module, `src/blas_threads.py`, does `os.environ.setdefault("OPENBLAS_NUM_THREADS", "1")` - and nothing else. It is the first import of `core.py` and of each entry module that could pull in - numpy or faiss before `core`: `MCPServer.py`, `incremental_indexer.py`, `hybrid_retriever.py` and - `model_host.py`. `setdefault` keeps an explicit value. Child processes (the model host, the - summarizer worker) inherit the variable through the environment. -2. **The isolated summarizer worker lowers its own priority when it starts**: `os.nice(10)` on POSIX, - `SetPriorityClass(BELOW_NORMAL_PRIORITY_CLASS)` on Windows, before it imports torch. Not - configurable: nothing interactive ever waits on that process. -3. **The model host keeps normal priority.** Its one scheduler thread also embeds every search query. - See the alternatives below and the open questions. + and nothing else. It is the first import of `core.py` and of every entry module that could load + numpy or faiss before `core`: + - `MCPServer.py` (`code-indexer-serve`, and the TUI, which drives it); + - `incremental_indexer.py` (`code-indexer`), which imports faiss and numpy above `core`; + - `hybrid_retriever.py`, which does the same and is what the eval tools import first; + - `model_host.py`, which the clients start as a subprocess. + + `setdefault` keeps an explicit value. Child processes (the model host, the summarizer worker) + inherit the variable through the environment. It is not an `indexer.toml` knob: it has to be set + before config could be read, and the environment variable is already the override. +2. **The isolated summarizer worker lowers its own CPU priority when it starts**, before it imports + torch: `os.nice(10)` on POSIX and `SetPriorityClass(BELOW_NORMAL_PRIORITY_CLASS)` on Windows, both + through the standard library (`os`, `ctypes`), with no psutil. It is the pool's initializer + (`_isolated_worker_init`), because `ProcessPoolExecutor` takes no creation flags. A failure is + logged and the worker runs at normal priority. **Not configurable:** nothing interactive ever waits + on that process. +3. **The model host keeps normal priority, for now.** Its one scheduler thread runs summaries and + query embeds, and a user waits on every query embed. Lowering the host would make searches wait + behind the developer's foreground work too. Lowering only while summarizing is practical on + Windows (`SetThreadPriority` goes both ways) but not on Linux, where an unprivileged thread cannot + raise its priority back (`RLIMIT_NICE` defaults to 0). On a GPU, the CPU side of decode is mostly + kernel launches, so CPU priority is the wrong lever anyway; see the open questions. +4. **The stretch goal, loading `sentence_transformers`/`transformers` lazily in `core`, is not + built.** It would save nothing on the pinned transformers 4.53.3: `transformers.utils.generic` + imports torch whenever torch is installed, so `AutoTokenizer` alone loads it. The MCP server counts + tokens with `AutoTokenizer` on every `semantic_code_search` (`_pack_results`) and on every + watchdog reindex (`ast_chunker`), and the shipped `[model_host].enabled = "auto"` imports torch to + check for CUDA. Lazy imports would only move torch's load to the first search. See the open + questions for what would actually remove it. ## Consequences -To be completed as the build lands. +**Better:** +- On @edb's Windows stack the variable is the difference between 2.10 GB and 0.07 GB of commit for + `import faiss`, and about 2.5 GB per process for `import core` (B-062's table). With three MCP + servers, a model host and an indexer run, that is on the order of 12 GB of commit, and the page + file that backs it. **Not yet re-measured on Windows with this branch;** see the open questions. +- A reindex on CPU, the default host-off path, no longer competes at equal priority with the + developer's foreground work for the minutes its summaries take. +- A test now fails if a future import puts numpy or faiss ahead of `blas_threads` in any entry point. + +**Worse:** +- Batched numpy and faiss BLAS work in those processes is single-threaded unless the user sets the + variable. Nothing on the hot path is batched BLAS: a search sends one query per tier, and faiss + only uses BLAS from 20 queries up (`distance_compute_blas_threshold`). The flat-search timings + below show no change on Linux. +- On CPU, summaries take longer when the machine is busy. That is the point of B-013, but a reindex + started during heavy foreground work now finishes later. +- One more module in `src/`, and four entry modules carry an import whose only job is its side + effect. + +**Neutral:** +- torch's own CPU threads are not BLAS (`libgomp`/MKL), so in-process CPU embedding and + summarization keep their parallelism. `OMP_NUM_THREADS` was deliberately not touched: it would also + make torch single-threaded. +- On Linux the faiss-cpu wheel bundles an OpenMP-threaded OpenBLAS, which ignores + `OPENBLAS_NUM_THREADS` (it follows `OMP_NUM_THREADS`). There the variable reaches only numpy's and + scipy's OpenBLAS. @edb's numbers show that the Windows wheel honours it. ## Alternatives Considered @@ -60,15 +106,103 @@ To be completed as the build lands. |--------|-------------| | Set the variable in `core.py` only | Too late wherever numpy or faiss is imported before `core`, as `incremental_indexer.py` and `hybrid_retriever.py` do. | | `faiss.omp_set_num_threads` / `threadpoolctl` after import | The buffers are committed when the library loads; nothing after the import gets them back. | -| Lower the whole model host (`BELOW_NORMAL_PRIORITY_CLASS` in `model_client._spawn`) | The same thread embeds search queries, which a user waits on. Under foreground CPU load a query would wait too. | -| Lower only the summarizing thread in the host | See the Implementation Log. | +| Also set `OMP_NUM_THREADS=1` | It would reach faiss's OpenMP-built OpenBLAS on Linux, but it also makes torch single-threaded: in-process CPU embedding and summarization would slow down several times. | +| A `[performance]` knob in `indexer.toml` | It has to take effect before numpy loads, which is before config is read. The environment variable is already the override. | +| Lower the whole model host (`BELOW_NORMAL_PRIORITY_CLASS` in `model_client._spawn`, `os.nice` in `serve()`) | The same thread embeds search queries. A query would wait behind foreground CPU work too. Kept as the first thing to try if @edb's measurement shows the host summarizing on GPU does hurt foreground work. | +| Lower only the host's thread while it summarizes | Windows can drop and restore a thread's priority; unprivileged Linux cannot restore it. On a GPU the CPU side is mostly kernel launches, so it would buy little. | +| Run the host's summaries in a separate low-priority worker process | Two processes on the card, which is what ADR-028's one-model-at-a-time scheduler exists to prevent. | +| `psutil.Process().nice()` | psutil is not a dependency; `os.nice` and `ctypes` do the same in a few lines. | +| Lazy `sentence_transformers`/`transformers` imports in `core` (the stretch goal) | Saves nothing in a live MCP server on transformers 4.53.3; see Decision §4. | + +## Open questions for @edb + +1. **Re-measure B-062 on Windows** with this branch, one command each: + `python tools/footprint_probe.py` (PrivateUsage per import step, unset/4/1) and + `python tools/footprint_probe.py --search --threads unset 1` (flat-search timing). The probe also + prints each BLAS library's threading layer, which shows whether the Windows faiss wheel's OpenBLAS + is pthreads-threaded (it should be, given your numbers). +2. **B-013 on the GPU path is unmeasured.** It could not be measured here (no GPU). To decide whether + the host should be lowered too: run a full reindex with summaries through the host, and meanwhile + (a) time `semantic_code_search` calls, and (b) run a CPU-heavy foreground job (a build, or + `python -c "while 1: pass"` per core). Compare foreground job time and search latency with the host + at normal priority and at below-normal (Task Manager → Details → Set priority on the host's + `python.exe`). If below-normal costs searches nothing noticeable and helps the foreground job, + lowering the host in `_spawn` is a two-line follow-up. +3. **The CPU host-off path** is what this branch changes. Worth one look: during a CPU reindex with + summaries, the worker `python.exe` should show "Below normal" in Task Manager. +4. **Torch in the MCP server (the stretch goal).** Removing it would take two changes, both outside + this ADR: a torch-free token counter (the `tokenizers` library loading the embedder's + `tokenizer.json` directly, checked for identical counts against `AutoTokenizer` on real chunks, + which needs the model files), and a CUDA check for `[model_host].enabled = "auto"` that does not + import torch (or setting `enabled = true` explicitly on the GPU machine). Worth a backlog item? +5. **faiss's OpenMP pin covers only the thread that imported `core`.** `omp_set_num_threads` sets a + per-thread value, and the MCP server runs tools in worker threads (ADR-047), so their searches use + every core. Measured here: single-query flat search took 2,554 ms per 100 queries on the main + thread and 1,510 ms on a worker thread. That is probably fine and possibly good, but it is not what + B-062 assumed. A backlog item if you want the pin to mean what it says. ## Implementation Log -- [ ] B-062: leaf module, imported first in `core` and the entry modules -- [ ] B-062: subprocess test that the variable is set before numpy/faiss on every entry point -- [ ] B-062: Linux memory and FAISS-speed measurements, and a `tools/` probe for Windows -- [ ] B-013: lower the isolated summarizer worker; tests for both OS branches -- [ ] Stretch: lazy `sentence_transformers`/`transformers` imports in `core` +- [x] B-062: `src/blas_threads.py`, imported first in `core.py`, `MCPServer.py`, + `incremental_indexer.py`, `hybrid_retriever.py` and `model_host.py` (one line each) +- [x] B-062: `tests/test_footprint_blas_threads.py` (15 tests). Each imports an entry module in a + fresh interpreter with a `sys.meta_path` finder recording the variable's value when numpy, faiss or + torch is first looked up. The cases: every entry point, an explicit value kept, a negative control + (numpy imported before `core` is caught) and `blas_threads` staying a leaf. +- [x] B-062: `tools/footprint_probe.py`, with Linux measurements below +- [x] B-013: `_lower_priority` / `_isolated_worker_init` in `summarizer.py`; + `tests/test_footprint_summarizer_priority.py` (10 tests). It covers both OS branches with + `sys.platform`, `os.nice` and `ctypes.windll` mocked, failures logged rather than raised, the + executor built with the lowering initializer, a real worker on Linux reporting nice 10, and the + model host's spawn and summarizer load leaving priority alone on both OS branches. +- [x] Stretch: investigated and not built (Decision §4) **Notes:** + +2026-10-20, **memory, Linux.** Python 3.12.3, faiss-cpu 1.15.1, numpy 2.5.4, torch 2.14.1 (the +PyPI CUDA wheel; the CPU wheel's index was blocked), transformers 4.53.3, sentence-transformers +5.5.0, on a 4-CPU cloud container. `python tools/footprint_probe.py`, cumulative, in GB. **Linux +overcommits, so neither column is Windows commit charge:** VmSize is reserved address space, VmRSS +is pages touched. With 4 CPUs, "unset" and "4" are the same configuration and measured identically. + +| Step | VmSize unset | VmSize 1 | VmRSS unset | VmRSS 1 | +|---|---|---|---|---| +| bare python | 0.02 | 0.02 | 0.01 | 0.01 | +| `import numpy` | 0.21 | 0.10 | 0.03 | 0.03 | +| `import faiss` | 0.79 | 0.67 | 0.04 | 0.04 | +| `import torch` | 3.74 | 3.63 | 0.50 | 0.50 | +| `import transformers, sentence_transformers` | 4.30 | 4.07 | 0.76 | 0.76 | +| `import core` | 4.30 | 4.07 | 0.76 | 0.76 | + +The variable saves 0.23 GB of address space here and no resident memory. That is the expected Linux +result, not evidence against the Windows one: at 4 CPUs there are few buffers to save, Linux backs +them lazily, and faiss's own OpenBLAS here is the OpenMP build that ignores the variable (the +`import faiss` step costs +0.57 GB either way). The 0.11 GB saved at `import numpy` is numpy's +pthreads OpenBLAS, and the rest is scipy's, loaded via sentence-transformers. + +2026-10-20, **flat FAISS search, Linux**, same stack. `python tools/footprint_probe.py --search`: +synthetic `IndexIDMap(IndexFlatIP)`, 50,000 × 1536 float32, 100 queries, top-50, faiss OpenMP +pinned to 1 as `core` does, best of 5, run twice in each order: + +| `OPENBLAS_NUM_THREADS` | batch of 100 queries | 100 single queries | +|---|---|---| +| unset | 589 / 546 ms | 2,698 / 2,733 ms | +| 1 | 592 / 554 ms | 2,669 / 2,727 ms | + +No slowdown. On Linux that is partly by construction, since faiss's BLAS here does not read the +variable. The Windows question is the batch column only: single queries (what a search sends, one +per tier) stay under faiss's BLAS threshold and never reach BLAS. For scale, letting faiss use 4 +threads (threadpoolctl, which on this OpenMP build also lifts the OpenMP pin) gave 151–161 ms for +the batch and 1,701–1,715 ms for single queries. That is what lifting the pin would buy, not what +this variable costs. + +2026-10-20, **surprise:** the Linux faiss-cpu wheel's OpenBLAS (`libopenblaso`, threading layer +`openmp`) reports 4 threads with `OPENBLAS_NUM_THREADS=1` and 1 with `OMP_NUM_THREADS=1` +(threadpoolctl). That is why decision §1 gives no faiss saving on Linux and why `OMP_NUM_THREADS` was +considered and rejected (see the alternatives). + +2026-10-20, **stretch, investigated and dropped.** I built lazy imports in `core` first, with a module +`__getattr__` so tests could still patch `core.SentenceTransformer`, plus a test that `import core` +with the host on loads no torch. It passed, but tracing imports showed `from transformers import +AutoTokenizer` loads torch (`transformers/utils/generic.py:46`), and the MCP server calls it on its +first search. I reverted it rather than ship a change that saves memory only until the first search. From c73134356d95c39ede3b11dc37cb8bd5bf946a0b Mon Sep 17 00:00:00 2001 From: Ethan Blauw Date: Tue, 20 Oct 2026 15:43:02 -0500 Subject: [PATCH 6/8] Let the POSIX priority tests run on Windows, which has no os.nice Co-Authored-By: Claude Opus 5.5 (1M context) [ADR-054] --- tests/test_footprint_summarizer_priority.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/test_footprint_summarizer_priority.py b/tests/test_footprint_summarizer_priority.py index 223796e..44db27b 100644 --- a/tests/test_footprint_summarizer_priority.py +++ b/tests/test_footprint_summarizer_priority.py @@ -35,7 +35,8 @@ def GetLastError(self): # noqa: N802 def posix(monkeypatch): calls = [] monkeypatch.setattr(sys, "platform", "linux") - monkeypatch.setattr(os, "nice", lambda inc: calls.append(inc) or inc) + # raising=False: Windows has no os.nice, and these run there too (B-047) + monkeypatch.setattr(os, "nice", lambda inc: calls.append(inc) or inc, raising=False) return calls @@ -70,7 +71,7 @@ def test_posix_failure_is_logged_not_raised(monkeypatch, caplog): def refuse(_inc): raise PermissionError("not allowed") - monkeypatch.setattr(os, "nice", refuse) + monkeypatch.setattr(os, "nice", refuse, raising=False) sm._lower_priority() assert "could not lower its priority" in caplog.text From e3e9696135005838bf31d7a97450b785089a9fa2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 20 Oct 2026 21:15:11 +0000 Subject: [PATCH 7/8] Log ADR-054's gate results and Inspector check Co-Authored-By: Claude [ADR-054] --- docs/adr/ADR-054-lean-and-polite-processes.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/adr/ADR-054-lean-and-polite-processes.md b/docs/adr/ADR-054-lean-and-polite-processes.md index 19b635f..d86188f 100644 --- a/docs/adr/ADR-054-lean-and-polite-processes.md +++ b/docs/adr/ADR-054-lean-and-polite-processes.md @@ -156,6 +156,16 @@ developer is doing. Summarization runs in one of two places: executor built with the lowering initializer, a real worker on Linux reporting nice 10, and the model host's spawn and summarizer load leaving priority alone on both OS branches. - [x] Stretch: investigated and not built (Decision §4) +- [x] MCP Inspector 2.8.0, run from an empty temp directory: this branch changes no tool, so only + `tools/list --strict` was run to show the server still starts with `blas_threads` as its first + import. It exits 0 with 14 tools, and the JSON is identical to `871b55c`'s listing. +- [x] Gates, 2026-10-20, on the Linux stack above: flake8 clean; 729 tests collected (704 on + `871b55c`); conformance, resolution and L5X fixture-teeth baselines hold. `pytest` gives 679 passed, + 48 failed, 2 skipped, against 654 / 48 / 2 on `871b55c`. The 48 failures are the same tests on + both. The container cannot reach huggingface.co, so these tests cannot load the `BAAI/bge-code-v1` + tokenizer: 31 fail on that `OSError`, and the rest fail on assertions downstream of it. The run used + `HF_HUB_OFFLINE=1`; without it, each of those tests waits out download retries, and the suite did + not finish in 30 minutes. **Notes:** From 9ef269b0041b2a23edc1999286e8dac9731cb54d Mon Sep 17 00:00:00 2001 From: Ethan Blauw Date: Tue, 20 Oct 2026 16:39:47 -0500 Subject: [PATCH 8/8] ADR-054 accepted Co-Authored-By: Claude Opus 5.5 (1M context) [ADR-054] --- docs/adr/ADR-054-lean-and-polite-processes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/ADR-054-lean-and-polite-processes.md b/docs/adr/ADR-054-lean-and-polite-processes.md index d86188f..8c4b6aa 100644 --- a/docs/adr/ADR-054-lean-and-polite-processes.md +++ b/docs/adr/ADR-054-lean-and-polite-processes.md @@ -1,6 +1,6 @@ # ADR-054: Lean Indexer Processes and a Polite Background Summarizer -**Status:** proposed +**Status:** accepted **Date:** 2026-10-20 **Branch:** `feature/adr-054-process-footprint` **Reviewer:** @edb