Write the design and implementation document

[charitarthchugh]

Question

Not a decision — the destination. Write the single design and implementation document that
this map exists to produce, synthesized from every closed ticket.

It must be readable start to finish by someone who has not read the map, and complete enough
to implement from without re-deriving anything. Cover:

• the domain and its vocabulary (cross-reference CONTEXT.md)
• the adapter contract and what it is allowed to know
• chunking, pooling, and the uniform output shape
• both sinks, with their exact formats
• identity and resume, including the user-facing warning about re-OCR
• the TUI panel
• concurrency, batching, and failure handling
• the implementation plan: module layout and a commit sequence

Decide while writing where it lives — docs/, a top-level design file, or the README — and
whether the pinned-model rationale belongs in it or in user-facing docs.

Blocked by

Every other ticket on this map. When this closes, the map closes. Four remain open, all raised
by an independent reconstruction of this document from the recorded decisions alone — the
reconstruction succeeded, which is why these are gaps in the connective tissue rather than in
any decision:

• The Adapter's concrete contents, and the Chunk budget as numbers
• The four handoffs #30 never picked up — itself blocked by The Adapter's concrete contents, and the Chunk budget as numbers #37
• The panel's server group: who produces its rows
• Clear the stale text from the closed record

A full independent draft already exists and should be read before writing this one — it is
faithful to every decision and its gaps section is the source of the four tickets above. It is
not a substitute for this ticket: it inherits the holes rather than filling them.

[charitarthchugh]

From #29 (resolved: variant A). The implementation plan must include three prerequisite changes to src/paperscale/tui.py, none of which are embed-specific bugs but all of which block the chosen panel: (1) _stat_columns() hardcodes group order ("run", "vllm", "issues"), which renders embed's groups as run | issues | server — throughput lands last; (2) _layout_budget() pads the event pane with blank rows on a quiet run; (3) resumed skips must be visually distinct from work performed, since a large skip count is the only symptom of the stale-output failure that standing decision 7 deliberately leaves undetected.

[charitarthchugh]

From The LanceDB schema and its namespacing, now closed — two things for the write-up.

The dependency list is now fixed: numpy (from #26), lancedb and pyarrow (from #27). None of them is a paperscale dependency today — all are absent from dependencies, from [project.optional-dependencies] (which holds only tui = ["rich"]) and from the dev group. Whether embed ships them as hard dependencies or as an extra following the tui precedent is on the map under Not yet specified. Everything measured for #26 and #27 was against lancedb 0.37.1 / pyarrow 25.0.1 / numpy 2.5.2, and two API notes belong in the plan: create_scalar_index is deprecated in favour of create_index(col, config=BTree()), and create_index(metric=...) in favour of create_index(col, config=IvfPq(...)).

Two CLI items surfaced. The LanceDB database path is a new argument with no prior art in this repo. And run labels are neither path-safe nor validated — _parse_runs at src/paperscale/cli.py:65 only strips whitespace and checks non-empty and unique, while standing decision 6 and #22 put the label into a filesystem path, so --run 'legal/2024=...' silently creates a nested directory. Both are recorded on the map under Not yet specified alongside the rest of the CLI surface.