Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .harness/docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ The drift source that paragraph used to name is closed: `planner.md` no longer s
The fourth, **`assignment_problems(story, stages)`**, is plan-time-only for exactly the same reason and is otherwise unlike its neighbour — which is why the module docstring now distinguishes the two rather than blending them. It reports one problem per `technical_plan.likely_file_changes` entry whose `file` falls under a prefix the entry's own `stage` is restricted from creating under, unless a grant on that stage covers the file. It is called from `artifact_problems` beside `strictness_problems`, below the `read_story` gate, so the existing order is unchanged. Both halves come off `stage_restrictions` and the grant is decided by `grant_covers`, so the module still names no stage and no prefix; a story with no `technical_plan`, and an entry missing `file` or `stage`, yield nothing rather than raising. It is **structural** — two literals compared, one in the artifact and one in the workflow definition — and inherits none of the third check's hedging: no clause split, no vocabulary of scoping words, no paraphrase that evades it, nothing it declines to decide. See "Work a stage cannot own" below for why the subject is `likely_file_changes` and what the check costs.

The fifth, **`naming_problems(story)`**, is plan-time-only for exactly the reason the third and the fourth are — committed artifacts that have already run name their modules for story numbers, and refusing them at pre-flight would make those stories unrunnable. It reports one problem per `technical_plan.likely_file_changes` entry whose `file` has a basename matching `STORY_NUMBERED_MODULE` (`^test_story_\d+`), and the message says what to name the module instead rather than only that the name is wrong. Its subject is `likely_file_changes` for the reason `assignment_problems` takes it: that is where a plan states which files it expects written, so a name is decidable there, before anything exists under it. It is **structural** like its neighbour — one literal in the artifact against one pattern — and inherits none of the third check's hedging. It is the only one of the five that reads no workflow, so it takes `story` alone; a story with no `technical_plan`, and an entry with no `file`, yield nothing rather than raising. See "A test module is named for what it checks" below for the convention it holds and the two other mechanisms holding it.
- `harness_source.py` — a **declaration the suite asserts against**, added by story-040, and the one module here that no run-time path reads. It declares where the harness's own source lives (`HARNESS_SOURCE_DIRS`: orchestration, prompts, workflows, schemas, rules, hooks, scripts, templates), which of those a target reads or describes (`TARGET_FACING_DIRS`: prompts and workflows), the tokens naming a language or toolchain (`STACK_TOKENS`), the directory shapes a target's test layout would use (`TARGET_LAYOUT_PATHS`), the one file exempt from its own scan (`EXEMPT_FILES`), and the directory names that are not source (`SKIP_DIR_NAMES`). `scan(root=None)` walks the declared directories of a repository root — this repository by default, resolved relative to the module the way `schema_validator` resolves `schemas/`, and a throwaway one built by a test otherwise, so both go through one code path — and returns sorted frozen `Finding(path, line_number, line, token, rule)` records. A directory the root does not have is skipped rather than raising, so a throwaway root need only carry what a test is about. Two rules: `STACK_RULE` reads all eight directories, `LAYOUT_RULE` reads `TARGET_FACING_DIRS` only. The coordinator does not call it, nothing routes on it, and no run is refused because of what it reports — see "No target-stack literal in harness source" below for what it is for and what it does not catch.
- `run_status.py` — read-only status snapshot backing `l5-status`. Lists every run under the configured runs directory (story id, status, current stage, retry count, sorted by story id) or shows one run's full `RunState` plus the last 10 lines of its `events.log`. Reuses `story_coordinator.load_state` for state parsing (never duplicated); a run with a missing or unparseable `state.json` is flagged `unreadable` in the listing without aborting it, while the detail view fails loudly (stderr, exit 1). Never writes to run directories or anywhere else.

### Tool allowlist
Expand Down Expand Up @@ -402,6 +403,24 @@ story-031 escalated seventeen minutes into its implementer on a conflict that wa

**This story's own plan was not governed by the check it adds**, for the eighth time in this pattern's history, and the cause is a new one worth naming: the artifact was written and committed by an `l5-plan` session that ran before `assignment_problems` existed. It happens to be clean — it assigns its test module to the stage that owns validation, which is the convention this story writes down — but it was not checked. Enforcement begins with the next plan, which is one step earlier than the stale-workflow and stale-import cases, since the artifact rather than the run is what escapes.

## No target-stack literal in harness source

The harness runs against any repository. Its only tie to a target's language, toolchain or directory layout belongs in `.harness/`, which is that target's own configuration. An audit on 2026-08-15 found five ties outside it: a Python snippet executed as a version probe and a configuration key whose name is a language, both in `orchestration/story_coordinator.py`; the `may_not_create` line of `workflows/story-workflow.json`; and two lines of prose in `prompts/tester.md` naming a pytest layout.

**The rule is a scan, not a paragraph, and the choice has a record.** Two rules of exactly this shape were written down here before: the `git diff HEAD` baseline rule was documented *and* injected into every stage prompt and shipped five more times, and the git-history-loader rule was documented and shipped three more times. Both stopped when a scan landed. Two of the five ties also sit in a prompt, which no fixture reaches — `prompts/tester.md` is prose an agent reads — so "every configurable value is proven configurable" could vary every key in existence and still not notice a pytest filename sitting in it. Only a reader or a scan sees that, and readers had missed it since the file was written.

**Where the two halves live.** The declaration and the scan are `orchestration/harness_source.py` (above); the judgement is `tests/test_no_target_stack_in_harness_source.py`. The module reports mentions and judges none of them — the test module holds **two allowlists**, keyed by repository-relative path and the exact text of the matched line rather than by line number, so an unrelated edit above a tie does not churn the list and read as burn-down. `TEMPORARY_TIES` (36 entries) holds the mentions that name or assume a target's stack or layout. `PERMANENT_MENTIONS` (9 entries) holds the mentions that are the opposite of a tie — a docstring saying every scalar `story_parser` produces is a Python `str` is a fact about this code, and the coordinator docstring saying a target's test command need not be a Python interpreter is the harness declining the assumption — and **every entry carries a one-line reason**, so the permanent half is a judgement on the record rather than a suppression list. The two together equal exactly what the scan reports against this repository (45 distinct `(path, line text)` pairs from 46 findings — `prompts/tester.md:47` matches both rules), asserted in both directions and asserted disjoint. The classification rule sits in the module docstring beside the declaration so the next one is not re-argued.

**Why they are two lists and not one.** `TEMPORARY_TIES` reaching empty is the completion signal for `.harness/requests/the-interpreter-is-not-assumed-to-be-python.md` and `.harness/requests/the-test-location-comes-from-configuration.md` — the two stories queued behind this one, which fix the ties this one grandfathers. The artifact-schema mentions of `python`, `python_version` and `clean_clone_python` in `schemas/clean-clone-result.schema.json`, `schemas/revert-check-result.schema.json` and `schemas/harness-config.schema.json` are on the temporary list for that reason: the interpreter story names those schemas and they burn down with it. Merge the two lists and the signal is gone — a list that stops shrinking cannot be told from work that finished.

**story-040 fixes none of the five**, deliberately. `orchestration/story_coordinator.py`, `workflows/story-workflow.json` and `prompts/tester.md` carry no edit on its branch, and no run-time behaviour changed.

**What the scan does not catch, stated where a reader meets it.** `STACK_TOKENS` is a guess about languages nobody has tried and is incomplete by construction — a Ruby `Gemfile` or an Elixir `mix.exs` is reported by nothing here, which the suite demonstrates by planting both and finding them invisible. It is worth having anyway because it catches the *shape* of the mistake, which recurs, rather than every instance of it. The layout half **cannot read `orchestration/` at all**: this repository's own suite is called `tests/`, so in Python source a `tests/` literal cannot be told from an honest reference to it, and the half therefore reads only the target-facing files where a path can only mean a target's. `.harness/` and `tests/` are outside the scan entirely. The module is exempt from its own scan **by name and by nothing else** — the token list would otherwise report itself — and a second file carrying the same tokens is reported. And the scan is not tamper-proof: deleting it alongside a forced repair is not caught here at any granularity, and no assertion in the module implies otherwise. Every one of those limits is asserted by reading the module's text rather than by trusting it to be there.

**The matcher's boundary is "not a letter or a digit", not `\b`.** That is what makes `clean_clone_python` and `platform.python_version()` visible — `_` and `.` are not alphanumeric — while `pipeline` and `pipe` are not, because `e` is; `\b` inverts both. The alternation is longest-token-first so `python3` claims its own match rather than being read as `python` followed by a digit, which the boundary would then reject and hide entirely. The layout pattern keeps the left boundary and drops the right one, because each path already ends in a slash and requiring a non-alphanumeric after it would hide `tests/conftest.py` — the exact shape being looked for. All four constructed cases are run through `scan` rather than reasoned about.

**Every absence assertion is paired with a demonstration it can fail.** A tie planted in a file that is clean today is reported and turns the equality assertion red; a list entry left behind after its tie is removed turns it red the other way; the identical text that is invisible under `.harness/`, under `tests/` and in `orchestration/` is reported when planted in `prompts/`; the exemption is paired with a twin file carrying the module's full text, which *is* reported. All of it runs against a throwaway root rather than by editing this repository, in the style `tests/test_schema_inventory_location.py` established.

## Run directory anatomy

.harness/runs/story-001/
Expand Down Expand Up @@ -580,5 +599,6 @@ It is narrower than Chapter 18's **checkpoints**, and the difference is the reas
- **Two literals, two homes, held equal by a test.** `STORY_NUMBERED_MODULE = re.compile(r"^test_story_\d+")` is defined in both `orchestration/plan_validation.py` and `tests/test_baseline_honesty.py`, because the plan-time refusal ships with the harness and the scan belongs to the suite; neither may import the other. `tests/test_validation_module_naming.py` holds the two to the same verdict over a shared name set, which is what stops the plan-time refusal and the standing scan from drifting into two conventions. If a third reader appears, that is the point at which the pattern should get one home rather than a third copy.
- **A reference naming a path at a pinned historical revision keeps its historical spelling.** The rename sweep rewrote only references naming a file *as it exists now* — in `tests/`, `orchestration/`, `scripts/`, `prompts/` and this document. A reference that reads a path's text at a pinned revision was left spelled as it was there, because that is the name the object has at that revision and rewriting it breaks the read; `tests/test_baseline_honesty.py`'s regression set and `STORY_ORIGINS` itself are full of these. Nothing under `.harness/stories/` or `.harness/runs-archive/` was touched: both record what was true when they were written, and history describing the past accurately is not drift.
- **A suite count recorded in a story is a measurement of the tree it was taken from, and the tree moves.** story-038's acceptance criterion recorded 2002 tests "before the rename", and reconciling the finished suite against it took two attempts. The number was correct when taken — the plan commit `abce051` collects exactly 2002 — but `426fe1f` ("Grant the verifier a self-route budget of 1") landed between the plan and the work, moving one declaration out of `BUDGETLESS` into `BUDGETED` and shrinking a parametrization by 3, so the branch base collects 1999. Every item of the difference to the finished count is accounted for by module: +3 for the naming scan and its two controls, +109 for this story's own validation. **The reconciliation that settles it is per-module collection counts between the two trees, not a total**, since a total can absorb a lost module against an added one; here 32 of the 34 renamed modules collect an identical count and the two that differ are the two named. Five test *names* disappeared and all five are merge disambiguations rather than deletions — two collisions in the story-008/story-009 merge renamed, three in the story-014/story-033 merge suffixed `_by_story_033`. When a story records a count, record the revision it was measured at with it; when a later stage cannot reproduce it, compare per module before concluding anything was lost.
- **story-040 forced the standing-assertion repair pattern twice more, and the collision was unavoidable rather than incidental.** A module whose deliverable is to declare `tests/` and `pytest` cannot do it without spelling them, so `tests/test_stage_output_ownership.py::test_no_path_prefix_is_named_in_orchestration_code` and `tests/test_clean_clone_check.py::test_no_test_command_string_appears_in_orchestration_code` both went red on `orchestration/harness_source.py` the moment it existed. Each was repaired the way the bullet above prescribes — a by-name exemption held shut from both sides: the exempt module must exist *and* must actually contain the literal, or the exemption is stale and the test goes red; every other module is held to the original assertion unchanged; and neither exemption widens past the one literal it is about, so `harness_source.py` is still held to `may_not_create` in the first and to `-m pytest` and `unittest` in the second. Both repairs were folded into the existing test functions rather than added as new ones, because `tests/test_shared_baseline_resolution.py` pins `test_stage_output_ownership.py`'s test-name set exactly. No assertion was weakened, skipped or deleted and no test function was added or removed — the pre-story total of 2196 was unchanged by the repairs, and the finished suite is 2248 with story-040's own 52 cases. **The general shape worth carrying forward:** a story that adds a *declaration* of forbidden literals will collide with every standing scan that forbids them, and the collision is the declaration working rather than a defect in either.
- Verification rules never change between retries; retries narrow scope, they do not restart the workflow.
- Capacity exhaustion (rate limits) is a reason to wait, not to fail; budget ceilings are a reason to stop.
Loading
Loading