Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .harness/config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Target-repository harness configuration
project: level-five
workflow: story-workflow
branch_prefix: story/
# The branch a story's branch is cut from, and the branch l5-plan expects to
Expand Down
10 changes: 7 additions & 3 deletions .harness/docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,13 @@ Since story-032 the `create` value is **a path at or beneath** one of that stage

One file in the inventory declares a *contract* rather than an artifact shape. Since story-039 the set of configuration keys the harness reads is declared in **`schemas/harness-config.schema.json`** — one property per key, thirteen of them (`allowed_tools`, `architecture_docs`, `base_branch`, `branch_prefix`, `logs_dir`, `model`, `permission_mode`, `runs_dir`, `standards_dir`, `stories_dir`, `test_command`, `verification_runner`, `workflow`), each typed as `load_config` produces it and described by what it governs and what it falls back to. It is in `schemas/manifest.json` like every other file there, and passes the same parametrized draft-2020-12, unsupported-keyword and no-`additionalProperties` checks, but no stage is asked to satisfy it because no agent produces a config file. `harness_config.declared_config_keys(harness_root=None)` is its **only reader**, resolving it relative to its own module through `schema_validator.load_schema` so `schemas/` keeps one reader, and raising `ValueError` naming the path on a missing, unparseable or wrong-shaped schema rather than degrading to an empty or partial tuple — a degraded return would make the coverage below vacuous instead of red.

The declaration is **not a run-time check**. Nothing calls `declared_config_keys` while a run executes, no target's `.harness/config.yaml` is validated against it, and no unknown key is refused — a target carrying an extra key runs exactly as it did before the file existed. `project` is the live example of a key a config file carries and nothing reads, and it is deliberately undeclared: the declared set *is* the set of keys the harness reads, so a key nothing reads is out of the set by construction rather than by oversight.
Since story-043 the declaration is also a **run-time check, and it is strict**: `harness_config.undeclared_config_problems(config, harness_root=None)` calls `declared_config_keys` and returns one problem per key a loaded config carries that the schema does not declare, in the order the config carries them. `run_story` calls it immediately after `load_config` and refuses through `_refuse_undeclared_config_keys` and the shared `refuse()` — **above** the workflow load and above every other pre-flight, so a refused run creates no run directory, no state file, no log, no branch, and invokes no agent. Only key *names* are examined; no value is validated, coerced or constrained, and `load_config`'s parsing is untouched, so a comment naming an unknown key is stripped before any key is seen and refuses nothing. `tests/test_undeclared_config_keys.py` holds the refusal's coverage — it replaced the retired-key module story-041 added and this story deleted, whose subject no longer exists — and includes a sweep asserting that no fixture configuration under `tests/` carries an undeclared key, which is what keeps the rest of the suite runnable under the strict rule.

**A *retired* key is the one exception, and it refuses rather than being ignored.** Since story-041 `harness_config` also declares `RETIRED_CONFIG_KEYS`, a mapping from a name the harness once read to the name that replaced it, and `retired_config_problems(config)` returns one problem per retired key a loaded config still carries, naming both. `run_story` calls it immediately after `load_config` and refuses through the shared `refuse()` — **above** the workflow load and above the routing pre-flight, so a refused run creates no run directory, no state file, no log, no branch, and invokes no agent. The reasoning is the difference between a key nothing has ever read and a key that was read yesterday: an unknown key is inert, but a config still carrying `clean_clone_python` after the rename would fall through to a *different* fallback and quietly change what the clean-clone check exercises. Silently accepting it as the new key is worse still — it would make the rename undiscoverable. Declaring the mapping beside `declared_config_keys` keeps the config vocabulary, what is read and what used to be, in one module. The refusal message is composed from the mapping rather than written out, so retiring the next key is a one-line edit and adds no prose to the coordinator.
**Each problem names the offending key and lists the declared set** — `'<key>' is not a key the harness reads; it reads: …` — the shape story-028's routing refusal takes, because a bare "unknown key" would leave the developer to find the vocabulary themselves. The declared set is composed into each problem rather than appended by the coordinator, so the function is self-contained and the refusal is actionable without opening the schema.

**This replaced a by-name retirement mapping rather than joining it.** story-041 left `RETIRED_CONFIG_KEYS` — one entry, `clean_clone_python` → `verification_runner` — and refused that name alone; story-043 deleted the mapping and its function, so the retired key is now refused as one more undeclared key and the literal went with the mechanism. That is what took the last target-stack language name out of `orchestration/`, and it is the *reason* a sibling mapping must not come back: any retirement mapping reintroduces the tie it removed. Two things fell out of the one change. A mistyped key stops being silently ignored — `branch_prefixx: story/` used to run, quietly take the default, and be discovered from the branch name. And the cost, stated plainly: a config carrying `clean_clone_python` is told the key is not one the harness reads rather than being told what replaced it; someone upgrading finds the replacement in the schema instead.

**`project` was removed rather than declared**, from `.harness/config.yaml`, `templates/config.yaml`, the `{project}` substitution in `scripts/l5-init`, and every fixture configuration under `tests/`. It was the one key a shipped config carried that nothing reads, and under the strict rule it would have refused this repository's own config. Declaring it was not available: the declared set *is* the set of keys the harness reads, and story-039's three coverage checks require every declared key to be read, to carry a proof, and for that proof to go red when its read is replaced by its fallback — none of which a key nothing reads can satisfy. Adding a key to `templates/config.yaml` therefore now means adding it to the schema and giving it a proof, or not adding it at all.

What the declaration is for is coverage, and the coverage is **set equality in both directions**, twice, in `tests/test_config_keys_are_obeyed.py`. Against `KEY_PROOFS` — a key declared with no proof fails, a proof naming an undeclared key fails — and against an AST scan of `orchestration/` and `scripts/` collecting every `config.get("...")` and `config["..."]`, extensionless scripts included, so a key the harness reads and the schema does not declare fails. Neither comparison is against a second maintained list. Every declared key carries a proof that **varies** it: the value is one the harness would never pick (each carries the token `xyzzy`, and an assertion checks that none coincides with the key's default or with this repository's own configured value), and ten keys are proven behaviourally while `model`, `permission_mode` and `allowed_tools` are proven on the invocation built for a fake runner, because those three are handed to the agent runner and observable nowhere else. `KEY_PROOFS` records which of the two each key gets, so a reader knows what *proven* means for it. A mutation control then replaces each key's read with its fallback literal in a throwaway copy of `orchestration/` and requires that key's proof to go red there, so a proof that sets a key and asserts nothing about its effect cannot survive. This subsumes but does not retire story-028's absence assertions: proving a literal is *absent* and proving the configured value *governs* are different claims, and both are kept.

Expand Down Expand Up @@ -421,7 +425,7 @@ The harness runs against any repository. Its only tie to a target's language, to

**The rule is a scan, not a paragraph, and the choice has a record.** Two rules of exactly this shape were written down here before: the `git diff HEAD` baseline rule was documented *and* injected into every stage prompt and shipped five more times, and the git-history-loader rule was documented and shipped three more times. Both stopped when a scan landed. Two of the five ties also sit in a prompt, which no fixture reaches — `prompts/tester.md` is prose an agent reads — so "every configurable value is proven configurable" could vary every key in existence and still not notice a pytest filename sitting in it. Only a reader or a scan sees that, and readers had missed it since the file was written.

**Where the two halves live.** The declaration and the scan are `orchestration/harness_source.py` (above); the judgement is `tests/test_no_target_stack_in_harness_source.py`. The module reports mentions and judges none of them — the test module holds **two allowlists**, keyed by repository-relative path and the exact text of the matched line rather than by line number, so an unrelated edit above a tie does not churn the list and read as burn-down. `TEMPORARY_TIES` (3 entries since story-041, 36 before it) holds the mentions that name or assume a target's stack or layout. `PERMANENT_MENTIONS` (9 entries) holds the mentions that are the opposite of a tie — a docstring saying every scalar `story_parser` produces is a Python `str` is a fact about this code, and `RETIRED_CONFIG_KEYS` spelling a retired key is the harness *refusing* the tie rather than carrying one — and **every entry carries a one-line reason**, so the permanent half is a judgement on the record rather than a suppression list. The two together equal exactly what the scan reports against this repository (12 distinct `(path, line text)` pairs from 13 findings — `prompts/tester.md:47` matches both rules), asserted in both directions and asserted disjoint. The classification rule sits in the module docstring beside the declaration so the next one is not re-argued.
**Where the two halves live.** The declaration and the scan are `orchestration/harness_source.py` (above); the judgement is `tests/test_no_target_stack_in_harness_source.py`. The module reports mentions and judges none of them — the test module holds **two allowlists**, keyed by repository-relative path and the exact text of the matched line rather than by line number, so an unrelated edit above a tie does not churn the list and read as burn-down. `TEMPORARY_TIES` (3 entries since story-041, 36 before it) holds the mentions that name or assume a target's stack or layout. `PERMANENT_MENTIONS` (8 entries since story-043, 9 before it) holds the mentions that are the opposite of a tie — a docstring saying every scalar `story_parser` produces is a Python `str` is a fact about this code, and the shebang each `scripts/l5-*` entry point carries names the interpreter *this* harness runs under — and **every entry carries a one-line reason**, so the permanent half is a judgement on the record rather than a suppression list. The ninth entry was `orchestration/harness_config.py`, whose `RETIRED_CONFIG_KEYS` spelled a retired key; story-043 deleted the mapping, and the allowlist entry had to go with it or the two-way set equality would fail. Only two entries name a file under `orchestration/` now, both docstrings about this code: `story_parser.py`'s type-coercion note and `story_coordinator.py`'s `self_route_problems`. The two lists together equal exactly what the scan reports against this repository (11 distinct `(path, line text)` pairs from 12 findings — `prompts/tester.md:47` matches both rules), asserted in both directions and asserted disjoint. The classification rule sits in the module docstring beside the declaration so the next one is not re-argued.

**Why they are two lists and not one.** `TEMPORARY_TIES` reaching empty is the completion signal for `.harness/requests/the-interpreter-is-not-assumed-to-be-python.md` and `.harness/requests/the-test-location-comes-from-configuration.md` — the two stories queued behind story-040, which fix the ties it grandfathers. Merge the two lists and the signal is gone — a list that stops shrinking cannot be told from work that finished.

Expand Down
48 changes: 24 additions & 24 deletions orchestration/harness_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,34 +13,34 @@
import schema_validator

# The declaration of which keys the harness reads, beside the artifact
# schemas. It is a declaration and not a run-time check: nothing here
# validates a target's config file against it, and no unknown key is
# refused.
# schemas. It is also what a target's config file is checked against at
# pre-flight: a key the schema does not declare refuses the run.
CONFIG_SCHEMA_NAME = "harness-config"

#: Keys the harness once read, mapped to the key that replaced each. A
#: retired key is refused rather than ignored: a config still carrying one
#: would fall through to the replacement's fallback and quietly change what
#: the harness does, which is the drift the declaration exists to stop. It
#: lives beside `declared_config_keys` so the config vocabulary — what is
#: read, and what used to be — has one home.
RETIRED_CONFIG_KEYS: dict[str, str] = {
"clean_clone_python": "verification_runner",
}

def undeclared_config_problems(
config: dict, harness_root: Path | None = None
) -> list[str]:
"""One problem per key a loaded config carries that the schema does not declare.

def retired_config_problems(config: dict) -> list[str]:
"""One problem per retired key a loaded config still carries.
The declared set is the set of keys the harness reads, so a key outside
it is a key nothing will ever act on — a retired name left behind after
a rename, or a mistyping of a declared one. Either is refused rather
than ignored, because ignoring it lets the run fall through to a
default and quietly do something other than what the config asked for.

Each names the retired key and the key that replaced it, so the refusal
is actionable without opening the schema. An empty list is the whole of
Each problem names the offending key and lists the declared set, the
shape the routing refusal takes: a bare "unknown key" would leave the
developer to find the vocabulary themselves. Problems come back in the
order the config carries the keys, and an empty list is the whole of
"this config carries none".
"""
declared = declared_config_keys(harness_root)
listed = ", ".join(declared)
return [
f"'{key}' is no longer read by the harness; it was replaced by "
f"'{replacement}'"
for key, replacement in RETIRED_CONFIG_KEYS.items()
if key in config
f"'{key}' is not a key the harness reads; it reads: {listed}"
for key in config
if key not in declared
]


Expand Down Expand Up @@ -90,10 +90,10 @@ def declared_config_keys(harness_root: Path | None = None) -> tuple[str, ...]:
this module's package exactly as the artifact schemas are, and read
through schema_validator.load_schema so schemas/ keeps one reader.

The declaration is not a run-time check. Nothing calls this while a run
is executing, no target's config file is validated against it, and no
unknown key is refused; what reads it is the coverage that asserts set
equality against the keys the harness actually reads.
Two things read it. The coverage asserts set equality against the keys
the harness actually reads, and `undeclared_config_problems` checks a
loaded config against it at pre-flight, so a key the schema does not
declare refuses the run rather than being silently ignored.

A missing, unparseable or wrong-shaped schema raises ValueError naming
the path, rather than degrading to an empty or partial tuple, which
Expand Down
27 changes: 14 additions & 13 deletions orchestration/story_coordinator.py
Original file line number Diff line number Diff line change
Expand Up @@ -2329,18 +2329,18 @@ def _refuse_bad_self_routes(workflow: dict, problems: list[str]) -> int:
)


def _refuse_retired_config_keys(target_root: Path, problems: list[str]) -> int:
"""Refuse a run whose configuration still carries a retired key.
def _refuse_undeclared_config_keys(target_root: Path, problems: list[str]) -> int:
"""Refuse a run whose configuration carries a key the harness does not read.

Thin, like every other caller of `refuse`. The configuration is wrong, not
the story and not the tree, so the guidance names the file to edit and the
edit to make.
"""
return refuse(
f"{target_root / '.harness' / 'config.yaml'} carries configuration keys "
f"the harness no longer reads:",
f"the harness does not read:",
problems,
"Rename each key to its replacement before running a story.",
"Remove or correct each key before running a story.",
)


Expand Down Expand Up @@ -2532,15 +2532,16 @@ def run_story(
"""
config = harness_config.load_config(target_root)

# Pre-flight: a retired configuration key is refused rather than ignored.
# Ignoring one lets the run fall back to the replacement's default and
# quietly exercise something other than what the config asked for. Above
# every other pre-flight, because it is decidable the moment the config
# loads: a refusal here leaves no run directory, no state.json, no log, no
# new branch, and invokes no agent.
retired = harness_config.retired_config_problems(config)
if retired:
return _refuse_retired_config_keys(target_root, retired)
# Pre-flight: a key the schema does not declare is refused rather than
# ignored. Ignoring one lets the run fall back to a default and quietly
# exercise something other than what the config asked for — a retired name
# left after a rename, or a mistyping of a declared key. Above every other
# pre-flight, because it is decidable the moment the config loads: a
# refusal here leaves no run directory, no state.json, no log, no new
# branch, and invokes no agent.
undeclared = harness_config.undeclared_config_problems(config, harness_root)
if undeclared:
return _refuse_undeclared_config_keys(target_root, undeclared)

workflow = harness_config.load_workflow(harness_root, config.get("workflow", "story-workflow"))
rules = harness_config.load_rules(harness_root)
Expand Down
Loading
Loading