Repository navigation
fix(playbook): fill and declare-check a route condition's params - #1058
Conversation
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
The route-condition references now follow the same declaration, filling, and secret-withholding path as node prompt references, while unchanged routes retain their prior wire shape. I covered the full diff, repository rules, the validation and dispatch callers through the DAG judge, the feature's originating history, backward compatibility, architecture boundaries, and whether tests were weakened; I found no issue worth raising.
Verification: PYTHONUTF8=1 uv run --frozen --python 3.12 --extra dev --dev pytest tests/test_playbook_executor.py tests/test_playbook_validate.py -x (134 passed); Ruff check and format checks passed for all changed files; source-language, commit-message, and git diff --check checks passed. A separate all-extras setup attempt could not build datrie because this host has no GCC, before tests began.
|
Not a blocker (review of Severity of this one finding, not a verdict on the pull request. Any blocking findings from this pass are review threads on the changed files; GitHub renders those collapsed, as a file name with no text.
|
A route condition reaches the judge where a prompt reaches its node, but
only prompts were filled and only promptTemplate and inputs were checked:
'the deal is above ${params.threshold}' reached the judge as written,
hinging the decision on a placeholder no reader can evaluate, and an
undeclared ${params.ghost} in one passed validation. Conditions are
filled on the same terms as prompts now -- secrets withheld included --
and an undeclared reference in a route is refused with the route named
(item 2 of EverMind-AI#1008).
…texts A secret referenced from a route condition is withheld today because the fill goes through _fill_text, but nothing proved it: a mutant that fills routes with the plain secret-substituting fill_param_refs passed every test, while the route text reaches the judge -- a refactor away from sending a credential to a model with the suite green. The fill test now carries a secret param in the route and asserts the withheld marker and the absence of the value; the mutant fails it on the leaked token. _fill_text's docstring counts the paths through it again, and the playbooks doc names route conditions where parameter references belong.
The first attempt at this sentence was lost to a scripted edit that overwrote it a beat later; this is the same text, verified in the file this time.
0435fe9 to
7333c47
Compare
|
Pinned: the fill test now carries a secret param in the route and asserts the withheld marker and the absence of the value, and I ran your mutant to prove the pin -- filling routes with the plain secret-substituting fill_param_refs fails exactly on the leaked token. _fill_text's docstring counts its paths again, and the playbooks doc names route conditions where parameter references belong. 275 across the playbook suites; the branch is rebased onto current main. |
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
I reviewed the new regression-test and documentation delta, the full branch diff after its rebase, the surrounding secret-fill and judge-dispatch callers, relevant history, repository rules, backward compatibility, architecture boundaries, and whether the tests were weakened. The new assertion pins both the withheld marker and absence of the supplied secret from dispatched nodes, while the refreshed texts accurately describe route conditions.
Verification: PYTHONUTF8=1 uv run --frozen --python 3.12 --extra dev --dev pytest tests/test_playbook_*.py -x (472 passed). Ruff check and format, source-language, commit-message, large-file, and git diff --check checks also passed.
|
No new Windows compatibility regression found in the inspected changes. Reviewed head Native Windows results (Python 3.13.12):
Code reading: the new route substitution reuses the existing callable regex replacement, which does not reinterpret backslashes in supplied text; no new filesystem, shell, process or dependency behavior is introduced. Tests use a fake DAG dispatcher, so they do not verify a live judge/model or remote execution. No full repository suite was run. This is a Windows-scoped assessment, not a merge verdict. Exact verification commandsRuns used detached head/base checkouts, the existing frozen dev environment, source imports pinned to each checkout, and isolated profile/Raven/OAuth state. # native; exit 1; cwd C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\pr1058\head
uv run --frozen --no-sync pytest tests/test_playbook_executor.py tests/test_playbook_validate.py -n 0 -x --tb=short --basetemp C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native\pytest --junitxml C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native\junit.xml
# base-reference; exit 1; cwd C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\base431599
uv run --frozen --no-sync pytest tests/test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots -n 0 -x --tb=short --basetemp C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-base-reference\pytest --junitxml C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-base-reference\junit.xml
# native-rest; exit 0; cwd C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\pr1058\head
uv run --frozen --no-sync pytest tests/test_playbook_validate.py --deselect=tests/test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots -n 0 -x --tb=short --basetemp C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native-rest\pytest --junitxml C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native-rest\junit.xml |
ypflll
left a comment
There was a problem hiding this comment.
Approving. The defect is real and the fix is the narrow one.
A route condition reaching the judge as the deal is above ${params.threshold} hinged a decision on text no reader can evaluate, and filling it through the same _fill_text as a prompt is what makes the secret withholding come for free rather than needing its own proof. Keeping the fill at its existing site, and saying plainly that composed-graph nodes were never filled either and stay that way here, is the right scope.
Nonblocking -- lock the secret withholding with a test of its own.
The Risk note says a secret param in a condition is withheld rather than exposed, and that is true today by construction: the condition goes through _fill_text. But nothing fails if that stops being true. A later change that fills a route by any other route -- a direct format, a new helper, an inlined substitution -- leaves the existing tests green while a secret reaches the judge, which is the one reader this PR exists to give readable text to.
The test is close to free: the route version of whichever prompt-side secret test already exists, asserting the withheld marker lands in the condition and the secret itself does not.
Minor, take it or leave it: **({"routes": {...}} if n.routes else {}) builds a conditional dict to avoid setting a key that already has a default. {name: ... for name, condition in (n.routes or {}).items()} is the same result flat -- and is the form validate_graph_nodes uses two files over, so the two would read alike.
… the docs (#1064) ## Summary Items 1 to 6 of #1008, the follow-ups the review of #1001 left out. Item 7, file references on Windows, is handled separately. 1. **A decision node's route is no longer the first thing cut on a tasks-board card.** The agent's name kept its full width (`min-width: auto`) and the tool-count chip never shrinks, so the selection took all of the line's overflow: with "Raven-Research" and "3 tools" a 196px card showed only the arrow. The selection now renders as the arrow with its first route, which keeps its width, then the routes after it. Those, and the instance handle, start from nothing and grow into the room the line has left (`flex: 1 1 0; max-width: max-content`), and the agent's name gives way after them, with an ellipsis. The later routes keep room for a letter and an ellipsis: narrower, a line's first character is clipped rather than replaced by one. The 6px spacing separates fields only: the two selection spans have no additional gap, and the comma-space separator stays with the first route, outside the text that can be cut. A card with no selection lays out as before, a long minted handle included. 2. **Route conditions are filled and validated for params.** The executor filled `${params.X}` only in `promptTemplate`, and `validate_graph_nodes` checked only `promptTemplate` and `inputs`, so "the deal is above ${params.threshold}" reached the judge as written and `${params.nobody}` passed validation. Conditions now go through the same `_fill_text` as the prompt, a secret withheld with the same log line, naming the route, and a reference to no declared param is an error. The filling and the check are on `main` now, carried by #1058; this branch holds the executor's docstring and the tests that pin both. 3. **Hand-written route names `yes`, `no`, `on`, `off`, `true`, `false` and bare numbers load.** The store parsed the spec block with `yaml.safe_load`; YAML 1.1 read those keys as booleans or integers, validation refused them, and the runtime dropped the playbook from the library. Every mapping is now keyed by a name, and the value keeps its YAML type. The loader is the one #1057 drafted, so the store carries one key loader rather than two: a node that is not a mapping is left to PyYAML to answer, which keeps the error the validate paths catch, and a key node is retagged through a copy, so an anchor that shares one scalar between a key and a value leaves the value's type alone. #1057 is closed as superseded by this branch. 4. **The generator's dag-mode instructions describe branches.** That section called `dependsOn` the whole graph language and never named `routes`, `select` or `when`; it now describes a decision node and the rule that a node depending on one names it in its `when`, and says a condition may use a param. A new test reads that section on its own. 5. **An ordinary node's judge request is held to a literal golden.** `test_judge_without_routes_sends_the_old_request_verbatim` compared the request with the module's own constants, so changing the instruction or a tool schema's description passed. It now compares the whole request -- messages, tools, model, tool choice -- with the fence nonce fixed, against `tests/data/dag_verdict_plain_request.json`. The golden is the request from before #1001: the tree before it and this one send byte-identical requests over 32 input combinations. 6. **Domain terms around branches.** `CONTEXT.md` gains three `_Avoid_` notes: "decision" alone for a decision node or for what it chose, since the word already names the main agent's answer to an exception report in a DAG run and a choice the host puts to the user; "skipped" for a node a branch left out; and "branch" for any other part of a graph. The uses those notes retire are rewritten in the files this pull request touches: the DAG orchestration skill (the `abandon` paragraphs and the conditional-branch section), the orchestration and playbook pages in both languages, a runner docstring, and test docstrings, one test helper and one test name; two ui-tui test comments that called a path through a drawn graph a branch now say path and edge. The playbook page's "not a runtime branching language" had read, since #1001, as saying a prompt-mode playbook cannot branch; it now says the guidance decides nothing while the graph runs, and a branch is chosen by a decision node in the graph it composes. Left as they are, outside the files this pull request touches: "decision" alone still names a decision node or its selection in text a model reads -- the `routes` description in the `run_subagent_dag` schema and on `DagNodeSpec`, which the playbook generator's tool carries; the guard-size error in `raven/agent/subagent/dag_guard.py`; the note `raven/agent/subagent/dag_render.py` writes into a sub-agent's prompt; the workflow compiler's prompt -- and in the RPC model's field description and both generated clients, the "Decision missing" reason in `i18n/messages.json`, the DAG Panel entry of `ui-tui/CONTEXT.md` (which `test_the_tui_glossary_says_what_the_turned_in_line_carries` quotes), and docstrings, comments and test names across the DAG modules and both UIs. Retiring those reaches the RPC schema, the generated clients, a translated label and pinned tests, so it is left to #1065. Python test function names, which predate the note, were left alone in these files too. Two things the review of this pull request left open are closed here: - **The card's layout test now lives where a real-environment test belongs.** `tests/test_web_task_card_route_real_browser.py` measured a card in a real Chromium, so it moved to `tests/integration/test_web_task_card_real_browser.py`, and a new `web-browser` CI job runs it on a runner that ships a system Chrome. It is the only place the tasks board card's agent line is measured: happy-dom lays nothing out, so the stylesheet gate beside it can only read the declarations. Without a browser it skips with the reason, except under `CI`, where a skip would leave the line unmeasured on every push and so fails instead. - **The store's loader answers a node that is not a mapping the way PyYAML does.** `raven playbook validate` now answers a YAML syntax error in a spec block as a finding, the way the RPC validator already did, instead of a traceback. The branch is rebased onto `main` at `471842132` (#1114). The `CHANGELOG.md` conflict that rebasing raised is resolved by keeping both sides' entries, and the branch merges clean. ## Type - [x] Fix - [ ] Feature - [ ] Docs - [ ] CI / tooling - [ ] Refactor - [ ] Other ## Verification - [x] Relevant tests pass locally - [x] Relevant lint / type checks pass locally - [x] User-facing docs or screenshots are updated when needed Native Windows 11, CPython 3.12.13: - `uv run --frozen --no-sync pytest tests/test_playbook_store.py tests/test_playbook_executor.py tests/test_playbook_validate.py tests/test_subagent_dag_verdict.py tests/test_subagent_dag_core.py tests/test_cli_playbook_commands.py tests/integration/test_web_task_card_real_browser.py -m "integration or not integration" -q -p no:randomly --tb=short`: 770 passed, 11 failed, 15 skipped. The same 11 fail at the merge base on this host, with the same IDs: `test_playbook_executor.py::test_a_playbook_run_s_node_attempts_nest_under_the_call_that_ran_it`, `test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots`, `test_subagent_dag_core.py::test_replanning_refuses_a_redeclared_id`, `test_subagent_dag_core.py::test_a_refused_submission_still_records_the_link_as_unstarted`, the five `test_subagent_dag_core.py::test_a_reference_landing_in_either_root_is_accepted` parameters, and `test_cli_playbook_commands.py::test_run_judges_its_nodes_so_a_branching_playbook_takes_its_branch` and `::test_run_judges_on_the_users_own_subagent_dag_settings`. The first group is the file-reference rules #1066 fixes, and the last two shell out to `cat`. The branch adds the passing cases. - The baseline comparison above, run over the same six files at the merge base: 746 passed, 11 failed, 15 skipped -- the same failing IDs. - `uv run --frozen --no-sync pytest tests/integration/test_web_task_card_real_browser.py -m integration -q -n 0 -p no:randomly --tb=short`: 10 passed, the board's real 196px card measured in a real Chromium. Playwright could not create its artifacts directory under this host's default temp path, so the run needed a writable `TEMP`; with the default one the nine browser cases skip and the path-lookup case still passes. - After the rebase onto `main` at `471842132`, with `-n 0`: the six playbook/DAG files (`tests/test_playbook_store.py`, `tests/test_playbook_executor.py`, `tests/test_playbook_validate.py`, `tests/test_subagent_dag_verdict.py`, `tests/test_subagent_dag_core.py`, `tests/test_cli_playbook_commands.py`): 790 passed; `tests/integration/test_web_task_card_real_browser.py -m integration`: 10 passed; `scripts/check_commit_messages.py`, `scripts/check_source_language.py` and `scripts/check_large_files.py` over `origin/main..HEAD`, and `git diff --check`: passed. - `uv run --frozen --no-sync ruff check raven/playbook/store.py raven/cli/playbook_commands.py tests/test_playbook_store.py tests/test_cli_playbook_commands.py tests/integration/test_web_task_card_real_browser.py` and `ruff format --check` over the same files: passed. - `uv run --frozen --no-sync ty check raven/playbook/store.py raven/cli/playbook_commands.py`: passed. - `uv run --frozen --no-sync lint-imports`: 10 kept, 0 broken. - `uv run --frozen --no-sync --extra dev python scripts/check_commit_messages.py origin/main..HEAD`, `scripts/check_source_language.py origin/main..HEAD` and `scripts/check_large_files.py origin/main..HEAD`: passed. - `git merge-tree` against `origin/main`: no conflict, through `--write-tree` on the Windows host and through the three-argument form on the Linux workstation, whose git predates `--write-tree`. Before the rebase the same check reported one conflict, in `CHANGELOG.md`. ui-web, run in `ui-web/`: - `node node_modules/vitest/vitest.mjs run src/features/tasks/BoardCard.test.tsx scripts/gates/task-card-selection-css.test.mjs --no-file-parallelism`: 2 files, 36 tests passed. - `node node_modules/vitest/vitest.mjs run src/features/tasks scripts/gates --no-file-parallelism --testTimeout=60000`: 62 files, 560 tests passed. The same command over `main` at the time of the run reported 61 files and 552 tests passed; the branch adds the stylesheet gate and the cases beside it. Without the longer timeout two of the gates time out on a loaded host; they pass on their own and the files they read are byte-identical to `main`. At head `211d226d`, the branch's CI run: 23 checks passed, 2 skipped (`claude`, `docs deploy`), 0 failed. The rebased head re-runs the same workflow, including the `web layout in a real browser` job, which runs the browser cases on a runner that ships a system Chrome. Not run here: macOS, a browser other than Chromium, and the full repository suite. ## Risk A stored playbook whose route condition names an undeclared param now fails validation, and so leaves the library, as one doing the same in `promptTemplate` already did. On the board, a crowded agent line now shortens the agent's name where it used to cut the selection. The store's loader now reads the frontmatter through the same path as the spec block; a frontmatter key that YAML 1.1 would have resolved to a boolean or a number is now read as the text it was written as, which the frontmatter's two string fields never were. - [x] Security impact considered - [x] Backward compatibility considered - [x] Rollback path is clear for risky changes ## Related Issues #1008 (items 1 to 6)
Summary
promptTemplateand validation checked references only inpromptTemplateandinputs:the deal is above ${params.threshold}reached the judge as written, hinging the decision on a placeholder no reader can evaluate, and an undeclared${params.ghost}in a condition passed validation -- item 2 of DAG conditional branches: seven follow-ups left out of #1001 #1008._fill_textdoes the withholding); and an undeclared reference in a route is refused at validation with the route named:node 'a': params.ghost in route 'go' names no declared param.promptTemplateeither and are unchanged by this PR; the fill stays where it was, now coveringroutesbeside it.Type
Verification
uv run pytest tests/test_playbook_validate.py tests/test_playbook_executor.py tests/test_playbook_store.py tests/test_unified_playbook.py tests/test_cli_playbook_commands.py tests/test_agent_loop_playbook_entry.py -q-> 288 passed'the deal is above ${params.floor}'verbatim on the base and the validation test accepted an undeclared reference; both assert the filled and refused outcomes on this headruff checkandruff format --checkon all four files,make lint-types-> cleanRisk
_fill_textthat withholds secrets from prompts, so a secret param in a condition is withheld, not exposedRelated Issues
Item 2 of #1008