Skip to content

fix(playbook): fill and declare-check a route condition's params - #1058

Merged
ypflll merged 3 commits into
EverMind-AI:mainfrom
yudongyouqing:fix/playbook_route_condition_params
Oct 23, 2026
Merged

ypflll merged 3 commits into
EverMind-AI:mainfrom
yudongyouqing:fix/playbook_route_condition_params

Conversation

@yudongyouqing

Copy link
Copy Markdown
Contributor

Summary

  • A route condition reaches the judge where a prompt reaches its node, but the executor filled parameters only in promptTemplate and validation checked references only in promptTemplate and inputs: the deal is above ${params.threshold} reached the judge as written, hinging the decision on a placeholder no reader can evaluate, and an undeclared ${params.ghost} in a condition passed validation -- item 2 of DAG conditional branches: seven follow-ups left out of #1001 #1008.
  • Conditions are filled on the same terms as prompts now, at the same fill site, secrets withheld included (_fill_text does the withholding); and an undeclared reference in a route is refused at validation with the route named: node 'a': params.ghost in route 'go' names no declared param.
  • Composed-graph nodes were never filled for promptTemplate either and are unchanged by this PR; the fill stays where it was, now covering routes beside it.

Type

  • Fix

Verification

  • Relevant tests pass locally: uv run pytest tests/test_playbook_validate.py tests/test_playbook_executor.py tests/test_playbook_store.py tests/test_unified_playbook.py tests/test_cli_playbook_commands.py tests/test_agent_loop_playbook_entry.py -q -> 288 passed
  • New tests failed first: the fill test dispatched 'the deal is above ${params.floor}' verbatim on the base and the validation test accepted an undeclared reference; both assert the filled and refused outcomes on this head
  • Relevant lint / type checks pass locally: ruff check and ruff format --check on all four files, make lint-types -> clean
  • User-facing docs or screenshots are updated when needed: no user-facing copy changes; the judge now reads a condition it can evaluate

Risk

  • Security impact considered: filling uses the same _fill_text that withholds secrets from prompts, so a secret param in a condition is withheld, not exposed
  • Backward compatibility considered: a playbook whose conditions name only declared params loads and runs identically; one that named undeclared references was already broken at run time and now says so at load
  • Rollback path is clear: plain revert restores the unfilled, unchecked condition

Related Issues

Item 2 of #1008

@LivXue
LivXue requested review from 0xKT and gloryfromca October 22, 2026 06:58

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

The route-condition references now follow the same declaration, filling, and secret-withholding path as node prompt references, while unchanged routes retain their prior wire shape. I covered the full diff, repository rules, the validation and dispatch callers through the DAG judge, the feature's originating history, backward compatibility, architecture boundaries, and whether tests were weakened; I found no issue worth raising.

Verification: PYTHONUTF8=1 uv run --frozen --python 3.12 --extra dev --dev pytest tests/test_playbook_executor.py tests/test_playbook_validate.py -x (134 passed); Ruff check and format checks passed for all changed files; source-language, commit-message, and git diff --check checks passed. A separate all-extras setup attempt could not build datrie because this host has no GCC, before tests began.

@0xKT

0xKT commented Oct 22, 2026

Copy link
Copy Markdown
Member

Not a blocker (review of 0435fe95, main 43159933): the route fill and the route declare-check do what the commit says, and a secret referenced from a route is withheld today, because the fill goes through _fill_text. Two small things for a follow-up:

Severity of this one finding, not a verdict on the pull request. Any blocking findings from this pass are review threads on the changed files; GitHub renders those collapsed, as a file name with no text.

  1. Nothing pins that a route withholds secrets. If the route fill at raven/playbook/executor.py:549 is switched to the plain params.fill_param_refs(condition, values) -- which does substitute secrets (fill_param_refs("tok=${params.TOKEN}", {"TOKEN": "s3cr3t"}) gives tok=s3cr3t) -- tests/test_playbook_executor.py and tests/test_playbook_validate.py still pass (134 passed). The route text goes into the judge's request, so a later refactor could send a secret to a model with every test green. test_a_secret_never_reaches_a_dispatched_work_order covers only promptTemplate; adding a secret param to the route in test_a_route_condition_is_filled_before_it_reaches_the_judge and asserting its value is absent from the dispatched nodes would close it.
  2. Two texts are now behind the code. _fill_text's docstring (executor.py:60) still says the paths through it are "three"; with the route fill there are more. And docs-site/docs/playbooks.md still describes parameter references as belonging in promptTemplate or prompts only.

A route condition reaches the judge where a prompt reaches its node, but
only prompts were filled and only promptTemplate and inputs were checked:
'the deal is above ${params.threshold}' reached the judge as written,
hinging the decision on a placeholder no reader can evaluate, and an
undeclared ${params.ghost} in one passed validation. Conditions are
filled on the same terms as prompts now -- secrets withheld included --
and an undeclared reference in a route is refused with the route named
(item 2 of EverMind-AI#1008).
…texts

A secret referenced from a route condition is withheld today because the
fill goes through _fill_text, but nothing proved it: a mutant that fills
routes with the plain secret-substituting fill_param_refs passed every
test, while the route text reaches the judge -- a refactor away from
sending a credential to a model with the suite green. The fill test now
carries a secret param in the route and asserts the withheld marker and
the absence of the value; the mutant fails it on the leaked token.
_fill_text's docstring counts the paths through it again, and the
playbooks doc names route conditions where parameter references belong.
The first attempt at this sentence was lost to a scripted edit that
overwrote it a beat later; this is the same text, verified in the file
this time.
@yudongyouqing
yudongyouqing force-pushed the fix/playbook_route_condition_params branch from 0435fe9 to 7333c47 Compare October 22, 2026 11:40
@yudongyouqing

Copy link
Copy Markdown
Contributor Author

Pinned: the fill test now carries a secret param in the route and asserts the withheld marker and the absence of the value, and I ran your mutant to prove the pin -- filling routes with the plain secret-substituting fill_param_refs fails exactly on the leaked token. _fill_text's docstring counts its paths again, and the playbooks doc names route conditions where parameter references belong. 275 across the playbook suites; the branch is rebased onto current main.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

I reviewed the new regression-test and documentation delta, the full branch diff after its rebase, the surrounding secret-fill and judge-dispatch callers, relevant history, repository rules, backward compatibility, architecture boundaries, and whether the tests were weakened. The new assertion pins both the withheld marker and absence of the supplied secret from dispatched nodes, while the refreshed texts accurately describe route conditions.

Verification: PYTHONUTF8=1 uv run --frozen --python 3.12 --extra dev --dev pytest tests/test_playbook_*.py -x (472 passed). Ruff check and format, source-language, commit-message, large-file, and git diff --check checks also passed.

@LivXue

LivXue commented Oct 22, 2026

Copy link
Copy Markdown
Member

No new Windows compatibility regression found in the inspected changes.

Reviewed head 7333c47f9f911d8aac7988eca65c51f4d78a11ed, base/merge base 4315993300e538ec954c79565cbc8585006c8053: complete five-file diff, route declaration/fill/secret-withholding logic, parameter substitution, dispatch/judge callers, documentation, tests and existing discussions.

Native Windows results (Python 3.13.12):

  • The executor/validator run stopped at 81 passed, 1 failed: tests/test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots. All 79 executor cases passed, including the new route fill and secret-withholding regression case.
  • The failing path-confinement case also fails at the exact base. The unchanged raven/agent/subagent/prompt_paths.py resolves a drive-qualified path as relative using POSIX path rules, then rejects the resulting combined path. This affects absolute Windows DAG file references, predates this PR, and is covered separately by #1066; it is not re-filed as a route-parameter regression.
  • Rerunning the validator with only that confirmed baseline failure deselected: 54 passed, 1 deselected. Thus the two inspected test files have 133 distinct passing cases and the one inherited failure.

Code reading: the new route substitution reuses the existing callable regex replacement, which does not reinterpret backslashes in supplied text; no new filesystem, shell, process or dependency behavior is introduced. Tests use a fake DAG dispatcher, so they do not verify a live judge/model or remote execution. No full repository suite was run. This is a Windows-scoped assessment, not a merge verdict.

Exact verification commands

Runs used detached head/base checkouts, the existing frozen dev environment, source imports pinned to each checkout, and isolated profile/Raven/OAuth state.

# native; exit 1; cwd C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\pr1058\head
uv run --frozen --no-sync pytest tests/test_playbook_executor.py tests/test_playbook_validate.py -n 0 -x --tb=short --basetemp C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native\pytest --junitxml C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native\junit.xml

# base-reference; exit 1; cwd C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\base431599
uv run --frozen --no-sync pytest tests/test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots -n 0 -x --tb=short --basetemp C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-base-reference\pytest --junitxml C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-base-reference\junit.xml

# native-rest; exit 0; cwd C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\pr1058\head
uv run --frozen --no-sync pytest tests/test_playbook_validate.py --deselect=tests/test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots -n 0 -x --tb=short --basetemp C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native-rest\pytest --junitxml C:\Users\livxu\AppData\Local\Temp\raven-windows-review\93f210e71847404cb53894de7ac36f62\checks\1058-native-rest\junit.xml

@ypflll ypflll left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The defect is real and the fix is the narrow one.

A route condition reaching the judge as the deal is above ${params.threshold} hinged a decision on text no reader can evaluate, and filling it through the same _fill_text as a prompt is what makes the secret withholding come for free rather than needing its own proof. Keeping the fill at its existing site, and saying plainly that composed-graph nodes were never filled either and stay that way here, is the right scope.

Nonblocking -- lock the secret withholding with a test of its own.

The Risk note says a secret param in a condition is withheld rather than exposed, and that is true today by construction: the condition goes through _fill_text. But nothing fails if that stops being true. A later change that fills a route by any other route -- a direct format, a new helper, an inlined substitution -- leaves the existing tests green while a secret reaches the judge, which is the one reader this PR exists to give readable text to.

The test is close to free: the route version of whichever prompt-side secret test already exists, asserting the withheld marker lands in the condition and the secret itself does not.

Minor, take it or leave it: **({"routes": {...}} if n.routes else {}) builds a conditional dict to avoid setting a key that already has a default. {name: ... for name, condition in (n.routes or {}).items()} is the same result flat -- and is the form validate_graph_nodes uses two files over, so the two would read alike.

@ypflll
ypflll merged commit 2582e79 into EverMind-AI:main Oct 23, 2026
25 checks passed
0xKT pushed a commit that referenced this pull request Oct 27, 2026
… the docs (#1064)

## Summary

Items 1 to 6 of #1008, the follow-ups the review of #1001 left out. Item
7, file references on Windows, is handled separately.

1. **A decision node's route is no longer the first thing cut on a
tasks-board card.** The agent's name kept its full width (`min-width:
auto`) and the tool-count chip never shrinks, so the selection took all
of the line's overflow: with "Raven-Research" and "3 tools" a 196px card
showed only the arrow. The selection now renders as the arrow with its
first route, which keeps its width, then the routes after it. Those, and
the instance handle, start from nothing and grow into the room the line
has left (`flex: 1 1 0; max-width: max-content`), and the agent's name
gives way after them, with an ellipsis. The later routes keep room for a
letter and an ellipsis: narrower, a line's first character is clipped
rather than replaced by one. The 6px spacing separates fields only: the
two selection spans have no additional gap, and the comma-space
separator stays with the first route, outside the text that can be cut.
A card with no selection lays out as before, a long minted handle
included.
2. **Route conditions are filled and validated for params.** The
executor filled `${params.X}` only in `promptTemplate`, and
`validate_graph_nodes` checked only `promptTemplate` and `inputs`, so
"the deal is above ${params.threshold}" reached the judge as written and
`${params.nobody}` passed validation. Conditions now go through the same
`_fill_text` as the prompt, a secret withheld with the same log line,
naming the route, and a reference to no declared param is an error. The
filling and the check are on `main` now, carried by #1058; this branch
holds the executor's docstring and the tests that pin both.
3. **Hand-written route names `yes`, `no`, `on`, `off`, `true`, `false`
and bare numbers load.** The store parsed the spec block with
`yaml.safe_load`; YAML 1.1 read those keys as booleans or integers,
validation refused them, and the runtime dropped the playbook from the
library. Every mapping is now keyed by a name, and the value keeps its
YAML type. The loader is the one #1057 drafted, so the store carries one
key loader rather than two: a node that is not a mapping is left to
PyYAML to answer, which keeps the error the validate paths catch, and a
key node is retagged through a copy, so an anchor that shares one scalar
between a key and a value leaves the value's type alone. #1057 is closed
as superseded by this branch.
4. **The generator's dag-mode instructions describe branches.** That
section called `dependsOn` the whole graph language and never named
`routes`, `select` or `when`; it now describes a decision node and the
rule that a node depending on one names it in its `when`, and says a
condition may use a param. A new test reads that section on its own.
5. **An ordinary node's judge request is held to a literal golden.**
`test_judge_without_routes_sends_the_old_request_verbatim` compared the
request with the module's own constants, so changing the instruction or
a tool schema's description passed. It now compares the whole request --
messages, tools, model, tool choice -- with the fence nonce fixed,
against `tests/data/dag_verdict_plain_request.json`. The golden is the
request from before #1001: the tree before it and this one send
byte-identical requests over 32 input combinations.
6. **Domain terms around branches.** `CONTEXT.md` gains three `_Avoid_`
notes: "decision" alone for a decision node or for what it chose, since
the word already names the main agent's answer to an exception report in
a DAG run and a choice the host puts to the user; "skipped" for a node a
branch left out; and "branch" for any other part of a graph. The uses
those notes retire are rewritten in the files this pull request touches:
the DAG orchestration skill (the `abandon` paragraphs and the
conditional-branch section), the orchestration and playbook pages in
both languages, a runner docstring, and test docstrings, one test helper
and one test name; two ui-tui test comments that called a path through a
drawn graph a branch now say path and edge. The playbook page's "not a
runtime branching language" had read, since #1001, as saying a
prompt-mode playbook cannot branch; it now says the guidance decides
nothing while the graph runs, and a branch is chosen by a decision node
in the graph it composes.

Left as they are, outside the files this pull request touches:
"decision" alone still names a decision node or its selection in text a
model reads -- the `routes` description in the `run_subagent_dag` schema
and on `DagNodeSpec`, which the playbook generator's tool carries; the
guard-size error in `raven/agent/subagent/dag_guard.py`; the note
`raven/agent/subagent/dag_render.py` writes into a sub-agent's prompt;
the workflow compiler's prompt -- and in the RPC model's field
description and both generated clients, the "Decision missing" reason in
`i18n/messages.json`, the DAG Panel entry of `ui-tui/CONTEXT.md` (which
`test_the_tui_glossary_says_what_the_turned_in_line_carries` quotes),
and docstrings, comments and test names across the DAG modules and both
UIs. Retiring those reaches the RPC schema, the generated clients, a
translated label and pinned tests, so it is left to #1065. Python test
function names, which predate the note, were left alone in these files
too.

Two things the review of this pull request left open are closed here:

- **The card's layout test now lives where a real-environment test
belongs.** `tests/test_web_task_card_route_real_browser.py` measured a
card in a real Chromium, so it moved to
`tests/integration/test_web_task_card_real_browser.py`, and a new
`web-browser` CI job runs it on a runner that ships a system Chrome. It
is the only place the tasks board card's agent line is measured:
happy-dom lays nothing out, so the stylesheet gate beside it can only
read the declarations. Without a browser it skips with the reason,
except under `CI`, where a skip would leave the line unmeasured on every
push and so fails instead.
- **The store's loader answers a node that is not a mapping the way
PyYAML does.** `raven playbook validate` now answers a YAML syntax error
in a spec block as a finding, the way the RPC validator already did,
instead of a traceback.

The branch is rebased onto `main` at `471842132` (#1114). The
`CHANGELOG.md` conflict that rebasing raised is resolved by keeping both
sides' entries, and the branch merges clean.

## Type

- [x] Fix
- [ ] Feature
- [ ] Docs
- [ ] CI / tooling
- [ ] Refactor
- [ ] Other

## Verification

- [x] Relevant tests pass locally
- [x] Relevant lint / type checks pass locally
- [x] User-facing docs or screenshots are updated when needed

Native Windows 11, CPython 3.12.13:

- `uv run --frozen --no-sync pytest tests/test_playbook_store.py
tests/test_playbook_executor.py tests/test_playbook_validate.py
tests/test_subagent_dag_verdict.py tests/test_subagent_dag_core.py
tests/test_cli_playbook_commands.py
tests/integration/test_web_task_card_real_browser.py -m "integration or
not integration" -q -p no:randomly --tb=short`: 770 passed, 11 failed,
15 skipped. The same 11 fail at the merge base on this host, with the
same IDs:
`test_playbook_executor.py::test_a_playbook_run_s_node_attempts_nest_under_the_call_that_ran_it`,
`test_playbook_validate.py::test_path_confinement_waits_for_dispatch_roots`,
`test_subagent_dag_core.py::test_replanning_refuses_a_redeclared_id`,
`test_subagent_dag_core.py::test_a_refused_submission_still_records_the_link_as_unstarted`,
the five
`test_subagent_dag_core.py::test_a_reference_landing_in_either_root_is_accepted`
parameters, and
`test_cli_playbook_commands.py::test_run_judges_its_nodes_so_a_branching_playbook_takes_its_branch`
and `::test_run_judges_on_the_users_own_subagent_dag_settings`. The
first group is the file-reference rules #1066 fixes, and the last two
shell out to `cat`. The branch adds the passing cases.
- The baseline comparison above, run over the same six files at the
merge base: 746 passed, 11 failed, 15 skipped -- the same failing IDs.
- `uv run --frozen --no-sync pytest
tests/integration/test_web_task_card_real_browser.py -m integration -q
-n 0 -p no:randomly --tb=short`: 10 passed, the board's real 196px card
measured in a real Chromium. Playwright could not create its artifacts
directory under this host's default temp path, so the run needed a
writable `TEMP`; with the default one the nine browser cases skip and
the path-lookup case still passes.
- After the rebase onto `main` at `471842132`, with `-n 0`: the six
playbook/DAG files (`tests/test_playbook_store.py`,
`tests/test_playbook_executor.py`, `tests/test_playbook_validate.py`,
`tests/test_subagent_dag_verdict.py`, `tests/test_subagent_dag_core.py`,
`tests/test_cli_playbook_commands.py`): 790 passed;
`tests/integration/test_web_task_card_real_browser.py -m integration`:
10 passed; `scripts/check_commit_messages.py`,
`scripts/check_source_language.py` and `scripts/check_large_files.py`
over `origin/main..HEAD`, and `git diff --check`: passed.
- `uv run --frozen --no-sync ruff check raven/playbook/store.py
raven/cli/playbook_commands.py tests/test_playbook_store.py
tests/test_cli_playbook_commands.py
tests/integration/test_web_task_card_real_browser.py` and `ruff format
--check` over the same files: passed.
- `uv run --frozen --no-sync ty check raven/playbook/store.py
raven/cli/playbook_commands.py`: passed.
- `uv run --frozen --no-sync lint-imports`: 10 kept, 0 broken.
- `uv run --frozen --no-sync --extra dev python
scripts/check_commit_messages.py origin/main..HEAD`,
`scripts/check_source_language.py origin/main..HEAD` and
`scripts/check_large_files.py origin/main..HEAD`: passed.
- `git merge-tree` against `origin/main`: no conflict, through
`--write-tree` on the Windows host and through the three-argument form
on the Linux workstation, whose git predates `--write-tree`. Before the
rebase the same check reported one conflict, in `CHANGELOG.md`.

ui-web, run in `ui-web/`:

- `node node_modules/vitest/vitest.mjs run
src/features/tasks/BoardCard.test.tsx
scripts/gates/task-card-selection-css.test.mjs --no-file-parallelism`: 2
files, 36 tests passed.
- `node node_modules/vitest/vitest.mjs run src/features/tasks
scripts/gates --no-file-parallelism --testTimeout=60000`: 62 files, 560
tests passed. The same command over `main` at the time of the run
reported 61 files and 552 tests passed; the branch adds the stylesheet
gate and the cases beside it. Without the longer timeout two of the
gates time out on a loaded host; they pass on their own and the files
they read are byte-identical to `main`.

At head `211d226d`, the branch's CI run: 23 checks passed, 2 skipped
(`claude`, `docs deploy`), 0 failed. The rebased head re-runs the same
workflow, including the `web layout in a real browser` job, which runs
the browser cases on a runner that ships a system Chrome.

Not run here: macOS, a browser other than Chromium, and the full
repository suite.

## Risk

A stored playbook whose route condition names an undeclared param now
fails validation, and so leaves the library, as one doing the same in
`promptTemplate` already did. On the board, a crowded agent line now
shortens the agent's name where it used to cut the selection. The
store's loader now reads the frontmatter through the same path as the
spec block; a frontmatter key that YAML 1.1 would have resolved to a
boolean or a number is now read as the text it was written as, which the
frontmatter's two string fields never were.

- [x] Security impact considered
- [x] Backward compatibility considered
- [x] Rollback path is clear for risky changes

## Related Issues

#1008 (items 1 to 6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants