Skip to content

feat(testing): resolve a browser_driver setting and stop by name when its tool is missing #8539

Closed

Description

@kyle-sexton

Refs: #7580

Agent Brief

Type: Feature
Summary: Add a browser_driver setting, default playwright-cli, that resolves through the testing plugin's config cascade, and make /testing:run-e2e stop with a named message when the resolved tool is not installed.

Current behavior:

  • The testing plugin resolves /testing:run-e2e's keys with plugins/testing/scripts/resolve-config.sh e2e: e2e_driver (auto | harness | run | playwright | chrome), reuse_running_instance and feature_map_dir (resolve-config.sh:50-69, key loop at resolve-config.sh:284-288). Layers, highest first: .claude/testing.local.yaml, docs/conventions/testing.yaml, ~/.claude/testing.yaml, then the --user value from userConfig.
  • e2e_driver says who drives the flow. Nothing names which browser tool a playwright drive uses, and no skill checks that the tool resolves before a run. /playwright:check checks command -v playwright-cli only when someone runs it (plugins/playwright/skills/check/SKILL.md:26).
  • The testing scan's config loader skips run-e2e's top-level keys by name (plugins/testing/skills/audit/scripts/adapter-load.awk:311); a new top-level key not in that list is parsed as a scan key.
  • The config-cascade Implementers row for testing (run-e2e) lists its keys (docs/conventions/config-cascade/README.md:649).
  • The gating probe is done (debugging: probe disallowed-tools MCP pattern and large-heap converter #7593, Probe result 2026-10-16, Claude Code 2.1.301): a skill's disallowed-tools entry mcp__context7__* or mcp__context7 removed that server's tools while a tool from another server stayed allowed; and analyze-profile's converter handled a 510.9 MiB heap in 59 s at 3.73 GiB peak memory.

Desired behavior:

  • browser_driver is a top-level key of the testing plugin's YAML layers and a userConfig option, default playwright-cli, scope repo. Its value is the command name of one browser tool (letters, digits, ., _, -); any other value is named on stderr and the key takes its default, the same rule e2e_driver follows.
  • resolve-config.sh e2e prints a browser_driver <tab> <value> <tab> <source> line, with the same layer order as e2e_driver.
  • /testing:run-e2e, when the driver chosen for the run is playwright (pinned, or picked by auto), checks that the resolved browser_driver command resolves on PATH before the first drive. When it does not, the run stops before driving anything with a message that names the setting, the value, where the value came from, and the two ways forward: install that tool, or set browser_driver to a browser tool the user has. The message says other browser tools are not used as a silent substitute. The run never switches tools partway through.
  • The check is a script (or a resolve-config.sh mode) the skill calls, so a test can drive it; the skill text gains the step and does not remove any existing text.
  • testing.schema.json, the plugin's userConfig, adapter-load.awk:311's skip list, plugins/testing/skills/run-e2e/context/e2e-config.md and the Implementers row at docs/conventions/config-cascade/README.md:649 all name the new key.

Known call (2026-10-20):

  • The PATH check runs whenever the driver chosen for the run is playwright, whether pinned or picked by auto (auto ranks harness, then run, then playwright: plugins/testing/skills/run-e2e/context/e2e.md:85-95).
  • run-e2e/SKILL.md edits are added lines only. Leave line 139 ("Five keys govern this run") and the resolve-config.sh e2e --user ... command line below it intact, and add the browser_driver resolution as new lines (for example a separate --user 'browser_driver=${user_config.browser_driver}' call). Changing a word in an existing line counts as a deleted line.
  • The Implementers-row edit touches the location cell only, so the generated glance table does not change and is not regenerated (only the Who wins and Merge form cells render, per scripts/sync-config-cascade-semantics.py's docstring).
  • "Scope repo" means a repository-scope surface (docs/conventions/config-cascade/README.md:354).
  • Test suites and CI-only gates are verified by the PR's CI, not run in the lane.
  • Uncertainty is Known: the three open questions (whether auto counts, whether the glance table regenerates, what scope repo means) have cited answers above, per poteto's slice research of 2026-10-20.

Key interfaces:

  • plugins/testing/scripts/resolve-config.sh (e2e mode) and plugins/testing/scripts/resolve-config.test.sh.
  • plugins/testing/.claude-plugin/plugin.json (userConfig.browser_driver), plugins/testing/schemas/testing.schema.json.
  • plugins/testing/skills/run-e2e/SKILL.md (the step that resolves the five keys, around SKILL.md:139-165) and context/e2e-config.md.
  • plugins/testing/skills/audit/scripts/adapter-load.awk:311.
  • docs/conventions/config-cascade/README.md Implementers row (location cell only; the glance table is unchanged).

Acceptance criteria:

  • resolve-config.test.sh has cases for: no layer sets the key, so it prints browser_driver, playwright-cli, default; docs/conventions/testing.yaml sets it, so that file is the source; an invalid value is named on stderr and the default stands.
  • resolve-config.test.sh has a case where browser_driver names a command not on PATH and the check exits non-zero with a message containing browser_driver, the command name and the source; and a control case where the command is a stub on PATH and the check exits 0.
  • The PR's CI runs resolve-config.test.sh (test-bash) and it passes.
  • Each of plugins/testing/skills/run-e2e/SKILL.md, plugins/testing/schemas/testing.schema.json, plugins/testing/.claude-plugin/plugin.json, plugins/testing/skills/audit/scripts/adapter-load.awk and docs/conventions/config-cascade/README.md mentions browser_driver.
  • The PR deletes no line from plugins/testing/skills/run-e2e/SKILL.md.
  • run-e2e's evals/evals.json gains one case where browser_driver resolves to a missing tool and the expected output is the named stop with no drive attempted.
  • The PR's CI runs python3 scripts/sync-config-cascade-semantics.py --check and scripts/check-convention-yaml.sh --check, and both pass.
  • scripts/check-changelog-fragments.sh --check passes, with one minor fragment for testing. The PR's CI runs scripts/check-changelog-fragments.sh --check-required origin/main and it passes.
  • CI's test-bash (resolve-config suite), lint-repo (cascade glance table, convention YAML) and check-skills (skill gate, eval sets) pass.

Out of scope:

Origin:

  • One named tool per job, no MCP handoff; the skill names the command, checks it resolves and stops if missing: hers (Lauren's Playwright plus CDP) and the owner's ("follow Lauren or better"), Q561, D55, D116.
  • A setting with a named default read through the cascade; use the tool the user has, one per run, never switch mid-run: owner's, Q569. This brief meets "use the tool the user has" by naming the setting in the stop message instead of switching tools on its own.
  • Setting scope repo: ours, from the thread's Key interfaces.
  • Known call (check on the chosen driver, added lines only, no glance-table regeneration): poteto's slice research, 2026-10-20, recorded by Issue Triage.

Do not run with: #7581 slices (testing and app-control), and any Phase 2 slice still open that edits plugins/testing/skills/audit/ (PLAN's file-overlap matrix pairs Phases 2 and 13).


Slice of #7580, drafted by the Issue Triage session from the thread's recorded decisions.

Activity

  1. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Oct 20, 2026
  2. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 20, 2026
  3. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 20, 2026
  4. melodic-automation-lanes commented on Oct 20, 2026

    @melodic-automation-lanes
    Contributor

    The lanes App's pull request #8720 links this issue with Refs:, not Closes, so the merge lane never merges it. report-outcome added needs-human so a person merges or closes it. When it merges, work-unblock removes this needs-human and agent-ready and adds needs-triage, so triage decides what work is left. To keep the issue held after the merge, add needs-owner-decision.

  5. kyle-sexton commented on Oct 20, 2026

    @kyle-sexton
    MemberAuthor

    Delivered by #8720 (merged), which referenced this issue with Refs: rather than a closing keyword.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.status: readyTriaged, unblocked, and fully specified; eligible to pick up.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions