Skip to content

feat: add passive CLI doctor diagnostics and on-demand agent - #380

Merged
Brian Krabach (bkrabach) merged 2 commits into
mainfrom
feat/passive-cli-doctor
Oct 24, 2026
Merged

Brian Krabach (bkrabach) merged 2 commits into
mainfrom
feat/passive-cli-doctor

Conversation

@bkrabach

@bkrabach Brian Krabach (bkrabach) commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a CLI-local diagnostic path that can collect useful evidence without activating the normal application runtime:

  • A reusable, bounded passive collector in amplifier_app_cli/lib/diagnostics.py, exposed through the thin Click command amplifier doctor [--format text|json] [--session EXACT_ID].
  • Runtime/distribution identity plus per-scope, redacted declaration shapes and opaque report-local equality references. These are not merged/effective settings or live-session observations; unknown state stays explicit.
  • An on-demand app-cli:cli-doctor agent. It consults the existing amplifier-config skill only for configuration cases. It adds no duplicate skill, always-on instruction body, or replacement root.

This separates passive evidence collection from configuration resolution, live probes, and repair, so a diagnosis request does not silently authorize environment changes.

Scope and compatibility

  • No automatic repairs, fetches, installations, module mounting, credential loading, or live probes. health.status is always not_assessed; collection complete means collection completed, not that the environment is healthy.
  • Exact leading doctor bypasses normal startup for the published console, package-module entry, and legacy main:main launcher. A lightweight dispatcher and guarded, in-place ordinary initialization preserve TLS setup, home-ownership enforcement, shell completion, and existing global patch points. Preceding launcher arguments do not take the passive route.
  • Static public package/command exports lazily resolve the real Click objects, including with foreign arguments or submodule-first imports. Narrow, supported ModuleType attribute hooks on the existing module objects resolve collisions with the actual same-named imported children; they do not replace modules or introduce aliases.
  • The bounded diagnostic YAML parser reports aliases, explicit tags, and duplicate keys as unsupported_yaml. This is a collection limit, not a claim that runtime settings are invalid or an instruction to repair them. Runtime settings parsing is unchanged.
  • Paths in reports can identify the local environment. Review before sharing; reports are not uploaded automatically. Python interpreter startup and its initial import-cache handling are outside the application passivity guarantee.

Breaking changes: none intended. The published console target moves to the lightweight dispatcher; the legacy amplifier_app_cli.main:main entry remains supported.

Verification

Pre-publication results below are from isolated Linux ARM64, Python 3.11.14 environments with locked dependencies, not a claim about other platforms:

Check Result and limit
Focused doctor/startup/compatibility tests 300 passed
Integration selection 13 passed
Full default suite 2,893 passed, 122 failed, 4 skipped, 13 deselected, 1 xfailed, 2 warnings
Independently installed clean base 4801c9d7bdf7e60159e21234b22e0c50c0bb214e 2,759 passed; the same 122 failed test IDs
Non-editable installed wheel, outside the checkout Passed for the new console, package -m, legacy module -m, and an actual installer-generated legacy main:main console
Static checks Ruff 0.15.5 passed all 13 changed/new Python files; Python 3.11 syntax and git diff --check passed

The full suite is not green. No new failed IDs were observed against that independently installed base. The matching failures were in offline provider-source normalization/resolution paths: 121 direct/captured failures plus one concurrency assertion after two provider-resolution thread errors. This comparison does not authorize ignoring a remote failure.

The installed-wheel checks cover ownership refusal and explicit override, argument errors/help, and exact-session selection. The safety probe records forbidden attempts even when application code swallows their exceptions, so refusal cannot be mistaken for successful passive collection.

The raw strict syscall harness remains false, not an all-process zero-write pass: it observed 3 console / 15 module standard-library bytecode-cache EACCES write attempts, all before the first package access. Paths and counts matched the corresponding python -c / python -m controls. No application-owned writes, network activity, extra subprocesses, or persistent changes were observed. This is bounded evidence, not universal passivity.

One prior real-provider observation exercised actual delegation, the child's doctor Bash tool call, and loading the installed amplifier-config skill. That observation kept effective/live state unknown and gave no health or repair verdict. It is not evidence of universal model reliability; the agent was unchanged afterward and no additional live-model trial is claimed.

The only post-runtime-test changes were two inline lint annotations for intentional bootstrap imports in tests/doctor_safety_probe.py; the tested and committed Python ASTs match exactly, and all other file hashes match the tested snapshot. No full Pyright result is claimed.

An independent source/privacy review and separate Astra and Fable packet reviews found no demonstrated remaining publication blocker. Committed material is source and synthetic fixtures only: no actual diagnostic reports, private evidence links, or raw research logs are included.

To verify in an isolated development environment:

uv sync --locked --all-extras --dev
uv run pytest tests/test_doctor.py tests/test_doctor_entrypoint.py tests/test_cli_doctor_agent.py -q
uv run pytest -q
uv run pytest -m integration -q

The added CI step builds and installs a non-editable wheel, then runs tests/check_doctor_wheel.py with tests/doctor_safety_probe.py from outside the checkout, including the installer-generated legacy launcher.

Remaining checks

Remote verification is complete for exact head 3b92f17fea1f4ce4c634b2f35c3be465950292eb: CI run 40064089760 passed all eight jobs (Ubuntu/macOS/Windows Python 3.11/3.12 unit tests and both POSIX integration jobs), including the Ubuntu Python 3.11 installed-wheel step. The CLA check passed. GitHub still requires review; the owner explicitly authorized an admin merge to bypass that review-policy gate. The historical local-baseline, interpreter-startup passivity, and live-agent coverage limitations above remain unchanged.

Windows CI follow-up

Initial CI run 40063047913 failed on Windows Python 3.11/3.12 during pytest setup/teardown: auto-generated parameter IDs for oversized YAML payloads made PYTEST_CURRENT_TEST exceed Windows' 32,767-character environment-variable limit. Commit 3b92f17fea1f4ce4c634b2f35c3be465950292eb adds 15 short, explicit IDs to test_bounded_settings in tests/test_doctor.py. All 15 payloads and assertions are unchanged; there are no new skips, xfails, or exclusions, and no production-code changes. This is an additional test-only change after the runtime verification documented above. Whole-file AST equality holds after removing only the new ids keyword. A standalone pytest 9.0.3 metadata smoke using the actual decorator inputs passed all 15 cases with PYTEST_CURRENT_TEST below 1,024 characters; this is not a collector test or proof on Windows. Full remote CI for this exact new head has now passed all eight jobs in run 40064089760, including both Windows versions and the installed-wheel step; CLA also passed. The earlier local-baseline and passivity/model-coverage limitations still apply.

Add a bounded provenance collector with a thin Click command, an on-demand diagnostic agent, compatibility-preserving entry points, and installed-wheel safety coverage.

Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@bkrabach
Brian Krabach (bkrabach) merged commit b7e533a into main Oct 24, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants