[liwenjie200543]

What

The setup preflight now detects the active harness from its environment markers and reads only that harness's installed versions. Missing, failed, empty or malformed listings stay unknown rather than becoming evidence that the whole project floor is absent.

Codex reads the installed array in its installed / available JSON envelope. A redacted live Codex 0.149.1 format fixture is included; the same envelope was also checked with the desktop-bundled 0.163 CLI. Non-Claude listings need an exact floor-plugin name (Codex) or the exact magpie extension name (Gemini) before they are trusted. Gemini's single extension version is compared against every family in the floor, so an older extension reports outdated, while an extension at or above the floor does not report missing families. Gemini uses the exact-name safeguard pending a live-format fixture.

The Codex refresh command is consistently codex plugin marketplace upgrade apache-magpie, verified with CLI help and the official plugin documentation. The missing-CLI print path has its own eval again, alongside separate Codex/Gemini cases. Main's five newer welcome cases are retained, with 18 preflight-floor cases in total. The no-CLI case supplies a captured listing through --plugin-list and a reconciliation stamp, isolating printing from unrelated setup proposals. The unadopted fixture explicitly requests adoption information while only permitting personal configuration. The fixture output contract explicitly excludes welcome-recording writes from installation commands.

How checked

• prek run --all-files on Ubuntu 24.04.5: 40 passed, 0 failed, including workspace Ruff, Mypy, pytest, link checks, the rendered-fixture drift guard and token stamps.
• Independent preflight-floor evaluation: GPT-5.6 Terra, desktop-bundled Codex CLI 0.163.0-alpha.10.2. The model received only the runner's system/user prompts in an isolated empty directory; expected answers were used only by the grader. The original suite assertions were retained.

PASS    preflight-floor/case-1-at-floor
PASS    preflight-floor/case-10-utilities-first-run
PASS    preflight-floor/case-11-mentoring-first-run
PASS    preflight-floor/case-12-pairing-first-run
PASS    preflight-floor/case-13-issue-first-run
PASS    preflight-floor/case-14-pr-management-first-run
PASS    preflight-floor/case-15-contributor-growth-first-run
PASS    preflight-floor/case-16-below-floor-codex
PASS    preflight-floor/case-17-below-floor-gemini
PASS    preflight-floor/case-18-at-floor-gemini
PASS    preflight-floor/case-2-below-floor
PASS    preflight-floor/case-3-plugin-missing
PASS    preflight-floor/case-4-foreign-marketplace
PASS    preflight-floor/case-5-no-claude-cli
PASS    preflight-floor/case-6-unadopted
PASS    preflight-floor/case-7-legitimate-url-alarming-context
PASS    preflight-floor/case-8-unknown-plugin-list
PASS    preflight-floor/case-9-first-magpie-run
Ran 18 cases: 18 passed, 0 failed, 0 manual, 0 errored

AI assistance: this update was prepared and validated with Codex; the independent behavioral evaluation used GPT-5.6 Terra.

[Kaap10]

Thanks @liwenjie200543 for working on #1641! Broadening setup-preflight version floor detection across other harnesses is an important improvement.

There are a few functional blockers with the current implementation that prevent it from working in live environments, plus a couple of edge cases to address:

---

1. Major: Missing JSON flags for gemini and codex listing commands

In plugins/magpie-setup/skills/setup/setup_preflight/cli.py:

HARNESS_LISTINGS: tuple[tuple[str, list[str]], ...] = (
    ("claude-code", ["claude", "plugin", "list", "--json"]),
    ("codex", ["codex", "plugin", "list"]),
    ("gemini", ["gemini", "extensions", "list"]),
)

• Gemini CLI: The default output format for gemini extensions list is human-readable text (--output-format choices: ["text", "json"], default: "text"). Without -o json (or --output-format json), it prints plain text (e.g. No extensions installed. or text lines).
• Codex CLI: codex plugin list also prints a formatted terminal summary by default. It requires --json (codex plugin list --json) to emit a JSON array.

Impact: In actual environments, completed.stdout contains plain text. _parse_listing catches json.JSONDecodeError and returns None (unknown). As a result, the floor check continues to be silently skipped on live Codex and Gemini runs. The unit tests in test_cli.py only passed because the mock returned JSON for a command that does not emit JSON in reality.

---

2. Major: Sequential PATH probing vs detecting the active running harness

In plugins/magpie-setup/skills/setup/setup_preflight/cli.py:

for _harness, argv in HARNESS_LISTINGS:
    ...

• Issue #1641 specifically directed: "Detect the running harness, and read its installed versions with its own listing command".
• When multiple AI CLIs are installed on the same machine (e.g. claude installed globally via npm/brew, but the user is currently working inside a Gemini CLI or Codex session), the sequential loop finds claude on PATH first and queries it. If Claude doesn't have the Magpie plugin installed, it returns None and halts, completely ignoring the running Gemini or Codex session.
• Magpie already has canonical harness-detection patterns (see tools/adversarial-review/src/adversarial_review/detect.py) using ambient environment variables:
  • CLAUDECODE == "1" ➔ Claude Code
  • CODEX_SANDBOX or CODEX_THREAD_ID ➔ Codex
  • GEMINI_CLI == "1" or GEMINI_PROJECT_DIR in env ➔ Gemini CLI
  • COPILOT_CLI ➔ Copilot CLI

The active running harness should be detected first; sequential PATH probing should only be an ambiguous fallback if no active harness environment is detected.

---

3. Major: Missing preflight-floor eval suite coverage

Issue #1641 asked to:

"Add preflight-floor cases for a below-floor Codex and Gemini install."

tools/skill-evals/evals/setup/preflight-floor/fixtures/case-5-no-claude-cli/report.md currently records the previous behavior where Gemini 0.2.0 was ignored. Updating this fixture or adding dedicated eval cases (and validating with python3 tools/skill-evals/render-preflight-fixtures.py --check) is needed to protect against future regressions.

---

4. Minor / Nits:

• Copilot CLI: Mentioned in the title and description of #1641 ("setup-preflight: the version floor is never checked on Codex, Gemini CLI or Copilot CLI"), but omitted from HARNESS_LISTINGS.
• Unit test mock fidelity (test_cli.py): In test_a_missing_claude_binary_falls_through_to_codex, the mock should assert that --json / -o json is present in argv so realistic CLI arguments are verified.
• CLI help string in build_parser():

  parser.add_argument(
      "--plugin-list",
      metavar="PATH",
      help="`claude plugin list --json` output to use instead of running it ('-' for stdin)",
  )

  Now that listing is harness-agnostic, the help text should say: "harness plugin listing JSON to use instead of running it ('-' for stdin)".

[potiuk]

Thanks for picking up #1641 — the shared _parse_listing and the tests are a good base, but the probe can answer from the wrong harness, and the Codex and Gemini listings will most likely never parse, so the floor stays unchecked where the issue needs it.

Why an unreadable result is unknown rather than absent. (plugins/magpie-setup/skills/setup/setup_preflight/cli.py:111)

Why an unreadable result is unknown rather than absent. Inside a
sandboxed session the plugin cache is read-denied and claude plugin list --json returns [] there — that reads exactly like "nothing installed"
but is not. Acting on it would propose installing a project's entire floor
on every sandboxed run.

        except (OSError, subprocess.SubprocessError):
            # The binary is not on PATH: this is not the running harness,
            # so the next harness's own command decides.
            continue

The handler treats every OSError and SubprocessError as "binary not on PATH", but that set also covers a claude that exists and times out (TimeoutExpired) or that the sandbox refuses to execute (PermissionError). Both then fall through to the next harness, which contradicts the rule the PR states itself ("the first binary that exists decides the answer") and borrows another harness's listing for an unreadable one. Checked against the PR head: with claude raising TimeoutExpired or PermissionError and codex returning a listing, read_installed(None) returns the Codex map after probing ['claude', 'codex'].

Suggested fix: Fall through only on FileNotFoundError; any other OSError or SubprocessError from a binary that exists returns None. Add a test for each (TimeoutExpired, PermissionError) asserting None and that codex is never probed.

The actions. (plugins/magpie-setup/skills/setup/setup_preflight/cli.py:56)

The actions.

• a floor plugin absent → claude plugin install <plugin>@apache-magpie;
• a floor plugin below min_version → claude plugin update <plugin>@apache-magpie.

Where there is no such CLI, run nothing and print the commands instead.

HARNESS_LISTINGS: tuple[tuple[str, list[str]], ...] = (
    ("claude-code", ["claude", "plugin", "list", "--json"]),
    ("codex", ["codex", "plugin", "list"]),
    ("gemini", ["gemini", "extensions", "list"]),
)

The first binary on PATH is not the running harness. Many maintainers have several harnesses installed, and the floor is then checked against whichever comes first in this tuple. This is not hypothetical: in the sandboxed Claude Code session this review ran in, claude is not on PATH while codex and gemini are, so the Codex listing would answer for a Claude Code session. The finding that follows also proposes the claude plugin install/update commands from step-3, whichever harness's listing produced it, and the _harness id is discarded, so nothing downstream can tell which install was measured. Issue #1641 asks to "detect the running harness, and read its installed versions with its own listing command".

Suggested fix: Detect the running harness first (the signals in install.md Step M2, for example the harness's own environment variables), run only that harness's listing, and return the harness id with the map so the findings name that harness's commands. When the harness cannot be detected, stay unknown rather than probing in a fixed order.

Smaller observations

• See inline comments on plugins/magpie-setup/skills/setup/setup_preflight/cli.py:58, tools/setup-preflight/tests/test_cli.py:147.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
Contributing guide.

[liwenjie200543]

Pushed a rework addressing both reviews:

• Detection, not probing. The running harness is now detected from
  its own environment markers — the same innermost-first order
  tools/adversarial-review/backends.py uses, CLAUDECODE last — and
  only that harness's listing runs. read_installed returns
  (harness, versions); the below-floor finding carries harness and
  the cache key includes it, so a verdict computed under one harness is
  never served to another. An undetectable session stays unknown; no
  fixed-order probing remains.
• Listings that parse. Codex gets --json, Gemini gets -o json,
  and the shared parser reads both. Copilot has no listing CLI (it
  installs from a repo URL), so its entry is None and the floor stays
  unknown there rather than guessing.
• Unreadable is unknown, never borrowed. The listing command runs
  once; missing, timed-out, refused, non-zero and unparsable all
  collapse to (None, None) — TimeoutExpired and PermissionError
  each have a test asserting no other harness is probed.
• Per-harness actions. step-3.md's actions are now a table naming
  each harness's own install/update commands (Codex plugin add/update, Gemini extensions install/update), with the Gemini
  one-extension note.
• Eval coverage. preflight-floor gains the below-floor Codex and
  Gemini cases from issue setup-preflight: the version floor is never checked on Codex, Gemini CLI or Copilot CLI #1641: case-5 is now a Gemini below-floor
  update, and case-9 adds a below-floor Codex install.

Verification: tools/setup-preflight pytest 147 passed, ruff check +
format clean, mypy clean (local toolchain). Happy to adjust anything
the detection contract should spell differently.

[potiuk]

Thanks for this round. Every code point from my two earlier reviews is addressed:

• the running harness is detected from its own markers;
• a missing, timed-out or refused binary stays unknown and never falls through to another harness's answer;
• Gemini's single magpie extension is compared against every floor entry and reported as outdated, never missing, with the parametrised above-floor CLI test;
• Codex reads only the installed array, backed by a real 0.149.1 fixture, and needs an exact floor name before its listing is believed;
• the Codex update command has one spelling in every doc and in the eval;
• case-5 is the print-path eval again, and the Gemini cases are new cases.

What was left was small, so rather than send it back I pushed a fixup commit (d503ae16) to your branch:

• setup_preflight/cli.py: the --plugin-list path now detects the running harness too, so a supplied Gemini listing keeps the single-extension rule and a supplied Codex envelope is unwrapped. Before, it always got harness=None, and a Gemini magpie entry read as every floor plugin missing. Two new tests in tests/test_cli.py cover it.
• sections/step-3.md: one paragraph saying what to print when facts.harness is null, which until now the evals could only infer from a case's prose. The pre-flight eval reports were re-rendered to match.
• case-17-below-floor-gemini and case-18-at-floor-gemini got the same local-smoke / smoke tags as the Codex case, so the Gemini cases run in smoke runs.

I also trimmed the labels to what the change implements; most of the 24 came in from main through the merge commits.

One note for whoever lands #1724, #1725 or #1726 next: they add case-16-* fixtures and edit the same case-count lines, so they will need a rebase and renumbering.

Thanks!

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
Contributing guide.