test(mcp): pin official 2026-07-28 conformance baseline - #1808
Merged
groupthinking merged 6 commits intoSep 12, 2026
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update baseline for MCP conformance on 2026-07-28
test(mcp): pin official 2026-07-28 conformance baseline
Sep 8, 2026
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Owner
Strategic compatibility receipt — 2026-09-09 — WATCHNew primary evidence: MCP merged the Filesystems Working Group charter on 2026-09-08, commit aa8ce049. The charter itself was initially authored August 24. VERIFIED: the WG plans a Resources write extension (create/update/delete/stat), optimistic concurrency/create-if-absent, and notification/cache interaction. The deliverable is explicitly Ideating. This is not an accepted write-method contract, SDK release, or evidence of production enterprise adoption. The charter excludes a parallel files/* primitive, host sandbox/local-disk semantics, and application write-authorization policy. Interpretation for #1661 / Agent Factory: retain an adapter boundary for future resource writes and durable receipts, while preserving independent authorization, origin isolation, conflict handling and replay protection. Do not implement guessed wire methods or label this fixture baseline as conformance for future writes. Queued acceptance gate: only add write fixtures after an accepted, pinned Extensions Track SEP and supported SDK contract; require negative tests for lost updates, create-if-absent conflicts, cross-origin writes, revocation and stale notifications. Existing pinned baseline remains unchanged. Fixtures certify their tested contract, not the deployed MCP surfaces. Action: attached this dated scope/acceptance note to the existing conformance PR; no new competing issue or runtime activation. Follow-up effort: 1–2 hours for contract review when the SEP advances. Current Copilot-head CI reports action_required, not passed. |
Owner
Upstream pin now predates official Skills conformanceVerified: official MCP conformance added eight executable SEP-2640 Skills scenarios on 2026-09-11 in commit 7169291 / PR #330. This PR currently pins Before approval:
Canonical crosswalk and scope are recorded on issue #1640. No merge or runtime activation requested. |
…s exit code, so a REQUIRED scenario whose runner reports failure but emits no FAILURE/WARNING checks is recorded as passing (fail-open).
This commit fixes the issue reported at scripts/testing/official_mcp_conformance.py:311
## Bug
`summarize_checks()` in `scripts/testing/official_mcp_conformance.py` computed `ok = not blocking`, where `blocking` was derived **only** from parsed `checks.json` statuses (`FAILURE`/`WARNING` for required scenarios). The conformance runner's process exit code (`result.returncode`), although captured into `record["exit_code"]`, was never consulted when computing `summary.ok` or `overall_ok`.
The official `CheckStatus` vocabulary is `SUCCESS | FAILURE | WARNING | SKIPPED | INFO`. A runner-level non-zero exit that produces only `SUCCESS`/`INFO`/`SKIPPED` checks therefore slips through as passing.
### Concrete trigger (already in the committed baseline)
`tests/fixtures/mcp_conformance/official-2026-07-28-receipt.json` demonstrates the failure mode. All four REQUIRED client auth scenarios report:
```
client auth/metadata-var2 exit=1 ok=True blocking=[]
client auth/token-endpoint-auth-basic exit=1 ok=True blocking=[]
client auth/token-endpoint-auth-post exit=1 ok=True blocking=[]
client auth/token-endpoint-auth-none exit=1 ok=True blocking=[]
```
Their checks contain only `INFO` and `SUCCESS` entries (e.g. `auth/metadata-var2`: `{'INFO': 22, 'SUCCESS': 18}`), so the FAILURE/WARNING blocklist is empty and each run is marked `ok: true`, yielding `overall_ok: true`. The runner signaled failure (exit 1) and the harness masked it — directly contradicting the acceptance criterion "fail closed on required checks; do not hide warnings in aggregate success."
`main()` returns `0 if receipt["overall_ok"] else 1`, so this fail-open behavior also masks real failures in CI.
## Fix
`summarize_checks()` now takes an `exit_code` parameter. For **required** scenarios, a non-zero exit code appends a synthetic blocking entry (`runner-exit-code:<n>`), forcing `ok = False`, which in turn drives `overall_ok = False`. Optional (non-required) scenarios and clean (exit 0) runs are unaffected.
Both `_run_server_scenario` and `_run_client_scenario` now pass `exit_code=result.returncode` into `summarize_checks`.
Verified behavior:
```
req exit1 no-fail: ok=False blocking=['runner-exit-code:1']
req exit0: ok=True blocking=[]
opt exit1: ok=True blocking=[]
```
Note: the committed receipt fixture still records `summary.ok: true`/`overall_ok: true` for the failing client runs; regenerating it (or reconciling why those clients exit 1) is follow-up work, but the harness now fails closed on the next run.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: groupthinking <garveyht@gmail.com>
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…keyword-only `exit_code` argument, so `test_required_checks_fail_closed_on_warning` and `test_unscored_failures_do_not_block_receipt` raise `TypeError` on every run. This commit fixes the issue reported at tests/unit/test_official_mcp_conformance.py:47 ## Bug Commit `258243d` changed the signature of `summarize_checks()` in `scripts/testing/official_mcp_conformance.py` to add a required keyword-only parameter: ```python def summarize_checks(checks, *, required: bool, exit_code: int) -> dict[str, Any]: ``` But the existing unit tests in `tests/unit/test_official_mcp_conformance.py` were not updated. Both `test_required_checks_fail_closed_on_warning` and `test_unscored_failures_do_not_block_receipt` invoke the function with only `required=...`, so they fail with: ``` TypeError: summarize_checks() missing 1 required keyword-only argument: 'exit_code' ``` **Trigger:** running either of these two tests. They fail unconditionally at the call site (collection-time TypeError), so this is a deterministic, always-failing test regression. ## Fix - Added `exit_code=0` to both existing calls, preserving the original assertions (a `WARNING` on a required check still blocks; an unscored `FAILURE` still does not block). - Added `test_required_checks_fail_closed_on_runner_exit_code`, which exercises the new behavior introduced by the fix commit: a required scenario with no `FAILURE`/`WARNING` checks but a non-zero runner exit code must fail closed with `blocking == ["runner-exit-code:1"]`. I verified all three scenarios by loading the module and calling `summarize_checks` directly (pytest is not installed in this environment); the outputs match the test assertions exactly. Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com> Co-authored-by: groupthinking <garveyht@gmail.com>
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches. Re-running this action after a short time may resolve the issue. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Contributor
🔍 PR Validation |
groupthinking
marked this pull request as ready for review
September 12, 2026 10:05
groupthinking
deleted the
copilot/test-mcp-baseline-official-2026-07-28-conformance
branch
September 12, 2026 10:05
groupthinking
approved these changes
Sep 12, 2026
Contributor
🔴 E2E Test Results: FAILURE DETECTED
Test Output |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canonical issue
Outcome
Adds a pinned, local-only official MCP conformance baseline for the
2026-07-28wire. The baseline certifies the implemented fixture surfaces, fails closed on required warnings/failures, preserves bare-origin PRMresourcevalues exactly through authorization and token exchange, and writes a machine-readable receipt for audit and CI use.Scope
Included:
a983ba93c91e0bb31d0b6849eeb52f0ad1083107tests/fixtures/mcp_conformance/official-2026-07-28-receipt.jsontools/list,tools/call, initialize/notification behavior, and OAuth metadata/token flowsresourceidentifier byte-for-byte instead of normalizing a bare origintools/listorder across three consecutive probesExplicitly excluded:
Risk
Verification
Checks run on head
edfbc7b31b534e60001eda6369edd2e2f180664d.pytest tests/unit/test_official_mcp_conformance.py --no-cov -qpython scripts/testing/official_mcp_conformance.py"overall_ok": truewith empty requiredwarnings/failuresProduction evidence
Not applicable. This PR adds a local-only conformance harness, fixture endpoints, and receipt generation; it does not change a production runtime path or deployment target.
Agent handoff