feat: lock complete tool annotations and output schemas (v4)

.github/workflows/integrity-gate.yml

@@ -96,7 +96,7 @@ jobs:
         run: pip show mcp | grep -i '^version'
 
       - name: Capture / parity / pagination tests under mcp 1.x
-        run: pytest -q tests/test_capture_model_dump.py tests/test_capture_pagination.py tests/test_e2e_pin_check.py tests/test_capture_http.py
+        run: pytest -q tests/test_capture_model_dump.py tests/test_capture_pagination.py tests/test_e2e_pin_check.py tests/test_capture_http.py tests/test_tool_integrity.py tests/test_tool_integrity_guard.py
 
   # --------------------------------------------------------------------------
   # Job 1b: lint gate (ruff)

CHANGELOG.md

@@ -30,6 +30,30 @@ Streamable HTTP; the v0.3 `guard` proxy adds deterministic runtime *result* insp
 
 ## [Unreleased]
 
+### Fixed — release validation
+
+- Refresh the dev/CI and Action dependency locks from PyJWT 2.13.0 to 2.15.1
+  with artifact hashes; the audited closure passes without advisory suppression.
+- Re-capture the three committed public examples at schema level 4 using the same
+  pinned server versions. Their prior fields are unchanged; new commitments are
+  reviewed example baselines, not transferred historical signatures.
+
+### Added — tool metadata integrity (DSE-1539)
+
+- Schema level **4** commits complete tool annotations and output schemas, including
+  output structural skeletons. Hint flips/removal and output changes cause drift;
+  schema-out-* classes distinguish structural/cosmetic changes. Python/TypeScript
+  share the vectors. The runtime tools/list gate compares new commitments for v4.
+- Missing/null hashes JSON null; {} is distinct. Invalid object fields and v4 locks
+  omitting commitments are refused. Raw annotations are not rendered, and hints
+  grant no capability or proof of behavioral safety.
+- V1–v3 locks remain readable. Approved legacy locks retain unapproved-change plus
+  migration; unapproved old locks also fail migration rather than claiming coverage.
+  Review/re-pin/re-approve for v4; historical signatures do not carry across.
+  Lock rotation preserves recorded schema.
+- The human checkpoint and protocol-neutral Warden direction over prompts, retrieval,
+  code, and serverless adapters are documented proposals, not shipped guarantees.
+
 ### Changed
 
 - **`mcp` SDK 2.x is now supported — the `<2` cap from #92 is lifted to `<3` (supersedes

DOCUMENTATION_INDEX.md

@@ -1,5 +1,7 @@
 # Documentation Index — mcp-warden
 
+Last Updated: 2026-10-02
+
 Master index of every document in this repository. The `docs/` files are the
 **security contract and source of truth** for all algorithms; the three core docs
 describe and visualize the implementation that satisfies that contract.
@@ -16,13 +18,17 @@ describe and visualize the implementation that satisfies that contract.
 
 ## Lock Format v1 conformance (`vectors/` + `@mcp-warden/lock` — DSE-1513)
 
+Schema level 4 adds annotation/output commitments (DSE-1539). The
+[upgrade plan and Warden checkpoint proposal](docs/plans/2026-10-02-tool-integrity-upgrade.md)
+separates implemented surface integrity from proposed protocol-neutral enforcement.
+
 The format is a standard, not a tool: a language-neutral corpus defines conformance and two
 implementations (Python reference, zero-dependency TypeScript) prove it in CI.
 
 | Artifact | Purpose |
 |----------|---------|
 | [`vectors/README.md`](vectors/README.md) | Consumer contract: manifest schema, the four vector kinds, the surface document shape, how a third implementation runs the corpus |
-| [`vectors/manifest.json`](vectors/manifest.json) + [`vectors/cases/`](vectors/cases/) | 77 generated vectors — canonical (RFC 8785), digest, drift (every `WRD-DRIFT-*` class), malformed |
+| [`vectors/manifest.json`](vectors/manifest.json) + [`vectors/cases/`](vectors/cases/) | 111 generated vectors — canonical (RFC 8785), digest, drift (every `WRD-DRIFT-*` class), malformed |
 | [`vectors/tools/generate.py`](vectors/tools/generate.py) | Regenerates the corpus from the Python reference; a corpus diff = a hashed-derivation change = a `schema_version` bump (SPEC §14.2) |
 | [`tests/test_spec_vectors.py`](tests/test_spec_vectors.py) | Python harness over the manifest (honours `MCP_LOCK_VECTORS_DIR`) |
 | [`packages/lock-ts/`](packages/lock-ts/README.md) | `@mcp-warden/lock` — verify-only TypeScript: hand-written JCS, SHA-256, capability + skeleton derivation, drift classifier; `npm test` runs the same corpus |
@@ -156,7 +162,7 @@ scope-honesty box and makes no compliance/regulatory claim.
 | [`docs/AGENT_TRUST_KERNEL.md`](docs/AGENT_TRUST_KERNEL.md) | **(DSE-714, design contract)** Normative invariants for the future deterministic Agent Trust Kernel: trust boundaries, complete mediation, default deny, non-overridable critical classes, evidence-before-effect, offline operation, residual risks, and bindings for DSE-715 through DSE-717. MCP-Warden v1.1 is explicitly not yet ATK-conformant |
 | [`docs/CONTENT_ENVELOPE.md`](docs/CONTENT_ENVELOPE.md) | **(DSE-715, implemented foundation)** Strict immutable V1 content envelope, domain-separated exact-byte digests, canonical metadata boundary, bounded one-hop lineage, monotonic taint, stable code-only errors, and secret-safe public projection. Evidence only; no authority or whole-ATK conformance claim |
 | [`docs/POLICY_ENFORCEMENT.md`](docs/POLICY_ENFORCEMENT.md) | **(DSE-716, implemented foundation)** Versioned signed policy/runtime/adapter/executable-bundle activation, exact adapter/bundle-bound leases, mechanically derived frozen handler identity, deterministic default-deny PDP, evidence-gated structural PEP, stable reason/recovery matrix, caps, and non-optional fixed-corpus adapter harness. DSE-717 durable evidence is in progress but remains required for any whole-ATK claim |
-| [`docs/WARDEN_LOCK_SCHEMA.md`](docs/WARDEN_LOCK_SCHEMA.md) | **mcp-warden implementation of [`docs/SPEC.md`](docs/SPEC.md) (MCP Lock Format v1).** `warden.lock` format, RFC 8785 canonicalization + SHA-256 hashing, field/entry/overall digests, the normative drift definition + severities; **§5.1/§6.2 structural schema diff** (normalized per-tool `schema_skeleton`, `schema_version` 3 — skeleton added at v2, in-document `$ref` resolution at v3 (#29), granular `WRD-DRIFT-SCHEMA-*` taxonomy + severities, v1 fallback); **§8.1/§8.2 (v0.3, #19)** structured out-of-digest provenance (`pinner` / `attestations` / `rotation_count`, `PROVENANCE_VERSION`, B4 `bound_digest` format) + `lock rotate` digest-invariant semantics + the #16 signing implication; **§11 (v0.2)** optional per-tool inspection policy (`expected_output_charset` / `may_return_urls` / `secret_echo_applies`, fail-safe defaults, digest impact) |
+| [`docs/WARDEN_LOCK_SCHEMA.md`](docs/WARDEN_LOCK_SCHEMA.md) | **mcp-warden implementation of [`docs/SPEC.md`](docs/SPEC.md) (MCP Lock Format v1).** `warden.lock` format, RFC 8785 canonicalization + SHA-256 hashing, field/entry/overall digests, the normative drift definition + severities; **§5.1/§6.2 structural schema diff** (normalized per-tool `schema_skeleton`, `schema_version` 4 — annotations/output added at v4, skeleton added at v2, in-document `$ref` resolution at v3 (#29), granular `WRD-DRIFT-SCHEMA-*` taxonomy + severities, v1 fallback); **§8.1/§8.2 (v0.3, #19)** structured out-of-digest provenance (`pinner` / `attestations` / `rotation_count`, `PROVENANCE_VERSION`, B4 `bound_digest` format) + `lock rotate` digest-invariant semantics + the #16 signing implication; **§11 (v0.2)** optional per-tool inspection policy (`expected_output_charset` / `may_return_urls` / `secret_echo_applies`, fail-safe defaults, digest impact) |
 | [`docs/WARDEN_LOCK_EXAMPLE.md`](docs/WARDEN_LOCK_EXAMPLE.md) | Illustrative full `warden.lock` + a post-`lock rotate` `pin` block (archived from WARDEN_LOCK_SCHEMA §9 to keep that core doc under the line cap) |
 | [`docs/CHECKS.md`](docs/CHECKS.md) | The deterministic `WRD-*` static-check catalog (capability/secret/supply/robustness), the shared tokenizer, severity→SARIF mapping, redaction rule, CUT list. **Reused by v0.2** `WRD-RES-SECRET-ECHO` (the `WRD-SEC-*` patterns + redaction) |
 | [`docs/POLICY_MODEL.md`](docs/POLICY_MODEL.md) | Policy schema, the four high-risk shapes, constraint vocabulary, fail-closed defaults, SSRF deny ranges, lint + single-sample eval semantics. **Enforced at runtime by v0.2 `guard`** on live `tools/call` requests |
@@ -167,6 +173,8 @@ scope-honesty box and makes no compliance/regulatory claim.
 
 ## Non-normative design and implementation plans
 
+[Runtime CLI examples](docs/archive/2026-10-02-runtime-cli-examples.md) retain the detailed guard commands moved from README.
+
 | Plan | Purpose |
 |---|---|
 | [`docs/plans/2026-07-18-agent-trust-kernel-design.md`](docs/plans/2026-07-18-agent-trust-kernel-design.md) | **Non-normative execution record.** Records the DSE-714 design decision and verification plan; binding requirements live in `docs/AGENT_TRUST_KERNEL.md` |
@@ -217,6 +225,7 @@ scope-honesty box and makes no compliance/regulatory claim.
 | `src/mcp_warden/signing.py` | **(#16)** Sigstore keyless sign/verify primitives (guarded import; `build_statement` / `sign_statement` / `verify_statement` — verify raises on failure, returns None on success) | SIGNING.md |
 | `src/mcp_warden/cli_sign.py` | **(#16)** `pin --sign` / `check --verify` CLI control flow: fixed-sidecar verify, atomic bundle write, fail-closed exits | SIGNING.md |
 | `src/mcp_warden/drift.py` | Per-class drift/diff engine + severities | WARDEN_LOCK_SCHEMA §6.2 |
+| `src/mcp_warden/drift_tool_metadata.py` | v4 annotation drift and structural output-schema classification without raw annotation disclosure | SPEC v4 extension |
 | `src/mcp_warden/schema_diff.py` | Deterministic structural `inputSchema` skeleton extractor + per-fact diff classifier (`WRD-DRIFT-SCHEMA-*`; `$ref`/cyclic/malformed-safe) | WARDEN_LOCK_SCHEMA §5.1, §6.2 |
 | `src/mcp_warden/checks.py` | Static-check orchestrator (deterministic sort) | CHECKS §4–§5 |
 | `src/mcp_warden/checks_secret.py` | `WRD-SEC-*` vendor + entropy + redaction | CHECKS §4.2 |
@@ -260,7 +269,7 @@ scope-honesty box and makes no compliance/regulatory claim.
 | `tests/test_capture_http.py` | **(#74, DSE-57)** Async/sync Streamable HTTP capture, protocol/list normalization, timeout handling, and connection errors |
 | `tests/test_diff.py` | **(v0.3)** `warden diff` renderer: identical→"no differences", tool add/remove + schema change rows, **redaction-leak guard** (secret in `server.args` absent from human/`--json`/`--sarif` incl. parsed-JSONL `detail`), provenance-only section vs empty integrity drift, `--exit-code` (1 on integrity drift / 0 on provenance-only), `--no-provenance` M6 message, fail-closed on missing/invalid lock |
 | `tests/test_result_inspection.py` | **(v0.2)** `WRD-RES-*`: ANSI codepoint match (incl. extended/binary-ok), secret-echo reuse + redaction, exfil host/subdomain boundary + path-qualified, injection exact-phrase (no broad-regex FP), URL/uninspectable notes |
-| `tests/test_inspection_policy.py` | **(v0.2)** §11 per-tool policy fail-safe defaults, byte-identical-to-v0.1 digest when absent, inspection-policy drift, pin-time validation, reader fallback + LOCK-INVALID |
+| `tests/test_inspection_policy.py` | **(v0.2)** §11 per-tool policy fail-safe defaults, inspection omission/None digest parity in the current format, inspection-policy drift, pin-time validation, reader fallback + LOCK-INVALID |
 | `tests/test_wire_block.py` | **(v0.2)** `-32001` error-response shape, block-mode mapping, ANSI strip-in-place `_meta.warden.modified`, secret redact-in-place |
 | `tests/test_framing.py` | **(v0.2)** newline + Content-Length framing, chunk-split reads, original-bytes pass-through, malformed-frame parse capture |
 | `tests/test_guard_posture.py` | **(v0.2/v0.3)** fail-open (inspector exception/malformed → pass-through) vs fail-closed (policy deny → block under `armed_policy`), audit-only precedence over default-on |

README.md

@@ -1,5 +1,7 @@
 # mcp-warden
 
+Last Updated: 2026-10-02
+
 [![CI](https://github.com/DataScience-EngineeringExperts/mcp-warden/actions/workflows/integrity-gate.yml/badge.svg)](https://github.com/DataScience-EngineeringExperts/mcp-warden/actions/workflows/integrity-gate.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
@@ -10,11 +12,22 @@
 tool/resource/prompt surface into a signed `warden.lock`, then fails CI when that surface
 drifts.** `pin` and `check` support stdio and Streamable HTTP; `guard` is stdio-only.
 
+**Schema level 4** also locks complete tool annotations and output schemas. Changing
+destructiveHint, removing a result schema, or widening its types now triggers drift;
+the existing runtime tools/list gate checks these fields against v4 locks too.
+Older locks stay readable, but require review and re-pinning for this coverage.
+Hints are server claims, not proof of safety, and schemas do not certify content.
+
+The proposed next direction is a protocol-neutral **Warden**: human-approved tool
+versions and bounded actions, with untrusted inputs kept separate from authority.
+The [upgrade plan and checkpoint proposal](docs/plans/2026-10-02-tool-integrity-upgrade.md)
+maps prompts, retrieval, code execution, and serverless adapters to existing Agent
+Trust Kernel work. Those broader checkpoints are proposals, not shipped guarantees.
+
 > ⚠️ **Install `mcp-warden-cli`, not `mcp-warden`.** The PyPI name `mcp-warden` is
 > an **unrelated package by a different author** — it is not this project. The
 > correct install is `pip install mcp-warden-cli` (the CLI command is still
-> `mcp-warden`). Or use the [GitHub Action](#github-action-one-step-drop-in) / a
-> git-pinned install.
+> `mcp-warden`). Or use the [GitHub Action](#github-action-one-step-drop-in) / a git-pinned install.
 
 If you already follow the published guidance — *pin versions, hash tool
 definitions, alert on drift* — mcp-warden is the deterministic tool that does it.
@@ -380,51 +393,13 @@ For stdio, `<server-cmd...>` is passed to the OS as an **argv array, never throu
 shell.** `--url` instead connects to an already-running Streamable HTTP endpoint and
 is mutually exclusive with a server command. Set `WARDEN_LOG_LEVEL=INFO` for diagnostics.
 
-### Runtime result inspection (v0.3 — blocks by default)
-
-`guard` sits transparently between an MCP client and server and inspects tool *results*.
-**As of v0.3 the deterministic tier blocks out of the box** (council-established field
-false-positive rate ~0):
-
-```bash
-# Default: ANSI is stripped in place; echoed secrets + exfil domains are error-replaced;
-# a mid-session tools/list swap that diverges from warden.lock is blocked (needs --lock);
-# an argument-policy deny is blocked (needs --policy). The fuzzy injection tier stays log-only.
-mcp-warden guard node ./build/index.js --lock warden.lock --policy policy.yaml --sarif guard.sarif
-
-# Observe-first rollout: --audit-only restores full v0.2 shadow in one flag (detect + log only).
-mcp-warden guard node ./build/index.js --lock warden.lock --audit-only
-
-# Opt a single category back to shadow (still detected/logged/SARIF, frame forwarded):
-mcp-warden guard node ./build/index.js --no-block-ansi --allow-exfil-domain
-# Or shadow the whole deterministic tier + both gates:
-mcp-warden guard node ./build/index.js --no-block-deterministic
-# Opt INTO the fuzzy injection tier (never default):
-mcp-warden guard node ./build/index.js --block-inject-phrase
-
-# Fail-CLOSED (high-security): TERMINATE the session (exit 3, -32003 to the client) if an
-# internal inspection (result / argument-policy / tools-list) cannot complete, instead of the
-# default fail-open pass-through. Opt-in; integrity over availability.
-mcp-warden guard node ./build/index.js --lock warden.lock --policy policy.yaml --strict
-
-# Re-analyze a recorded session offline with the identical rule catalog (always report-only):
-mcp-warden inspect session.trace.jsonl --lock warden.lock --sarif inspect.sarif
-```
+### Runtime result inspection
 
-**Flag scheme:** opt-out is canonical `--no-block-<category>`
-(`ansi|secret-echo|exfil-domain|list-changed|policy`, plus `--no-block-deterministic` for the
-whole tier); `--allow-exfil-domain` is the sole affirmative alias. Precedence:
-`--audit-only` > `--no-block-*` > default-block / `--block-inject-phrase`. The v0.2
-`--block-*` enable flags are accepted but **inert no-ops** (one-line stderr deprecation note),
-so old scripts keep working. **`--strict`** (opt-in, default off) trades availability for
-integrity: an internal inspection error at the result / argument-policy / tools-list layer
-**terminates the session** (exit `3`, `-32003` non-retriable error to the client) instead of
-failing open — framing/EOF/over-cap stay fail-open in all modes (known limitation). Reserved
-error codes: **`-32001`** (policy/result block), **`-32002`** (transport/lifecycle), **`-32003`**
-(`--strict` abort, non-retriable). See
-[`docs/RESULT_INSPECTION.md`](docs/RESULT_INSPECTION.md),
-[`docs/GUARD_PROXY.md`](docs/GUARD_PROXY.md), and
-[`docs/GUARD_PROXY_V3.md`](docs/GUARD_PROXY_V3.md).
+`guard` inspects stdio tool results and blocks deterministic hazards by default.
+`--audit-only` restores observation; `--strict` terminates on inspection errors.
+Framing errors still fail open. Prompt-injection phrase matching remains monitor-only.
+See [runtime examples](docs/archive/2026-10-02-runtime-cli-examples.md) and the
+[guard contract](docs/GUARD_PROXY_V3.md) for flags, reserved errors, and limitations.
 
 ---
 
@@ -487,15 +462,16 @@ fallback evidence, rollback-resistant state, the recovery latch, and any whole-k
 claim remain incomplete. See [`docs/POLICY_ENFORCEMENT.md`](docs/POLICY_ENFORCEMENT.md) and
 [`docs/AGENT_TRUST_KERNEL.md`](docs/AGENT_TRUST_KERNEL.md).
 
-See [`DOCUMENTATION_INDEX.md`](DOCUMENTATION_INDEX.md). The security-contract specs
+See [`DOCUMENTATION_INDEX.md`](DOCUMENTATION_INDEX.md) and
+[`SYSTEM_CONTEXT_DIAGRAM.md`](SYSTEM_CONTEXT_DIAGRAM.md). The security-contract specs
 under `docs/` (including [`GUARD_PROXY_V3.md`](docs/GUARD_PROXY_V3.md) for the v0.3
 default-block + lifecycle contract) are the source of truth for every algorithm; the
 schemas in `warden.lock` and the SARIF output match them byte-for-byte.
 
 ## Tests
 
 ```bash
-.venv/bin/python -m pytest -q
+PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest -q
 ```
 
 The headline test is a real stdio round-trip: spawn the clean fixture → `pin` →

SYSTEM_CONTEXT_DIAGRAM.md

@@ -1,5 +1,15 @@
 # mcp-warden — System Context Diagram
 
+Last Updated: 2026-10-02
+
+**Schema level 4** extends capture/lock/check to complete tool annotations and output
+schemas, with structural output drift and Python/TypeScript parity. The existing
+tools/list gate compares those commitments for v4 locks; legacy locks retain narrower
+runtime coverage and require re-pin for the new fields. Annotations grant no authority.
+The [Warden checkpoint proposal](docs/plans/2026-10-02-tool-integrity-upgrade.md)
+shows a future kernel beneath prompt/retrieval/code/serverless adapters with signing
+authority outside agent-editable state. It does not expand current runtime claims.
+
 Where mcp-warden sits, what it talks to, and where its outputs go. The **definition-only
 path introduced in v0.1** (`pin`/`check`/`policy`) is read-only: it captures the
 *declared* surface and writes a baseline + machine reports — no proxy, no runtime
@@ -177,8 +187,8 @@ sequenceDiagram
     end
 ```
 
-> `compute_drift` structurally classifies tool `inputSchema` changes via the normalized
-> `schema_skeleton` stored in the lock (`schema_version` 3 — skeleton added at v2, in-document
+> `compute_drift` structurally classifies tool `inputSchema` and v4 `outputSchema` changes via the normalized
+> `schema_skeleton` stored in the lock (`schema_version` 4 — annotations/output added at v4, skeleton added at v2, in-document
 > `$ref` resolution at v3, #29): each security-relevant mutation is a per-fact
 > `WRD-DRIFT-SCHEMA-*` item (`docs/WARDEN_LOCK_SCHEMA.md` §6.2). v1 locks fall
 > back to a single high-severity `schema-modified` until re-pinned.