fix(lock-ts): normative 512 nesting cap + verify() error contract (DSE-1527)

CHANGELOG.md

@@ -32,6 +32,19 @@ Streamable HTTP; the v0.3 `guard` proxy adds deterministic runtime *result* insp
 
 ### Fixed
 
+- **Normative nesting bound: 512 (DSE-1527).** The TypeScript verifier refused JSON nested
+  past 512 levels while the Python reference accepted anything up to the interpreter's
+  ~1000-frame recursion limit, so two conforming implementations disagreed on the same
+  document. `docs/SPEC.md` §4 now names 512 as the maximum depth (root = 0, each
+  enclosing array/object adds one); `canon()` and `read_lock()` enforce it with an
+  explicit, iterative check that raises a typed `DepthError` (a `ValueError`) before any
+  serialization; `vectors/` gains `canonical/depth-512-accepted` and
+  `malformed/depth-513-rejected` so both harnesses pin the exact bound. `@mcp-warden/lock`'s
+  `verify()` now throws only `LockFormatError` — a canonicalizer or depth failure on the
+  observed surface no longer leaks as `JcsError`/`DepthError` — and the Python vector
+  harness routes every `malformed` lock through `read_lock()`, the public reader, as
+  `vectors/README.md` asks of every implementation.
+
 - **`doctor` follow-ups from the security review of #98 (DSE-1529).** `#servers` is no
   longer a reserved namespace: config entries are keyed on `(map, name)` and the display
   key is derived afterwards, unique by construction, so a server the user really named

docs/SPEC.md

@@ -85,6 +85,17 @@ is hashed MUST first be canonicalized.
   canonicalizer. Therefore an optional field that is simply not present contributes
   nothing to a digest, and a baseline that omits it is byte-identical to one written
   before the field existed.
+- **Nesting bound (normative):** the *depth* of an element is the number of arrays and
+  objects enclosing it; the document root is depth 0. A conforming implementation MUST
+  accept any value whose deepest element is at depth **512** and MUST refuse — fail closed,
+  never hash — any value containing an element at depth greater than 512, both when
+  canonicalizing a surface value (§4–§5) and when reading a lock document (§3). An
+  implementation that merely inherits its host language's recursion limit is NOT
+  conformant: two such implementations disagree on the same document. The bound is
+  pinned by `vectors/cases/canonical-depth-512-accepted.json` and
+  `vectors/cases/malformed-depth-513-rejected.json`. This bound governs lock documents and
+  surface values only; the content-envelope profile (`docs/CONTENT_ENVELOPE.md`) keeps its
+  own, much smaller, nesting limit.
 
 ---
 
@@ -419,7 +430,7 @@ byte-for-byte:
 - `drift` — the ordered `(drift_class, severity, target, detail)` set for a baseline lock
   and an observed surface (§8.2–§8.3);
 - `malformed` — lock documents a conforming reader MUST reject, and JSON values a
-  conforming canonicalizer MUST refuse (unpaired surrogates, excessive nesting).
+  conforming canonicalizer MUST refuse (unpaired surrogates, nesting past the §4 bound).
 
 The corpus is generated from the reference implementation (`vectors/tools/generate.py`)
 and is regenerated only on a deliberate format change (§14). Two implementations ship

packages/lock-ts/README.md

@@ -37,7 +37,7 @@ console.log("surface matches baseline", digest(surface));
 
 | export | purpose |
 |--------|---------|
-| `verify(lock, surface) → { ok, findings, observed_digest }` | drift between a baseline lock document and an observed surface; always runs the strict reader first and throws `LockFormatError` on a malformed lock or one at a newer `schema_version` — it can never return `ok` for a document a conforming reader must reject |
+| `verify(lock, surface) → { ok, findings, observed_digest }` | drift between a baseline lock document and an observed surface; always runs the strict reader first and throws `LockFormatError` — and only `LockFormatError` — on a malformed lock, one at a newer `schema_version`, or an observed surface it cannot canonicalize (unpaired surrogate, nesting past 512); it can never return `ok` for a document a conforming reader must reject |
 | `digest(surface) → "sha256:…"` | the `overall_digest` a conforming writer would store for that surface |
 | `parseLock(doc)` | strict structural reader; throws `LockFormatError` (fail closed) |
 | `buildFromSurface(surface)` | the hashed entries (`entry_digest`, `capabilities`, `schema_skeleton`) for a surface |

packages/lock-ts/src/index.ts

@@ -8,15 +8,17 @@
  * what drifted — with the same rule ids, severities and ordering as the Python
  * reference. Conformance is defined by the corpus under `vectors/`.
  *
- * Fail-closed contract: `verify()` ALWAYS runs the strict reader (`parseLock`) on
+ * Fail-closed contract: `verify()` throws `LockFormatError` and nothing else. It ALWAYS runs the strict reader (`parseLock`) on
  * the lock it is given — a document a conforming reader must reject (missing
  * `overall_digest`, a `schema_version` above the level this package implements,
  * malformed entries, excessive nesting) throws `LockFormatError` and can never
  * produce `ok: true`. There is no duck-typed "already parsed" shortcut.
  */
 
 import { computeDrift, type DriftItem } from "./drift.js";
-import { buildFromSurface, parseLock, type Surface } from "./lock.js";
+import { JcsError } from "./jcs.js";
+import { buildFromSurface, LockFormatError, parseLock, type Surface } from "./lock.js";
+import { DepthError } from "./py.js";
 
 export { canonicalize, hasUnpairedSurrogate, JcsError } from "./jcs.js";
 export { canon, hashArguments, hashDescription, hashInputSchema, hashValue, SHA256_PREFIX } from "./digest.js";
@@ -57,12 +59,25 @@ export function digest(surface: Surface): string {
  * Compare an observed surface against a baseline lock document.
  *
  * `lock` is the raw JSON document (or a `Lock` previously returned by `parseLock`,
- * which re-validates identically). Throws `LockFormatError` if the document is not
- * a structurally valid lock at a level this package implements — fail closed.
+ * which re-validates identically). Throws `LockFormatError` — and ONLY `LockFormatError`
+ * — if the document is not a structurally valid lock at a level this package implements,
+ * or if the observed surface cannot be canonicalized (unpaired surrogate, nesting past
+ * `MAX_JSON_DEPTH`). A `JcsError`/`DepthError` never escapes this entry point — it is
+ * wrapped with the original attached as `cause` (DSE-1527).
  */
 export function verify(lock: unknown, surface: Surface): VerifyResult {
   const baseline = parseLock(lock);
-  const current = buildFromSurface(surface);
-  const findings = computeDrift(baseline, current);
-  return { ok: findings.length === 0, findings, observed_digest: current.overall_digest };
+  try {
+    const current = buildFromSurface(surface);
+    const findings = computeDrift(baseline, current);
+    return { ok: findings.length === 0, findings, observed_digest: current.overall_digest };
+  } catch (e) {
+    // Only the two fail-closed refusals of the observed surface are folded into the
+    // contract error (with the original as `cause`); anything else is a programming
+    // error and must not be masked.
+    if (e instanceof JcsError || e instanceof DepthError) {
+      throw new LockFormatError(`observed surface is not verifiable: ${e.message}`, { cause: e });
+    }
+    throw e;
+  }
 }

packages/lock-ts/test/vectors.test.ts

@@ -221,3 +221,40 @@ test("depth: canonicalize, deepEqual and parseLock are bounded", () => {
   ((lock["tools"] as Doc[])[0] as Doc)["schema_skeleton"] = { props: { p: { constraints: { deep: deep(MAX_JSON_DEPTH + 2) } } } };
   assert.throws(() => verify(lock, {}), LockFormatError);
 });
+
+// --- DSE-1527: normative nesting bound + verify() error contract -----------------
+
+const nestedArrays = (n: number): unknown => {
+  // n enclosing arrays around an empty array: the innermost `[]` sits at depth n.
+  let v: unknown = [];
+  for (let i = 0; i < n; i++) v = [v];
+  return v;
+};
+
+test("DSE-1527: the canonicalizer bound is exactly 512 (root = 0)", () => {
+  assert.equal(MAX_JSON_DEPTH, 512);
+  assert.doesNotThrow(() => canonicalize(nestedArrays(MAX_JSON_DEPTH)));
+  assert.throws(() => canonicalize(nestedArrays(MAX_JSON_DEPTH + 1)), JcsError);
+  // A leaf counts, and objects count like arrays.
+  let obj: unknown = { k: "leaf" };
+  for (let i = 0; i < MAX_JSON_DEPTH; i++) obj = { k: obj };
+  assert.throws(() => canonicalize(obj), JcsError);
+});
+
+test("DSE-1527: verify() throws only LockFormatError, even when the observed surface is unverifiable", () => {
+  const lock = validLock();
+  const surface = { tools: [{ name: "t", inputSchema: { deep: nestedArrays(MAX_JSON_DEPTH + 5) } }] } as unknown as Surface;
+  let caught: unknown;
+  try {
+    verify(lock, surface);
+  } catch (e) {
+    caught = e;
+  }
+  assert.ok(caught instanceof LockFormatError, `expected LockFormatError, got ${String(caught)}`);
+  assert.ok(!(caught instanceof JcsError) && !(caught instanceof DepthError), "underlying error type must not leak");
+  assert.match((caught as Error).message, /observed surface is not verifiable/);
+  assert.ok((caught as Error).cause instanceof JcsError, "the original refusal travels as `cause`");
+  // Unpaired surrogate on the observed side takes the same path.
+  const bad = { tools: [{ name: "t", inputSchema: { d: "\ud800" } }] } as unknown as Surface;
+  assert.throws(() => verify(lock, bad), LockFormatError);
+});

src/mcp_warden/hashing.py

@@ -26,6 +26,38 @@
 #: Public prefix for every digest emitted by mcp-warden.
 SHA256_PREFIX = "sha256:"
 
+#: Normative nesting bound for LOCK/SURFACE canonicalization (docs/SPEC.md §4): the
+#: document root is depth 0 and every enclosing array/object adds one; an element at
+#: depth > MAX_CANON_DEPTH MUST be refused, never hashed. ``@mcp-warden/lock`` enforces
+#: the same constant. Distinct from ``content_models.MAX_JSON_DEPTH`` (16), which bounds
+#: the content-envelope profile. Without an explicit check the reference silently
+#: accepted 513–~990 levels (up to the interpreter's recursion limit) while the
+#: TypeScript verifier refused them — two conforming implementations disagreeing on one
+#: document (DSE-1527).
+MAX_CANON_DEPTH = 512
+
+
+class DepthError(ValueError):
+    """A JSON value nests deeper than :data:`MAX_CANON_DEPTH` (fail closed)."""
+
+
+def check_depth(value: Any, *, where: str = "value") -> None:
+    """Raise :class:`DepthError` if ``value`` nests deeper than :data:`MAX_CANON_DEPTH`.
+
+    Iterative (explicit stack) so the check itself can never trip the interpreter's
+    recursion limit on the very input it exists to refuse. Leaves count: a scalar at
+    depth 513 is as much a violation as a container there.
+    """
+    stack: list[tuple[Any, int]] = [(value, 0)]
+    while stack:
+        node, depth = stack.pop()
+        if depth > MAX_CANON_DEPTH:
+            raise DepthError(f"{where}: nesting deeper than {MAX_CANON_DEPTH} levels")
+        if isinstance(node, dict):
+            stack.extend((child, depth + 1) for child in node.values())
+        elif isinstance(node, (list, tuple)):
+            stack.extend((child, depth + 1) for child in node)
+
 
 def hash_bytes(payload: bytes, *, domain: DigestDomain) -> str:
     """Hash exact bytes under a closed content-envelope digest domain."""
@@ -48,8 +80,11 @@ def canon(value: Any) -> bytes:
         The canonical UTF-8 byte string.
 
     Raises:
+        DepthError: If ``value`` nests deeper than :data:`MAX_CANON_DEPTH` (SPEC.md §4);
+            a ``ValueError`` subclass, raised before any serialization is attempted.
         ValueError: If ``value`` is not JSON-serializable under JCS.
     """
+    check_depth(value)
     try:
         return rfc8785.dumps(value)
     except Exception as exc:  # rfc8785 raises a variety of types on bad input

src/mcp_warden/lockfile.py

@@ -17,6 +17,8 @@
 
 from . import SCHEMA_VERSION, __version__
 from .hashing import (
+    DepthError,
+    check_depth,
     hash_arguments,
     hash_description,
     hash_input_schema,
@@ -322,8 +324,18 @@ def read_lock(path: str | Path) -> WardenLock:
         raise FileNotFoundError(f"lock file not found: {p}")
     try:
         raw = json.loads(p.read_text(encoding="utf-8"))
-    except json.JSONDecodeError as exc:
+    except (json.JSONDecodeError, RecursionError) as exc:
+        # RecursionError: the C decoder refuses documents nested past the interpreter's
+        # limit (~1000 on 3.11) before check_depth() ever sees them — still the
+        # documented ValueError, never a bare RecursionError (CSO review of #102).
         raise ValueError(f"lock file {p} is not valid JSON: {exc}") from exc
+    try:
+        # SPEC.md §4 nesting bound — checked before validation so a hostile document is
+        # refused by an explicit, iterative check rather than by wherever the interpreter
+        # happens to run out of stack (DSE-1527).
+        check_depth(raw, where=f"lock file {p}")
+    except DepthError as exc:
+        raise ValueError(f"lock file {p} exceeds the nesting bound: {exc}") from exc
     try:
         return WardenLock.model_validate(raw)
     except Exception as exc:

tests/test_hashing.py

@@ -8,6 +8,8 @@
 
 import hashlib
 
+import pytest
+
 from mcp_warden.hashing import (
     canon,
     hash_arguments,
@@ -80,3 +82,42 @@ def test_schema_change_changes_digest():
     base = {"type": "object", "properties": {"path": {"type": "string"}}}
     changed = {"type": "object", "properties": {"path": {"type": "string"}, "enc": {"type": "string"}}}
     assert hash_input_schema(base) != hash_input_schema(changed)
+
+
+# --- DSE-1527: normative nesting bound (SPEC.md §4) ---------------------------
+
+
+def _nested_arrays(n: int) -> list:
+    """``n`` enclosing arrays around an empty array: the innermost ``[]`` sits at depth ``n``."""
+    v: list = []
+    for _ in range(n):
+        v = [v]
+    return v
+
+
+def test_canon_accepts_depth_512_and_refuses_513():
+    from mcp_warden.hashing import MAX_CANON_DEPTH, DepthError
+
+    assert MAX_CANON_DEPTH == 512
+    ok = _nested_arrays(MAX_CANON_DEPTH)
+    assert canon(ok) == b"[" * (MAX_CANON_DEPTH + 1) + b"]" * (MAX_CANON_DEPTH + 1)
+    with pytest.raises(DepthError):
+        canon(_nested_arrays(MAX_CANON_DEPTH + 1))
+    # A leaf counts, and objects count like arrays: "leaf" ends up at depth 513.
+    deep_obj: dict = {"k": "leaf"}
+    for _ in range(MAX_CANON_DEPTH):
+        deep_obj = {"k": deep_obj}
+    with pytest.raises(DepthError):
+        canon(deep_obj)
+
+
+def test_depth_error_is_a_value_error_raised_before_serialization():
+    from mcp_warden.hashing import DepthError, check_depth
+
+    assert issubclass(DepthError, ValueError)
+    with pytest.raises(DepthError, match="nesting deeper than 512"):
+        check_depth(_nested_arrays(600), where="probe")
+    # Iterative: far past any recursion limit, still a clean DepthError.
+    check = _nested_arrays(5000)
+    with pytest.raises(DepthError):
+        check_depth(check)

tests/test_lockfile.py

@@ -4,6 +4,8 @@
 
 import json
 
+import pytest
+
 from mcp_warden.lockfile import build_lock, lock_to_pretty_json, read_lock, write_lock
 from mcp_warden.models import (
     Attestation,
@@ -171,3 +173,42 @@ def test_lock_stores_hashes_not_raw_text():
     # Raw description text must NOT appear in the lock.
     assert "Read a file" not in text
     assert "description_hash" in text
+
+
+# --- DSE-1527: read_lock enforces the SPEC.md §4 nesting bound ------------------
+
+
+def _deep(n: int) -> list:
+    v: list = []
+    for _ in range(n):
+        v = [v]
+    return v
+
+
+def test_read_lock_refuses_nesting_past_the_bound(tmp_path):
+    from mcp_warden.hashing import MAX_CANON_DEPTH
+
+    lock = build_lock(_surface(), [])
+    doc = json.loads(lock_to_pretty_json(lock))
+    path = tmp_path / "warden.lock"
+
+    # Control: the document as written reads back.
+    path.write_text(json.dumps(doc), encoding="utf-8")
+    assert read_lock(path).overall_digest == lock.overall_digest
+
+    # One past the bound anywhere in the document -> refused by the explicit check,
+    # before schema validation, with an intelligible message.
+    hostile = dict(doc)
+    hostile["x"] = _deep(MAX_CANON_DEPTH)  # innermost [] at depth 512 + 1 (the "x" key)
+    path.write_text(json.dumps(hostile), encoding="utf-8")
+    with pytest.raises(ValueError, match="nesting deeper than 512"):
+        read_lock(path)
+
+    # Exactly at the bound the depth check is silent (whatever schema validation says).
+    at_bound = dict(doc)
+    at_bound["x"] = _deep(MAX_CANON_DEPTH - 1)
+    path.write_text(json.dumps(at_bound), encoding="utf-8")
+    try:
+        read_lock(path)
+    except ValueError as exc:
+        assert "nesting deeper" not in str(exc)

tests/test_spec_vectors.py

@@ -18,7 +18,7 @@
 
 from mcp_warden.drift import compute_drift
 from mcp_warden.hashing import canon, hash_value
-from mcp_warden.lockfile import build_lock
+from mcp_warden.lockfile import build_lock, read_lock
 from mcp_warden.models import (
     CapturedPrompt,
     CapturedResource,
@@ -108,17 +108,20 @@ def test_drift(entry):
 
 
 @pytest.mark.parametrize("entry", _by_kind("malformed"))
-def test_malformed_is_rejected(entry):
+def test_malformed_is_rejected(entry, tmp_path):
     v = _load(entry)
     assert v["expect"] == {"error": True}
     if "input_json" in v:
         # A JSON value the canonicalizer MUST reject: it parses, but is not Unicode text
-        # (unpaired surrogate) or nests past the recursion bound. Either the parser or
-        # canon() refusing it is a rejection; a wrong digest is not.
+        # (unpaired surrogate) or nests past the §4 bound. Either the parser or canon()
+        # refusing it is a rejection; a wrong digest is not.
         with pytest.raises((ValueError, RecursionError)):
             canon(json.loads(v["input_json"]))
         return
-    # json.JSONDecodeError and pydantic's ValidationError are both ValueError subclasses.
-    with pytest.raises((ValueError, TypeError)):
-        doc = json.loads(v["lock_text"]) if "lock_text" in v else v["lock"]
-        WardenLock.model_validate(doc)
+    # Through the PUBLIC reader, exactly as a consumer hits it (vectors/README.md step 4):
+    # read_lock() must refuse. JSON errors, schema errors (pydantic ValidationError) and
+    # the nesting bound (DepthError) all surface as ValueError from read_lock.
+    path = tmp_path / "warden.lock"
+    path.write_text(v["lock_text"] if "lock_text" in v else json.dumps(v["lock"]), encoding="utf-8")
+    with pytest.raises(ValueError):
+        read_lock(path)

vectors/README.md

@@ -80,9 +80,12 @@ fails **both** implementations.
   cannot even represent them exactly (`2**63` parses to `9223372036854776000`). Vectors
   never contain such integers; a surface that does is not portable and the reference
   will not digest it. Integer-valued *floats* such as `1e21` are fine and are pinned.
-- **Nesting is bounded.** The reference raises past its recursion limit; the TypeScript
-  verifier refuses anything deeper than 512 levels. `malformed/deep-nesting-2000` pins
-  that both reject, not the exact bound.
+- **Nesting is bounded at 512 — normative (SPEC.md §4).** Depth counts enclosing
+  arrays/objects with the root at 0. `canonical/depth-512-accepted` MUST canonicalize and
+  `malformed/depth-513-rejected` MUST be refused; both the Python reference (`canon()`,
+  `read_lock()`) and the TypeScript verifier enforce the same explicit bound.
+  `malformed/deep-nesting-2000` remains as the coarse guard. An implementation that only
+  relies on its runtime's recursion limit is not conformant.
 
 - Enum ordering inside `schema_skeleton` keys on Python's `json.dumps` text of each value.
   JavaScript cannot distinguish `1.0` from `1`, so vectors never put integer-valued