Mid-Flight Crash & Revocation Black-Box Verification Gap Across agent-manifest -> ca2a -> cmcp -> trace-spec (GapDisclosure/1.0 vs cross_check_trace_dag) #416

Summary

When a multi-agent workflow faults, or has a delegation revoked, part-way through, the evidence the stack produces cannot be verified end to end. The pieces each do their own job correctly, but there is no artefact that says "hop N never completed, and here is the signed statement of where", and no verifier that would accept one:

• ca2a_verify.cross_check_trace_dag requires len(records) == len(chain) and nothing else.
• ca2a_verify.verify_trace_dag validates every element of the DAG as a TRACE record, so a GapDisclosure/1.0 (trace-spec spec/trace-v0.2.md §3.3.4) cannot be a DAG element.
• cmcp emits a claim only from close_session; an orphaned session is logged, not disclosed.
• Neither the ca2a hop record nor the cMCP claim carries a manifest digest (see the cross-reference to trace-spec#483 below), so even a complete DAG cannot say which approved manifest each hop ran under.

The result is a black box exactly where a regulator or a counterparty most needs evidence: the failure window. A truthful emitter that discloses the loss gets the same verdict as one that silently dropped a hop, and a verifier operator has to choose between trimming the chain (which erases the revocation from the record) and failing the whole workflow.

I checked integrations (35 issues, 26 open PRs) before filing. #333 (revocation after dispatch, reference scenario) and #199 (acceptance harness for confidential workflows) are adjacent, and this issue is meant to give both of them a verifier-side contract to test against. Nothing open or closed covers the verifier gap itself, and GapDisclosure is not referenced anywhere in ca2a, cmcp or integrations on main.

Pinned at ca2a@945292b, cmcp@5c7ef0a, trace-spec@b938790.

Where it breaks

Reproduction

Compact form, from a ca2a checkout at 945292b with trace-spec@b938790 on PYTHONPATH (src and conformance/src):

python - <<'PY'
import sys,time
sys.path[:0]=['src','examples/trace-dag']
import demo
from ca2a_runtime.trace_binding import HopContext,HopSpec,digest,emit_dag
from ca2a_verify import cross_check_trace_dag,verify_trace_dag
chain,keys=demo.build_chain(demo.SCOPES)
def hop(i): return HopSpec(subject=demo.SUBJECTS[i],signing_key=keys[i],context=HopContext.software(model_provider='p',model_id='m',image_label='x',policy_bundle_hash=digest(b'p')),iat=int(time.time()),credential_id=chain[i].credential_id)
T=[k.public_key() for k in keys]
for name,hops in (('fault',[0,1]),('retry',[0,1,1,2])):
    r=emit_dag([hop(i) for i in hops]); print(name,'verify_trace_dag hops =',verify_trace_dag(r,trusted_keys=T).hops)
    try: cross_check_trace_dag(r,chain)
    except Exception as e: print(' ',type(e).__name__+':',e)
PY

fault verify_trace_dag hops = 2
  ProvenanceLinkBroken: DAG length 2 does not match chain length 3
retry verify_trace_dag hops = 4
  ProvenanceLinkBroken: DAG length 4 does not match chain length 3

Each DAG is individually valid, and the same DAG is rejected by the cross-check. Neither error says which of "crash", "revocation" or "replay" occurred, and neither can be resolved by anything the emitter can sign.

import base64
import hashlib
import importlib.util
import pathlib
import sys
import time

root = pathlib.Path(__file__).parent
sys.path[:0] = [str(root / "trace-spec/src"), str(root / "ca2a/src"), str(root / "trace-spec/conformance/src")]

import rfc8785  # noqa: E402
from ca2a_runtime.delegation.holder import build_holder_proof  # noqa: E402
from ca2a_runtime.errors import CA2AError  # noqa: E402
from ca2a_runtime.trace_binding import (  # noqa: E402
    HopContext,
    HopSpec,
    build_trace_record,
    digest,
    emit_dag,
    sign_trace_record,
    trace_record_hash,
)
from ca2a_verify import cross_check_trace_dag, verify_trace_dag  # noqa: E402

spec = importlib.util.spec_from_file_location("demo", root / "ca2a/examples/trace-dag/demo.py")
demo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(demo)

now = int(time.time())
chain, keys = demo.build_chain(demo.SCOPES)


def hop(i):
    return HopSpec(
        subject=demo.SUBJECTS[i],
        signing_key=keys[i],
        context=HopContext.software(
            model_provider="example-provider",
            model_id="example-model",
            image_label=f"ca2a-peer:{i}",
            policy_bundle_hash=digest(b"policy"),
        ),
        iat=now,
        credential_id=chain[i].credential_id,
    )


def attempt(label, fn):
    try:
        out = fn()
        print(f"{label}: OK {out if out is not None else ''}")
    except CA2AError as e:
        print(f"{label}: {type(e).__name__}: {e}")


trusted = [k.public_key() for k in keys]

# The leaf holder proof for hop 2 is built (ca2a step) ...
hp = build_holder_proof(
    keys[2], chain[2], audience="spiffe://ca2a.example/agent/retriever",
    challenge="c1", requested_capability="tool:search", record_id="r-2",
)
print("holder proof for hop 2 built:", bool(hp.signature))

# ... then the workflow faults / hop 2 is revoked, so only hops 0-1 emit TRACE records.
recs = emit_dag([hop(0), hop(1)])
print("\n== A. mid-flight fault or revocation: 2 records, 3 credentials ==")
attempt("verify_trace_dag(records)         ", lambda: verify_trace_dag(recs, trusted_keys=trusted).hops)
attempt("cross_check_trace_dag(rec, chain) ", lambda: cross_check_trace_dag(recs, chain))
attempt("cross_check_trace_dag(rec, chain[:2]) (verifier silently trims the chain)",
        lambda: cross_check_trace_dag(recs, chain[:2]))

# GapDisclosure/1.0 per trace-spec section 3.3.4, signed by the key that signed the predecessor
prev_hash = trace_record_hash(recs[1])
gap = {
    "type": "GapDisclosure/1.0",
    "previous_receipt_hash": prev_hash,
    "session_id": "wf-001",
    "issuer_key_id": keys[1].public_key().public_bytes_raw().hex(),
    "cause": "delegation_revoked",
}
gap["signature"] = base64.urlsafe_b64encode(keys[1].sign(rfc8785.dumps(gap))).rstrip(b"=").decode()
gap_hash = "sha256:" + hashlib.sha256(rfc8785.dumps(gap)).hexdigest()

print("\n== B. same fault, with a spec-conformant GapDisclosure spliced in ==")
attempt("verify_trace_dag([r0, r1, gap])   ", lambda: verify_trace_dag(recs + [gap], trusted_keys=trusted).hops)

# resumed hop 2 linked to the disclosure, as section 3.3.4 'Chain binding' requires
r2 = sign_trace_record(
    build_trace_record(
        subject=demo.SUBJECTS[2], iat=now, context=hop(2).context,
        credential_id=chain[2].credential_id, parent_record_hash=gap_hash,
    ),
    keys[2],
)
attempt("verify_trace_dag([r0, r1, r2->gap])", lambda: verify_trace_dag(recs + [r2], trusted_keys=trusted).hops)
attempt("verify_trace_dag([r0, r1, gap, r2])", lambda: verify_trace_dag(recs + [gap, r2], trusted_keys=trusted).hops)
attempt("cross_check_trace_dag([r0,r1,gap,r2], chain)", lambda: cross_check_trace_dag(recs + [gap, r2], chain))

print("\n== C. retry creates an extra hop: 4 records, 3 credentials ==")
retry = emit_dag([hop(0), hop(1), hop(1), hop(2)])
attempt("verify_trace_dag(records)         ", lambda: verify_trace_dag(retry, trusted_keys=trusted).hops)
attempt("cross_check_trace_dag(rec, chain) ", lambda: cross_check_trace_dag(retry, chain))

print("\n== D. no manifest binding anywhere on the DAG ==")
print("HopContext fields containing 'manifest':", [f for f in HopContext.__dataclass_fields__ if "manifest" in f])
print("'manifest' in any emitted record      :", any("manifest" in r for r in retry))

holder proof for hop 2 built: True

== A. mid-flight fault or revocation: 2 records, 3 credentials ==
verify_trace_dag(records)         : OK 2
cross_check_trace_dag(rec, chain) : ProvenanceLinkBroken: DAG length 2 does not match chain length 3
cross_check_trace_dag(rec, chain[:2]) (verifier silently trims the chain): OK 

== B. same fault, with a spec-conformant GapDisclosure spliced in ==
verify_trace_dag([r0, r1, gap])   : TraceRecordInvalid: record 2 is not a valid TRACE record
verify_trace_dag([r0, r1, r2->gap]): ProvenanceLinkBroken: record 2 parent link does not match the previous record's hash
verify_trace_dag([r0, r1, gap, r2]): TraceRecordInvalid: record 2 is not a valid TRACE record
cross_check_trace_dag([r0,r1,gap,r2], chain): ProvenanceLinkBroken: DAG length 4 does not match chain length 3

== C. retry creates an extra hop: 4 records, 3 credentials ==
verify_trace_dag(records)         : OK 4
cross_check_trace_dag(rec, chain) : ProvenanceLinkBroken: DAG length 4 does not match chain length 3

== D. no manifest binding anywhere on the DAG ==
HopContext fields containing 'manifest': []
'manifest' in any emitted record      : False

Why this matters beyond a length check

1. It rewards concealment, which is what §3.3.4 was written to prevent. The spec's own argument is that "an operator who backfills a lost receipt scores better than one who reports the loss." In ca2a today the best outcome for a disclosing emitter is a hard failure, and the best outcome for a silent one is a trimmed chain that verifies.
2. A revocation is erased by the workaround. Trimming chain to match records is the only way to get green, and it deletes the credential that was revoked from the evidence. The revocation snapshot is then never consulted for that hop, because the hop is no longer in the chain under test.
3. Retries are unclassifiable. A fenced retry after a crash and an attacker re-presenting a hop both produce the same extra element. Without a fence rule the verifier can only reject both or accept both.
4. The manifest is not in the picture at all. HopContext has no manifest member and no emitted record has a manifest key (case D). The v0.3 draft added manifest {id, digest, media_type} (trace-spec#483, #538), and trace-spec#483 has my analysis of why cMCP cannot emit it yet. For this issue the consequence is that a retry or a resumed hop can run under a different manifest than the original attempt and nothing in the DAG would show it.

Proposed contract

This is a verifier and emitter contract; I am deliberately not proposing code while the PR queue is being merged (per the maintainers' request on trace-spec#653).

Chain model. A workflow's evidence is an ordered list of elements, each either a TRACE hop record or a GapDisclosure/1.0. cross_check_trace_dag reconciles by credential identity, not by count: every chain[i] must be covered by exactly one of (a) a hop record carrying that credential_id, signed by chain[i].subject, or (b) a verified disclosure whose position in the chain replaces it.

Disclosure verification. Reuse the rules and the vectors under trace-spec/examples/action-receipts/gap-disclosure/ as a library (they exist today only inside a test module): stream binding on session_id, issuer = signer of the predecessor or a permitted ancestor, two-sided splice, tail disclosure reported as unverified rather than disclosed or invalid, and receipt_invalid when the claimed gap is contradicted by an element that is present.

Outcomes. verify_trace_dag and cross_check_trace_dag return a result object rather than only raising: complete, gap_disclosed (with each disclosure's predecessor, consecutive count and cause), or fail-closed errors as today. A profile that requires proven completeness MUST NOT accept gap_disclosed, exactly as §3.3.4 requires.

Retry rule. An additional hop for an already-covered credential_id is admissible only if it follows a disclosure (or an explicit supersession link) and carries a higher fencing epoch than the attempt it replaces. Otherwise it is a replay and is rejected. This is where the compensation lease belongs:

• draft-durga-agent-tx-compensation-lease-00 defines the Pre-Release Transactional Compensation Lease (PTCL), acquired before a state-mutating call, with RECOVERY_WINDOW_INEQUALITY (t_req_ms <= t_granted_ms, and expiry checked against a monotonic clock) and MONOTONIC_CAS_FENCING (abort with ERR_STALE_FENCE_EPOCH if the resource epoch is >= cas_fence_epoch or the pre-image digest differs). The lease carries trace_anchor_ref.
• For this issue the lease gives the verifier a producer-independent trigger and a replay discriminator. A hop whose lease expired unreleased is the crash case, so the disclosure's cause can name the lease_id. A retry is legitimate iff its cas_fence_epoch is strictly greater than the crashed attempt's. A replay of the first attempt carries the stale epoch and is refused by the same rule that protects the system of record.
• draft-durga-agenttrust-trace-verification-00 §5 specifies N-hop verification (signature validity, hash continuity hop[i].parent_record_hash == SHA256(JCS(hop[i-1])), scope attenuation, temporal monotonicity) but takes N from the hops themselves. It has no clause for a hop that is known to be missing. The disclosure gives that clause its input, and I would carry the fix into that draft at the same time.

The PTCL draft does not yet define how lease_id and the fencing epoch appear in a TRACE record or a GapDisclosure. That is a real open design question, and I would rather the maintainers choose the carrier (an optional cause detail on the disclosure, or a field on the hop record) than have me assume one.

Manifest binding. Once trace-spec#483 lands a carrier for manifest {id, digest, media_type} in ca2a hop records and cMCP claims, cross_check_trace_dag should also require that every hop of a workflow, and every retry, names the same manifest digest unless a signed supersession says otherwise. On the cMCP side, the orphaned-session path (find_orphaned_sessions) and the outcome_unknown recovery marker are the natural place to produce a signed disclosure for the claim that close_session never wrote.

Acceptance criteria

1. A library entry point in trace-spec or agentrust-trace that verifies a GapDisclosure/1.0 and passes every existing vector in that directory, so ca2a and cmcp do not each re-implement it.
2. ca2a_verify.verify_trace_dag accepts GapDisclosure/1.0 elements and reports each disclosed gap individually; cross_check_trace_dag reconciles by credential identity and no longer depends on len(records) == len(chain).
3. Case A, B and C above become conformance cases in the shared acceptance harness (Build a reproducible acceptance harness for confidential workflows across components #199) with expected verdicts: A and B gap_disclosed, C rejected unless fenced, and a profile requiring completeness rejects all of A and B.
4. Trimming the chain no longer silently succeeds: a verifier given a credential that has a revocation statement and no covering record or disclosure fails closed.
5. A cMCP orphaned session produces a signed disclosure, or the verifier documents that it will report receipt_missing_required for it.

I have not opened a PR for any of this. If the maintainers agree on the shape, I can start with item 1 as the smallest piece once the queue allows.