From 95029a318a78f423bcb562551badc838a6f45690 Mon Sep 17 00:00:00 2001 From: Srinivasa Consultants <78673153+dinakarjs@users.noreply.github.com> Date: Sun, 25 Oct 2026 03:50:20 +0530 Subject: [PATCH 1/2] docs: register measurement failure and guard its documentation Signed-off-by: Srinivasa Consultants <78673153+dinakarjs@users.noreply.github.com> --- docs/spec/error-codes.md | 1 + docs/spec/failure-modes.md | 575 +++++++++--------- tests/unit/test_measurement_error_registry.py | 14 + 3 files changed, 307 insertions(+), 283 deletions(-) create mode 100644 tests/unit/test_measurement_error_registry.py diff --git a/docs/spec/error-codes.md b/docs/spec/error-codes.md index c7a0dfd6..bd9ba3cd 100644 --- a/docs/spec/error-codes.md +++ b/docs/spec/error-codes.md @@ -8,6 +8,7 @@ This is the normative registry for all error codes used across the cMCP Runtime. | error_code | HTTP status | log level | description | spec reference | |---|---|---|---|---| +| `MEASUREMENT_UNAVAILABLE` | 500 / startup | FATAL | The gateway code/configuration/policy inputs could not be measured; startup aborts rather than extending an unestablished identity. Inspect the reported measurement detail and restore readable, supported inputs before retrying; do not rotate a signing key as a generic remedy | [failure-modes.md](failure-modes.md) | | `ATTESTATION_REPORT_UNAVAILABLE` | 503 | FATAL | TEE provider did not return an attestation report within timeout | [failure-modes.md FM-1](failure-modes.md) | | `ATTESTATION_PROVIDER_UNSUPPORTED` | 500 | FATAL | No supported TEE provider detected and `CMCP_DEV_MODE` is not set | [attestation.md §1.1](attestation.md) | | `POLICY_HASH_MISMATCH` | 500 | FATAL | Measured policy bundle hash does not match deployment manifest | [failure-modes.md FM-4](failure-modes.md) | diff --git a/docs/spec/failure-modes.md b/docs/spec/failure-modes.md index a0d3307f..124ac0a7 100644 --- a/docs/spec/failure-modes.md +++ b/docs/spec/failure-modes.md @@ -1,283 +1,292 @@ ---- -description: "Exact cMCP runtime behavior for each failure: attestation at startup and mid-session, TEE faults, policy hash mismatch, and MCP parse errors." ---- - -# Failure Mode Specification - ---- -Status: Draft v0.1 -Last updated: 2026-06-04 -Stability: Unstable , expect breaking changes before v1.0 ---- - -Documents exact runtime behavior for every failure scenario. - -In short, cMCP fails closed: when something goes wrong, such as a missing hardware report or a rule set that does not match the approved one, it refuses the affected calls instead of carrying on unchecked. Read this if you operate the gateway or need to know what evidence a failed call leaves behind. - -Closes #22. - ---- - -## Decision Table - -| Failure | Default Behavior | Configurable? | TRACE Claim Status | Log Format | -|---------|-----------------|---------------|-------------------|------------| -| Attestation failure at startup | Fail-closed. Runtime does not start. No traffic processed. | No | Not produced | `FATAL attestation_failure` + provider + error code | -| Attestation staleness mid-session | Fail-closed. Active sessions terminated. | Yes (see FM-2) | `attestation_stale: true` on claims produced at or after deadline | `WARN attestation_stale` + session_id + expiry_utc | -| TEE fault mid-invocation | Fail-closed on in-flight call. Structured error returned to agent. Session continues if TEE recovers. | No | `status: fault` | `ERROR tee_fault` + call_id + timestamp | -| Policy bundle hash mismatch at startup | Fail-closed. Runtime does not start. | No | Not produced | `FATAL policy_hash_mismatch` + expected_hash + actual_hash | -| MCP protocol parse failure | Fail-closed on the specific call. Structured error returned. Session continues. | No | `status: parse_error` | `WARN parse_failure` + call_id + payload_hash | - ---- - -## FM-1: Attestation Failure at Startup - -**Trigger**: The TEE cannot produce a valid attestation report at runtime startup. Causes include: TPM not present or disabled in firmware, SEV-SNP or TDX not enabled on the host, firmware version not matching a known-good measurement, or the attestation service is unreachable. - -**Decision**: Fail-closed. The runtime does not start. No MCP traffic is processed. No TRACE Claims are produced (there is no attested identity to sign with). - -**Behavior**: -1. Runtime startup sequence attempts attestation as the first step, before binding any network port. -2. If attestation fails, the runtime writes a structured log entry and exits with a non-zero status code. -3. The orchestrator (Kubernetes, systemd, ECS) observes the non-zero exit and does not route traffic to the pod/service. - -**Log entry format**: -```json -{ - "level": "FATAL", - "event": "attestation_failure", - "timestamp_utc": "2025-10-01T12:00:00.000Z", - "tee_provider": "sev-snp", - "error_code": "ATTESTATION_REPORT_UNAVAILABLE", - "error_detail": "AMD PSP did not return a report within 5000ms", - "cmcp_version": "0.3.0", - "action": "startup_aborted" -} -``` - -**Operator notification**: The orchestration platform health check fails (no process listening on the health port). Alerting on pod restart loops or task failure is the operator responsibility. The runtime does not attempt to send an outbound notification from a failed-attestation state, as there is no attested channel to send it on. - ---- - -## FM-2: Attestation Staleness Mid-Session - -**Trigger**: The attestation report has a validity period (typically 1-24 hours depending on provider). An active session is in progress when the attestation validity deadline passes. - -**Decision**: Configurable. Default is fail-closed: terminate the session. The session max duration is bounded by the attestation validity period. A new session requires a fresh attestation report. - -**Configurable option**: `attestation_staleness_policy: warn_only` allows sessions to continue past the validity deadline. This is not recommended for production. TRACE Claims produced after the deadline are marked `attestation_stale: true` to alert downstream verifiers. - -**Behavior (default, fail-closed)**: -1. Runtime tracks the attestation validity deadline per session. -2. At the deadline, the runtime closes the session: sends a structured close notification to the agent, writes a log entry, and marks the session closed in the audit chain. -3. In-flight tool calls at the moment of termination are treated as FM-3 (TEE fault mid-invocation): fail-closed, structured error returned. -4. No new sessions are accepted until attestation is renewed (runtime restarts with a fresh report, or a re-attestation flow completes if supported by the provider). - -**TRACE Claim field** (when `warn_only` is configured): -```json -"attestation_report": { - "provider": "tpm", - "measurement": "...", - "report_data": "...", - "attestation_stale": true, - "validity_expired_utc": "2025-10-01T13:00:00.000Z" -} -``` - -**Log entry format**: -```json -{ - "level": "WARN", - "event": "attestation_stale", - "timestamp_utc": "2025-10-01T13:00:01.000Z", - "session_id": "sess_01JABCDE", - "expiry_utc": "2025-10-01T13:00:00.000Z", - "staleness_policy": "fail_closed", - "action": "session_terminated" -} -``` - ---- - -## FM-3: TEE Fault Mid-Invocation - -**Trigger**: The TEE process (the runtime enclave context) crashes, becomes unresponsive, or returns an internal fault during an active tool call. This covers: enclave exception, out-of-memory inside the TEE, watchdog timeout, or unhandled panic in the gateway process. - -**Decision**: Fail-closed on the in-flight call. The agent receives a structured error, not the tool response (which was never obtained or cannot be trusted). The audit chain records the fault. If the TEE recovers (e.g., the runtime process restarts and re-attests), subsequent calls in a new session may proceed. - -**Behavior**: -1. In-flight call is abandoned. No tool response is forwarded to the agent. -2. A structured error is returned to the agent over the existing HTTP/SSE connection (if the connection is still alive). If the connection is also lost, the agent MCP client receives a connection error. -3. An audit chain entry is written for the failed call (see TRACE Claim structure below). -4. The runtime process exits if the TEE fault is unrecoverable. The orchestrator restarts it; a new attestation report is obtained on restart. - -**Structured error returned to agent**: -```json -{ - "jsonrpc": "2.0", - "id": "call-42", - "error": { - "code": -32099, - "message": "Runtime internal fault", - "data": { - "call_id": "call-42", - "fault_type": "tee_fault", - "timestamp_utc": "2025-10-01T12:05:00.000Z", - "trace_claim_available": false - } - } -} -``` - -**Audit chain entry for the failed call**: -```json -{ - "call_id": "call-42", - "session_id": "sess_01JABCDE", - "timestamp_utc": "2025-10-01T12:05:00.000Z", - "tool_name": "database_query", - "status": "fault", - "fault_type": "tee_fault", - "tool_response_logged": false, - "note": "In-flight call abandoned due to TEE fault. No tool response was received or forwarded." -} -``` - -**Log entry format**: -```json -{ - "level": "ERROR", - "event": "tee_fault", - "timestamp_utc": "2025-10-01T12:05:00.000Z", - "call_id": "call-42", - "session_id": "sess_01JABCDE", - "fault_detail": "Enclave watchdog timeout after 30000ms", - "action": "call_abandoned_runtime_restarting" -} -``` - ---- - -## FM-4: Policy Bundle Hash Mismatch at Startup - -**Trigger**: The runtime loads its Cedar policy bundle at startup and measures its hash. If the measured hash does not match the expected hash in the deployment manifest (set at deploy time, part of the TEE measurement or a separate signed manifest), the runtime refuses to start. - -**Decision**: Fail-closed. Runtime does not start. This prevents a swapped or tampered policy bundle from being used. A silently different policy would undermine the governance guarantee: the TRACE Claim asserts a specific policy bundle hash, and if that hash is wrong, every claim produced would be a lie. - -**Behavior**: -1. At startup, before binding any port, the runtime reads the policy bundle from disk and computes its SHA-256 hash. -2. The runtime compares the computed hash against the expected hash from the signed deployment manifest. -3. If they do not match, the runtime writes a log entry and exits with a non-zero status code. -4. No TRACE Claims are produced. - -**Log entry format**: -```json -{ - "level": "FATAL", - "event": "policy_hash_mismatch", - "timestamp_utc": "2025-10-01T12:00:00.000Z", - "expected_hash": "sha256:abc123...", - "actual_hash": "sha256:def456...", - "policy_bundle_path": "/etc/cmcp/policies/bundle.cedar", - "action": "startup_aborted" -} -``` - -**Operator notification**: Same as FM-1. Orchestrator health check fails. The log entry provides the expected and actual hashes for forensic comparison. - -**Note on deployment**: The expected hash must be set at deploy time and must be part of the TEE measurement or a signed manifest that the TEE verifies before trusting. If the expected hash itself can be tampered with, this check provides no protection. The deployment pipeline is responsible for ensuring the expected hash is authoritative. - ---- - -## FM-5: MCP Protocol Parse Failure - -**Trigger**: The runtime receives a message from the agent MCP client that is malformed JSON-RPC, uses an unsupported JSON-RPC version, violates the MCP protocol schema, or is crafted to exploit a parser (e.g., deeply nested structures, oversized payloads, binary data in a text field). - -**Decision**: Fail-closed on the specific call. The session continues; only the failing call is rejected. A structured parse error is returned to the agent. The raw payload hash (not the content) is logged for forensics. The full payload is not logged to avoid storing attacker-controlled data. - -**Behavior**: -1. Parser rejects the message before any tool lookup or policy evaluation. -2. Structured error is returned to the agent MCP client. -3. Log entry records the call_id (if parseable), session_id, error type, and a SHA-256 hash of the raw payload bytes. -4. The session remains open. Subsequent well-formed calls are processed normally. -5. Rate limiting: if a session sends more than N parse failures within a configurable window, the runtime closes the session (configurable threshold, default: 10 failures in 60 seconds). - -**Structured error returned to agent**: -```json -{ - "jsonrpc": "2.0", - "id": null, - "error": { - "code": -32700, - "message": "Parse error", - "data": { - "fault_type": "mcp_parse_failure", - "timestamp_utc": "2025-10-01T12:10:00.000Z" - } - } -} -``` - -**Log entry format**: -```json -{ - "level": "WARN", - "event": "parse_failure", - "timestamp_utc": "2025-10-01T12:10:00.000Z", - "session_id": "sess_01JABCDE", - "call_id": null, - "parse_error": "invalid JSON: unexpected token at position 42", - "payload_hash": "sha256:789abc...", - "payload_size_bytes": 1024, - "action": "call_rejected_session_continues" -} -``` - ---- - -## Failed-Call TRACE Claim JSON Structure - -For calls that are rejected or faulted (FM-3, FM-5), the TRACE Claim records the failure without including the tool response. The claim is still signed by the TEE if the TEE is operational; for FM-3 (TEE fault), the claim may be unsigned if the signing key is unavailable. - -```json -{ - "trace_version": "0.1", - "session_id": "sess_01JABCDE", - "timestamp_utc": "2025-10-01T12:05:00.000Z", - "tee_public_key": "...", - "attestation_report": { - "provider": "sev-snp", - "measurement": "...", - "report_data": "...", - "attestation_stale": false - }, - "policy_bundle": { - "hash": "sha256:abc123...", - "enforcement_mode": "enforcing", - "policy_version": "1.2.0" - }, - "tool_catalog": { - "hash": "sha256:def456..." - }, - "call_summary": { - "total": 1, - "allowed": 0, - "denied": 0, - "faulted": 1, - "tools_invoked": [] - }, - "failed_call": { - "call_id": "call-42", - "tool_name": "database_query", - "status": "fault", - "fault_type": "tee_fault", - "timestamp_utc": "2025-10-01T12:05:00.000Z", - "tool_response_logged": false, - "policy_evaluated": false - }, - "audit_chain_root": "...", - "audit_chain_tip": "...", - "signature": null, - "signature_note": "TEE fault prevented signing. This claim is for audit record only and must not be treated as an attested proof." -} -``` +--- +description: "Exact cMCP runtime behavior for each failure: attestation at startup and mid-session, TEE faults, policy hash mismatch, and MCP parse errors." +--- + +# Failure Mode Specification + +--- +Status: Draft v0.1 +Last updated: 2026-06-04 +Stability: Unstable , expect breaking changes before v1.0 +--- + +Documents exact runtime behavior for every failure scenario. + +In short, cMCP fails closed: when something goes wrong, such as a missing hardware report or a rule set that does not match the approved one, it refuses the affected calls instead of carrying on unchecked. Read this if you operate the gateway or need to know what evidence a failed call leaves behind. + +Closes #22. + +--- + +## Decision Table + +| Failure | Default Behavior | Configurable? | TRACE Claim Status | Log Format | +|---------|-----------------|---------------|-------------------|------------| +| Attestation failure at startup | Fail-closed. Runtime does not start. No traffic processed. | No | Not produced | `FATAL attestation_failure` + provider + error code | +| Attestation staleness mid-session | Fail-closed. Active sessions terminated. | Yes (see FM-2) | `attestation_stale: true` on claims produced at or after deadline | `WARN attestation_stale` + session_id + expiry_utc | +| TEE fault mid-invocation | Fail-closed on in-flight call. Structured error returned to agent. Session continues if TEE recovers. | No | `status: fault` | `ERROR tee_fault` + call_id + timestamp | +| Policy bundle hash mismatch at startup | Fail-closed. Runtime does not start. | No | Not produced | `FATAL policy_hash_mismatch` + expected_hash + actual_hash | +| MCP protocol parse failure | Fail-closed on the specific call. Structured error returned. Session continues. | No | `status: parse_error` | `WARN parse_failure` + call_id + payload_hash | + +--- + +## FM-1: Attestation Failure at Startup + +**Trigger**: The TEE cannot produce a valid attestation report at runtime startup. Causes include: TPM not present or disabled in firmware, SEV-SNP or TDX not enabled on the host, firmware version not matching a known-good measurement, or the attestation service is unreachable. + +**Decision**: Fail-closed. The runtime does not start. No MCP traffic is processed. No TRACE Claims are produced (there is no attested identity to sign with). + +**Behavior**: +1. Runtime startup sequence attempts attestation as the first step, before binding any network port. +2. If attestation fails, the runtime writes a structured log entry and exits with a non-zero status code. +3. The orchestrator (Kubernetes, systemd, ECS) observes the non-zero exit and does not route traffic to the pod/service. + +**Log entry format**: +```json +{ + "level": "FATAL", + "event": "attestation_failure", + "timestamp_utc": "2025-10-01T12:00:00.000Z", + "tee_provider": "sev-snp", + "error_code": "ATTESTATION_REPORT_UNAVAILABLE", + "error_detail": "AMD PSP did not return a report within 5000ms", + "cmcp_version": "0.3.0", + "action": "startup_aborted" +} +``` + +**Operator notification**: The orchestration platform health check fails (no process listening on the health port). Alerting on pod restart loops or task failure is the operator responsibility. The runtime does not attempt to send an outbound notification from a failed-attestation state, as there is no attested channel to send it on. + +--- + +## FM-2: Attestation Staleness Mid-Session + +**Trigger**: The attestation report has a validity period (typically 1-24 hours depending on provider). An active session is in progress when the attestation validity deadline passes. + +**Decision**: Configurable. Default is fail-closed: terminate the session. The session max duration is bounded by the attestation validity period. A new session requires a fresh attestation report. + +**Configurable option**: `attestation_staleness_policy: warn_only` allows sessions to continue past the validity deadline. This is not recommended for production. TRACE Claims produced after the deadline are marked `attestation_stale: true` to alert downstream verifiers. + +**Behavior (default, fail-closed)**: +1. Runtime tracks the attestation validity deadline per session. +2. At the deadline, the runtime closes the session: sends a structured close notification to the agent, writes a log entry, and marks the session closed in the audit chain. +3. In-flight tool calls at the moment of termination are treated as FM-3 (TEE fault mid-invocation): fail-closed, structured error returned. +4. No new sessions are accepted until attestation is renewed (runtime restarts with a fresh report, or a re-attestation flow completes if supported by the provider). + +**TRACE Claim field** (when `warn_only` is configured): +```json +"attestation_report": { + "provider": "tpm", + "measurement": "...", + "report_data": "...", + "attestation_stale": true, + "validity_expired_utc": "2025-10-01T13:00:00.000Z" +} +``` + +**Log entry format**: +```json +{ + "level": "WARN", + "event": "attestation_stale", + "timestamp_utc": "2025-10-01T13:00:01.000Z", + "session_id": "sess_01JABCDE", + "expiry_utc": "2025-10-01T13:00:00.000Z", + "staleness_policy": "fail_closed", + "action": "session_terminated" +} +``` + +--- + +## FM-3: TEE Fault Mid-Invocation + +**Trigger**: The TEE process (the runtime enclave context) crashes, becomes unresponsive, or returns an internal fault during an active tool call. This covers: enclave exception, out-of-memory inside the TEE, watchdog timeout, or unhandled panic in the gateway process. + +**Decision**: Fail-closed on the in-flight call. The agent receives a structured error, not the tool response (which was never obtained or cannot be trusted). The audit chain records the fault. If the TEE recovers (e.g., the runtime process restarts and re-attests), subsequent calls in a new session may proceed. + +**Behavior**: +1. In-flight call is abandoned. No tool response is forwarded to the agent. +2. A structured error is returned to the agent over the existing HTTP/SSE connection (if the connection is still alive). If the connection is also lost, the agent MCP client receives a connection error. +3. An audit chain entry is written for the failed call (see TRACE Claim structure below). +4. The runtime process exits if the TEE fault is unrecoverable. The orchestrator restarts it; a new attestation report is obtained on restart. + +**Structured error returned to agent**: +```json +{ + "jsonrpc": "2.0", + "id": "call-42", + "error": { + "code": -32099, + "message": "Runtime internal fault", + "data": { + "call_id": "call-42", + "fault_type": "tee_fault", + "timestamp_utc": "2025-10-01T12:05:00.000Z", + "trace_claim_available": false + } + } +} +``` + +**Audit chain entry for the failed call**: +```json +{ + "call_id": "call-42", + "session_id": "sess_01JABCDE", + "timestamp_utc": "2025-10-01T12:05:00.000Z", + "tool_name": "database_query", + "status": "fault", + "fault_type": "tee_fault", + "tool_response_logged": false, + "note": "In-flight call abandoned due to TEE fault. No tool response was received or forwarded." +} +``` + +**Log entry format**: +```json +{ + "level": "ERROR", + "event": "tee_fault", + "timestamp_utc": "2025-10-01T12:05:00.000Z", + "call_id": "call-42", + "session_id": "sess_01JABCDE", + "fault_detail": "Enclave watchdog timeout after 30000ms", + "action": "call_abandoned_runtime_restarting" +} +``` + +--- + +## FM-4: Policy Bundle Hash Mismatch at Startup + +**Trigger**: The runtime loads its Cedar policy bundle at startup and measures its hash. If the measured hash does not match the expected hash in the deployment manifest (set at deploy time, part of the TEE measurement or a separate signed manifest), the runtime refuses to start. + +**Decision**: Fail-closed. Runtime does not start. This prevents a swapped or tampered policy bundle from being used. A silently different policy would undermine the governance guarantee: the TRACE Claim asserts a specific policy bundle hash, and if that hash is wrong, every claim produced would be a lie. + +**Behavior**: +1. At startup, before binding any port, the runtime reads the policy bundle from disk and computes its SHA-256 hash. +2. The runtime compares the computed hash against the expected hash from the signed deployment manifest. +3. If they do not match, the runtime writes a log entry and exits with a non-zero status code. +4. No TRACE Claims are produced. + +**Log entry format**: +```json +{ + "level": "FATAL", + "event": "policy_hash_mismatch", + "timestamp_utc": "2025-10-01T12:00:00.000Z", + "expected_hash": "sha256:abc123...", + "actual_hash": "sha256:def456...", + "policy_bundle_path": "/etc/cmcp/policies/bundle.cedar", + "action": "startup_aborted" +} +``` + +**Operator notification**: Same as FM-1. Orchestrator health check fails. The log entry provides the expected and actual hashes for forensic comparison. + +**Note on deployment**: The expected hash must be set at deploy time and must be part of the TEE measurement or a signed manifest that the TEE verifies before trusting. If the expected hash itself can be tampered with, this check provides no protection. The deployment pipeline is responsible for ensuring the expected hash is authoritative. + +--- + +## FM-5: MCP Protocol Parse Failure + +**Trigger**: The runtime receives a message from the agent MCP client that is malformed JSON-RPC, uses an unsupported JSON-RPC version, violates the MCP protocol schema, or is crafted to exploit a parser (e.g., deeply nested structures, oversized payloads, binary data in a text field). + +**Decision**: Fail-closed on the specific call. The session continues; only the failing call is rejected. A structured parse error is returned to the agent. The raw payload hash (not the content) is logged for forensics. The full payload is not logged to avoid storing attacker-controlled data. + +**Behavior**: +1. Parser rejects the message before any tool lookup or policy evaluation. +2. Structured error is returned to the agent MCP client. +3. Log entry records the call_id (if parseable), session_id, error type, and a SHA-256 hash of the raw payload bytes. +4. The session remains open. Subsequent well-formed calls are processed normally. +5. Rate limiting: if a session sends more than N parse failures within a configurable window, the runtime closes the session (configurable threshold, default: 10 failures in 60 seconds). + +**Structured error returned to agent**: +```json +{ + "jsonrpc": "2.0", + "id": null, + "error": { + "code": -32700, + "message": "Parse error", + "data": { + "fault_type": "mcp_parse_failure", + "timestamp_utc": "2025-10-01T12:10:00.000Z" + } + } +} +``` + +**Log entry format**: +```json +{ + "level": "WARN", + "event": "parse_failure", + "timestamp_utc": "2025-10-01T12:10:00.000Z", + "session_id": "sess_01JABCDE", + "call_id": null, + "parse_error": "invalid JSON: unexpected token at position 42", + "payload_hash": "sha256:789abc...", + "payload_size_bytes": 1024, + "action": "call_rejected_session_continues" +} +``` + +--- + +## Failed-Call TRACE Claim JSON Structure + +For calls that are rejected or faulted (FM-3, FM-5), the TRACE Claim records the failure without including the tool response. The claim is still signed by the TEE if the TEE is operational; for FM-3 (TEE fault), the claim may be unsigned if the signing key is unavailable. + +```json +{ + "trace_version": "0.1", + "session_id": "sess_01JABCDE", + "timestamp_utc": "2025-10-01T12:05:00.000Z", + "tee_public_key": "...", + "attestation_report": { + "provider": "sev-snp", + "measurement": "...", + "report_data": "...", + "attestation_stale": false + }, + "policy_bundle": { + "hash": "sha256:abc123...", + "enforcement_mode": "enforcing", + "policy_version": "1.2.0" + }, + "tool_catalog": { + "hash": "sha256:def456..." + }, + "call_summary": { + "total": 1, + "allowed": 0, + "denied": 0, + "faulted": 1, + "tools_invoked": [] + }, + "failed_call": { + "call_id": "call-42", + "tool_name": "database_query", + "status": "fault", + "fault_type": "tee_fault", + "timestamp_utc": "2025-10-01T12:05:00.000Z", + "tool_response_logged": false, + "policy_evaluated": false + }, + "audit_chain_root": "...", + "audit_chain_tip": "...", + "signature": null, + "signature_note": "TEE fault prevented signing. This claim is for audit record only and must not be treated as an attested proof." +} +``` + +## Measurement unavailable at startup + +`MEASUREMENT_UNAVAILABLE` means the gateway could not derive its measurement. +Startup aborts; a caller should not treat this as a successfully measured gateway +or as evidence that its TPM changed. Inspect the diagnostic detail for unreadable +or unsupported measured inputs, restore the intended deployment inputs, then retry. +Do not bypass measurement or rotate a signing key merely to suppress this error. +This entry describes the existing runtime behavior; it adds no new recovery action. diff --git a/tests/unit/test_measurement_error_registry.py b/tests/unit/test_measurement_error_registry.py new file mode 100644 index 00000000..975c2ad5 --- /dev/null +++ b/tests/unit/test_measurement_error_registry.py @@ -0,0 +1,14 @@ +"""Keep the measurement failure emitted at startup discoverable by operators.""" +from pathlib import Path + +from cmcp_runtime.tee.measurement import MeasurementUnavailable + + +def test_measurement_failure_has_registry_row_and_recovery_guidance(): + root = Path(__file__).resolve().parents[2] + code = MeasurementUnavailable.code + registry = (root / 'docs/spec/error-codes.md').read_text(encoding='utf-8') + rows = [row for row in registry.splitlines() if row.startswith(f'| `{code}` |')] + assert len(rows) == 1 + assert '500' in rows[0] and 'FATAL' in rows[0] + assert code in (root / 'docs/spec/failure-modes.md').read_text(encoding='utf-8') From 694cd63ad1ae268581df03be50f2ff88966dff0d Mon Sep 17 00:00:00 2001 From: Srinivasa Consultants <78673153+dinakarjs@users.noreply.github.com> Date: Sun, 1 Nov 2026 08:49:37 +0530 Subject: [PATCH 2/2] docs: complete measurement failure recovery and mode coverage Cover TPM/NV failures and production versus dev-mode startup behavior in the registry and decision table. Protect recovery guidance and exact registry columns with focused consistency tests. Runtime behavior is unchanged. Prepared with OpenAI Codex. Signed-off-by: Srinivasa Dinakar Jainti --- docs/spec/error-codes.md | 2 +- docs/spec/failure-modes.md | 33 ++++++-- tests/unit/test_measurement_error_registry.py | 75 +++++++++++++++++-- 3 files changed, 93 insertions(+), 17 deletions(-) diff --git a/docs/spec/error-codes.md b/docs/spec/error-codes.md index 371d17cf..550f558d 100644 --- a/docs/spec/error-codes.md +++ b/docs/spec/error-codes.md @@ -8,7 +8,7 @@ This is the normative registry for all error codes used across the cMCP Runtime. | error_code | HTTP status | log level | description | spec reference | |---|---|---|---|---| -| `MEASUREMENT_UNAVAILABLE` | 500 / startup | FATAL | The gateway code/configuration/policy inputs could not be measured; startup aborts rather than extending an unestablished identity. Inspect the reported measurement detail and restore readable, supported inputs before retrying; do not rotate a signing key as a generic remedy | [failure-modes.md](failure-modes.md) | +| `MEASUREMENT_UNAVAILABLE` | 500 / startup | FATAL (production); WARN (dev mode) | Code/configuration/policy measurement or a TPM/NV operation failed; production startup aborts. `CMCP_DEV_MODE` warns and continues without established gateway measurement. Inspect diagnostic detail and recovery guidance; do not infer TPM replacement or rotate a signing key solely from this code | [failure-modes.md](failure-modes.md#measurement-unavailable-at-startup) | | `ATTESTATION_REPORT_UNAVAILABLE` | 503 | FATAL | TEE provider did not return an attestation report within timeout | [failure-modes.md FM-1](failure-modes.md) | | `ATTESTATION_PROVIDER_UNSUPPORTED` | 500 | FATAL | No supported TEE provider detected and `CMCP_DEV_MODE` is not set | [attestation.md §1.1](attestation.md) | | `POLICY_HASH_MISMATCH` | 500 | FATAL | Measured policy bundle hash does not match deployment manifest | [failure-modes.md FM-4](failure-modes.md) | diff --git a/docs/spec/failure-modes.md b/docs/spec/failure-modes.md index 8e323df6..4b009372 100644 --- a/docs/spec/failure-modes.md +++ b/docs/spec/failure-modes.md @@ -1,12 +1,12 @@ --- -description: "Exact cMCP runtime behavior for each failure: attestation at startup and mid-session, TEE faults, policy hash mismatch, and MCP parse errors." +description: "cMCP failure behavior and recovery: gateway measurement and TPM/NV operations, production/dev startup, attestation, TEE faults, policy mismatch, and MCP parse errors." --- # Failure Mode Specification --- Status: Draft v0.1 -Last updated: 2026-06-04 +Last updated: 2026-11-01 Stability: Unstable , expect breaking changes before v1.0 --- @@ -26,6 +26,7 @@ Closes #22. | Attestation staleness mid-session | Fail-closed. Active sessions terminated. | Yes (see FM-2) | `attestation_stale: true` on claims produced at or after deadline | `WARN attestation_stale` + session_id + expiry_utc | | TEE fault mid-invocation | Fail-closed on in-flight call. Structured error returned to agent. Session continues if TEE recovers. | No | `status: fault` | `ERROR tee_fault` + call_id + timestamp | | Policy bundle hash mismatch at startup | Fail-closed. Runtime does not start. | No | Not produced | `FATAL policy_hash_mismatch` + expected_hash + actual_hash | +| Gateway measurement or TPM/NV operation unavailable at startup | Production: fail-closed; runtime does not start. `CMCP_DEV_MODE`: warn and continue without established gateway measurement. | Dev mode only (see Measurement unavailable at startup) | No claim in production; dev continuation is not evidence of a successfully measured gateway | Production: `FATAL MEASUREMENT_UNAVAILABLE` + detail; dev mode: `WARN` measurement unavailable | | Sealed signing key unavailable at startup | Fail-closed. Runtime does not start. Nothing is minted to recover. | No | Not produced | `FATAL SEALED_KEY_*` + detail (see FM-1b) | | MCP protocol parse failure | Fail-closed on the specific call. Structured error returned. Session continues. | No | `status: parse_error` | `WARN parse_failure` + call_id + payload_hash | @@ -301,9 +302,25 @@ For calls that are rejected or faulted (FM-3, FM-5), the TRACE Claim records the ## Measurement unavailable at startup -`MEASUREMENT_UNAVAILABLE` means the gateway could not derive its measurement. -Startup aborts; a caller should not treat this as a successfully measured gateway -or as evidence that its TPM changed. Inspect the diagnostic detail for unreadable -or unsupported measured inputs, restore the intended deployment inputs, then retry. -Do not bypass measurement or rotate a signing key merely to suppress this error. -This entry describes the existing runtime behavior; it adds no new recovery action. +`MEASUREMENT_UNAVAILABLE` covers failures deriving code/configuration/policy +measurements and failures completing the TPM-backed measurement path. + +**Triggers**: unreadable or unsupported measured inputs, TPM open errors, NV index +define/read/validation errors, `TPM2_NV_Extend` failure, chain-value mismatch, +truncated NV values, and certify failures. Inspect the reported diagnostic detail +rather than assuming an input-file problem. + +**Production**: startup aborts with `FATAL MEASUREMENT_UNAVAILABLE`; no traffic is +processed from this failed startup. + +**Development**: with `CMCP_DEV_MODE`, the runtime logs a warning and continues. +This continuation is not evidence of a successfully measured gateway. + +**Recovery**: restore readable, supported measurement inputs when they are the +reported cause. For a TPM/NV failure, investigate device access, library support, +NV index configuration and the reported chain or certify operation against the +intended deployment state before restarting. This code alone is not evidence +that the TPM changed. Do not bypass measurement or rotate a signing key merely +to suppress this error. Sealed-key recovery remains a separate decision (FM-1b). + +This entry describes existing runtime behavior; it adds no new recovery action. diff --git a/tests/unit/test_measurement_error_registry.py b/tests/unit/test_measurement_error_registry.py index 975c2ad5..b4c435cd 100644 --- a/tests/unit/test_measurement_error_registry.py +++ b/tests/unit/test_measurement_error_registry.py @@ -1,14 +1,73 @@ -"""Keep the measurement failure emitted at startup discoverable by operators.""" +"""Keep startup measurement diagnostics and operator recovery discoverable.""" + from pathlib import Path from cmcp_runtime.tee.measurement import MeasurementUnavailable +ROOT = Path(__file__).resolve().parents[2] + -def test_measurement_failure_has_registry_row_and_recovery_guidance(): - root = Path(__file__).resolve().parents[2] - code = MeasurementUnavailable.code - registry = (root / 'docs/spec/error-codes.md').read_text(encoding='utf-8') - rows = [row for row in registry.splitlines() if row.startswith(f'| `{code}` |')] +def _row(text, first_column): + rows = [line for line in text.splitlines() if line.startswith(f"| {first_column} |")] assert len(rows) == 1 - assert '500' in rows[0] and 'FATAL' in rows[0] - assert code in (root / 'docs/spec/failure-modes.md').read_text(encoding='utf-8') + columns = [column.strip() for column in rows[0].split("|")[1:-1]] + assert len(columns) == 5 + return columns + + +def test_measurement_failure_registry_columns_match_runtime_and_modes(): + registry = (ROOT / "docs/spec/error-codes.md").read_text(encoding="utf-8") + columns = _row(registry, f"`{MeasurementUnavailable.code}`") + assert columns[1] == f"{MeasurementUnavailable.http_status} / startup" + assert columns[2] == "FATAL (production); WARN (dev mode)" + for phrase in ( + "TPM/NV", + "production startup aborts", + "`CMCP_DEV_MODE`", + "without established gateway measurement", + "do not infer TPM replacement", + "rotate a signing key solely from this code", + ): + assert phrase in columns[3] + assert columns[4] == "[failure-modes.md](failure-modes.md#measurement-unavailable-at-startup)" + + +def test_measurement_failure_decision_table_covers_both_modes(): + failure_modes = (ROOT / "docs/spec/failure-modes.md").read_text(encoding="utf-8") + assert "## Decision Table" in failure_modes + table = failure_modes.split("## Decision Table", 1)[1].split("\n## ", 1)[0] + columns = _row(table, "Gateway measurement or TPM/NV operation unavailable at startup") + assert "runtime does not start" in columns[1] + assert ( + "`CMCP_DEV_MODE`: warn and continue without established gateway measurement" in columns[1] + ) + assert columns[2] == "Dev mode only (see Measurement unavailable at startup)" + assert "not evidence of a successfully measured gateway" in columns[3] + assert "`FATAL MEASUREMENT_UNAVAILABLE`" in columns[4] + assert "dev mode: `WARN`" in columns[4] + + +def test_measurement_failure_recovery_guidance_is_retained(): + failure_modes = (ROOT / "docs/spec/failure-modes.md").read_text(encoding="utf-8") + assert "## Measurement unavailable at startup" in failure_modes + section = failure_modes.split("## Measurement unavailable at startup", 1)[1].split("\n## ", 1)[ + 0 + ] + prose = " ".join(section.split()) + for phrase in ( + "TPM open errors", + "NV index define/read/validation errors", + "`TPM2_NV_Extend` failure", + "chain-value mismatch", + "truncated NV values", + "certify failures", + "**Production**: startup aborts", + "**Development**: with `CMCP_DEV_MODE`, the runtime logs a warning and continues.", + "not evidence of a successfully measured gateway", + "restore readable, supported measurement inputs", + "investigate device access, library support, NV index configuration", + "This code alone is not evidence that the TPM changed.", + "Do not bypass measurement or rotate a signing key merely to suppress this error.", + "Sealed-key recovery remains a separate decision (FM-1b).", + ): + assert phrase in prose