Skip to content

feat: add provider-neutral conformance contract - #6

Merged
jonathanjasare merged 2 commits into
masterfrom
feature/m3-conformance-contract
Aug 12, 2026
Merged

feat: add provider-neutral conformance contract#6
jonathanjasare merged 2 commits into
masterfrom
feature/m3-conformance-contract

Conversation

@jonathanjasare

@jonathanjasare jonathanjasare commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Purpose

Establish the provider-neutral conformance boundary required before implementing the first cloud renderer. This gives future provider work an exact local-engine reference without changing the frozen TW-001 rule or its spent holdout.

What changed

  • Added the fixed sequence-v1 corpus with 37 synthetic normalized events across 12 sessions.
  • Added exact expected detections for ordering, session isolation, ordinal gaps, candidate selection, missing timestamps, and the 300-second window boundary.
  • Added a test-only bounded rule outside rules/; TW-001 remains the only published measured detector.
  • Added tripwire conformance evaluate and tripwire conformance verify.
  • Extended tripwire verify to reproduce both the committed AgentDojo result and the conformance result without credentials.
  • Made rule and conformance inputs reject unknown or malformed fields so renderer typos and invalid metadata fail closed.
  • Recorded the contract and provider-status boundary in ADR-002 and the decision log.

Root cause addressed during review

The shared event deserializer treats non-string timestamps as missing timestamps. Without validation at the conformance boundary, a number, Boolean, array, or object could silently change bounded-window behavior. The conformance parser now requires observed_at to be a string or null and also rejects Boolean manifest schema versions.

Measured impact

  • Frozen TW-001 holdout is unchanged: TP=6, FP=1, TN=36, FN=11.
  • Conformance reference is unchanged: 37 events, 12 sessions, 2 rules, 14 exact detections.
  • The bounded conformance rule owns 1 positive and 1 negative fixture and makes no detection-effectiveness or performance claim.
  • No provider SDK, query renderer, infrastructure, credentials, ingestion, alerting, cost, or latency claim is introduced.

Developer impact

uv run tripwire verify is now the single credential-free gate for the measured result and the provider-neutral sequence contract. A future provider may be called conformant for a rule only after an executed result reproduces the committed rule hash, session IDs, matched ordinals, and tool names.

Verification

  • Clean clone: uv sync --frozen
  • Clean clone: uv run tripwire verify
  • uv run pytest (37 passed)
  • uv run ruff check .
  • uv run mypy
  • uv build
  • git diff --check
  • Public-content scan: no credentials, personal paths, email addresses, or known trace PII

Risks and limitations

  • No provider backend has been rendered, executed, or shown conformant.
  • Historic AgentDojo events still do not measure elapsed-time behavior; the timestamp cases are synthetic and separate.
  • The new bounded rule is test-only and must not be presented as an additional Tripwire detector.
  • The first real provider consumer should define the mapping and rendering API. This PR intentionally avoids a placeholder backend abstraction.

Review guidance

Please focus on whether the fixed suite captures the local engine semantics a provider must reproduce and whether the test-only rule remains clearly separated from TW-001's measured claims.

@jonathanjasare

Copy link
Copy Markdown
Owner Author

Verification recorded for commit 3b20f2a:

  • Fresh local clone on Python 3.11.15: uv sync --frozen
  • Fresh clone: uv run tripwire verify reproduced TP=6 FP=1 TN=36 FN=11 and sequence-v1: rules=2 sessions=12 detections=14
  • uv run pytest: 35 passed
  • Ruff, mypy, package build, and git diff --check: passed

Review context: rules/TW-001.yaml, the AgentDojo corpus manifest/split, and results/agentdojo-banking-v1.json are unchanged. The 300-second rule is test-only and lives outside the published rule directory. No provider implementation or provider status is claimed in this PR.

@jonathanjasare jonathanjasare changed the title feat: add provider-neutral M3 conformance contract feat: add provider-neutral conformance contract Aug 12, 2026
@jonathanjasare

Copy link
Copy Markdown
Owner Author

Merge gate passed for commit 1551868.

  • Review finding resolved: non-string observed_at values and Boolean manifest schema versions now fail closed.
  • Fresh clone on Python 3.11.15: uv sync --frozen and uv run tripwire verify passed without credentials.
  • Frozen TW-001 result remains TP=6, FP=1, TN=36, FN=11.
  • Provider-neutral result remains 37 events, 12 sessions, 2 rules, 14 detections.
  • Full suite: 37 tests passed; Ruff, mypy, build, and git diff --check passed.
  • Public-content scan found no credentials, personal paths, email addresses, or known trace PII.
  • GitHub Actions verify run #21 completed successfully.

Final review found no remaining merge blocker.

@jonathanjasare
jonathanjasare marked this pull request as ready for review August 12, 2026 17:17
@jonathanjasare
jonathanjasare merged commit 89709e3 into master Aug 12, 2026
2 checks passed
@jonathanjasare
jonathanjasare deleted the feature/m3-conformance-contract branch August 12, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant