Skip to content

feat: evidence bundles for public conformance claims and a release process - #14

Merged
requie merged 2 commits into
mainfrom
feat/release-process
Sep 30, 2026
Merged

requie merged 2 commits into
mainfrom
feat/release-process

Conversation

@requie

@requie requie commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two commits, both about making CAAP results publishable and trackable.

Evidence bundles for public conformance claims (5fac026)

  • caap attest create packages a graded agent-native session (--session) or an observed caap run report (--report) into caap-evidence.zip. Its submission.json names the assurance tier with its fixed labels and claim boundary, the taxonomy and benchmark versions taken from the evidence, the subject, the scorecard and integrity-layer summaries, every evidence file with its SHA-256 and size, a badge text, and its own canonical hash. The tier is derived from the evidence, never chosen by the submitter. Bundles are unsigned; the labels say so.
  • caap attest verify is the reference verifier a registry or third party runs. It recomputes every hash, rejects undeclared files and altered claim text, re-grades an assessment from the bundled cases and responses and compares the scorecard, layers, and per-case states, recomputes an observed scorecard from the bundled results, and reports every check as pass, fail, or warn (--json for machines). A differing taxonomy version is a warning so a registry can mark entries stale rather than reject them.
  • New evidence-bundle schema, bundled with the package. Observed reports now record the taxonomy and benchmark versions, which move into a shared versions module.
  • docs/EVIDENCE_SUBMISSION.md specifies the bundle layout, every field, every check, the claim rule, and what the website's submission form, server-side acceptance, registry entry, badge, and policy text need. CONFORMANCE.md and the README point to it.

Release process (4486925)

  • Releases are git tags vX.Y.Z on main. The release workflow runs on a tag push: it checks that the tag matches the version in pyproject.toml, versions.py, and the README, runs lint, validation, tests, compileall, and the generated-files check, builds the wheel and sdist, and publishes a GitHub release with the distribution, the taxonomy JSON and YAML, a zip of the schemas, SHA256SUMS, and notes taken from the changelog section for that version. A pre-release suffix marks the release as a pre-release. Nothing is published if any step fails. The workflow uses the runner's gh with the workflow token, so no new third-party action is introduced.
  • scripts/release.py provides bump (set the version everywhere and move the unreleased changelog entries into a dated section), check (readiness), and notes. Repository validation fails on version drift. caap --version reports the package and taxonomy versions from one source.
  • docs/RELEASING.md documents the two version numbers, the cutting procedure, and how a defective release is superseded. GOVERNANCE.md, CONTRIBUTING.md, the README status section, the Makefile, and the changelog are updated.

No tag is pushed by this PR; cutting v0.1.0 is a separate step after merge.

Pattern or implementation impact

No pattern, case, registry, or generated-file changes beyond the packaged copy of the new schema. Observed caap-report.json files gain two informational fields (taxonomy_version, caap_benchmark_version); the report schema does not restrict additional properties, so existing consumers are unaffected. The release workflow's permission changes from contents: read to contents: write, which publishing a release requires; it runs only on tag pushes.

Safety impact

  • Authorized targets only
  • Synthetic data and identities only
  • Mock tools, sinks, and actuators only
  • No destructive payload, real exfiltration, persistence, or approval bypass

Bundles package results that already exist; nothing here runs an evaluation. The verifier extracts bundle contents only into a temporary directory and re-grades with the existing grader. Verification cannot establish who produced the responses or that the evaluation was authorized, which is why the tiers are named self-assessed and observed and the independent tier is reserved.

Validation

  • Generated files are current (generator run twice; packaged schema matches)
  • Repository validation passes, including the new version-consistency check
  • Unit tests pass: 92 with jsonschema, 92 with 8 skipped without it. New tests cover both bundle tiers, schema conformance, tampered responses, edited submissions, undeclared and missing files, ungraded or stale sessions, malformed bundles, a vulnerable run bundling honestly, the CLI round trip, and the release script's bump, check, and notes on a copy of the real files
  • Secure mock passes; an observed bundle from a safe mock run verifies with 12 checks, an assessment bundle with 13
  • Vulnerable synthetic behavior fails when an executable case changes (no executable case changes; a vulnerable run bundles with score 0.0 and still verifies as internally consistent)
  • Wheel installed outside the checkout: attest create and verify work and the schema is packaged
  • Dry run of release.py bump 0.2.0, check, and notes on a scratch copy of the tree
  • CI creates and verifies a bundle for each tier
  • Commits include DCO sign-off

Give operators a single verifiable artifact to publish a CAAP result,
and give the registry a reference verifier, so public conformance
claims are machine-checkable rather than self-described.

- `caap attest create` packages a graded agent-native session or an
  observed `caap run` report into caap-evidence.zip. Its submission.json
  names the assurance tier with its fixed labels and claim boundary,
  the taxonomy and benchmark versions taken from the evidence, the
  subject, the scorecard and integrity-layer summaries, every evidence
  file with its SHA-256 and size, a badge text, and its own canonical
  hash. The tier is derived from the evidence, never chosen.
- `caap attest verify` recomputes every hash, rejects undeclared files
  and altered claim text, re-grades an assessment from the bundled
  cases and responses and compares scorecard, layers, and per-case
  states, recomputes an observed scorecard from the bundled results,
  and reports each check as pass, fail, or warn (`--json` for a
  registry). A differing taxonomy version is a warning so a registry
  can mark entries stale rather than reject them.
- New `evidence-bundle` schema, bundled with the package. Observed
  reports now record the taxonomy and benchmark versions, which move
  to a shared `versions` module.
- docs/EVIDENCE_SUBMISSION.md specifies the bundle layout, the fields,
  every verification check, the claim rule, and what the website's
  submission form, server-side acceptance, registry entry, badge, and
  policy text need. CONFORMANCE.md and the README point to it.
- Tests cover both tiers: content, schema conformance, tampered
  responses, edited submissions, undeclared and missing files, ungraded
  or stale sessions, malformed bundles, a vulnerable run bundling
  honestly, and the CLI round trip. CI creates and verifies a bundle
  for each tier.

Signed-off-by: requie <tarique.smith@gmail.com>
Track releases as git tags vX.Y.Z on main, published on the GitHub
Releases page by the release workflow.

- The release workflow now runs on a tag push: it checks that the tag
  matches the version in pyproject.toml, versions.py, and the README,
  runs lint, validation, tests, compileall, and the generated-files
  check, builds the wheel and sdist, and publishes a GitHub release
  with the distribution, the taxonomy JSON and YAML, a zip of the
  schemas, SHA256SUMS, and notes taken from the changelog section for
  that version. A version with a pre-release suffix is marked as a
  pre-release. Nothing is published if any step fails.
- scripts/release.py provides `bump` (set the version everywhere and
  move the unreleased changelog entries into a dated section), `check`
  (readiness, including that nothing is left under Unreleased), and
  `notes` (the release body naming the software and taxonomy versions).
- Repository validation fails on version drift between pyproject.toml,
  versions.py, and the README. `caap --version` reports the package and
  taxonomy versions from versions.py instead of a hard-coded string.
- docs/RELEASING.md documents the two version numbers, the cutting
  procedure, what the check enforces, and how a defective release is
  superseded. GOVERNANCE, CONTRIBUTING, the README status section, the
  Makefile, and the changelog are updated. Tests cover the script and
  a bump on a copy of the real files.

Signed-off-by: requie <tarique.smith@gmail.com>
@requie requie changed the title Add evidence bundle creation and verification for public CAAP claims feat: evidence bundles for public conformance claims and a release process Sep 30, 2026
@requie
requie merged commit 4a64fc7 into main Sep 30, 2026
8 checks passed
@requie
requie deleted the feat/release-process branch September 30, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant