Repository navigation
feat: evidence bundles for public conformance claims and a release process - #14
Merged
Merged
Conversation
Give operators a single verifiable artifact to publish a CAAP result, and give the registry a reference verifier, so public conformance claims are machine-checkable rather than self-described. - `caap attest create` packages a graded agent-native session or an observed `caap run` report into caap-evidence.zip. Its submission.json names the assurance tier with its fixed labels and claim boundary, the taxonomy and benchmark versions taken from the evidence, the subject, the scorecard and integrity-layer summaries, every evidence file with its SHA-256 and size, a badge text, and its own canonical hash. The tier is derived from the evidence, never chosen. - `caap attest verify` recomputes every hash, rejects undeclared files and altered claim text, re-grades an assessment from the bundled cases and responses and compares scorecard, layers, and per-case states, recomputes an observed scorecard from the bundled results, and reports each check as pass, fail, or warn (`--json` for a registry). A differing taxonomy version is a warning so a registry can mark entries stale rather than reject them. - New `evidence-bundle` schema, bundled with the package. Observed reports now record the taxonomy and benchmark versions, which move to a shared `versions` module. - docs/EVIDENCE_SUBMISSION.md specifies the bundle layout, the fields, every verification check, the claim rule, and what the website's submission form, server-side acceptance, registry entry, badge, and policy text need. CONFORMANCE.md and the README point to it. - Tests cover both tiers: content, schema conformance, tampered responses, edited submissions, undeclared and missing files, ungraded or stale sessions, malformed bundles, a vulnerable run bundling honestly, and the CLI round trip. CI creates and verifies a bundle for each tier. Signed-off-by: requie <tarique.smith@gmail.com>
Track releases as git tags vX.Y.Z on main, published on the GitHub Releases page by the release workflow. - The release workflow now runs on a tag push: it checks that the tag matches the version in pyproject.toml, versions.py, and the README, runs lint, validation, tests, compileall, and the generated-files check, builds the wheel and sdist, and publishes a GitHub release with the distribution, the taxonomy JSON and YAML, a zip of the schemas, SHA256SUMS, and notes taken from the changelog section for that version. A version with a pre-release suffix is marked as a pre-release. Nothing is published if any step fails. - scripts/release.py provides `bump` (set the version everywhere and move the unreleased changelog entries into a dated section), `check` (readiness, including that nothing is left under Unreleased), and `notes` (the release body naming the software and taxonomy versions). - Repository validation fails on version drift between pyproject.toml, versions.py, and the README. `caap --version` reports the package and taxonomy versions from versions.py instead of a hard-coded string. - docs/RELEASING.md documents the two version numbers, the cutting procedure, what the check enforces, and how a defective release is superseded. GOVERNANCE, CONTRIBUTING, the README status section, the Makefile, and the changelog are updated. Tests cover the script and a bump on a copy of the real files. Signed-off-by: requie <tarique.smith@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two commits, both about making CAAP results publishable and trackable.
Evidence bundles for public conformance claims (
5fac026)caap attest createpackages a graded agent-native session (--session) or an observedcaap runreport (--report) intocaap-evidence.zip. Itssubmission.jsonnames the assurance tier with its fixed labels and claim boundary, the taxonomy and benchmark versions taken from the evidence, the subject, the scorecard and integrity-layer summaries, every evidence file with its SHA-256 and size, a badge text, and its own canonical hash. The tier is derived from the evidence, never chosen by the submitter. Bundles are unsigned; the labels say so.caap attest verifyis the reference verifier a registry or third party runs. It recomputes every hash, rejects undeclared files and altered claim text, re-grades an assessment from the bundled cases and responses and compares the scorecard, layers, and per-case states, recomputes an observed scorecard from the bundled results, and reports every check as pass, fail, or warn (--jsonfor machines). A differing taxonomy version is a warning so a registry can mark entries stale rather than reject them.evidence-bundleschema, bundled with the package. Observed reports now record the taxonomy and benchmark versions, which move into a sharedversionsmodule.docs/EVIDENCE_SUBMISSION.mdspecifies the bundle layout, every field, every check, the claim rule, and what the website's submission form, server-side acceptance, registry entry, badge, and policy text need.CONFORMANCE.mdand the README point to it.Release process (
4486925)vX.Y.Zonmain. The release workflow runs on a tag push: it checks that the tag matches the version inpyproject.toml,versions.py, and the README, runs lint, validation, tests, compileall, and the generated-files check, builds the wheel and sdist, and publishes a GitHub release with the distribution, the taxonomy JSON and YAML, a zip of the schemas,SHA256SUMS, and notes taken from the changelog section for that version. A pre-release suffix marks the release as a pre-release. Nothing is published if any step fails. The workflow uses the runner'sghwith the workflow token, so no new third-party action is introduced.scripts/release.pyprovidesbump(set the version everywhere and move the unreleased changelog entries into a dated section),check(readiness), andnotes. Repository validation fails on version drift.caap --versionreports the package and taxonomy versions from one source.docs/RELEASING.mddocuments the two version numbers, the cutting procedure, and how a defective release is superseded.GOVERNANCE.md,CONTRIBUTING.md, the README status section, the Makefile, and the changelog are updated.No tag is pushed by this PR; cutting
v0.1.0is a separate step after merge.Pattern or implementation impact
No pattern, case, registry, or generated-file changes beyond the packaged copy of the new schema. Observed
caap-report.jsonfiles gain two informational fields (taxonomy_version,caap_benchmark_version); the report schema does not restrict additional properties, so existing consumers are unaffected. The release workflow's permission changes fromcontents: readtocontents: write, which publishing a release requires; it runs only on tag pushes.Safety impact
Bundles package results that already exist; nothing here runs an evaluation. The verifier extracts bundle contents only into a temporary directory and re-grades with the existing grader. Verification cannot establish who produced the responses or that the evaluation was authorized, which is why the tiers are named self-assessed and observed and the independent tier is reserved.
Validation
jsonschema, 92 with 8 skipped without it. New tests cover both bundle tiers, schema conformance, tampered responses, edited submissions, undeclared and missing files, ungraded or stale sessions, malformed bundles, a vulnerable run bundling honestly, the CLI round trip, and the release script's bump, check, and notes on a copy of the real filesattest createandverifywork and the schema is packagedrelease.py bump 0.2.0,check, andnoteson a scratch copy of the tree