Skip to content

feat: enforce the published JSON Schemas in the runner and CI - #7

Merged
requie merged 1 commit into
mainfrom
feat/json-schema-validation
Sep 30, 2026
Merged

requie merged 1 commit into
mainfrom
feat/json-schema-validation

Conversation

@requie

@requie requie commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The four schemas under schemas/ were published but never applied. The runner used a hand-rolled required-field check that required three fewer fields than the test-case schema (pattern_version, target_profile, expected_secure_behavior), adapter payloads were never validated, and the registry was never checked against the taxonomy schema. A malformed adapter payload could also raise an uncaught KeyError from Event.from_dict.

The schemas are now enforced. Full validation uses the optional jsonschema package through a new schema extra that CI installs. Without it, a structural fallback reads its required fields and safety keys from the schema itself, so the two can no longer diverge.

Pattern or implementation impact

Technical review. No taxonomy, case, or schema content changes; all 200 cases and the registry already validated under the published schemas, so this adds enforcement only.

  • src/caap_benchmark/schemas.py: new. Loads the bundled schemas, reports whether jsonschema is available, and returns schema violations with JSON paths.
  • src/caap_benchmark/loaders.py: validate_test_case uses the test-case schema; structural_test_case_errors is the dependency-free fallback driven by the schema's required list, pattern_id pattern, and safety keys.
  • src/caap_benchmark/adapters/http.py: _normalize (shared with the command adapter) validates every payload against the adapter-response schema and returns test_error for a malformed one instead of raising.
  • src/caap_benchmark/cli.py: caap validate reports which validator ran.
  • scripts/generate_catalog.py: copies the schemas into the package data. scripts/validate_repository.py: checks those copies match, and validates the registry and all 200 cases against the schemas when jsonschema is available (prints a note otherwise).
  • pyproject.toml: schema extra; dev extra now includes PyYAML and jsonschema; schemas added to package data.
  • .github/workflows/ci.yml: installs [yaml,schema]. .github/workflows/release.yml: previously installed only build and then ran the unit tests without the package importable, so it would have failed on the first tag; it now installs [dev] and runs ruff before the checks.
  • tests/test_schemas.py: new, 13 tests. Docs: docs/CONFORMANCE.md (schema enforcement section), README.md (extras), CHANGELOG.md.

Safety impact

Not applicable. No fixture, capability, sink, network boundary, persistence boundary, or public procedure changes. The change makes the runner stricter about what it accepts: a malformed adapter payload yields test_error, never pass.

  • Authorized targets only
  • Synthetic data and identities only
  • Mock tools, sinks, and actuators only
  • No destructive payload, real exfiltration, persistence, or approval bypass

Validation

  • Generated files are current (regenerated twice; no drift; four schema files bundled)
  • Repository validation passes with full schema validation of the registry and 200 cases, and passes with the structural fallback when jsonschema is blocked
  • Unit tests pass: 41 with jsonschema installed, 37 plus 4 skipped with its import blocked
  • Secure mock passes (unchanged; the mock's response also validates against the adapter-response schema)
  • Vulnerable synthetic behavior fails when an executable case changes (no executable case changed)
  • caap validate on a case missing target_profile with an invalid severity rating reports both violations with their JSON paths
  • ruff check src tests scripts examples is clean
  • Commit includes DCO sign-off

The schemas under schemas/ were published but never applied: the
runner used a hand-rolled required-field check that required three
fewer fields than the test-case schema, adapter payloads were never
validated, and the registry was never checked against the taxonomy
schema. A malformed adapter payload could also raise an uncaught
KeyError from Event.from_dict.

Add caap_benchmark.schemas, which loads the schemas bundled with the
package and validates against them with the optional jsonschema
package (new schema extra, installed in CI). Without it the loader
applies a structural subset that reads its required-field list and
safety keys from the schema itself, so the two can no longer diverge.
The HTTP and command adapters validate every payload against the
adapter-response schema and return test_error for a malformed one.
Repository validation checks the registry and all 200 cases against
the schemas when jsonschema is available. caap validate reports which
validator ran. The generator copies the schemas into the package data
and validation checks those copies match.

The release workflow installed only build and then ran the unit tests
without the package importable; it now installs the dev extra and runs
ruff before the checks.

Tests cover every schema-required field through the fallback, the
three previously missing fields through validate_test_case, a
constraint only the full schema catches, the registry, a written
report, the mock adapter's response, and six malformed adapter
payloads becoming test_error. Verified with jsonschema installed (41
tests) and with its import blocked (37 tests, 4 skipped).

Signed-off-by: requie <tarique.smith@gmail.com>
@requie requie changed the title Enforce published JSON Schemas for test cases and adapter responses feat: enforce the published JSON Schemas in the runner and CI Sep 30, 2026
@requie
requie merged commit 7778a69 into main Sep 30, 2026
8 checks passed
@requie
requie deleted the feat/json-schema-validation branch September 30, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant