Skip to content

feat: add relay conform --manifest (relay-conform/1), spec §17.2, Requirement 15 - #180

Merged
SoundMatt merged 2 commits into
mainfrom
feat/relay-139-conformance-manifest
Aug 21, 2026
Merged

feat: add relay conform --manifest (relay-conform/1), spec §17.2, Requirement 15#180
SoundMatt merged 2 commits into
mainfrom
feat/relay-139-conformance-manifest

Conversation

@SoundMatt

Copy link
Copy Markdown
Owner

Summary

Implements issue #139 (NEW-SPEC-1): turns relay conform's self-reported conformance into a machine-checkable, committable artifact instead of only an on-demand CLI report.

What's added

  • New §17 Requirement 15 — Conformance manifest. The implementation MUST commit a relay-conform/1 manifest (§17.2), generated by relay conform --manifest against its own built binary; CI MUST regenerate the manifest on every change and fail the build on any diff from the committed copy, or on any requirement entry whose status is FAIL.
  • New §17.2 defines the manifest JSON shape (kind, manifest_version, tool, binary_version, spec_version, git_sha, capabilities_sha256, requirements[], overall) and a 4-value status vocabulary — PASS/FAIL/SHAPE_ONLY/NOT_OBSERVABLE — deliberately distinct from relay conform's existing PASS/WARN/FAIL finding severities. Core rule: a manifest MUST NOT report PASS for a requirement relay conform cannot actually observe.
  • New optional commit field on the §12.1 version document (backward-compatible — not in the schema's required array) so an implementation can populate the manifest's git_sha.
  • New §20.1 CI gate: regenerate + diff the manifest, fail on drift or any FAIL entry.
  • New embedded schema: spec/schemas/relay-conform-manifest.json.
  • Reference implementation: --manifest flag + buildManifest() in cmd/relay/conform.go. Only Requirements 1, 6, 7, 12 get real PASS/FAIL/SHAPE_ONLY statuses, mirroring §17's existing narrative of what relay conform can observe black-box. Everything else — including Requirement 15's own manifest-generation entry — is honestly NOT_OBSERVABLE, per this PR's own new rule (no self-congratulatory PASS).
  • New requirement REQ-RELAY-095 in .fusa-reqs.json — a fresh ID rather than reusing REQ-RELAY-052, since --manifest is materially new behavior (new output format, new status vocabulary, new artifact type), not the base findings-report path.

Version

SpecVersion bumped 2.22.3 (MINOR), matching the established precedent from Requirements 13 (v2.1) and 14 (v2.2): a new numbered §17 requirement is MINOR-worthy even where relay conform cannot fully observe it.

Verification

  • 7 new tests in cmd/relay/conform_test.go: manifest schema validity (against the new embedded schema), capabilities_sha256 independently recomputed and compared, all-15-requirements-present with correct NOT_OBSERVABLE defaults, FAIL propagation to overall, and CLI-level exit codes for both --manifest success and failure paths.
  • go build ./... && go vet ./... && go test ./... — all packages green.
  • gofusa check (pinned v0.48.0, matching CI): 0 errors.
  • gofusa trace -req-coverage 100: 100% (95/95 requirements traced).
  • Manually verified capabilities_sha256 byte-for-byte via independent sha256sum comparison against a locally built binary.
  • Fixed a stray "§17.13" → "§17.2" citation left over from drafting.

Closes #139.

SoundMatt and others added 2 commits August 20, 2026 20:09
…uirement 15

- New §17 Requirement 15: implementations MUST commit a relay-conform/1
  manifest generated by `relay conform --manifest`; CI MUST regenerate
  and fail the build on any diff or any FAIL requirement entry.
- New §17.2 defines the manifest JSON shape and a 4-value status
  vocabulary (PASS/FAIL/SHAPE_ONLY/NOT_OBSERVABLE), deliberately distinct
  from relay conform's existing PASS/WARN/FAIL finding severities: a
  manifest MUST NOT report PASS for a requirement relay conform cannot
  actually observe.
- New optional `commit` field on the §12.1 version document (backward
  compatible) so a manifest can populate its own git_sha.
- New §20.1 CI gate for manifest regeneration/diffing.
- New embedded schema spec/schemas/relay-conform-manifest.json.
- Reference implementation: --manifest flag + buildManifest() in
  cmd/relay/conform.go. Only Requirements 1, 6, 7, 12 get real
  PASS/FAIL/SHAPE_ONLY statuses (mirroring what relay conform can
  actually observe black-box); everything else, including Requirement
  15's own entry, is honestly NOT_OBSERVABLE.
- New REQ-RELAY-095 (not a reuse of REQ-RELAY-052 — this is materially
  new behavior: new output format, new status vocabulary, new artifact).
  7 new tests cover schema validity, capabilities_sha256 correctness,
  NOT_OBSERVABLE defaults, FAIL propagation, and CLI exit codes.
- SpecVersion bumped 2.2 -> 2.3 (MINOR, matching the Requirement 13/14
  precedent: a new numbered §17 requirement is MINOR-worthy even where
  relay conform cannot fully observe it).

Closes #139.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[NEW-SPEC-1] Machine-checkable conformance manifest (conformance.json) to turn self-reported conformance into an enforced artifact

1 participant