Skip to content

feat: continuous example verification (§17 Requirement 19) - #187

Merged
SoundMatt merged 1 commit into
mainfrom
feat/relay-146-doctest-examples
Aug 21, 2026
Merged

feat: continuous example verification (§17 Requirement 19)#187
SoundMatt merged 1 commit into
mainfrom
feat/relay-146-doctest-examples

Conversation

@SoundMatt

Copy link
Copy Markdown
Owner

Summary

Closes #146 (NEW-SPEC-8). Adds a doctest mechanism: every normative
JSON example in the spec document is machine-extracted and validated
in CI, and README ## RELAY conformance sections must have every
literal executable command example continuously proven too — not just
true when someone last checked it by hand.

Background

§13.8 already requires a README's CLI round-trip to work; §19.5
already requires this document's own version literals to match
spec/version.json. Neither stopped a content example from silently
rotting — exactly the failure mode THEME-J and REL-SPEC-10 documented.

What's added

  • New §20.7 and new §17 Requirement 19.
  • <!-- doctest:KIND --> marker convention, applied to all 7 of
    this document's existing normative JSON examples (§12.1, §12.2 ×2,
    §12.3, §15.8, §17.2, §20.6). KIND selects the validator — the
    same one relay conform applies to a real binary's output, not a
    separate check invented for the doctest.
  • README extension: every literal, executable command in a
    README's ## RELAY conformance section must be machine-extracted
    and executed in CI. Network-fetching commands (go get/go install) and commands needing a published container image are
    explicitly exempt.
  • Reference implementation: extractDoctestExamples
    (cmd/relay/doctest.go), TestSpecDoctestExamplesValidate
    (cmd/relay/doctest_test.go). New REQ-RELAY-102.

Real bugs found and fixed while building this

  • Three of this document's own examples used truncated placeholder
    hex
    ("816b402d...") instead of a real 64-char SHA-256. Replaced
    with genuine values — two are the actual hashes of the referenced
    golden vectors.
  • RELAY's own JSON Schema validator never implemented the pattern
    keyword at all
    — silently ignored per its own documented keyword
    list. Every schema constraining a SHA-256 field with
    "^[0-9a-f]{64}$" was unenforced: relay conform would have
    accepted a malformed hash in any capabilities/manifest/attestation/
    vectors-manifest document without a single finding. Found because
    the truncated placeholders above initially validated cleanly — a
    false pass this mechanism exists to prevent. Fixed with a
    regexp-based pattern check; new TestSchemaPattern; mutation-
    tested.
  • README.md's own "Current" version line was six MINOR releases
    stale
    (v2.0 while spec/version.json was at v2.8) — the exact
    class of drift §19.5 already guards against, just not in README.md.
    Fixed, and TestReadmeVersionMatchesVersionJSON extends the §19.5
    mechanism so it can't silently recur.

Version

SpecVersion 2.8 → 2.9 (MINOR — new §17 requirement).

Verification

  • go build/vet/test clean, gofmt -l . clean, golangci-lint run 0
    issues
  • Coverage 90.4% (≥90% gate)
  • gofusa check 0 errors/PASS, gofusa trace -req-coverage 100
    100%/100% (102/102 requirements traced)
  • Manual end-to-end: built binary, relay conform --manifest reports
    19 total requirements with Requirement 19 correctly
    NOT_OBSERVABLE/implementation's own CI; README's version line
    confirmed matching
  • Mutation-tested the doctest mechanism itself: corrupted a hash
    pattern and a required commands entry in two separate examples,
    confirmed both caught with a precise violation message, restored and
    reconfirmed green. Separately mutation-tested the new pattern
    keyword support in validateSchema.

New §20.7 "Continuous example verification" and §17 Requirement 19:
every JSON example this document marks normative (<!-- doctest:KIND
--> marker convention) and every literal, executable command in an
implementation's README `## RELAY conformance` section MUST be
machine-extracted and checked in CI -- not merely true when someone
last looked. Extends §13.8's existing "the demo must work" and §19.5's
existing "version literals must match" to cover example *content*
more broadly, closing the same class of gap THEME-J and REL-SPEC-10
independently documented.

<!-- doctest:KIND --> applied to all 7 of this document's existing
normative JSON examples (§12.1 version, §12.2 capabilities x2, §12.3
status, §15.8 vectors-manifest, §17.2 manifest, §20.6 attestation).
Each is run through the exact same validator relay conform applies to
a real binary's output (cmd/relay/doctest.go's extractDoctestExamples
+ cmd/relay/doctest_test.go's TestSpecDoctestExamplesValidate).

Three real bugs found and fixed while building this:

- Three of this document's own examples used truncated placeholder
  hex ("816b402d...") instead of a real 64-char SHA-256. Replaced with
  genuine values -- two are the actual hashes of the referenced golden
  vectors.
- RELAY's own JSON Schema validator never implemented the `pattern`
  keyword at all (silently ignored per its own documented keyword
  list) -- every schema constraining a SHA-256 field with
  "^[0-9a-f]{64}$" was unenforced. relay conform would have accepted a
  malformed hash in any capabilities/manifest/attestation/
  vectors-manifest document without a single finding. Found because
  the truncated placeholders above initially validated cleanly -- a
  false pass this doctest mechanism exists to prevent. Fixed with a
  regexp-based pattern check in validateSchema; mutation-tested.
- README.md's own "Current" version line was six MINOR releases stale
  (v2.0 while spec/version.json was at v2.8) -- the exact class of
  drift §19.5 already guards against in relay-spec.md, just not in
  README.md. Fixed, and TestReadmeVersionMatchesVersionJSON extends
  the §19.5 mechanism so it can't silently recur (README.md now
  embedded evidence).

New REQ-RELAY-102. Deliberately out of scope: network-fetching
commands (go get/go install) and commands needing a published
container image are exempt from README example execution -- a
hermetic runner for those is a reasonable future addition, not
mandated here.

Mutation-tested the doctest mechanism itself: corrupted a hash pattern
and a required commands entry in two separate examples, confirmed both
caught with a precise violation message, restored and reconfirmed
green.

SpecVersion 2.8 -> 2.9 (MINOR, new §17 requirement).

Closes #146

Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
@SoundMatt
SoundMatt merged commit 6aefd24 into main Aug 21, 2026
8 checks passed
@SoundMatt
SoundMatt deleted the feat/relay-146-doctest-examples branch August 21, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[NEW-SPEC-8] Require spec and README code/CLI examples to be executable in CI (doctest / round-trip harness)

1 participant