feat: continuous example verification (§17 Requirement 19) - #187
Merged
Conversation
New §20.7 "Continuous example verification" and §17 Requirement 19:
every JSON example this document marks normative (<!-- doctest:KIND
--> marker convention) and every literal, executable command in an
implementation's README `## RELAY conformance` section MUST be
machine-extracted and checked in CI -- not merely true when someone
last looked. Extends §13.8's existing "the demo must work" and §19.5's
existing "version literals must match" to cover example *content*
more broadly, closing the same class of gap THEME-J and REL-SPEC-10
independently documented.
<!-- doctest:KIND --> applied to all 7 of this document's existing
normative JSON examples (§12.1 version, §12.2 capabilities x2, §12.3
status, §15.8 vectors-manifest, §17.2 manifest, §20.6 attestation).
Each is run through the exact same validator relay conform applies to
a real binary's output (cmd/relay/doctest.go's extractDoctestExamples
+ cmd/relay/doctest_test.go's TestSpecDoctestExamplesValidate).
Three real bugs found and fixed while building this:
- Three of this document's own examples used truncated placeholder
hex ("816b402d...") instead of a real 64-char SHA-256. Replaced with
genuine values -- two are the actual hashes of the referenced golden
vectors.
- RELAY's own JSON Schema validator never implemented the `pattern`
keyword at all (silently ignored per its own documented keyword
list) -- every schema constraining a SHA-256 field with
"^[0-9a-f]{64}$" was unenforced. relay conform would have accepted a
malformed hash in any capabilities/manifest/attestation/
vectors-manifest document without a single finding. Found because
the truncated placeholders above initially validated cleanly -- a
false pass this doctest mechanism exists to prevent. Fixed with a
regexp-based pattern check in validateSchema; mutation-tested.
- README.md's own "Current" version line was six MINOR releases stale
(v2.0 while spec/version.json was at v2.8) -- the exact class of
drift §19.5 already guards against in relay-spec.md, just not in
README.md. Fixed, and TestReadmeVersionMatchesVersionJSON extends
the §19.5 mechanism so it can't silently recur (README.md now
embedded evidence).
New REQ-RELAY-102. Deliberately out of scope: network-fetching
commands (go get/go install) and commands needing a published
container image are exempt from README example execution -- a
hermetic runner for those is a reasonable future addition, not
mandated here.
Mutation-tested the doctest mechanism itself: corrupted a hash pattern
and a required commands entry in two separate examples, confirmed both
caught with a precise violation message, restored and reconfirmed
green.
SpecVersion 2.8 -> 2.9 (MINOR, new §17 requirement).
Closes #146
Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #146 (NEW-SPEC-8). Adds a doctest mechanism: every normative
JSON example in the spec document is machine-extracted and validated
in CI, and README
## RELAY conformancesections must have everyliteral executable command example continuously proven too — not just
true when someone last checked it by hand.
Background
§13.8 already requires a README's CLI round-trip to work; §19.5
already requires this document's own version literals to match
spec/version.json. Neither stopped a content example from silentlyrotting — exactly the failure mode THEME-J and REL-SPEC-10 documented.
What's added
<!-- doctest:KIND -->marker convention, applied to all 7 ofthis document's existing normative JSON examples (§12.1, §12.2 ×2,
§12.3, §15.8, §17.2, §20.6).
KINDselects the validator — thesame one
relay conformapplies to a real binary's output, not aseparate check invented for the doctest.
README's
## RELAY conformancesection must be machine-extractedand executed in CI. Network-fetching commands (
go get/go install) and commands needing a published container image areexplicitly exempt.
extractDoctestExamples(
cmd/relay/doctest.go),TestSpecDoctestExamplesValidate(
cmd/relay/doctest_test.go). NewREQ-RELAY-102.Real bugs found and fixed while building this
hex (
"816b402d...") instead of a real 64-char SHA-256. Replacedwith genuine values — two are the actual hashes of the referenced
golden vectors.
patternkeyword at all — silently ignored per its own documented keyword
list. Every schema constraining a SHA-256 field with
"^[0-9a-f]{64}$"was unenforced:relay conformwould haveaccepted a malformed hash in any capabilities/manifest/attestation/
vectors-manifest document without a single finding. Found because
the truncated placeholders above initially validated cleanly — a
false pass this mechanism exists to prevent. Fixed with a
regexp-basedpatterncheck; newTestSchemaPattern; mutation-tested.
stale (
v2.0whilespec/version.jsonwas atv2.8) — the exactclass of drift §19.5 already guards against, just not in README.md.
Fixed, and
TestReadmeVersionMatchesVersionJSONextends the §19.5mechanism so it can't silently recur.
Version
SpecVersion2.8 → 2.9 (MINOR — new §17 requirement).Verification
go build/vet/testclean,gofmt -l .clean,golangci-lint run0issues
gofusa check0 errors/PASS,gofusa trace -req-coverage 100100%/100% (102/102 requirements traced)
relay conform --manifestreports19 total requirements with Requirement 19 correctly
NOT_OBSERVABLE/implementation's own CI; README's version lineconfirmed matching
pattern and a required
commandsentry in two separate examples,confirmed both caught with a precise violation message, restored and
reconfirmed green. Separately mutation-tested the new
patternkeyword support in
validateSchema.