feat: persisted, regression-gated interop matrix (§17 Requirement 18) - #186
Merged
Conversation
relay interop's --format json output is now a versioned relay-interop-matrix/1 artifact (kind, matrix_version, an explicit participants[] list, the equivalence matrix, result) instead of an ephemeral pass/fail whose result vanished after the CI run. New --baseline FILE flag: compares a fresh run against a previously- committed matrix and reports a regressions[] list -- one entry per (vector, participant) cell that was EQUIVALENT in the baseline but isn't now. A cell that was never EQUIVALENT to begin with is not itself a regression, matching the issue's precise wording (distinct from the existing any-mismatch-fails gate, which already covers that case independently). Deliberately NOT pairwise (N^2): every participant is compared only against the shared in-process reference. Byte-equality is transitive, so "A equivalent to reference" and "B equivalent to reference" jointly establish "A equivalent to B" already -- documented explicitly in §17 and §20.1 rather than silently narrowed from the issue's literal "every other sibling" wording. New §17 Requirement 18, new spec/schemas/relay-interop-matrix.json, new REQ-RELAY-101. §20.1 item 3 extended; also fixes a real citation bug found along the way (§20.4's "§20.1.2" -- §20.1 is a flat list, not subdivided). 9 new/changed interop tests, including a mutation test on the regression detector's core logic -- caught immediately by a pure unit test; the first integration-level attempt was fooled twice (the reject-path vectors' by-design leniency, then a substring collision between the test binary's name and the ever-equivalent 'relay (reference)' row) before correctly discriminating the mutation. SpecVersion 2.7 -> 2.8 (MINOR, new §17 requirement). Closes #145 Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
…ring branches CI's 90% coverage gate caught 3 genuinely-uncovered branches from the previous commit: malformed --baseline JSON (distinct from the file-not-found path, which was already covered), and the markdown/text renderers' new Regressions sections (only the JSON format's doc.Regressions content was exercised before). 3 new tests: TestInteropBaselineInvalidJSON, TestRenderInteropRegressions (both text and markdown). Total coverage 89.9% -> 90.3%. Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #145 (NEW-SPEC-7). Extends §20.1's behavioural-conformance gate
and the
relay interopcommand so cross-language equivalence is apersisted, diffable, regression-gated artifact instead of a one-shot
CI check whose result vanishes after the run.
Background
§20.1 item 3's existing gate checks EQUIVALENT-vs-reference every run,
but the result is never committed anywhere — a cross-language
equivalence regression has no artifact trail, only whatever the CI log
happened to say at the time.
What's added
relay interop's JSON output is now a versioned artifact:kind: "relay-interop-matrix",matrix_version: "relay-interop-matrix/1", and an explicitparticipants[]list —so the set of compared implementations is itself part of the
diffable artifact, not just the per-cell verdicts.
--baseline FILEflag: compares a fresh run against apreviously-committed matrix and reports
regressions[]— one entryper (vector, participant) cell that was
EQUIVALENTin the baselinebut isn't now. A cell that was never
EQUIVALENTto begin with(e.g. an implementation still catching up to a new golden vector) is
not reported as a regression — matches the issue's precise
wording, distinct from the existing any-mismatch-fails gate.
published sibling it can invoke) and MUST regenerate +
--baselineit on every change, failing the build on any regression.
spec/schemas/relay-interop-matrix.json, newREQ-RELAY-101.Deliberately not pairwise (N²)
The issue's literal wording ("compare against every other sibling
implementation, not only the reference") would mean diffing every
binary's
convertoutput against every other binary's directly. Notbuilt: byte-equality is transitive, so "A equivalent to the shared
reference" and "B equivalent to the shared reference" already jointly
establish "A equivalent to B" — the existing O(N) hub-spoke comparison
carries the same information an O(N²) pairwise matrix would, without
the redundant
convertinvocations. Documented explicitly in §17 and§20.1 rather than silently narrowed.
Bug found along the way
§20.4 cited
§20.1.2for a specific CI-gate item, but §20.1 is a flatnumbered list, not subdivided into
.1/.2/.3subsections — thesame class of citation drift the issue itself flags for the audit's
stale
§20.1.3. Corrected to "§20.1 item 2".Version
SpecVersion2.7 → 2.8 (MINOR — new §17 requirement).Verification
go build/vet/testclean,gofmt -l .clean,golangci-lint run0issues
gofusa check0 errors/PASS,gofusa trace -req-coverage 100100%/100% (101/101 requirements traced)
relay interop --format jsonagainst itself, fed the output back as
--baseline— cleanround-trip, no false regressions;
relay conform --manifestreports18 total requirements with Requirement 18 correctly
NOT_OBSERVABLE/implementation's own CIregression detector's core comparison logic — caught immediately by
a pure unit test on
interopRegressions; the firstintegration-level attempt was fooled twice before correctly
discriminating the mutation (the reject-path vectors' by-design
leniency toward a broken
convert, then a substring collisionbetween the built test binary's name and the ever-equivalent
relay (reference)row) — both false-pass paths fixed and reconfirmed.