Skip to content

feat: persisted, regression-gated interop matrix (§17 Requirement 18) - #186

Merged
SoundMatt merged 2 commits into
mainfrom
feat/relay-145-interop-matrix
Aug 21, 2026
Merged

feat: persisted, regression-gated interop matrix (§17 Requirement 18)#186
SoundMatt merged 2 commits into
mainfrom
feat/relay-145-interop-matrix

Conversation

@SoundMatt

Copy link
Copy Markdown
Owner

Summary

Closes #145 (NEW-SPEC-7). Extends §20.1's behavioural-conformance gate
and the relay interop command so cross-language equivalence is a
persisted, diffable, regression-gated artifact instead of a one-shot
CI check whose result vanishes after the run.

Background

§20.1 item 3's existing gate checks EQUIVALENT-vs-reference every run,
but the result is never committed anywhere — a cross-language
equivalence regression has no artifact trail, only whatever the CI log
happened to say at the time.

What's added

  • relay interop's JSON output is now a versioned artifact:
    kind: "relay-interop-matrix", matrix_version: "relay-interop-matrix/1", and an explicit participants[] list —
    so the set of compared implementations is itself part of the
    diffable artifact, not just the per-cell verdicts.
  • New --baseline FILE flag: compares a fresh run against a
    previously-committed matrix and reports regressions[] — one entry
    per (vector, participant) cell that was EQUIVALENT in the baseline
    but isn't now. A cell that was never EQUIVALENT to begin with
    (e.g. an implementation still catching up to a new golden vector) is
    not reported as a regression — matches the issue's precise
    wording, distinct from the existing any-mismatch-fails gate.
  • New §17 Requirement 18: CI MUST commit the matrix (covering every
    published sibling it can invoke) and MUST regenerate + --baseline
    it on every change, failing the build on any regression.
  • New spec/schemas/relay-interop-matrix.json, new REQ-RELAY-101.

Deliberately not pairwise (N²)

The issue's literal wording ("compare against every other sibling
implementation, not only the reference") would mean diffing every
binary's convert output against every other binary's directly. Not
built: byte-equality is transitive, so "A equivalent to the shared
reference" and "B equivalent to the shared reference" already jointly
establish "A equivalent to B" — the existing O(N) hub-spoke comparison
carries the same information an O(N²) pairwise matrix would, without
the redundant convert invocations. Documented explicitly in §17 and
§20.1 rather than silently narrowed.

Bug found along the way

§20.4 cited §20.1.2 for a specific CI-gate item, but §20.1 is a flat
numbered list, not subdivided into .1/.2/.3 subsections — the
same class of citation drift the issue itself flags for the audit's
stale §20.1.3. Corrected to "§20.1 item 2".

Version

SpecVersion 2.7 → 2.8 (MINOR — new §17 requirement).

Verification

  • go build/vet/test clean, gofmt -l . clean, golangci-lint run 0
    issues
  • gofusa check 0 errors/PASS, gofusa trace -req-coverage 100
    100%/100% (101/101 requirements traced)
  • Manual end-to-end: built binary, ran relay interop --format json
    against itself, fed the output back as --baseline — clean
    round-trip, no false regressions; relay conform --manifest reports
    18 total requirements with Requirement 18 correctly
    NOT_OBSERVABLE/implementation's own CI
  • 9 new/changed interop tests, including a real mutation test on the
    regression detector's core comparison logic — caught immediately by
    a pure unit test on interopRegressions; the first
    integration-level attempt was fooled twice before correctly
    discriminating the mutation (the reject-path vectors' by-design
    leniency toward a broken convert, then a substring collision
    between the built test binary's name and the ever-equivalent relay (reference) row) — both false-pass paths fixed and reconfirmed.

relay interop's --format json output is now a versioned
relay-interop-matrix/1 artifact (kind, matrix_version, an explicit
participants[] list, the equivalence matrix, result) instead of an
ephemeral pass/fail whose result vanished after the CI run.

New --baseline FILE flag: compares a fresh run against a previously-
committed matrix and reports a regressions[] list -- one entry per
(vector, participant) cell that was EQUIVALENT in the baseline but
isn't now. A cell that was never EQUIVALENT to begin with is not
itself a regression, matching the issue's precise wording (distinct
from the existing any-mismatch-fails gate, which already covers that
case independently).

Deliberately NOT pairwise (N^2): every participant is compared only
against the shared in-process reference. Byte-equality is transitive,
so "A equivalent to reference" and "B equivalent to reference" jointly
establish "A equivalent to B" already -- documented explicitly in §17
and §20.1 rather than silently narrowed from the issue's literal
"every other sibling" wording.

New §17 Requirement 18, new spec/schemas/relay-interop-matrix.json,
new REQ-RELAY-101. §20.1 item 3 extended; also fixes a real citation
bug found along the way (§20.4's "§20.1.2" -- §20.1 is a flat list,
not subdivided).

9 new/changed interop tests, including a mutation test on the
regression detector's core logic -- caught immediately by a pure unit
test; the first integration-level attempt was fooled twice (the
reject-path vectors' by-design leniency, then a substring collision
between the test binary's name and the ever-equivalent 'relay
(reference)' row) before correctly discriminating the mutation.

SpecVersion 2.7 -> 2.8 (MINOR, new §17 requirement).

Closes #145

Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
…ring branches

CI's 90% coverage gate caught 3 genuinely-uncovered branches from the
previous commit: malformed --baseline JSON (distinct from the
file-not-found path, which was already covered), and the markdown/text
renderers' new Regressions sections (only the JSON format's
doc.Regressions content was exercised before).

3 new tests: TestInteropBaselineInvalidJSON, TestRenderInteropRegressions
(both text and markdown). Total coverage 89.9% -> 90.3%.

Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
@SoundMatt
SoundMatt merged commit ba434c9 into main Aug 21, 2026
8 checks passed
@SoundMatt
SoundMatt deleted the feat/relay-145-interop-matrix branch August 21, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[NEW-SPEC-7] Require a persisted, CI-committed cross-implementation interop matrix (not just a one-shot pass/fail)

1 participant