Repository navigation
Run the frozen requirement sets in conformance CI - #638
Merged
koic merged 1 commit intoOct 30, 2026
Merged
Conversation
`rake conformance:test` (and the Conformance Tests workflow) runs the conformance CLI's default active suite, which does not cover the frozen SEP-1730 requirement sets: the scenarios a spec revision required when it shipped, run at that revision's own wire version. The 2026-07-28 set (`server-stateless`, `caching`, the `input-required-result-*` family) therefore had no CI leg, so the Tier 1 evidence of 100% on scored scenarios per revision was only verifiable by hand. - `rake conformance:test` accepts `REQUIREMENTS=<revision>`, forwarded as `--requirements` to both legs. On the client leg it replaces the `--suite all` selection, mirroring the CLI's own rule that a requirement set already fixes which scenarios run; combining it with `SCENARIO` or `SPEC_VERSION` aborts for the same reason. - `--expected-failures` stays on both legs: under a requirement set the CLI judges the baseline against scored scenarios only, so not_scored entries (extensions, post-anchor additions) keep not affecting the exit code. - The workflow gains a `requirements-conformance` matrix job over 2025-11-25 and 2026-07-28. Both legs pass today, and the job carries the same `continue-on-error` as the active-suite job. Testing: test/conformance/client_runner_test.rb and test/conformance/server_runner_test.rb (command construction); `REQUIREMENTS=2025-11-25` and `REQUIREMENTS=2026-07-28 bundle exec rake conformance:test` both exit 0 with every scored scenario passing on both legs (the failures they report are unscored extension and pending scenarios); `REQUIREMENTS=... SCENARIO=ping` aborts with the exclusivity message.
atesgoral
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rake conformance:test(and the Conformance Tests workflow) runs the conformance CLI's default active suite, which does not cover the frozen SEP-1730 requirement sets: the scenarios a spec revision required when it shipped, run at that revision's own wire version. The 2026-07-28 set (server-stateless,caching, theinput-required-result-*family) therefore had no CI leg, so the Tier 1 evidence of 100% on scored scenarios per revision was only verifiable by hand.rake conformance:testacceptsREQUIREMENTS=<revision>, forwarded as--requirementsto both legs. On the client leg it replaces the--suite allselection, mirroring the CLI's own rule that a requirement set already fixes which scenarios run; combining it withSCENARIOorSPEC_VERSIONaborts for the same reason.--expected-failuresstays on both legs: under a requirement set the CLI judges the baseline against scored scenarios only, so not_scored entries (extensions, post-anchor additions) keep not affecting the exit code.requirements-conformancematrix job over 2025-11-25 and 2026-07-28. Both legs pass today, and the job carries the samecontinue-on-erroras the active-suite job.Testing: test/conformance/client_runner_test.rb and test/conformance/server_runner_test.rb (command construction);
REQUIREMENTS=2025-11-25andREQUIREMENTS=2026-07-28 bundle exec rake conformance:testboth exit 0 with every scored scenario passing on both legs (the failures they report are unscored extension and pending scenarios);REQUIREMENTS=... SCENARIO=pingaborts with the exclusivity message.Checklist