Skip to content

Commit 14a7dd9

Browse files
koickoic
committed
Run the frozen requirement sets in conformance CI
`rake conformance:test` (and the Conformance Tests workflow) runs the conformance CLI's default active suite, which does not cover the frozen SEP-1730 requirement sets: the scenarios a spec revision required when it shipped, run at that revision's own wire version. The 2026-07-28 set (`server-stateless`, `caching`, the `input-required-result-*` family) therefore had no CI leg, so the Tier 1 evidence of 100% on scored scenarios per revision was only verifiable by hand. - `rake conformance:test` accepts `REQUIREMENTS=<revision>`, forwarded as `--requirements` to both legs. On the client leg it replaces the `--suite all` selection, mirroring the CLI's own rule that a requirement set already fixes which scenarios run; combining it with `SCENARIO` or `SPEC_VERSION` aborts for the same reason. - `--expected-failures` stays on both legs: under a requirement set the CLI judges the baseline against scored scenarios only, so not_scored entries (extensions, post-anchor additions) keep not affecting the exit code. - The workflow gains a `requirements-conformance` matrix job over 2025-11-25 and 2026-07-28. Both legs pass today, and the job carries the same `continue-on-error` as the active-suite job. Testing: test/conformance/client_runner_test.rb and test/conformance/server_runner_test.rb (command construction); `REQUIREMENTS=2025-11-25` and `REQUIREMENTS=2026-07-28 bundle exec rake conformance:test` both exit 0 with every scored scenario passing on both legs (the failures they report are unscored extension and pending scenarios); `REQUIREMENTS=... SCENARIO=ping` aborts with the exclusivity message.
1 parent 1126766 commit 14a7dd9

7 files changed

Lines changed: 171 additions & 10 deletions

File tree

‎.github/workflows/conformance.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,26 @@ jobs:
2727
with:
2828
node-version: '24' # Specify the latest Node.js version.
2929
- run: bundle exec rake conformance:test
30+
31+
# Frozen requirement sets for SEP-1730 tier scoring: each spec revision's scenarios run at
32+
# that revision's own wire version, unlike the moving active suite above.
33+
requirements-conformance:
34+
runs-on: ubuntu-latest
35+
continue-on-error: true
36+
name: requirements-conformance (${{ matrix.requirements }})
37+
strategy:
38+
fail-fast: false
39+
matrix:
40+
requirements: ['2025-11-25', '2026-07-28']
41+
steps:
42+
- uses: actions/checkout@v7
43+
- uses: ruby/setup-ruby@v1
44+
with:
45+
ruby-version: '4.0' # Specify the latest supported Ruby version.
46+
bundler-cache: true
47+
- uses: actions/setup-node@v7
48+
with:
49+
node-version: '24' # Specify the latest Node.js version.
50+
- run: bundle exec rake conformance:test
51+
env:
52+
REQUIREMENTS: ${{ matrix.requirements }}

‎Rakefile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,22 @@ RuboCop::RakeTask.new
1717
task default: [:rubocop, :test, "conformance:test"]
1818

1919
namespace :conformance do
20-
desc "Run MCP conformance tests (PORT, SCENARIO, SPEC_VERSION, VERBOSE)"
20+
desc "Run MCP conformance tests (PORT, SCENARIO, SPEC_VERSION, REQUIREMENTS, VERBOSE)"
2121
task :test do |t|
2222
next unless npx_available?(t.name)
2323

24+
if ENV["REQUIREMENTS"] && (ENV["SCENARIO"] || ENV["SPEC_VERSION"])
25+
abort("REQUIREMENTS cannot be combined with SCENARIO or SPEC_VERSION: a requirement set already fixes which scenarios run.")
26+
end
27+
2428
require_relative "conformance/server_runner"
2529
require_relative "conformance/client_runner"
2630

2731
options = {}
2832
options[:port] = Integer(ENV["PORT"]) if ENV["PORT"]
2933
options[:scenario] = ENV["SCENARIO"] if ENV["SCENARIO"]
3034
options[:spec_version] = ENV["SPEC_VERSION"] if ENV["SPEC_VERSION"]
35+
options[:requirements] = ENV["REQUIREMENTS"] if ENV["REQUIREMENTS"]
3136
options[:verbose] = true if ENV["VERBOSE"]
3237

3338
Conformance::ServerRunner.new(**options).run

‎conformance/README.md‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,17 +22,25 @@ are allowed to fail without affecting the exit code.
2222

2323
### Environment variables
2424

25-
| Variable | Description | Default |
26-
|----------------|---------------------------------------|---------|
27-
| `PORT` | Server port (server conformance only) | `9292` |
28-
| `SCENARIO` | Run a single scenario by name | (all) |
29-
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
30-
| `VERBOSE` | Show raw JSON output when set | (off) |
25+
| Variable | Description | Default |
26+
|----------------|------------------------------------------------------------------------------|---------|
27+
| `PORT` | Server port (server conformance only) | `9292` |
28+
| `SCENARIO` | Run a single scenario by name | (all) |
29+
| `SPEC_VERSION` | Filter scenarios by spec version | (all) |
30+
| `REQUIREMENTS` | Run a spec revision's frozen requirement set at that revision's wire version | (off) |
31+
| `VERBOSE` | Show raw JSON output when set | (off) |
32+
33+
`REQUIREMENTS` runs exactly the scenarios a spec revision required when it shipped, which is
34+
what SEP-1730 tier scoring measures. It cannot be combined with `SCENARIO` or `SPEC_VERSION`:
35+
a requirement set already fixes which scenarios run.
3136

3237
```bash
3338
# Run a single server scenario
3439
bundle exec rake conformance:test SCENARIO=ping
3540

41+
# Run the frozen 2026-07-28 requirement set
42+
bundle exec rake conformance:test REQUIREMENTS=2026-07-28
43+
3644
# Use a different port with verbose output
3745
bundle exec rake conformance:test PORT=3000 VERBOSE=1
3846

‎conformance/client_runner.rb‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,10 @@
55

66
module Conformance
77
class ClientRunner
8-
def initialize(scenario: nil, spec_version: nil, verbose: false)
8+
def initialize(scenario: nil, spec_version: nil, requirements: nil, verbose: false)
99
@scenario = scenario
1010
@spec_version = spec_version
11+
@requirements = requirements
1112
@verbose = verbose
1213
end
1314

@@ -34,7 +35,10 @@ def build_command
3435
"--command",
3536
"bundle exec ruby #{client_script}",
3637
]
37-
npx_command += if @scenario
38+
# A requirement set already fixes which scenarios run, so it replaces the suite selection.
39+
npx_command += if @requirements
40+
["--requirements", @requirements]
41+
elsif @scenario
3842
["--scenario", @scenario]
3943
else
4044
["--suite", "all"]

‎conformance/server_runner.rb‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,11 @@ class ServerRunner
1212
SERVER_POLL_INTERVAL = 0.5
1313
SERVER_HEALTH_CHECK_RETRIES = (SERVER_START_TIMEOUT / SERVER_POLL_INTERVAL).to_i
1414

15-
def initialize(port: Server::DEFAULT_PORT, scenario: nil, spec_version: nil, verbose: false)
15+
def initialize(port: Server::DEFAULT_PORT, scenario: nil, spec_version: nil, requirements: nil, verbose: false)
1616
@port = port
1717
@scenario = scenario
1818
@spec_version = spec_version
19+
@requirements = requirements
1920
@verbose = verbose
2021
end
2122

@@ -34,6 +35,7 @@ def build_command
3435
npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance@alpha", "server", "--url", "http://localhost:#{@port}/mcp"]
3536
npx_command += ["--scenario", @scenario] if @scenario
3637
npx_command += ["--spec-version", @spec_version] if @spec_version
38+
npx_command += ["--requirements", @requirements] if @requirements
3739
npx_command += ["--verbose"] if @verbose
3840
npx_command += ["--expected-failures", expected_failures_yml]
3941
npx_command

‎test/conformance/client_runner_test.rb‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# frozen_string_literal: true
2+
3+
require "test_helper"
4+
require_relative "../../conformance/client_runner"
5+
6+
module Conformance
7+
class ClientRunnerTest < ActiveSupport::TestCase
8+
EXPECTED_FAILURES_YML = File.expand_path("../../conformance/expected_failures.yml", __dir__)
9+
CLIENT_SCRIPT = File.expand_path("../../conformance/client.rb", __dir__)
10+
11+
test "build_command runs the full suite by default" do
12+
command = ClientRunner.new.send(:build_command)
13+
14+
assert_equal(
15+
[
16+
"npx",
17+
"--yes",
18+
"@modelcontextprotocol/conformance@alpha",
19+
"client",
20+
"--command",
21+
"bundle exec ruby #{CLIENT_SCRIPT}",
22+
"--suite",
23+
"all",
24+
"--expected-failures",
25+
EXPECTED_FAILURES_YML,
26+
],
27+
command,
28+
)
29+
end
30+
31+
test "build_command replaces the suite selection with a frozen requirement set" do
32+
command = ClientRunner.new(requirements: "2026-07-28").send(:build_command)
33+
34+
assert_equal(
35+
[
36+
"npx",
37+
"--yes",
38+
"@modelcontextprotocol/conformance@alpha",
39+
"client",
40+
"--command",
41+
"bundle exec ruby #{CLIENT_SCRIPT}",
42+
"--requirements",
43+
"2026-07-28",
44+
"--expected-failures",
45+
EXPECTED_FAILURES_YML,
46+
],
47+
command,
48+
)
49+
end
50+
51+
test "build_command keeps the single scenario selection" do
52+
command = ClientRunner.new(scenario: "tools_call").send(:build_command)
53+
54+
assert_equal(
55+
[
56+
"npx",
57+
"--yes",
58+
"@modelcontextprotocol/conformance@alpha",
59+
"client",
60+
"--command",
61+
"bundle exec ruby #{CLIENT_SCRIPT}",
62+
"--scenario",
63+
"tools_call",
64+
"--expected-failures",
65+
EXPECTED_FAILURES_YML,
66+
],
67+
command,
68+
)
69+
end
70+
end
71+
end

‎test/conformance/server_runner_test.rb‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# frozen_string_literal: true
2+
3+
require "test_helper"
4+
require_relative "../../conformance/server_runner"
5+
6+
module Conformance
7+
class ServerRunnerTest < ActiveSupport::TestCase
8+
EXPECTED_FAILURES_YML = File.expand_path("../../conformance/expected_failures.yml", __dir__)
9+
10+
test "build_command runs the active suite by default" do
11+
command = ServerRunner.new.send(:build_command)
12+
13+
assert_equal(
14+
[
15+
"npx",
16+
"--yes",
17+
"@modelcontextprotocol/conformance@alpha",
18+
"server",
19+
"--url",
20+
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
21+
"--expected-failures",
22+
EXPECTED_FAILURES_YML,
23+
],
24+
command,
25+
)
26+
end
27+
28+
test "build_command runs a frozen requirement set and keeps the baseline" do
29+
command = ServerRunner.new(requirements: "2026-07-28").send(:build_command)
30+
31+
assert_equal(
32+
[
33+
"npx",
34+
"--yes",
35+
"@modelcontextprotocol/conformance@alpha",
36+
"server",
37+
"--url",
38+
"http://localhost:#{Server::DEFAULT_PORT}/mcp",
39+
"--requirements",
40+
"2026-07-28",
41+
"--expected-failures",
42+
EXPECTED_FAILURES_YML,
43+
],
44+
command,
45+
)
46+
end
47+
end
48+
end

0 commit comments

Comments
 (0)