Publish the compatibility matrix on GitHub Pages

.github/workflows/compat.yml

@@ -1,6 +1,7 @@
 # Runs proof against every server with a profile and publishes the
-# compatibility matrix in the job summary. Servers failing checks is the
-# expected state of the world, so only harness problems fail this workflow.
+# compatibility matrix in the job summary and on GitHub Pages. Servers
+# failing checks is the expected state of the world, so only harness
+# problems fail this workflow.
 name: Compatibility
 
 on:
@@ -22,10 +23,14 @@ jobs:
       matrix:
         profile: [clojure, babashka, clojure-clr, basilisp, jank, dialtone, repartee]
     steps:
+      # The reports say which version of proof made them. Go 1.24 and newer
+      # work it out from the tags and the commit, like for go install.
       - uses: actions/checkout@v7
+        with:
+          fetch-depth: 0
       - uses: actions/setup-go@v7
         with:
-          go-version-file: go.mod
+          go-version: stable
       - run: go build -o bin/proof ./cmd/proof
 
       - if: matrix.profile == 'clojure'
@@ -127,7 +132,32 @@ jobs:
           done
           bin/proof matrix "${reports[@]}" > matrix.md
           cat matrix.md >> "$GITHUB_STEP_SUMMARY"
+          mkdir site
+          bin/proof matrix -html "${reports[@]}" > site/index.html
       - uses: actions/upload-artifact@v7
         with:
           name: matrix
           path: matrix.md
+      - if: github.ref == 'refs/heads/main' && needs.run.result == 'success'
+        uses: actions/upload-pages-artifact@v5
+        with:
+          path: site
+
+  pages:
+    # Only when every server's job succeeded, as a page that's a day old
+    # beats one that's missing a server.
+    needs: [run, matrix]
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      pages: write
+      id-token: write
+    concurrency:
+      group: pages
+    environment:
+      name: github-pages
+      url: ${{ steps.deploy.outputs.page_url }}
+    steps:
+      - id: deploy
+        uses: actions/deploy-pages@v5

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## main (unreleased)
 
+### Features
+
+- [#18](https://github.com/nrepl/proof/pull/18): The `version` snippet, so reports and the compatibility matrix show which version of a server proof checked.
+- [#18](https://github.com/nrepl/proof/pull/18): `proof matrix -html`, which builds the compatibility matrix as a web page, and the daily matrix of the bundled profiles on [GitHub Pages](https://nrepl.github.io/proof/).
+
 ## 0.1.0 (2026-10-26)
 
 Initial release.

PLAN.md

@@ -142,23 +142,36 @@ and it covers the rest of what every editor sends: `interrupt`,
 - [Spec Changes](doc/spec-changes.md) has everything the new checks found.
 - The docs cover the new checks and the release.
 
-## After 0.1.0
+## 0.2.0
 
-This is what's next for proof.
+The compatibility matrix gets a page of its own, and more servers and
+clients.
 
 ### The Matrix
 
-- Publish the compatibility matrix on GitHub Pages from the
-  Compatibility workflow, with the version of each server.
-- Add nbb and let-go, which are easy to install in CI, and then
-  shadow-cljs.
+- Done: the Compatibility workflow publishes the matrix on
+  [GitHub Pages](https://nrepl.github.io/proof/), with the version of
+  each server, which comes from the new `version` snippet. `proof matrix
+  -html` builds the same page out of any reports.
+- nbb and let-go, which are easy to install in CI.
 
 ### More Clients
 
 - [clojure-cli.repl](https://github.com/clojure/clojure-cli.repl), the
   new client of the Clojure team, which doesn't handle `need-input` as
   of 0.1.1.
 - `bb repl --connect`.
+
+## After 0.2.0
+
+This is what's next for proof.
+
+### The Matrix
+
+- shadow-cljs.
+
+### More Clients
+
 - The client in `nrepl.cmdline`, which expects an exact list of statuses
   when interrupting an idle session.
 - vim-iced and rebel-readline.
@@ -173,23 +186,41 @@ This is what's next for proof.
 
 ### Upstream
 
-- The open pull requests based on proof's findings:
-  [spec.nrepl.org#6-#9](https://github.com/nrepl/spec.nrepl.org/pulls),
-  [nrepl-beam#1](https://github.com/nrepl/nrepl-beam/pull/1) and
-  [#2](https://github.com/nrepl/nrepl-beam/pull/2),
-  [neat#2](https://github.com/nrepl/neat/pull/2) and
-  [mezcaml#2](https://github.com/nrepl/mezcaml/pull/2).
+- The pull requests for the spec based on proof's findings,
+  [spec.nrepl.org#6-#9](https://github.com/nrepl/spec.nrepl.org/pulls).
+  The fixes for nrepl-beam, neat and mezcaml got merged on 2026-10-02,
+  except for [nrepl-beam#2](https://github.com/nrepl/nrepl-beam/pull/2),
+  which was closed when the branch it was based on went away and has to
+  be opened again.
 - Issues for the spec about the rest of [Spec Changes](doc/spec-changes.md),
   once the spec pull requests are settled.
 - nREPL: a release with
   [nrepl#498](https://github.com/nrepl/nrepl/pull/498), then
   [nrepl#477](https://github.com/nrepl/nrepl/issues/477) and
   [nrepl#183](https://github.com/nrepl/nrepl/issues/183), which has the
-  same cause as #498.
+  same cause as #498. The next eval after a client goes away in the
+  middle of one loses its `done` for the same reason as
+  [nrepl#509](https://github.com/nrepl/nrepl/issues/509). Babashka has
+  the same problem, which is why it fails `robust.disconnect` in a full
+  run, but not on its own.
 - CIDER and Calva accepting `ops` as a list, after which
   `describe.ops-dict` can allow both formats.
-- Issues for what the matrix finds in Basilisp and ClojureCLR, a few at
-  a time. jank's can go in a reply to jank#1104.
+- Reported on 2026-10-29:
+  [basilisp#1356](https://github.com/basilisp-lang/basilisp/issues/1356)
+  (no `no-info` for unknown symbols),
+  [basilisp#1357](https://github.com/basilisp-lang/basilisp/issues/1357)
+  (columns from 0),
+  [jank#1169](https://github.com/jank-lang/jank/issues/1169) (a request
+  that throws stops its connection from answering),
+  [conjure#802](https://github.com/Olical/conjure/issues/802) (no
+  `interrupt-id` in `interrupt`) and
+  [cider#4224](https://github.com/clojure-emacs/cider/issues/4224) (async
+  eldoc ignores `no-eldoc`).
+- ClojureCLR doesn't say `no-info` for unknown symbols either, but
+  clr.tools.nrepl takes bug reports on Ask Clojure rather than GitHub.
+- Calva waits for an eval of `java.version` in the main session before
+  it interrupts, so it looks like it can't interrupt an eval there. That
+  needs a run in VS Code before it gets reported.
 
 ## Further Down the Road
 

README.md

@@ -50,7 +50,8 @@ The [profiles](profiles) folder has complete profiles for nREPL itself,
 Babashka, Basilisp, ClojureCLR, jank, dialtone (Erlang) and repartee
 (Elixir), which you can use as a starting point. They come with proof as
 well, so `proof run babashka` will check Babashka (provided you have it
-installed).
+installed). The [compatibility matrix](https://nrepl.github.io/proof/)
+shows how all of them do, and it's updated every day.
 
 Now you can run the checks:
 

cmd/proof/main.go

@@ -12,6 +12,7 @@ import (
 	"os/signal"
 	"path"
 	"runtime/debug"
+	"strconv"
 	"strings"
 	"syscall"
 	"time"
@@ -22,6 +23,7 @@ import (
 	"github.com/nrepl/proof/internal/report"
 	"github.com/nrepl/proof/internal/serve"
 	"github.com/nrepl/proof/internal/server"
+	"github.com/nrepl/proof/nrepl"
 )
 
 // version is set for releases, with -ldflags "-X main.version=...".
@@ -52,7 +54,7 @@ Usage:
   proof run [flags] PROFILE           run the checks against the server a profile describes
   proof proxy [flags] [PROFILE]       check what a client sends to a server, by sitting between them
   proof serve [flags] [SCENARIO...]   be a server for a client's tests, acting like other servers where asked
-  proof matrix REPORT...              build a Markdown compatibility matrix from JSON reports
+  proof matrix [-html] REPORT...      build a compatibility matrix (Markdown or a web page) from JSON reports
   proof list                          list every check, rule and scenario
   proof version
 
@@ -208,12 +210,17 @@ func run(args []string) int {
 		addr = srv.Addr
 	}
 
+	ver, err := serverVersion(p, addr)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "proof: couldn't get the version of the server:", err)
+	}
 	env := &check.Env{Profile: p, Addr: addr}
 	started := time.Now()
 	results := check.Run(env, selected, checks.WireRules())
 	outcome := check.Apply(results, p.ExpectedFailures, opts.only == "")
 
-	r := report.Run{Proof: proofVersion(), Server: p.Name, Address: addr, Started: started, Results: results}
+	r := report.Run{Proof: proofVersion(), Server: p.Name, ServerVersion: ver, Address: addr, Started: started,
+		Results: results}
 	report.Text(os.Stdout, r, opts.verbose)
 	for _, id := range outcome.Stale {
 		fmt.Printf("%s is listed in expected-failures but didn't fail; remove it from the profile\n", id)
@@ -260,6 +267,38 @@ func startServer(p *profile.Profile) (*server.Server, error) {
 	return srv, err
 }
 
+// serverVersion evaluates the version snippet of the profile, if it has
+// one, in a session of its own. A string comes back without its quotes,
+// and on a single line.
+func serverVersion(p *profile.Profile, addr string) (string, error) {
+	sn, ok := p.Snippets["version"]
+	if !ok {
+		return "", nil
+	}
+	c, err := nrepl.Dial(addr, 5*time.Second)
+	if err != nil {
+		return "", err
+	}
+	defer c.Close()
+	clone, err := c.Request(nrepl.Message{"op": "clone"}, p.Timeout)
+	if err != nil {
+		return "", err
+	}
+	resp, err := c.Request(nrepl.Message{"op": "eval", "code": sn.Code, "session": clone.Str("new-session")}, p.Timeout)
+	if err != nil {
+		return "", err
+	}
+	values := resp.Values()
+	if len(values) == 0 {
+		return "", fmt.Errorf("the version snippet gave no value (status %v)", resp.Status())
+	}
+	v := values[len(values)-1]
+	if s, err := strconv.Unquote(v); err == nil {
+		v = s
+	}
+	return strings.Join(strings.Fields(v), " "), nil
+}
+
 // showDeath shows the output of a server that exited while proof was
 // using it, as that explains whatever went wrong afterwards.
 func showDeath(w io.Writer, srv *server.Server) {
@@ -311,7 +350,13 @@ func allIDs() []string {
 	return ids
 }
 
-func matrix(paths []string) int {
+func matrix(args []string) int {
+	fs := newFlags("matrix", "REPORT...", os.Stderr)
+	html := fs.Bool("html", false, "write a web page instead of a Markdown table")
+	paths, err := parseArgs(fs, args)
+	if err != nil {
+		return flagStatus(err)
+	}
 	if len(paths) == 0 {
 		fmt.Fprintln(os.Stderr, "proof matrix: expected one or more JSON reports (from proof run -json)")
 		return 2
@@ -325,6 +370,13 @@ func matrix(paths []string) int {
 		}
 		reports = append(reports, r)
 	}
+	if *html {
+		if err := report.HTML(os.Stdout, reports); err != nil {
+			fmt.Fprintln(os.Stderr, "proof:", err)
+			return 2
+		}
+		return 0
+	}
 	report.Matrix(os.Stdout, reports)
 	return 0
 }

cmd/proof/main_test.go

@@ -3,6 +3,10 @@ package main
 import (
 	"runtime/debug"
 	"testing"
+	"time"
+
+	"github.com/nrepl/proof/internal/profile"
+	"github.com/nrepl/proof/internal/serve"
 )
 
 func TestVersion(t *testing.T) {
@@ -24,3 +28,34 @@ func TestVersion(t *testing.T) {
 		}
 	}
 }
+
+func TestServerVersion(t *testing.T) {
+	srv, err := serve.Listen("127.0.0.1:0", "0.1.0-dev", nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	go srv.Serve()
+	t.Cleanup(func() { srv.Stop(0) })
+	cases := []struct {
+		code, want string
+	}{
+		{`"1.2.3"`, "1.2.3"},
+		{`"1.2.3\nSNAPSHOT"`, "1.2.3 SNAPSHOT"},
+		{"(+ 1 2)", "3"},
+		{"", ""},
+	}
+	for _, c := range cases {
+		p := &profile.Profile{Timeout: time.Second}
+		if c.code != "" {
+			p.Snippets = map[string]profile.Snippet{"version": {Code: c.code}}
+		}
+		got, err := serverVersion(p, srv.Addr())
+		if err != nil || got != c.want {
+			t.Errorf("version snippet %q gave %q (%v), want %q", c.code, got, err, c.want)
+		}
+	}
+	p := &profile.Profile{Timeout: time.Second, Snippets: map[string]profile.Snippet{"version": {Code: "(throw (ex-info \"no\" {}))"}}}
+	if _, err := serverVersion(p, srv.Addr()); err == nil {
+		t.Error("expected an error for a snippet that throws")
+	}
+}

doc/hacking.md

@@ -308,6 +308,8 @@ To add a server to the compatibility matrix:
 
 1. Create `profiles/<server>.toml`. Start it with a comment explaining
    what needs to be installed. [Profiles](profiles.md) covers the format.
+   Unless the profile always starts the same version of the server, give
+   it a `version` snippet, so the matrix shows which version it checked.
 2. Run it locally and make sure the results make sense. It's fine for a
    server to fail checks (that's what the matrix is for), but make sure
    that it's really the server that's failing and not the profile. Running
@@ -362,9 +364,10 @@ proof uses a few GitHub Actions workflows:
   the way a server's CI would use it.
 - Compatibility runs all the profiles against their servers every day
   (and on every push to `main`) and builds the compatibility matrix in
-  the job summary. As servers failing checks is perfectly normal, this
-  workflow fails only when proof itself can't do its job (e.g. a server
-  can't be installed or doesn't start).
+  the job summary. When every server ran, it also publishes the matrix
+  on [GitHub Pages](https://nrepl.github.io/proof/). As servers failing
+  checks is perfectly normal, this workflow fails only when proof itself
+  can't do its job (e.g. a server can't be installed or doesn't start).
 - Release builds the binaries for every platform when a version gets
   tagged and publishes them in a GitHub release.
 

doc/profiles.md

@@ -72,6 +72,9 @@ value = "println"
 [snippets.lookup]
 code = "println"
 
+[snippets.version]
+code = "fizz.core/version"
+
 [expected-failures]
 "eval.multiple-forms" = "only the last value is returned, see #42"
 ```
@@ -84,7 +87,7 @@ code = "println"
 
 | Option | Required | Description |
 |---|---|---|
-| `name` | yes | The name of the server in reports and in the compatibility matrix. It's a good idea to include the version you're testing. |
+| `name` | yes | The name of the server in reports and in the compatibility matrix. If the profile always starts the same version of the server, it's a good idea to include it here (otherwise see the `version` snippet). |
 | `homepage` | no | The homepage of the server. |
 | `address` | no | The `host:port` of a server that's already running. When it's set proof connects to the server instead of starting it, and `[launch]` is ignored. The `-address` command-line option does the same. |
 | `timeout` | no | How long proof waits for each request to finish (i.e. for its `done`). It's a duration like `"10s"` (which is also the default). |
@@ -148,6 +151,7 @@ snippet is missing, all the checks that need it are skipped.
 | `sleep` | `code`, `out` | print `out` and then keep running for longer than the `timeout` (e.g. a minute), so there's something to interrupt | all the `interrupt.*` checks, `robust.disconnect`, `robust.sessions-at-once` |
 | `completion` | `code`, `value` | `code` is the start of a name, which completes to `value` | all the `completion.*` checks |
 | `lookup` | `code` | `code` is the name of a function | `lookup.lookup`, `lookup.info`, `lookup.eldoc` |
+| `version` | `code` | evaluate to the version of the server | none, proof shows the version in the report and in the compatibility matrix |
 
 A few tips for writing snippets:
 
@@ -178,6 +182,8 @@ A few tips for writing snippets:
 - The checks of completions and lookups evaluate `value` first, and send
   the `ns` that comes with its value (if any) along with their requests,
   just like clients send the namespace of the code they complete.
+- proof evaluates `version` before the checks, on a connection of its
+  own. If the version is a string, the quotes are dropped.
 - proof interrupts `sleep` as soon as its `out` arrives, so make sure
   your server sends the output right away (in Clojure that takes a
   `flush`, as there's no newline).