Skip to content

supported-matrix.md

Latest commit

 

History

History
163 lines (137 loc) · 9.82 KB

supported-matrix.md

File metadata and controls

163 lines (137 loc) · 9.82 KB
Raw

Supported CLI / core / agent combinations

This page is the human-readable view of pkg/conformance/matrix.json, the machine-readable support matrix. The two cannot drift: pkg/conformance fails if a row here disagrees with the matrix, names a CI job that does not exist, or claims a status the repository cannot back.

Every row describes the CLI built from this checkout against the github.com/codefly-dev/core release line pinned in go.mod — the matrix records the line (v0.3.41), not the pseudo-version, so a core bump within a line does not invalidate the claim. Version skew between a released CLI and a released agent is not, by itself, a compatibility failure — a row names exactly the host behavior and infrastructure it exercises.

Required CLI gates use test-only protocol peers, never released service agents. The native-control row proves host selection, accepted endpoint/configuration propagation and teardown through gRPC and real sockets. It replaces the former Redis/Nix-specific row and makes no Nix or released-agent qualification claim. Provider runtime, packaging and language-toolchain qualification belong to their owners, not prerequisites for a CLI contract change.

Updating the core pin

When go.mod moves to a new core release line, update the core field in pkg/conformance/matrix.json in the same change. Run go test ./pkg/conformance/... before the full coverage and race gates. TestMatrixCoreMatchesGoMod refuses a stale declared version.

The current pin targets Core's published v0.19.15 release line, including proto companion 0.0.23 without Rust generation tooling and the Linux process-inspection correction: a missing executable link while process state remains is an inspection error, not successful proof of absence. The exact source revision is declared in go.mod; the CLI consumes Core's container parent attribution, release acquisition and explicit process-group/container recovery APIs. These source gates qualify the CLI against that selected revision. They do not qualify separately compiled agents or update the Core code embedded in their binaries. Agents creating sidecar containers must adopt the attribution API in their own source and qualify their lifecycle before consumers select the resulting release.

A missing receipt can be a consequence of an earlier failure: coverage and race run go test -failfast, so a failed package can stop the npm tests in pkg/librarystore before they emit linux-amd64-native-npm receipts. Each job produces and verifies its own receipts. The quality matrix also uses fail-fast, so coverage failure can cancel the race job. Inspect the first failed test and the cancelled job's check annotation before diagnosing the final receipt check. Keep the required rows and receipt verifier enabled; absence is still a failure.

What the statuses mean

Status Meaning
qualified A CI job drives this row on every change and lists it in CODEFLY_CONFORMANCE_REQUIRED, so its tests cannot skip themselves when the backend is missing. The job may pass over a change that touches only documentation, which cannot alter the row's outcome. Every qualified row must declare a gate call site — without one nothing could hold it to the claim.
not-yet-qualified Real tests and a real gate exist, but nothing proves the row outside a developer machine. This is not a support claim. Each row lists what blocks it.
unsupported Not shipped and not tested.

Rows

Row Status Platform Backend Prerequisites Evidence
linux-amd64-source qualified linux/amd64 none — go.yml — coverage, race, lint
linux-amd64-native-npm qualified linux/amd64 native npm go.yml — coverage, race
linux-amd64-native-control qualified linux/amd64 native protocol peers go go.yml — control-integration
linux-amd64-docker-generate qualified linux/amd64 docker docker, buf generate-qualification.yml — generate, reported as docker-generate
linux-amd64-docker-mcp-run not-yet-qualified linux/amd64 docker docker CODEFLY_MCP_RUN_QUALIFY=1, developer machine only
linux-amd64-network-mcp-agent not-yet-qualified linux/amd64 native — CODEFLY_MCP_AGENT_QUALIFY=1, developer machine only
linux-amd64-k3d-deploy not-yet-qualified linux/amd64 k3d docker, k3d, kubectl, ssh-keygen, git CODEFLY_GITOPS_K3D_QUALIFY=1, developer machine only
darwin-arm64-native-user-service not-yet-qualified darwin/arm64 native launchctl CODEFLY_SERVICE_INTEGRATION=1, developer machine only
windows-amd64-native-run unsupported windows/amd64 native — none

The Docker generate job runs every gate package of the row and every package holding a file built under its build_tag (proto_companion_required), with that tag and CODEFLY_GENERATE_QUALIFY=1. Both come from the matrix and the tagged sources through scripts/conformance-packages, never from a list in the script, and a test in pkg/conformance fails if the script's loop drifts from them. It pulls the published proto image resolved by Core once per runner, and extracts that image's buf for the SDK fixture. Each package is compiled once per shard; three runners split the discovered tests round-robin, and every top-level test runs in a fresh process with its own Codefly home, so package globals cannot leak between tests. The shards retain per-test output, timings and conformance receipts.

It runs on every pull request, every merge-queue entry and nightly. One job, docker-generate, always reports: it fails unless every shard passed and the receipts of all shards together prove the row. A change touching only documentation (docs/ or Markdown outside testdata/) cannot alter what generation compiles or reads, so there the shards are skipped and docker-generate passes without them; every other change runs them. scripts/generate-qualification-scope.sh decides, with rename detection off: a file moved into docs/ or renamed to .md counts by its old path too.

Qualification gaps

The matrix carries the blocker for every not-yet-qualified row. The two that matter most for a support claim:

  • No deployment row is qualified. linux-amd64-k3d-deploy is the only row that reaches a cluster, and no CI job provisions k3d. It also creates its cluster at k3d's default k3s image; a qualified cluster row must pin an explicit Kubernetes version, which Validate enforces.
  • No macOS row is qualified. CI has no macOS runner, so every released darwin binary ships without lifecycle evidence.

How the gate works

conformancetest.Gate(t, "<row>") replaces the ad-hoc t.Skip that used to guard each of these tests:

  • The row is listed in CODEFLY_CONFORMANCE_REQUIRED → the test runs. A missing prerequisite, or a runner whose OS/architecture is not the row's, fails. It never skips.
  • The row's own opt-in is set (CODEFLY_GITOPS_K3D_QUALIFY=1, …) → the test runs, and a missing prerequisite fails: you asked for that backend.
  • Neither → the test skips, naming the row and how to enable it.

A row with no gate call site cannot be required at all (Row.Requirable), and Validate refuses to call such a row qualified — so a job cannot claim a row nothing enforces, and a row cannot claim support nothing can check.

Prerequisites are a union; a caller may narrow

A row's prerequisites are the union over its tests, and no single test needs all of them: only the solution-SDK fixture runs buf, and only the two ArgoCD shapes need kubectl and ssh-keygen. A call site names the subset it uses — conformancetest.Gate(t, "linux-amd64-k3d-deploy", "docker", "k3d", "git") — and outside a CI claim only that subset must be present, so a partial local toolchain still qualifies the tests it can run. A claimed row is always held to every prerequisite it declares: narrowing is a local convenience, never a way to call a row qualified on a runner missing a tool. Naming a prerequisite the row does not declare is an error, so the two cannot drift apart.

Receipts

When CODEFLY_CONFORMANCE_RECEIPTS names a directory, each admitted test drops <row>.<package>.<test>.json there with the row, its status, the calling package, the resolved prerequisite paths, the core pin, the phase duration and the outcome. The package is part of the name because every package gating one row writes into the same directory and two packages may hold a same-named test. CI uploads the directory as an artifact and runs scripts/verify-conformance-receipts.sh in an always-run step, which fails when a required row produced no receipt, or produced one recording a failed run. That is what closes the last silent-pass hole: a go test -run filter that matches nothing exits 0, but it leaves no receipt — and checking the recorded outcome means the claim does not rest on the test step's exit code.

Qualifying a new row

  1. Add the row to pkg/conformance/matrix.json as not-yet-qualified, with its blockers, prerequisites and gate packages.
  2. Call conformancetest.Gate(t, "<row>", …) at the top of every test that is evidence for it, replacing any bespoke skip, naming the prerequisites that test actually uses.
  3. Add the row to this table.
  4. When a CI job provisions the backend, add the job to the row's ci list, name the row in that gate's CODEFLY_CONFORMANCE_REQUIRED (or its conformance_rows matrix value), drop the blockers, and flip the status to qualified. The tests in pkg/conformance fail if any of those four are missing — the ci entry is resolved against the workflow's real jobs and matrix gates, not matched as a substring.