You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(tests): let external compute drivers run the OpenShell conformance suite #4848
As a maintainer, I field questions from teams building external compute drivers
against compute_driver.proto. They want to check, in their own CI, that their
driver behaves like the built-in ones.
Problem Statement
There is no supported way for an external compute driver to run OpenShell's
conformance tests against its own gateway. The only option today is running the
CLI conformance suite from a source checkout (see TESTING.md), which isn't
documented for external use and is being replaced by testkit suites.
Impact / Why This Matters
External driver authors can't tell whether their driver meets the same bar as
the built-in drivers.
Without a shared suite, each driver writes its own checks and expectations
drift.
We have no single answer to point people at.
Proposed Design
Goal: an external driver can point OpenShell's conformance tests at its own
gateway, from its own CI, without building OpenShell.
Direction, details to be worked out:
Conformance means behavior every driver must pass. Tests specific to a driver
or a setup are kept separate.
The tests can be run against any existing gateway; they don't provision
environments.
They are packaged so an external project can get them for a given OpenShell
version.
Acceptance Criteria
An external driver's CI can run OpenShell conformance against its own gateway
without checking out or building OpenShell.
Docs explain how to do it.
Alternatives Considered
Document the existing from-source path as the supported way. Works today, but
ties external CI to building OpenShell and to a suite being replaced.
Run external drivers in OpenShell's CI. Doesn't scale.
Checklist
I've reviewed existing issues and the published docs
This is a design proposal, not a "please build this" request
Hey, I’d love to work on this if it’s still available! I’m planning to first understand the existing conformance test infrastructure, identify the driver-agnostic tests, and package them into a reusable runner that external compute-driver projects can run against their own gateways in CI, without needing to build OpenShell. I’ll also add a minimal CI example and documentation.
Let me know if I can take this up!
Activity
jyotir07 commentedon Oct 23, 2026
Hey, I’d love to work on this if it’s still available! I’m planning to first understand the existing conformance test infrastructure, identify the driver-agnostic tests, and package them into a reusable runner that external compute-driver projects can run against their own gateways in CI, without needing to build OpenShell. I’ll also add a minimal CI example and documentation.
Let me know if I can take this up!