Skip to content

ci: isolate dataplane conformance from control plane - #132

Draft
lucarlig wants to merge 3 commits into
mainfrom
user/luca/standalone-conformance
Draft

ci: isolate dataplane conformance from control plane#132
lucarlig wants to merge 3 commits into
mainfrom
user/luca/standalone-conformance

Conversation

@lucarlig

@lucarlig lucarlig commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Run make conformance and make conformance-bless through the standalone external lane: Redis, dataplane, nginx, and the official fixture, without control-plane startup or publisher-schema timing. The workflow installs cf-integration 0.3.1.

The companion harness change discovers all fixture routes and preserves the real tool schemas instead of maintaining an incomplete hardcoded catalog. Regenerating the baseline removes four false findings: the extra-parameter warning and three capability checks that previously hit missing tool routes. The external server baseline now contains 35 findings, down from 39; fixture-direct and client baselines are unchanged.

Document the limits of the baseline in the testing wiki. Strict matching detects regressions; it does not establish full conformance. Catalog-dependent scenarios cannot exercise checks after the dataplane rejects tools/list, and the pinned fixture lacks a custom-header-annotated tool. Those coverage gaps remain explicit alongside existing protocol failures.

Dependency

Depends on contextforge-org/contextforge-dev-tools#13. Keep this PR draft until cf-integration 0.3.1 is published with the fixture discovery fix.

Validation

  • make conformance-bless, followed by clean strict make conformance, using the companion PR's local binary and an image built from this PR's dataplane code.
  • All 84 scenarios completed: 40 fixture-direct server, 40 external server, and 4 scoped external client scenarios. Strict result: 44 passed, 40 expected failures, no unexpected failures or baseline drift.
  • No missing tool or prompt route errors remain in the external report.
  • Repository commit hooks passed: format, clippy, dependency/license audit, nextest, and workspace build.
  • mdbook build _context/wiki and mdbook test _context/wiki passed.

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig force-pushed the user/luca/standalone-conformance branch from de756e6 to 754a713 Compare September 4, 2026 15:24
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant