Reduce CI and release critical-path duplication without weakening validation

[Teakowa]

Goal

Reduce Wright CI and release wall-clock time and redundant runner work while preserving the validation contracts that protect real projects, public CLI behavior, distribution artifacts, and release provenance.

Context

The current workflow performs several expensive checks through overlapping execution paths. Full real-project Workshop coverage is exercised at multiple layers, while the release workflow repeats broad quality/build work before building the native release matrix. Some jobs also install toolchains that are not the toolchain actually selected by the repository.

The optimization target is redundant work, not weaker validation.

Wright should implement the WrightKit CI reuse principles concretely:

• For one workflow revision, a compatible build identity should have one producer and multiple consumers; downstream jobs should reuse the exact current-run artifact/result rather than invoke the same build again.
• Dependency/download cache, compiled dependency/build cache, and current-run artifacts are distinct layers and should be used for their respective purposes rather than conflated.
• Cache/artifact compatibility must include the material build identity: revision/source inputs, Cargo.lock/dependency state, Rust toolchain/compiler, target, profile, feature/configuration set, relevant compiler flags/environment, and other material build inputs.
• Rust incremental/intermediate state should only be retained when measured hosted-runner savings exceed restore/save/storage cost; the goal is less work, not a larger cache.

Scope

• Keep expensive semantic and real-project evidence at the layer that owns the corresponding contract, and avoid repeating the same corpus coverage at higher layers unless it protects an independent failure mode.
• Preserve independent public CLI/integration coverage while allowing representative fixtures where full-corpus repetition is not required by that contract.
• Identify Wright build identities that are currently rebuilt by multiple jobs and convert compatible cases to single-producer/current-run artifact reuse.
• Reuse Cargo dependency downloads and compatible compiled dependency/build state across jobs and runs with bounded cache writers and identity-safe invalidation.
• Measure Rust incremental/intermediate workspace caching separately from dependency caching; adopt it only where before/after evidence demonstrates a net benefit.
• Separate release-only identity/package validation from correctness work that is already independently proven for the exact release revision when that reuse can be made explicit and trustworthy.
• Reuse release/native build outputs across package/distribution consumers when target/profile/toolchain/configuration identity is the same rather than rebuilding per distribution channel.
• Avoid irrelevant capability/distribution jobs when changed-path evidence proves they are unnecessary.
• Remove redundant toolchain setup and cache fragmentation.

Non-goals

• Removing meaningful correctness, compatibility, real-project, packaging, or smoke validation solely to reduce runtime.
• Weakening diagnostics, assertions, failure handling, or release provenance.
• Sharing outputs across incompatible toolchain, target, profile, feature/configuration, lockfile, compiler flag, or revision identities.
• Treating a build cache as exact current-revision evidence where a current-run artifact/result is required.
• Caching the entire Rust target tree or forcing incremental compilation without evidence that transfer/storage cost is lower than the work avoided.
• Introducing an arbitrary-command CI abstraction or a repository-specific CI framework.
• Moving to self-hosted or larger runners before redundant work is addressed.

Acceptance criteria

• Comparable before/after workflow evidence demonstrates a material reduction in the CI and release critical paths.
• Full-corpus Workshop semantic/owner contract coverage remains blocking in the appropriate owning/integration layer.
• Public Wright CLI behavior retains independent regression coverage without duplicating full-corpus semantics unless an independent failure mode requires it.
• For every compatible build identity used by multiple jobs in one workflow revision, one producer builds the output and downstream consumers reuse that exact artifact/result, unless a concrete independent rebuild requirement is documented.
• Cargo dependency downloads are not repeatedly fetched when a compatible cache can safely satisfy the job.
• Compatible compiled dependency/build cache state is reused where measurements show a net benefit, with cache identity covering all material build inputs.
• Rust incremental/intermediate workspace caching is either adopted with before/after evidence showing net benefit or deliberately omitted because restore/save/storage cost does not justify it.
• Release tag/commit/version identity remains explicitly verified.
• Native release artifacts for every supported target remain built, packaged, checksummed, and smoke-tested before publication.
• Distribution/package consumers reuse an already-built native artifact when they require the same target/profile/toolchain/configuration identity instead of rebuilding it independently.
• No expensive validation/build step is repeated on the same revision without either an independent failure mode or an explicit provenance/isolation requirement.
• Cache/artifact reuse cannot silently cross incompatible toolchain, lockfile, profile, target, feature/configuration, compiler flag, or revision identities.
• Reduced or relocated validation is independently ablated so the retained evidence is shown to catch the failures it is intended to protect.

[Teakowa]

Baseline evidence from 2026-09-14/15 profiling:

• Main CI run 34816417724 completed in about 12m26s. Its Workshop real-project integration job was the critical path: provider_matches_released_workshop_contract took 172.87s and real_projects_run_check_and_lint_through_wright took 419.57s. Cache/setup/compilation were comparatively small.
• The CLI real-project test walks the full released Workshop corpus and, for every file, computes owner expectations and launches separate wright check and wright lint subprocesses. The driver contract test separately walks the same full corpus and compares WorkshopProvider::check with the owner API. This is the first duplication candidate; reducing it requires ablation to preserve independent CLI failure coverage.
• Release v0.2.23 run 34816417262 took about 10m25s. Release gates alone took about 4m04s. Inside scripts/release.sh, cold dev/test/release compilation accounted for roughly 3m23s; the semantic/scenario/benchmark/standalone checks themselves were short.
• .github/workflows/release.yml currently has no Rust cache in either release-gates or the four native release-build jobs, and the native build matrix waits for the broad release gate before rebuilding release binaries per target.
• workshop-integration installs Rust 1.85.0, but the repository rust-toolchain.toml selects stable and the subsequent cargo test commands do not use +1.85.0; the run log showed both toolchains in the environment. This is a small setup/cache-fragmentation cleanup, not the main bottleneck.
• Pushes to main currently force every path filter to true, so all capability/distribution checks run after every non-doc merge even though PR runs are path-selected.

Likely optimization order: (1) remove redundant full-corpus CLI coverage while retaining an independently ablated CLI contract, (2) retain full-corpus owner/provider evidence at the appropriate layer and profile/shard it only if still necessary, (3) eliminate release correctness re-runs by gating publication on exact-SHA CI evidence where feasible, with release-only identity/package checks retained, (4) add target-correct Rust caching/artifact reuse, then (5) refine main/distribution path selection. Faster runners should come only after these sources of duplicate work are removed.