Pre-bake ProtoFleet E2E runner dependencies

[illegalprime]

Reviewable diff: +1552/-12 across 8 files (excludes generated, test, and story files).

Summary

ProtoFleet E2E can now run on an opt-in GitHub-hosted custom VM image that preloads the exact client dependency tree, matching Playwright browser, and Docker Compose images. Hash and presence checks keep stale image components from being used, while the existing ubuntu-latest, Actions cache, and Docker artifact paths remain the default and fallback. This also checks in the timing analyzer and baseline used to measure whether the runner reduces overhead or causes regressions.

How it works

A trusted weekly or manual workflow on the default branch prepares a Linux x64 image-generation runner, writes a manifest containing the source SHA plus hashFiles digests, and snapshots it as protofleet-e2e. Once an administrator provisions a runtime larger runner and sets PROTOFLEET_E2E_RUNNER, the E2E build, 32 functional shards, and two visual jobs land on that image.

After checkout, a local action compares the current server and lockfile hashes with the manifest. Matching client dependencies are copied into the workspace, Playwright is pointed at the baked Chromium tree, and Docker images are accepted only if every recorded image is still present in the host daemon. Each component falls back independently; scheduled E2E runs always rebuild Docker images against fresh upstream bases, and the build job still publishes an image artifact so older runner instances can fall back safely during image rollouts.

The timing analyzer reads completed PR Gate runs and latest-attempt job data from GitHub, groups the newest successful runs per PR, classifies step time as test, validation/build, or overhead, and emits Markdown/JSON plus a regression comparison against the checked-in baseline.

Diagrams

flowchart LR
  A["Trusted default-branch image workflow"] --> B["Image-generation larger runner"]
  B --> C["Install Node dependencies and Chromium"]
  B --> D["Pull and build Compose images"]
  C --> E["Write hash manifest"]
  D --> E
  E --> F["Snapshot protofleet-e2e"]
  F --> G["Runtime larger runner"]
  H["PR E2E job"] --> I["Hash-aware restore action"]
  G --> I
  I --> J{"Component matches and exists?"}
  J -->|"Yes"| K["Use pre-baked component"]
  J -->|"No"| L["Use existing cache or artifact path"]
  K --> M["Build, functional shards, and visual tests"]
  L --> M

Loading

sequenceDiagram
  participant Admin as "CI administrator"
  participant Builder as "Image workflow"
  participant GitHub as "Custom image store"
  participant E2E as "PR E2E job"
  Admin->>Builder: "Configure builder variable and run on main"
  Builder->>GitHub: "Snapshot tools, dependencies, Docker images, and manifest"
  Admin->>E2E: "Configure runtime runner variable"
  E2E->>E2E: "Compare current hashes and inspect Docker images"
  alt "Valid pre-baked component"
    E2E->>E2E: "Restore or reuse local component"
  else "Missing or stale component"
    E2E->>E2E: "Restore Actions cache/artifact or rebuild"
  end

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
.github/workflows/protofleet-e2e-runner-image.yml	Adds trusted weekly/manual custom-image generation with GitHub snapshot	Verify the image lifecycle, default-branch restriction, and organization runner dependency
.github/actions/protofleet-e2e-runner/	Exposes independently validated client, Playwright, and Docker hits	This is the boundary that decides whether setup steps may be skipped
.github/scripts/prepare_protofleet_e2e_runner.sh	Installs exact dependencies and Compose images, then writes the root-owned manifest	Review image contents, hash inputs, and snapshot safety
.github/scripts/restore_protofleet_e2e_runner.sh	Validates the manifest, restores dependencies, and inspects host Docker images	Review fail-safe behavior for stale, missing, and partially available content
.github/workflows/protofleet-e2e-tests.yml	Selects the configured runner and conditionally skips redundant setup	Verify the normal runner path is unchanged on misses and scheduled rebuilds stay fresh
.github/scripts/analyze_ci_timings.py and baseline	Adds repeatable historical step timing, Markdown/JSON reports, caching, and regression thresholds	Review sampling semantics and name-based timing classification
docs/plans/2026-08-31-protofleet-e2e-prebaked-runner-plan.md	Documents administrator activation, canary measurement, and rollback	Organization-level runners cannot be provisioned by repository code
Python tests	Covers run selection/classification and runner hit/miss/fallback behavior	Test-only — focus on edge-case coverage

Key technical decisions & trade-offs

• Use a custom runner VM rather than a job container, because the measured hotspot is hydrating the host Docker daemon used by Compose.
• Validate server and client inputs independently rather than invalidating the entire image, so client-only and server-only changes retain safe partial hits.
• Keep publishing the Docker artifact on a pre-baked hit rather than maximizing the single build-job saving, so mixed image versions can fall back during rollout.
• Force scheduled runs down the Docker rebuild path rather than using a weekly snapshot, preserving the existing fresh-upstream-base behavior.
• Keep ubuntu-latest as the variable fallback rather than requiring organization infrastructure at merge time; activation and rollback are configuration-only.
• Classify timing by normalized step name rather than workflow-specific IDs, which supports reusable workflows but surfaces unmatched steps explicitly for review.

Testing & validation

• server/sdk/v1/python/.venv/bin/python -B .github/scripts/protofleet_e2e_runner_test.py — 5 tests passed.
• server/sdk/v1/python/.venv/bin/python -B .github/scripts/analyze_ci_timings_test.py — 7 tests passed.
• server/sdk/v1/python/.venv/bin/python -B .github/scripts/evaluate_review_policy_test.py — 72 tests passed.
• Ruff check/format, shell syntax, YAML parsing, Prettier, and git diff --check passed.
• Pre-commit hooks passed Ruff and protected-branch checks; pre-push hooks passed client typecheck and server lint.
• Not covered yet: generating the real GitHub custom image and running the E2E canary. Those require an organization administrator to provision the image-generation/runtime larger runners and set the two documented variables.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9dbac9369e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Require the default branch ref before snapshotting

For workflow_dispatch, the selected ref may be a tag, and github.ref_name strips the refs/tags/ prefix. A writable tag named like the default branch therefore satisfies this condition, after which checkout and the preparation script bake that tag's dependencies and Docker images into the shared E2E runner image. This can persist unreviewed code in later CI runs; compare the full github.ref against refs/heads/<default_branch> instead.

Useful? React with 👍 / 👎.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (89dc5a5f5300bcab3271c12b294ce80e0e562fef...9dbac9369e3a112beb8083f7a92cc98a844a1018, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: HIGH

Findings

[HIGH] Automated review incomplete

• Category: Other
• Description: The automated review produced no usable result for 89dc5a5f5300bcab3271c12b294ce80e0e562fef...9dbac9369e3a112beb8083f7a92cc98a844a1018 (workflow run 33428036438; reason: codex-job-timeout, elapsed: unknown, budget: 9 minutes).
• Impact: The pull request has not received complete automated security, correctness, and reliability analysis.
• Recommendation: Require human review before merging. Do not treat this result as approval-free or low risk.

Notes

Human review is required because the bounded automated review was incomplete.

---

_{Generated by Codex Security Review |
Triggered by: @illegalprime |
Review workflow run}

[ankitgoswami]

seems like a good idea if we can get a custom runner