Skip to content

feat(agents): support Kubernetes Gators with reusable tooling images - #4744

Open
johntmyers wants to merge 2 commits into
mainfrom
feat/3469-gator-kubernetes/johntmyers
Open

johntmyers wants to merge 2 commits into
mainfrom
feat/3469-gator-kubernetes/johntmyers

Conversation

@johntmyers

@johntmyers johntmyers commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Enable the agent launcher to run Gator on Kubernetes gateways, using explicit local k3d imports, registry publishing, or reusable digest-pinned images. Separate per-run assignments from immutable tooling so different PRs no longer require task-specific image contents.

Related Issue

Related to #3469.

Changes

  • Keep the prompt template, skills, reviewer contract, and runtime read-only in the image. Supply the assignment and supervision settings at runtime; the Gator agent supervisor captures the assembled prompt in memory and passes it to every harness cycle on stdin.
  • Add --k3d-cluster for Docker builds imported into an explicitly selected running k3d cluster with a matching kube context. Do not infer cluster locality from a loopback gateway endpoint.
  • Add --publish-image for registry delivery with digest resolution, --image for trusted digest-pinned tooling reuse, and --build-only for building/delivering without provider configuration or sandbox creation.
  • Validate the reusable image's runtime-task protocol, agent ID, and payload version before entering the agent runtime. Old images with baked assignments cannot silently execute their previous assignment.
  • Preserve Docker/Podman engine selection, gateway-managed credential refresh, supervised watch cycles, and ephemeral defaults.
  • Bump the Gator payload to 13; document the new options and their limitations in the README and launch skill.

Unlike the original issue's prompt-baking assumption, the image now contains only task-independent instructions and tooling. PR assignments and operator test authorization travel in the runtime startup command, not Docker build inputs. Runtime command metadata can contain the assignment; credentials must continue to use providers. Prompt isolation is not a security boundary.

Release Notes

Gator can run through Kubernetes-backed gateways with explicit image delivery or a reusable tooling image. Per-run tasks are no longer baked into its images.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated
  • Automated sandbox E2E tests added/updated (manual Kubernetes smoke tests performed below)

Passed all nine focused suites: launcher/build-context isolation, image transport, tooling-image compatibility, supervisor, Codex harness, feedback ledger, GitHub write guard, thread resolution, and finding validation. Shell syntax, git diff --check, and Markdown lint also pass.

Manual validation on an ARM64 local k3d/Helm deployment:

  • Built the primary Gator Dockerfile, imported it into k3d, created a sandbox, and ran two supervised cycles with a mock Codex binary. Both received identical runtime instructions and completed successfully; temporary resources were removed.
  • Built and pushed to a temporary loopback registry, resolving the published image to a digest. Temporary registry and test image references were removed.
  • Launched a real GPT-6.1 Sol / medium Gator against already-merged PR fix(vm): flush guest filesystem before a planned sandbox stop #4195. After refreshing host Codex auth and reseeding the new gateway with --reset-refresh, gateway-owned token rotation succeeded. The agent verified authenticated GitHub access, fetched the published payload version, confirmed the merged PR, and exited 0 with merged_pr_connectivity_smoke_passed. No PR comments, reviews, labels, or workflows were changed. Initial Codex stream reconnects recovered. The completed sandbox was retained intentionally with --keep.

A remote Kubernetes deployment and a live Podman gateway were not exercised; registry/prebuilt paths and Docker/Podman selection have mocked coverage. Builds use the local builder's platform, and registry pull access remains deployment configuration.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Agent README and launch skill updated; no gateway configuration or architecture changes

Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
@purp purp added the test:e2e Requires end-to-end coverage label Oct 23, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for c8afb73. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@purp

purp commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

gator-agent

PR Review Status

The independent code review found no blocking findings across all 13 changed files. @johntmyers, I checked the reusable-tooling and runtime-assignment behavior you described, including image compatibility checks, Docker/Podman selection, and the updated README and launch skill.

Action required: A maintainer must open Branch E2E Checks run 39409836733 and select Re-run all jobs with test:e2e applied. The previous run skipped the runtime suites; the label-help bot confirms that a rerun is required. Workflow reruns were not authorized in this invocation.

Blocking findings: None.

Carried findings: None.

Non-blocking suggestions: None.

Gator metadata
  • Validation: Implements accepted issue feat(agents): support Kubernetes-driver gateways in the agent launcher #3469; maintainer-authored agent launcher work.
  • Docs: Agent README and launch skill updated for repository-internal tooling; the author's stated scope introduces no published CLI, gateway configuration, or API changes requiring Fern updates.
  • Checks: Branch Checks, Helm Lint, Trivy Changes, DCO, and vouch checks passed on the current head. No local tests were run during this code-only review.
  • E2E: Applied test:e2e because sandbox startup and supervised runtime assignment changed. The mirror matches the current head, but the existing E2E run skipped its suites before this label was present. Required runtime testing has not started.
  • Head SHA: c8afb7321ad49b8e5b7f947d344cde174f1a671e
  • Base SHA: 292c9c8d4865ad570d9a0d2f0994221bcd2ab9af
  • Merge base SHA: 7417f75384a02a12b30b736cc185bb08b814bd40
  • Patch ID: be9e5aefed4a38b0411d5d45c53f9227127bde13
  • Gator payload: 13
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:blocked
  • Blocked reason: test_dispatch_required

@purp purp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at c8afb7321. The design is solid, and the five launcher suites (run_test.sh, image_test.sh, launch_test.sh, supervisor_test.sh, codex/exec_test.sh) all pass locally. Nothing blocking; one unverified Podman risk is below.

Key design decisions

  • No guessing which cluster: a Kubernetes gateway with no delivery option fails with guidance. A loopback endpoint isn't treated as local, because it could be a port-forward (scripts/agents/image.sh:57).
  • Task kept out of the image: runtime placeholders survive the build as {{…}} (run.sh:635). The supervisor fills them in once, in memory, and fails if any are missing (supervisor.sh:34). It then unsets the task variable and feeds the prompt to every cycle on stdin (supervisor.sh:55, :312). A cycle that modifies the prompt file can't change what later cycles receive.
  • Compatibility check before startup: the build writes a small tooling.json with the agent ID and payload version (run.sh:597). Sandboxes now start through a new launch.sh, which checks it before handing off to entrypoint.sh. An old image doesn't have launch.sh, so it can't silently run its baked-in task.
  • Launches by digest, never by tag: --image must include @sha256: (image.sh:22), and registry delivery looks up the pushed digest and refuses a tag-only launch (image.sh:86).
  • --build-only stops early: it exits after delivering the image (run.sh:734), before settings, providers or sandbox creation.

Potential concern

  • --publish-image with Podman may launch a digest that doesn't exist in the registry (unverified). The registry digest comes from image inspect … RepoDigests (scripts/agents/image.sh:86). Podman can recompress or convert the manifest during a push, so the pushed digest may differ from the local one; that's why podman push has a --digestfile option. If that happens, the launcher prints a Tooling image: reference that nodes can't pull. The tests cover Podman only through a Docker stub symlinked to podman, and the PR body says a live Podman gateway wasn't exercised. main has no registry push path, so this path is new here.

Context (not blocking)

  • Remote Kubernetes is demonstrated on this very PR. Gator, running from this PR's payload-13 image on a remote Kubernetes gateway, posted the PR Review Status comment on this PR. (obligatory gif) That run produced two issues: #4884 (pushes aren't retried) and #4885 (*.oaiusercontent.com is denied, and only the first proposal chunk gets auto-approved).
  • On shared gateways, the launcher writes gateway-global settings; #4888 fixes it. This PR lets the launcher run on Kubernetes gateways, which are usually shared. There it still runs settings set --global for proposal_approval_mode=auto and agent_policy_proposals_enabled=true (scripts/agents/run.sh:742), so every other sandbox on the gateway gets auto-approved proposals too. #4888 applies the settings per sandbox instead, and the two PRs merge cleanly in either order.
  • CI doesn't run any of the scripts/agents tests. That was already true before this PR, so the new suites only run when someone invokes them by hand.

@purp

purp commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Follow-up on the Podman concern: confirmed with Podman 6.1.4 against a local registry:2. After podman build and podman push, podman image inspect … RepoDigests still reports the local digest (sha256:e8242466…), while the registry stores the pushed manifest as sha256:edb374f1…, matching podman push --digestfile. The selection in scripts/agents/image.sh:86 returns the local digest, and podman pull <repo>@sha256:e8242466… fails with manifest unknown. Podman fills RepoDigests before any push, so the "push did not produce a matching repository digest" guard doesn't fire. Reading the digest from podman push --digestfile on Podman would fix it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:watch-pipeline Gator is monitoring PR CI/CD status test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants