Skip to content

Specialist runs keep their frozen tool bundle after a hub rebuild and never pick up fixes #67

Closed

Description

@brianjfox

A designer or coder run bundles its tool code when the run is created and keeps that bundle for its whole life. After the hub is rebuilt with a fix, every live specialist keeps failing exactly as before, and nothing in the product replaces it.

What happened. On 2026-11-01 and 2026-11-02 the Compose hub was rebuilt four times to fix sandbox faults (see #66). Each time, the live designers and later the coder kept reporting the old failure. Their closures under local-sidecars/sidecar/workflow-definition-closures/<run>/packages/ still held the package version pinned at creation (for example designer-native-1.0.0-sha-999417dd while the hub had published ...-sha-5c235a32). The coordinator's design_app and deploy_coder route an instruction to a specialist already working rather than starting a new run, so asking it to retry re-ran the stale code.

Why nothing replaces them. apps/hub/src/projects/coordinator-upgrade.ts replaces specialists only as part of upgrading the coordinator, and only when the coordinator's own pin is stale. A fix to apps/hub/src/builds/* changes the designer and coder bundles but not the coordinator's, so the pin matches and no upgrade fires. The coordinator cannot cancel a specialist (its token only reads runs), and there is no operator action for it. The only way out was writing an operator cancellation into workflow_run by hand and asking the coordinator to start the specialists again.

Expected. One of:

  • The reconcile pass compares each live specialist's pin with the installed descriptor and, when a specialist is parked (on input or an approval) and stale, cancels it and has the coordinator redeploy it with the same concept or instruction. Designers already seed from the previous designer's draft (previousDesignDraft), so the work carries over; a coder's workspace would need the same care.
  • Or the coordinator upgrade also fires when any specialist's pin is stale, not only the coordinator's.
  • At minimum, an operator route or UI action to stop a specialist run, so an operator can do the replacement without touching the database.

Seen on the Compose hub built from main at 0e292be plus the #66 commits, Colima 0.10.3 (Ubuntu 24.04.4, arm64).

Activity

  1. TheGreatAxios commented on Nov 3, 2026

    @TheGreatAxios
    Contributor

    Resolved in PR #112 (commit d946201).

  2. TheGreatAxios commented on Nov 3, 2026

    @TheGreatAxios
    Contributor

    Code & Commit Verification Proof

    Implemented in PR #112 (commit d946201):

    • apps/hub/src/projects/coordinator-upgrade.ts:
      • staleRuns: Inspects both the coordinator workflow run and specialist workflow runs, comparing their pinned closure hashes against the current built workflow packages.
      • update: Replaces stale coordinator and specialist runs in-place while preserving project state and mail routing.
    • apps/hub/src/projects/routes.ts:
      • Exposes POST /api/projects/:projectId/update to trigger the coordinator + specialist workflow upgrade for an existing project.
    • apps/web/src/pages/project-menu.tsx:
      • Adds the Update… action to the project menu in the web UI so users can upgrade a project's running workflows when stale pins are detected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions