Skip to content

Track starting and suspending states for the team broker agent - #8917

Merged
cstns merged 1 commit into
mainfrom
8619-team-broker-agent-states
Oct 14, 2026
Merged

cstns merged 1 commit into
mainfrom
8619-team-broker-agent-states

Conversation

@cstns

@cstns cstns commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Closes #8619

Merge order: FlowFuse/driver-localfs#363, FlowFuse/driver-docker#254 and FlowFuse/driver-k8s#415 go first and get released, then this one together with bumping the driver versions. On its own this PR is no worse than today, but it only closes the race once the drivers wait for the launch.

Starting the team broker and suspending it right after could leave the schema agent running with no record of it. The suspend landed while start was still launching the agent, so there was no pid to kill yet, and then the record got deleted. Only possible when the two calls overlap within a few ms, so API callers, not the UI.

The agent row now carries the state through the gap. Start creates it as starting, waits for the driver to launch the agent, then sets running. A suspend while it's starting gets a 409 agent_starting instead of trying to stop something that isn't there yet. Suspend marks it suspending before stopping it, and a start in that window (before the row is removed) gets a 409 agent_suspending rather than seeing a "running" agent and doing nothing. If the launch fails the row is removed so it can't get stuck in starting, and a second start while one is already starting is a no-op.

Things worth a look:

  • I went with the row state over a lock in the cache driver since the DB is already shared by every replica and nothing new is needed.
  • Two concurrent starts can still create two rows, there's no unique index on TeamId. Adding one needs a look at existing data first (a migration would fail on startup if prod has duplicates), so left out.
  • Moving out of suspending still relies on the existing 1.5s timer that deletes the row.

Added route tests for the starting/suspending states, the 409s and the failed launch, all failing before the change. Team broker specs pass. Also raced start and suspend against the local app (localfs) at offsets from 2 to 30ms: with this plus the driver change 0 orphans out of 24 tries and the overlapping suspends got 409s. With only the driver change it was 3 orphans out of 24, with only this one 1 out of 18, so both are needed.

A suspend arriving while start was still launching the agent found nothing to stop, then deleted the record, leaving the agent running untracked. The agent now stays starting until the driver has launched it and suspend replies 409 meanwhile; suspend marks it suspending first so a start in that window gets a 409 too. A failed launch removes the record instead of leaving it stuck in starting.
@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 79.15%. Comparing base (fe92734) to head (fe230dd).

Files with missing lines Patch % Lines
forge/ee/routes/teamBroker/3rdPartyBroker.js 93.75% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8917      +/-   ##
==========================================
+ Coverage   79.07%   79.15%   +0.07%     
==========================================
  Files         476      476              
  Lines       25890    25903      +13     
  Branches     6886     6889       +3     
==========================================
+ Hits        20473    20504      +31     
+ Misses       5417     5399      -18     
Flag Coverage Δ
backend 79.15% <93.75%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cstns
cstns merged commit 94b1c15 into main Oct 14, 2026
29 checks passed
@cstns
cstns deleted the 8619-team-broker-agent-states branch October 14, 2026 14:02

This branch was successfully deployed

1 active deployment
staging — fe230dda Deployed Oct 14, 2026 by cstns via Remove application #12484
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Suspending the team broker can leave its agent process running with no record

2 participants