Repository navigation
Flaky backend tests: billing trial state, 3rd party broker topics, device APIs #8478
Description
While watching CI on open PRs we hit a batch of test failures that look unrelated to the PR under test (frontend-only change, PR #8467, run 34595722439). All 9 failed in the Postgres tests job and passed elsewhere, and the assertions look timing dependent.
- Billing routes > Can suspend/resume a trial project without touching billing (
test/unit/forge/ee/routes/billing/index_spec.js:1453, expected undefined to be 'starting') - 3rd Party Broker API > Verify topics all correct (
test/unit/forge/ee/routes/teamBroker/3rdPartyBroker_spec.js:443) - 3rd Party Broker API > Get Topics for 3rd Party broker as a Team Owner (
3rdPartyBroker_spec.js:467) - 3rd Party Broker API > Add Metadata to a Topic (
3rdPartyBroker_spec.js:485) - 3rd Party Broker API > Cannot add metadata to a topic for a broker in a different team (
3rdPartyBroker_spec.js:516) - 3rd Party Broker API > Delete Topic (
3rdPartyBroker_spec.js:543) - 3rd Party Broker API > Get Topics for team-broker as a Team Owner (
3rdPartyBroker_spec.js:697, expected [] to have length 1) - Device API > can unassign from a project (
test/unit/forge/routes/api/device_spec.js:1358, null read) - Project/Device API > returns a list of devices for team owner (
test/unit/forge/routes/api/projectDevices_spec.js:168, expected [] to have length 1)
The broker topic tests failing as a block might share one root cause (broker state not settling before assertions). Might be worth starting there.
Other issues have been and will be added as comments
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
cstns commentedon Sep 14, 2026
Another one from the same family, hit on PR #8467 (run 34836147852, Postgres tests job):
test/unit/forge/ee/routes/teamBroker/index_spec.js:1322, expected false to be true)Same pattern as the other broker topic ones, an assertion on topic state right after adding topics. Re-ran the job.
EDIT by @Steve-Mcl 2026/09/28
Got same error AFTER merge (on main branch)
in action https://github.com/FlowFuse/flowfuse/actions/runs/36427799841/job/108946858426
cstns commentedon Sep 14, 2026
One more, again on PR #8467 (run 34839432240, Postgres tests job):
test/unit/forge/routes/api/projectDevices_spec.js:200, expected 400 got 404)Same spec file as the
projectDevices_spec.js:168entry above. Re-ran the job.cstns commentedon Sep 14, 2026
Two Cypress (UI EE) flakes this time, on PR #8502 (run 34854653678, UI EE tests job). Both in
test/e2e/frontend/cypress/tests-ee/devices/developer-mode.spec.js, both acy.click()timing out because a dialog overlay (ff-dialog-container--open) was still covering the tab link, each failed all 3 attempts:The PR only touches backend MCP tools, so unrelated. Looks like the dialog close animation racing the next click.
cstns commentedon Sep 14, 2026
Another device API one, on PR #8503 (run 34854920756, Postgres tests job):
test/unit/forge/routes/api/device_spec.js:1639, expected 400 got 200)Third distinct flake in
device_spec.js/projectDevices_spec.jsterritory, looks like shared fixture state leaking between tests. Re-ran the job.[edit by @Steve-Mcl] same test fail in #8652 (action https://github.com/FlowFuse/flowfuse/actions/runs/36401134932/job/108861401213?pr=8652)
andypalmi commentedon Sep 24, 2026
Another one
Search API
Simple search
search by device id returns only one device & does not query other tables:
AssertionError: expected 0 to be 1
-0
+1
andypalmi commentedon Sep 24, 2026
Another one
DeviceCommsHandler
Device Status
suppresses the transient stopped broadcast while restarting, but still records the state:
AssertionError: expected 'restarting' to be 'stopped'
-restarting
+stopped
**EDIT by @Steve-Mcl **
Got this too - in PR #8713 on run https://github.com/FlowFuse/flowfuse/actions/runs/36696069757/job/109824909510?pr=8713
Steve-Mcl commentedon Sep 29, 2026
Another one - occurred AFTER merging #8677 in run https://github.com/FlowFuse/flowfuse/actions/runs/36539851763/job/109312964381
Quick looksee:
the PUT
/:instanceIdhandler withsourceProjectin the body replies 200 immediately, then runsexportProjectToExistingProject(...)in the background, unawaitedproject.js:391-393.That background job suspends the target project (stub STOP_DELAY = 250ms) → copies node/flow/credential/settings data across several sequential DB reads and writes → resumes it (kicks off START_DELAY = 500ms, though the DB row's state actually flips to 'running' earlier, inside
unSuspendProject, before the 500ms driver delay even starts).The test waits a fixed STOP_DELAY + START_DELAY + 50 = 800ms
project_spec.js:2146and assumes that's enough for the whole background job to finish. That budget only accounts for the two stub driver timers - it has no slack for the half-dozen-plus real DB round trips in between (StorageSettings.byProjectx2, a save/create,getSettingx2 more,updateSetting,project.save(),project.reload(), two audit log writes). On SQLite those are practically instant. On Postgres, under CI connection-pool contention, they can add extra latency to blow through the 800ms window, so the GET still sees state: 'suspended'.This same sleep(STOP_DELAY + START_DELAY + ...) pattern shows up roughly a dozen times in project_spec.js (lines 1359, 1426, 1611, 1680, 1718, 2061, 2146, 2218, 2279, 2327, 2381...), all racing the same fixed duration against variable Postgres latency. This one just has the least headroom, since it's the only call site with a full settings/flow/credential copy pipeline stuffed into the wait.
Steve-Mcl commentedon Sep 29, 2026
This one should be fettled by #8677
Steve-Mcl commentedon Sep 29, 2026
This is covered in #8677
cstns commentedon Sep 29, 2026
New one, on PR #8681 (run 36555287765, Postgres tests job):
test/unit/forge/ee/routes/ha/index_spec.js:115, expected 'starting' to be undefined)The PR only touches MCP snapshot tools, so unrelated. Same fixed
sleep(START_DELAY + 100)pattern Steve described forproject_spec.js: the restart hasn't finished on Postgres when the inflight state gets checked. Followed by aConnectionManager.getConnection was called after the connection manager was closedfrom theproject.startedaudit log write inforge/ee/routes/ha/index.js:101, so the background restart was still running when the app closed. Re-running the job.EDIT by @Steve-Mcl
Got this flake again in PR #8736 (run: https://github.com/FlowFuse/flowfuse/actions/runs/36750176035/job/110007319876?pr=8736)
cstns commentedon Sep 29, 2026
A Cypress (UI OS) one, on PR #8682 (run 36555697806, UI OS tests job), failed all 3 attempts:
test/e2e/frontend/cypress/tests/team/instances.spec.js:85, timed out after 4000ms looking for 'instance-1-1' in the dashboards table)The PR only touches the MCP applications tool, so unrelated. The test only waits on
@getTeambefore looking in the table, not on thedashboard-instancesrequest that fills it, so a slow response loses the race. Might be worth adding an intercept +cy.waitfor that call like the test above it does. Re-running the job.cstns commentedon Sep 29, 2026
New one, on PR #8691 (run 36556568977, Postgres tests job):
test/unit/forge/db/controllers/AccessToken_spec.js:479, expected null to exist)The PR only touches MCP broker tools, so unrelated to it. This one might not be just a test timing thing though: two concurrent
refreshToken()calls on Postgres and one of them came back null, which is exactly what the test says shouldn't happen. Might be worth checking whether the row lock / conditional update in the refresh path holds up under Postgres isolation. Re-running the job.Also a repeat of an existing entry, on PR #8689 (run 36556397782, Postgres tests job): 3rd Party Broker API > Topic Storage API > Verify topics all correct (
3rdPartyBroker_spec.js:443, expected length 4 got 2). Still happening.Steve-Mcl commentedon Oct 1, 2026
on PR #8719 (run https://github.com/FlowFuse/flowfuse/actions/runs/36840866078/job/110300052886?pr=8719)
1 remaining item
cstns commentedon Oct 1, 2026
New one, on PR #8684 (run 36867544025, Postgres tests job):
test/unit/forge/comms/devices_spec.js:192,activeSnapshotIdwas still null after the status message)The PR only touches the MCP device group tool, so unrelated. The test emits a
status/devicemessage and then does a fixedsleep(100)before reloading the device. On Postgres the handler was still mid flight, theDevice.findOneinsidesendDeviceUpdateCommandran after the app closed and took theafter allhook down with it (done() called multiple times). Third distinct test in thisDeviceCommsHandler > Device Statusblock to show up here, after the two andypalmi and Steve posted. Same fixed sleep vs background work pattern as the rest. Re-running the job.andypalmi commentedon Oct 2, 2026
Another flaky: Owner can add and remove devices from an existing group. Run and PR #8751
Details
Update Device Group Membership
Owner can add and remove devices from an existing group:
AssertionError: expected Device {
dataValues: Object {
id: 35,
name: 'device-117',
type: 'unnamed-type',
lastSeenAt: null,
state: '',
mode: 'autonomous',
TeamId: 2,
ApplicationId: 38,
ProjectId: null
},
_previousDataValues: Object {
id: 35,
name: 'device-117',
type: 'unnamed-type',
lastSeenAt: null,
state: '',
mode: 'autonomous',
TeamId: 2,
ApplicationId: 38,
ProjectId: null
},
uniqno: 1,
_changed: Set {},
_options: Object {
isNewRecord: false,
_schema: null,
_schemaDelimiter: '',
include: undefined,
includeNames: undefined,
includeMap: undefined,
includeValidated: true,
raw: true,
attributes: Array [
'hashid',
'id',
'name',
'type',
'links',
'lastSeenAt',
'state',
'mode',
'TeamId',
'ApplicationId',
'ProjectId',
'ownerType'
]
},
isNewRecord: false
} to have property id of 33 (got 35)
at Assertion.fail (node_modules/should/cjs/should.js:275:17)
at Assertion.value [as property] (node_modules/should/cjs/should.js:356:19)
at Context. (test/unit/forge/ee/routes/api/applicationDeviceGroups_spec.js:1038:56)
at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
andypalmi commentedon Oct 9, 2026
The AccessToken one again, on PR #8823 (run 37928022373, Postgres tests job). It failed on 3 attempts in a row:
test/unit/forge/db/controllers/AccessToken_spec.js:621, expected null to exist)The PR doesn't touch tokens, so it's unrelated. Following up on the earlier comment that this might be a real bug: it is, a race in
refreshToken()inforge/db/controllers/AccessToken.js.UPDATE, then writes theAccessTokenRefreshRotationrecord in a separate query afterwards.UPDATEmatches 0 rows, so it looks the presented token up in the rotation table.null. A real client refreshing at the same moment would see that as a failed refresh.It only shows on Postgres because the test SQLite database runs every query over one connection in the order issued, so the winner's insert always lands first. On Postgres the queries run on separate pooled connections, in parallel.
Possible fix: run the rotate
UPDATEand the rotation insert in one transaction. The losingUPDATEthen waits on the row lock until the winner commits, so the rotation record is there when it looks.cstns commentedon Oct 14, 2026
New one, on PR #8920 (run 38565957659, Postgres tests job):
test/unit/forge/routes/api/team_spec.js:1802, onlyteam.agent-auto-deploy.enabledwas in the audit log,team.settings.updatedmissing)The PR only adds a pipeline stage deleted audit event, so unrelated. The same commit sits under #8924 and #8925 and passed Postgres there, and SQLite passed on this run too. Already hit once on 2026-10-13 on a dependabot branch (run 38444754408). The test is new, came in with #8862 last week.
Looks like a timing thing in the route: the
agentAutoDeploy.enabledentry is awaited inforge/routes/api/team.js:940, but the genericteam.settings.updatedwrite at the end of the handler (team.js:980) isn't, so the reply can go out before that row lands and the test'sfindAllright after only sees the first entry. On SQLite everything goes through one connection in order so it always wins. Might be worth awaiting that call like the other one. Re-ran the job.cstns commentedon Oct 14, 2026
Two Cypress (UI OS) ones, on PR #8915 (run 38538676049, UI OS tests job). Both in
test/e2e/frontend/cypress/tests/devices.spec.js, both failed all 3 attempts, and both passed clean on re-run:devices.spec.js:510, timed out after 4000ms looking for[data-action="device-assign-to-instance"]in the kebab menu)devices.spec.js:600, same thing for[data-action="device-assign-to-application"])The PR only touches the Brokers page and the 3rd party broker route, so unrelated. The test opens the kebab on the unassigned device row and immediately looks for the option in the kebab options list, so it reads like the menu didn't finish opening (or the row's kebab click landed on the wrong element after the scrollTo) before the lookup gave up. First time I've seen this one. Re-ran the job and it went green.
cstns commentedon Oct 15, 2026
New one, on PR #8914 (run 38719485082, Backend tests job, so SQLite this time):
test/unit/forge/ee/routes/teamBroker/index_spec.js:28, uncaughtConnectionManager.getConnection was called after the connection manager was closed, then theafter allhook fell over onapp.close()and mocha bailed withdone() called multiple times)The PR only touches the device group description view and MCP tool, so unrelated. The error isn't from this spec at all. The stack goes back to the
setTimeoutin the team broker suspend route (forge/ee/routes/teamBroker/3rdPartyBroker.js:534), which deletes the agent row 1.5s after replying. Thesuspend marks the team-broker agent suspending before stopping ittest in3rdPartyBroker_spec.jsis one of the last tests in that file and drives that route, so the timer is still pending when the file'saftercloses the app, and it fires into whichever test is running next. It landed inindex_spec.jsthis time. Those suspend tests came in with #8917 yesterday, and main passed on the same code this morning, so it comes down to whether the last test and theafterhooks finish inside the 1.5s. Might be worth keeping a handle on that timer soapp.close()can clear it, or having the test wait the delete out. Re-ran the job.cstns commentedon Oct 15, 2026
Repeat of the AccessToken one, on PR #8930 (run 38730098693, Postgres tests job):
test/unit/forge/db/controllers/AccessToken_spec.js:620, expected null to exist)The PR only touches the instance rollback route and its tests, so unrelated. Same race andypalmi wrote up above, still happening. Re-ran the job.
Edit: hit again fifteen minutes later on PR #8924 (run 38730818569, Postgres tests job), same test, same assertion. That PR only touches pipeline stage audit logging. Two in one hour, so it's getting more frequent. Re-ran that one too.
Edit 2: and a third time today on PR #8937 (run 38749673728, Postgres tests job), same test. That PR only touches the tables route and MCP tool. Three hits in about three hours across unrelated PRs. Re-ran it.
cstns commentedon Oct 16, 2026
Two repeats this morning, both on the stacked pipeline audit PRs:
test/unit/forge/db/controllers/AccessToken_spec.js:621, expected null to exist), on PR Write an audit log entry when a pipeline stage is updated #8924 (run 38877965894, Postgres tests job). Same race andypalmi wrote up, fourth hit in two days. Re-ran the job.test/unit/forge/ee/routes/teamBroker/index_spec.js:37, uncaughtConnectionManager.getConnection was called after the connection manager was closed, thendone() called multiple timesin theafter allhook), on PR Write an audit log entry when a pipeline stage is deleted #8920 (run 38877966477, Backend tests job). Same leaked suspend timer from3rdPartyBroker.jsas yesterday, second hit. Re-ran the job.Edit: the AccessToken one failed again on the re-run of that same job (attempt 2,
AccessToken_spec.js:620, same assertion). Two in a row on one PR. Re-ran it once more.Edit 2: and again on PR #8943 (run 38897005227, Postgres tests job),
AccessToken_spec.js:621, same assertion. That PR only touches device target snapshot validation. Four Postgres runs hit this today out of the six I watched. Re-ran the job.cstns commentedon Oct 16, 2026
Repeat of the broker topic one, on PR #8949 (run 38916285914, Postgres tests job):
test/unit/forge/ee/routes/teamBroker/3rdPartyBroker_spec.js:722, expected [] to have length 1)The PR only touches pipeline stage audit log values, so unrelated. Same test as the
:697entries above, the line just moved with the suspend tests that came in last week. Re-ran the job.cstns commentedon Oct 19, 2026
Repeat of the Search API one andypalmi posted, on PR #8960 (run 39234578629, Backend tests job, so SQLite):
test/unit/forge/routes/api/search_spec.js:269, expected 0 to be 1 onresult.count)The PR only touches the MCP HTTP endpoint auth and its tests, so unrelated. One guess at why it comes and goes: the route in
forge/routes/api/search.js:63tries to decode the query as an Application hashid first and only falls through to the Device decode if that fails. The hashid salt includes the platforminstanceId, which is random per test app, so every now and then a device hashid also happens to decode as a "valid" Application id, the route takes the application branch with a bogus id and finds nothing. Might be worth checking which branch it took when it fails, or decoding against the expected model only. Re-running the job once the Postgres job on that run finishes.