Skip to content

feat(daemon): opt-in inactivity expiry for idle local device claims (#2833) - #2988

Open
thymikee wants to merge 2 commits into
mainfrom
feat/session-idle-expiry
Open

thymikee wants to merge 2 commits into
mainfrom
feat/session-idle-expiry

Conversation

@thymikee

@thymikee thymikee commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes #2833

On a host shared by several agents, a session that never runs close keeps its host-global device claim until its daemon stops, so other agents read DEVICE_IN_USE and cannot tell active from abandoned.

AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS=900000 agent-device open --platform ios

Off by default. When set, the owning daemon expires a claim-holding session that has taken no attached commands for that long: it settles under that session's own execution lock pair, releases the claim, and leaves a bounded marker so the next command answers SESSION_NOT_FOUND with details.reason: SESSION_IDLE_EXPIRED, naming the window and device. A session holding a remote lease (ADR 0007) or an active capture is excluded.

A settle that cannot confirm its claim gone holds the record back and retries a full window after it ends: forgetting it would strand a claim owned by a process that no longer knows it holds one. The lock pair belongs to the release, not to the sweep's wait for it.

Validation

tsc, format, lint, build, check:layering, check:affected --run, gate fallow, gate production-exports and the size ratchet green at 6eeab911b5. 33 files, ~1210 production lines.

Live run at ca9f8ff74e, disposable simulator, 8000ms window: a snapshot re-stamped and the claim survived 5s of an 8s window; expiry fired at idleForMs: 8004 with claim: "deleted", freeing the claim file and writing idle-expiry.json; close answered SESSION_IDLE_EXPIRED; a reopen reclaimed the device with no stale marker. Review rounds since touched shutdown reporting and test timing only.

Size · post-review findings

Review in cubic

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.85 MB 4.86 MB +9.2 kB
Package (unpacked) 4.85 MB 4.86 MB +9.2 kB
Package (download) 1.45 MB 1.45 MB +2.8 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 19.0 ms 19.6 ms +0.6 ms
CLI --help 55.6 ms 56.9 ms +1.4 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 22 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/daemon/session-store.ts">

<violation number="1" location="src/daemon/session-store.ts:315">
P2: The idle-expiry marker is not actually keyed uniquely by the session address: `a/b` and `a_b` both resolve to the same session directory. An expiry for one session can therefore overwrite or explain a later `SESSION_NOT_FOUND` for the other with the wrong owner, timeout, and released device; use an injective session-directory encoding for the marker path and keep read/write/clear consistent.</violation>
</file>

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread src/daemon/request-router.ts Outdated
Comment thread src/daemon/session-idle-tombstone.ts Outdated
Comment thread src/commands/schema/cli-help.ts Outdated
Comment thread src/daemon/session-idle-expiry.ts Outdated
Comment thread src/daemon/server/daemon-session-idle-expiry.test.ts Outdated
Comment thread src/daemon/server/daemon-session-idle-expiry.ts Outdated
Comment thread src/daemon/server/daemon-session-idle-expiry.ts Outdated
Comment thread src/daemon/__tests__/session-store.test.ts
Comment thread src/daemon/__tests__/session-idle-expiry.test.ts
Comment thread src/daemon/__tests__/session-idle-activity.test.ts Outdated
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at a8a764b. This needs another pass before merge, mainly around lock lifetime and the shutdown ledger.

In settleUnderBudget (https://github.com/callstack/agent-device/blob/a8a764b/src/daemon/server/daemon-session-idle-expiry.ts#L336), the settle races against settleBudgetMs inside the locked callback. When the budget wins, the callback returns and both locks are released, but settleExpiredSession keeps running underneath. A request that then takes those same locks — for example the returning client's open on the same session — can end up with a claim the late settle is still tearing down: buildNextOpenSession carries the existing deviceClaim forward (I'm inferring this from the if (deviceClaim) … pattern, not from reading that function directly), so the late settle's clearDeviceClaim/sessionStore.delete can delete the record the reopened session now depends on, and a snapshot admitted in the same window can have its session vanish under it. The guard at lines 415-423 only catches a remover that already deleted the record, not one that re-occupied it, and this path also skips onSessionExpired, so idleReap never gets re-armed. Every mutation a settle makes — teardown, claim clear, store delete, tombstone write — should happen while the reaper still holds the session and device lock pair; can the budget instead bound only the sweep's wait outside the locks, with onSessionExpired fired from the settle's own completion? A test that admits a request on the same lock pair right after the budget fires, and asserts it doesn't run until the settle ends, would pin this down.

Following on from that: if a stuck settle outlives one retry window plus the budget, nextDueMs (https://github.com/callstack/agent-device/blob/a8a764b/src/daemon/server/daemon-session-idle-expiry.ts#L242) returns a past-due time for an address still in settling, so arm() schedules a 0 ms timer, the sweep bails at the settling.has(address) check without moving retryNotBeforeMs, and the finally re-arms immediately — a tight loop that also forces a diagnostics flush each iteration. The existing test at daemon-session-idle-expiry.test.ts:698-703 reaches this state but only checks settleCalls===1, so the spin isn't caught. Shouldn't nextDueMs simply exclude any address currently in settling, with the settle's own completion doing the re-arm? A bounded sweep-count assertion in that over-budget test would confirm the loop is gone.

The new unattributable claim outcome also flows into the shutdown ledger's switch (https://github.com/callstack/agent-device/blob/a8a764b/src/daemon/device/daemon-shutdown-claims.ts#L66-L75), which sends any outcome it doesn't list to orphaned. Allocator-held or undecodable claims used to report as superseded and now report as orphaned, and the CLI then tells the operator to run device release --stale (https://github.com/callstack/agent-device/blob/a8a764b/src/cli/commands/daemon.ts#L81-L88) — a command that refuses allocator-held records with allocator-held-owner. Is this reclassification intentional, or should unattributable stay local to the idle reaper? Either keep the shutdown ledger mapping this case to superseded, or give it its own bucket with advice that doesn't point at --stale, and add a ledger test for whichever mapping is chosen.

Not blocking: the tombstone lookup in request-router.ts reads the unscoped session key while sessions are stored under a tenant-scoped name, the help text undercounts which capture kinds block expiry, several idle-expiry tests share a fixture that hides the no-claim branch, and a good chunk of daemon-session-idle-expiry.ts is control-flow narration rather than encoded invariants — these can be picked up or left as is.

Is there a simpler shape here? The linked issue asks to close an idle session "the normal way." Dispatching an internal close through the existing request pipeline under req.internal, re-checking isSessionIdleExpired inside the lock, and writing the tombstone on success would hold the lock pair for the whole close by construction, removing the first finding outright, and would let this drop settleUnderBudget, clearExpiredClaim, the CLAIM_GONE table, the unattributable outcome, and the daemonLeaving parameter — closer to 300 lines than the current ~1018. Proxy-lease expiry can't substitute since it's lazy at admission and never fires for a quiet client, so a timer still has to stay. The one thing that would need to change first: close needs a typed "only if still idle-expired" precondition and outcome (closed, not-expired, claim-unconfirmed) before the reaper can lean on it, and if close is to stop forgetting a session with an unconfirmed claim clear, that has to change for every caller, not just this one.

I didn't re-run tsc, lint, layering, fallow, or the affected tests, and the PR body's validation claims are unverified from my side. The live simulator run described in the PR body doesn't say whether the daemonLeaving:false finalize actually stopped the iOS runner, or whether any settle exceeded the 5 s budget on a real device — that's the precondition for the first two findings above. Smoke Tests was in progress with no failure excerpt at review time; this diff touches the smoke path through the activity stamp in request-execution-scope.ts's runAdmitted and the per-request reaper arming in daemon-runtime.ts, so a failure there would most likely trace to that change, though the reaper stays inert with the env var unset.

The main things to resolve before merge: keep the session and device locks held until the settle finishes, with the budget only bounding the sweep's wait outside the locks; stop the zero-delay re-arm while a settle is still in flight; and scope the unattributable outcome so daemon stop doesn't reclassify allocator-held claims.

@thymikee
thymikee force-pushed the feat/session-idle-expiry branch from a8a764b to ae5d445 Compare September 26, 2026 06:47

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files (changes from recent commits).

Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.

Fix all with cubic | Re-trigger cubic

Comment thread src/daemon/__tests__/request-router-idle-expired.test.ts
Comment thread src/daemon/server/daemon-session-idle-expiry.ts Outdated
@thymikee
thymikee force-pushed the feat/session-idle-expiry branch from ae5d445 to 9ad0a2b Compare September 26, 2026 09:46
@thymikee

Copy link
Copy Markdown
Member Author

This is a follow-up on the earlier review (a8a764b, #2988 (comment)), reviewed at 9ad0a2b. Finding 3 from that round is still open and hasn't been answered.

clearDeviceClaim (src/daemon/device/device-claims.ts:362-368) now returns 'unattributable' when a conflicting record has no decoded claim, meaning it's allocator-held or undecodable; on main this case returned 'ownership-changed'. The shutdown ledger's switch (https://github.com/callstack/agent-device/blob/9ad0a2b/src/daemon/server/daemon-shutdown-claims.ts#L74) only lists deleted, absent, and ownership-changed, so 'unattributable' falls to default and lands in claims.orphaned. That means on daemon stop, regardless of whether the idle-expiry option is on, an allocator-held or undecodable claim that used to be reported as superseded is now reported as orphaned, and orphanedClaimWarnings (src/cli/commands/daemon.ts:81-88) tells the operator to run device release --stale, which refuses allocator-held records with 'allocator-held-owner'. So a default-off option changes behavior on an existing route, with no docs and no test. Every consumer of DeviceClaimClearOutcome needs to classify each member explicitly, and that consumer set is the shutdown ledger plus the idle reaper's CLAIM_GONE table; session-close.ts, lease-lifecycle.ts, and device-claim-admission.ts ignore the result today. Can the ledger get an exhaustive switch, or a Record<DeviceClaimClearOutcome, bucket> like CLAIM_GONE, with 'unattributable' mapped to superseded (matching main) or to its own bucket whose advice doesn't point at --stale? A daemon-shutdown-claims.test.ts case with an allocator-held record asserting the bucket would pin this down.

Not blocking: the comment above sessionStore.delete at src/daemon/server/daemon-session-idle-expiry.ts#L479 still says a remover can get there after a settle budget already released the lock and left the task running unheld, but the delta makes that impossible since the lock pair now outlives the budget and daemon shutdown is the only lock-free remover, so that clause can be dropped, and this can be taken or left.

CI is green, all 19 checks pass, so there's no failing job to attribute here.

I did not run tsc, lint, layering, fallow, or the affected tests, and judged regression validity by reading the pre-delta a8a764b code against each new test. The PR body doesn't name the commit of the live simulator run, doesn't say whether the daemonLeaving:false finalize stopped the iOS runner, and doesn't say whether any settle exceeded the teardown budget on a real device; the over-budget lock-hold path is covered only by unit tests. I didn't verify that isSafeSessionSegment accepts the tenant-scoped address form ('tenant-a:idle-x') outside the added router test. I also didn't examine how long a returning client's request queues behind a stuck release that now holds the lock pair; that's the intended trade-off the prior review asked for, and I didn't measure it.

Before this can merge, 'unattributable' needs to be classified explicitly in the shutdown claim ledger and pinned with a test, so daemon stop stops sending allocator-held claims to device release --stale.

@thymikee
thymikee force-pushed the feat/session-idle-expiry branch from 9ad0a2b to ca9f8ff Compare September 26, 2026 13:41
@thymikee

Copy link
Copy Markdown
Member Author

Size justification

~1150 net production lines crosses the guide's ~700 threshold, so this went to an independent design review asking whether a smaller owning interface would suffice. It would not, and the two masses that make it big are both load-bearing:

Eager reclaim cannot be replaced by a lazy answer. A ~50-line design (activity field + error graft on DEVICE_IN_USE) explains an occupant but never frees it. device release --stale proves staleness from the owner's liveness, so it refuses exactly the record this reaper refuses to forget — a live daemon that lost track of its own claim. #1320 forbids reclaiming a verified live foreign owner, so only the owning daemon can end its own session, and a check at the next command never runs: the abandoning agent is by definition the one that stopped sending commands. Freeing the device requires a timer.

The lock/budget/deferral machinery is the feature, not its scaffolding. Settling outside the session+device lock pair either releases a device under a command that still holds it or deadlocks against a request holding the other key. Bounding the wait without tracking the expiry runs two teardowns of one session; bounding it with the locks detached lets a retried close join a teardown in progress. Each of those was reached through this review, not predicted.

Two reductions the review proposed were taken: repair finalization moved to the commit point (see the review-round comment), and the retry clock's threading collapsed to stamping at remember-time, which removed a parameter path across three functions and fixed a defect.

What the review offered as remaining candidates are not reductions. SessionIdleExpiryOutcome is the reaper's only outward claim about a release; controller.idleExpiryMs is read by the off-state test; noteSessionActivity's atMs default matches the sibling predicates it sits beside. A third-party seam the review flagged — three session-teardown variants — is real but predates this PR; consolidating it is separate work, and folding idle expiry into teardownDaemonSession would import the exact behaviors that make idle expiry correct (successor hand-off, delete-on-failure).

@thymikee

Copy link
Copy Markdown
Member Author

Findings fixed after the review round (ca9f8ff74e)

Two further review comments came in (both applied: error.code pinned on the marker tests, and the JSDoc moved onto the field it documents). Reading the whole feature afterward turned up three more, none caught by the bot:

A held-back settle published the repair transaction it was supposed to leave alone. settleIdleExpiredSession called finalizeRepairTeardown as part of releasing resources, before the reaper's claim-confirmed gate. That call commits the healed .ad and stamps COMMITTED on the record — and a write onto a committed transaction is an idempotent no-op (isRepairArmedWriteBlocked). So for a repair-armed session whose clear could not confirm, the settle's own promise that it "changes nothing, not even the session record" was false: the session survived with its transaction marked published, and no later teardown would ever publish it. Finalize now runs after the gate, at the point the expiry is committed to. Mutation: finalize back before the gate fails the new test; deleting it fails its sibling.

unattributable reached daemon stop through default:. The new DeviceClaimClearOutcome member silently reclassified shutdown reporting from superseded to orphaned. That happens to be right here — an exiting daemon's owner identity dies with the process, so --stale does resolve it — but the reason is the opposite of why idle expiry holds the same verdict back, and a default: arm hid that. The ledger's switch is now exhaustive with a never tail, and CLEAR_UNRECORDED is a named sentinel rather than an absent map entry. Verified: adding a hypothetical member to the union now fails to compile in both this switch and CLAIM_GONE.

A settle that outlives its window retried with no gap. The retry clock was captured before the locks, so a release running longer than the window wrote a deferral already in the past: the reaper armed at zero delay and re-ran a teardown that had only just failed, overwriting the fresher deferral a sweep that found the release in flight had computed for the same address. Deferrals are now stamped when the attempt ends, which also removed a parameter threaded through three functions. Mutation: an already-elapsed deferral fails the new test.

clearDeviceClaim's unattributable also taught me the same record means different things to a process that is leaving and one that is staying; that distinction is now written down where both readers can find it.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/daemon/server/daemon-shutdown-claims.ts">

<violation number="1" location="src/daemon/server/daemon-shutdown-claims.ts:60">
P3: The new `CLEAR_UNRECORDED` branch (sentinel set here and classified at `case CLEAR_UNRECORDED:`) has no test: `daemon-shutdown-claims.test.ts` only covers released, failed-teardown, superseded, undecodable ('unattributable'), and no-claim sessions, and `releaseClaim` has no other callers. Add a test that forces `clearDeviceClaim` to reject (e.g., `vi.mock`/spy the module, or arrange claim file state so `fs.unlinkSync` throws a non-ENOENT error) and asserts the session lands in `orphaned` via the sentinel.</violation>
</file>

<file name="src/daemon/server/daemon-session-idle-expiry.ts">

<violation number="1" location="src/daemon/server/daemon-session-idle-expiry.ts:479">
P3: `finalizeRepairTeardown` runs before the `delete` guard, so a settle whose delete reports false (superseded by shutdown) has already published the healed `.ad`, possibly written a repair tombstone, and stamped COMMITTED before learning the session was ended by someone else. Today that is harmless because the only racing remover (daemon shutdown at daemon-runtime.ts:200) finalizes the same live object idempotently, but it contradicts the settle's documented contract that "a failed settle... changes nothing" — this path returns `undefined` yet changes the record and publishes. Placing the finalize after the `delete` guard keeps the one-way commit conditional on this settle actually ending the session.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/daemon/server/daemon-shutdown-claims.test.ts Outdated
outcomes.set(session.name, await clearDeviceClaim(session.deviceClaim));
} catch (error) {
// An unrecorded outcome stays orphaned: the claim may still be on disk.
outcomes.set(session.name, CLEAR_UNRECORDED);

@cubic-dev-ai cubic-dev-ai Bot Sep 26, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new CLEAR_UNRECORDED branch (sentinel set here and classified at case CLEAR_UNRECORDED:) has no test: daemon-shutdown-claims.test.ts only covers released, failed-teardown, superseded, undecodable ('unattributable'), and no-claim sessions, and releaseClaim has no other callers. Add a test that forces clearDeviceClaim to reject (e.g., vi.mock/spy the module, or arrange claim file state so fs.unlinkSync throws a non-ENOENT error) and asserts the session lands in orphaned via the sentinel.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/server/daemon-shutdown-claims.ts, line 60:

<comment>The new `CLEAR_UNRECORDED` branch (sentinel set here and classified at `case CLEAR_UNRECORDED:`) has no test: `daemon-shutdown-claims.test.ts` only covers released, failed-teardown, superseded, undecodable ('unattributable'), and no-claim sessions, and `releaseClaim` has no other callers. Add a test that forces `clearDeviceClaim` to reject (e.g., `vi.mock`/spy the module, or arrange claim file state so `fs.unlinkSync` throws a non-ENOENT error) and asserts the session lands in `orphaned` via the sentinel.</comment>

<file context>
@@ -42,7 +57,7 @@ export function createDaemonShutdownClaimLedger(): DaemonShutdownClaimLedger {
         outcomes.set(session.name, await clearDeviceClaim(session.deviceClaim));
       } catch (error) {
-        // An unrecorded outcome stays orphaned: the claim may still be on disk.
+        outcomes.set(session.name, CLEAR_UNRECORDED);
         emitDiagnostic({
           level: 'warn',
</file context>
Fix with cubic

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 3f1b9e7: a claim clear that throws is reported orphaned, the one bucket with a working remedy.

It pins the sentinel the way you suggested second — arranging claim-file state so the unlink throws a non-ENOENT error — by spying fs.unlinkSync to throw EACCES inside the claim lock. That is exactly the branch releaseClaim's catch writes CLEAR_UNRECORDED for, and it is what your first suggestion (vi.mock/spy the module) also lands on; spying fs keeps it below the module seam so the real clearDeviceClaim still runs.

I measured the chmod variant you implied rather than guessing, and it does reach the sentinel (4ms, orphaned), so it would have worked. I still preferred the spy because the chmod reaches it through acquireProcessLock's mkdirSync on resolveDeviceClaimPath(key) + '.lock' — that is, through the lock, not through the clear. The assertion would then pass for a fault in lock acquisition, which is a different invariant than "the clear reported no verdict". The spy puts the fault on the unlink itself, inside the held lock.

The test asserts the full four-bucket shape, so the sentinel cannot migrate to another bucket silently. Its title is deliberately the inverse of the new unattributable case: orphaned is kept precisely because our own owner identity does die with the exiting daemon, which is the one proof --stale can still use here.

Comment thread src/daemon/server/daemon-session-idle-expiry.test.ts
// stamps COMMITTED onto the record, and a write onto an already-committed transaction is an
// idempotent no-op. Finalizing a settle that is being held back would therefore mark a still-live
// session's healed script as already published, and no later teardown would ever publish it.
params.sessionStore.finalizeRepairTeardown(session);

@cubic-dev-ai cubic-dev-ai Bot Sep 26, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: finalizeRepairTeardown runs before the delete guard, so a settle whose delete reports false (superseded by shutdown) has already published the healed .ad, possibly written a repair tombstone, and stamped COMMITTED before learning the session was ended by someone else. Today that is harmless because the only racing remover (daemon shutdown at daemon-runtime.ts:200) finalizes the same live object idempotently, but it contradicts the settle's documented contract that "a failed settle... changes nothing" — this path returns undefined yet changes the record and publishes. Placing the finalize after the delete guard keeps the one-way commit conditional on this settle actually ending the session.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/server/daemon-session-idle-expiry.ts, line 479:

<comment>`finalizeRepairTeardown` runs before the `delete` guard, so a settle whose delete reports false (superseded by shutdown) has already published the healed `.ad`, possibly written a repair tombstone, and stamped COMMITTED before learning the session was ended by someone else. Today that is harmless because the only racing remover (daemon shutdown at daemon-runtime.ts:200) finalizes the same live object idempotently, but it contradicts the settle's documented contract that "a failed settle... changes nothing" — this path returns `undefined` yet changes the record and publishes. Placing the finalize after the `delete` guard keeps the one-way commit conditional on this settle actually ending the session.</comment>

<file context>
@@ -473,6 +469,14 @@ async function settleExpiredSession(params: {
+  // stamps COMMITTED onto the record, and a write onto an already-committed transaction is an
+  // idempotent no-op. Finalizing a settle that is being held back would therefore mark a still-live
+  // session's healed script as already published, and no later teardown would ever publish it.
+  params.sessionStore.finalizeRepairTeardown(session);
   // `delete` reports whether a record was still here to remove. The usual request-path removers —
   // `close`, a replacing `open`, a lease-expiry teardown — all remove a session from inside
</file context>
Fix with cubic

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not moving it, but you are right that the comment above it was wrong. Corrected in 3f1b9e7.

The suggested placement is not safe. finalizeRepairTeardown does not only write files under the session dir: for a committable transaction it calls recordRepairFinalizeCloseIfCommitting → recordAction → resolveStoredSessionName(session) (src/daemon/session-store.ts:433), which finds the store address by map identity and falls back to session.name once the record is gone. Run after delete, that fallback names the wrong artifact directory for any session whose store address is tenant- or cwd-scoped — so the placement you propose moves the commit to the wrong path precisely in the case the guard detects.

And the cost of keeping it where it is, is nil: the only remover that can win this race is daemon shutdown, which tears the session set down by finalizing the same live SessionState object, and a write onto an already-committed transaction is the idempotent no-op described. Verified by mutation: moving the finalize below the delete guard changes no test outcome; moving it above the claim gate fails the held-back test. The gate worth pinning is the claim gate, and it is pinned.

What was genuinely stale is the prose, and I fixed the sentence you and the review both caught: a settle budget bounds only the sweep's wait, never the lock pair, so the "any of those three after a settle budget already released this lock" remover cannot happen. The comment now says shutdown is the only way here, and records why publishing precedes the guard. The settleExpiredSession header's "changes nothing, not even the session record" was overclaiming for the same reason and now describes the claim/record invariant it actually guarantees.

@thymikee
thymikee force-pushed the feat/session-idle-expiry branch from ca9f8ff to 1e92484 Compare September 26, 2026 13:56

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/daemon/__tests__/session-idle-expiry-harness.ts">

<violation number="1" location="src/daemon/__tests__/session-idle-expiry-harness.ts:113">
P2: `runUntilIdle` only sleeps a fixed `ms` after triggering the sweep; it never actually waits for the expiry to complete. The sweep is scheduled with a real `setTimeout(..., 0)` in `arm()` and its settle chain performs real filesystem work (`withDeviceClaimLock`/`acquireProcessLock`, claim unlink, record delete, tombstone write), so the downstream assertions in both test files (`claimFileHeld(...) === false`, `sessionStore.get(...) === undefined`, tombstone existence) can run before the settle finishes and flake under CI load. This also runs against the repo's explicit policy "unit tests must not wait real time" (vitest.config.ts), enforced by `slowTestThreshold: 500` and the ratcheting slow-test reporter — the two new test files together add ~20 fixed 10–60 ms sleeps. Await completion instead: poll for the expected outcome with an overall budget, or await a settle-completion signal (the `settleSession` seam already returns a promise), or drive the controller with fake timers via `vi.advanceTimersByTimeAsync` as the long-window test already does.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

claimFileHeld: (deviceClaim) => fs.existsSync(resolveDeviceClaimPath(deviceClaim.deviceKey)),
runUntilIdle: (controller, ms) => {
controller.noteSessionsChanged();
return new Promise((resolve) => setTimeout(resolve, ms));

@cubic-dev-ai cubic-dev-ai Bot Sep 26, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: runUntilIdle only sleeps a fixed ms after triggering the sweep; it never actually waits for the expiry to complete. The sweep is scheduled with a real setTimeout(..., 0) in arm() and its settle chain performs real filesystem work (withDeviceClaimLock/acquireProcessLock, claim unlink, record delete, tombstone write), so the downstream assertions in both test files (claimFileHeld(...) === false, sessionStore.get(...) === undefined, tombstone existence) can run before the settle finishes and flake under CI load. This also runs against the repo's explicit policy "unit tests must not wait real time" (vitest.config.ts), enforced by slowTestThreshold: 500 and the ratcheting slow-test reporter — the two new test files together add ~20 fixed 10–60 ms sleeps. Await completion instead: poll for the expected outcome with an overall budget, or await a settle-completion signal (the settleSession seam already returns a promise), or drive the controller with fake timers via vi.advanceTimersByTimeAsync as the long-window test already does.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/__tests__/session-idle-expiry-harness.ts, line 113:

<comment>`runUntilIdle` only sleeps a fixed `ms` after triggering the sweep; it never actually waits for the expiry to complete. The sweep is scheduled with a real `setTimeout(..., 0)` in `arm()` and its settle chain performs real filesystem work (`withDeviceClaimLock`/`acquireProcessLock`, claim unlink, record delete, tombstone write), so the downstream assertions in both test files (`claimFileHeld(...) === false`, `sessionStore.get(...) === undefined`, tombstone existence) can run before the settle finishes and flake under CI load. This also runs against the repo's explicit policy "unit tests must not wait real time" (vitest.config.ts), enforced by `slowTestThreshold: 500` and the ratcheting slow-test reporter — the two new test files together add ~20 fixed 10–60 ms sleeps. Await completion instead: poll for the expected outcome with an overall budget, or await a settle-completion signal (the `settleSession` seam already returns a promise), or drive the controller with fake timers via `vi.advanceTimersByTimeAsync` as the long-window test already does.</comment>

<file context>
@@ -0,0 +1,116 @@
+    claimFileHeld: (deviceClaim) => fs.existsSync(resolveDeviceClaimPath(deviceClaim.deviceKey)),
+    runUntilIdle: (controller, ms) => {
+      controller.noteSessionsChanged();
+      return new Promise((resolve) => setTimeout(resolve, ms));
+    },
+  };
</file context>
Fix with cubic

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3f1b9e7, using the barrier option rather than polling everywhere.

runUntilIdle's fixed sleep is no longer load-bearing on any assertion that a settle landed. Both files now drive completion through withinDiagnosticsScope: the scope wraps a sweep and awaits it, and a sweep configured with no settle budget awaits every session's settle to the end, so a scope whose run() resolved is a sweep whose work is finished. createSweepBarrier in the harness exposes that as await sweeps.swept(n).

Two cases cannot use it and are handled honestly:

  • Where a settle budget is configured, the scope resolves when the wait ends, not when the release does, so the barrier would resolve early. Those poll the outcome they assert via waitFor.
  • Where the assertion is that something did not happen, there is no fact to poll. Those keep a bounded wait, but each now says so at the call site and is paired with a completion assertion that proves the loop was live — e.g. the lock tests assert the settle count is 0 while held and then await the settle that happens after the lock is released, so the zero means "waited", not "never started". I had to learn this the hard way: a first cut converted one such bound to sweeps.swept(1) and it passed against a sweep that had merely skipped the address, so the barrier now waits for the settle itself there.

Inventory: the two files went from 43 fixed sleeps (16 + 27) to 16 bounded waits, and each of those now guards an assertion that something did not happen (settleCalls === 0, expired still empty, acquiredWhileStuck === false, "and it stays that way" after cancel()). Every assertion that something arrived is preceded by a barrier or a waitFor, so none of them can be reading a half-finished settle. Slowest test went 470ms → 111ms, and the ~20ms-margin "outlives its window" test is now fake-timer driven (vi.advanceTimersByTimeAsync) and has moved to the scheduling file, where it belongs by the file headers. It still fails on the pre-fix ordering, which I checked by routing both rememberRetry calls through a clock captured before the locks.

Comment thread src/daemon/server/daemon-session-idle-expiry-scheduling.test.ts Outdated
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 1e92484. This is a follow-up to the earlier review (#2988 (comment)).

The fix is only half done. unattributable is now named explicitly, but src/daemon/server/daemon-shutdown-claims.ts#L92 still routes it into claims.orphaned, and orphanedClaimWarnings in src/cli/commands/daemon.ts then tells the operator to run device release --stale. That command refuses every such record: releaseInspectedStaleClaim (src/daemon/device/device-claims.ts#L273) returns refused whenever !claim, and an allocator-held or undecodable record never carries a decoded claim. The stated rationale, that owner identity dies with the process so --stale proves it stale, does not hold here, because liveness is never consulted when there is no decoded claim to check. On main the same record was reported superseded. So on the default daemon stop route, an allocator-held or corrupt claim now surfaces as not released cleanly, with remediation advice that cannot succeed. Each DeviceClaimClearOutcome's shutdown bucket needs to name advice its owning release route can actually carry out, as enumerated by STALE_RELEASE_REFUSAL_REASONS / deviceClaimOwnerCannotRelease: map unattributable back to superseded to match main, or give it its own bucket pointing at device status --stale or the allocator instead of release --stale, and update the comments at lines 41-48 and 93-97 to match.

The new test in src/daemon/server/daemon-shutdown-claims.test.ts#L100 writes '{bad json', which is the inconsistent case, and asserts orphaned; it does not cover the allocator-held case the earlier review asked for. Its comment claims device release --stale resolves this state, but that route refuses it, so the test pins the wrong bucket and would also pass unchanged on 9ad0a2b, since default: already sent the outcome to orphaned there. Could you add a case that writes an allocator-held record over the session's claim file, the same shape inspectClaimContents classifies as allocator-held, and assert whichever bucket the fix settles on, keeping the undecodable case asserting that same bucket?

CI is green across all 19 checks. I did not run tsc, the affected tests, or the compile-fails-on-new-member mutation; exhaustiveness was judged by reading the never tail. The live run at ca9f8ff used a non-repair session with claim deleted, so it did not exercise repair finalize-after-gate or the unattributable shutdown bucket; that gap is not a device-facing difference beyond what the unit tests already cover. I also did not check whether the next open settles an undecodable claim, which orphanedClaimWarnings also names as a resolution path.

A few non-blocking notes: the "held-back settle leaves a repair transaction uncommitted" test at daemon-session-idle-expiry.test.ts#L267 stubs settleSession, so it doesn't reach the daemon-runtime settler where the pre-fix bug actually lived — the paired "committed" test is the one that fails on 9ad0a2b. The retry-after-end test uses a real clock with 20-60ms margins, which can flake under contention. That test also sits in the commit file, though the file header assigns scheduling to the other file. And the comment above sessionStore.delete still describes a lock-free remover running "after a settle budget already released this lock," which this change makes impossible — could you update it?

Before merge, unattributable needs to land in a daemon stop bucket whose advice route can actually release it (superseded, as on main, or a dedicated bucket), pinned with an allocator-held record test in daemon-shutdown-claims.test.ts.

@thymikee
thymikee force-pushed the feat/session-idle-expiry branch 2 times, most recently from 140a0e3 to 6453799 Compare September 27, 2026 06:32
#2833: on a host shared by several agents, a session that never runs `close`
keeps its host-global device claim until the daemon stops, and every other
agent reads DEVICE_IN_USE with no way to tell active from abandoned.

Set AGENT_DEVICE_SESSION_IDLE_TIMEOUT_MS to expire a claim-holding session with
no remote lease and no capture running for it, once it has taken no attached
commands for that long. The daemon that owns the session settles it under that
session's own execution lock pair, releases the claim, and leaves a bounded
marker so the next command answers SESSION_NOT_FOUND with details.reason
SESSION_IDLE_EXPIRED naming the window and the released device. Off by default;
a positive window below one millisecond is the shortest window, not off.

A settle that cannot confirm its claim gone holds the session record back and
retries a window later: this daemon stays alive, so forgetting the record would
strand a claim owned by a process that no longer knows what it holds, which
`device release --stale` cannot reclaim. `clearDeviceClaim` therefore gains
`unattributable` for a record that yields no attributable owner, because that
says nothing about a successor having taken the device.

The lock pair belongs to the release, not to the sweep's wait for it. A teardown
budget bounds only that wait, so one stuck recorder cannot hang a sweep while a
budget releasing the locks would let a retried `close` join a teardown in
progress, or a late release delete a session a retried `open` just created and
mark it expired. A sweep that finds a release in flight defers that address
rather than re-arming at a deadline already past, and a release that lands after
its budget still reports the session it freed.
…field

R7 holds every SessionState field to one declared writer. The #2833 request path
reports activity through `SessionStore.noteSessionActivity` rather than mutating
the record itself, so the store that owns the record is the only writer here and
the field now says so.
@thymikee
thymikee force-pushed the feat/session-idle-expiry branch from 6453799 to 6eeab91 Compare September 27, 2026 06:35
@thymikee

Copy link
Copy Markdown
Member Author

Re-reviewed and pushed as 3f1b9e7 (rebased on de00955358, same two-commit shape; enforcement edits still isolated to the chore(gates) commit).

The blocking item is fixed, with the dedicated bucket rather than superseded. You offered either mapping unattributable back to superseded to match main, or giving it its own bucket pointing at a route that works. I took the second, because the first restores the assertion the verdict exists to refuse: superseded means a successor owns the device, and supersededClaimWarnings tells the operator those devices "are now owned elsewhere" — for an undecodable or allocator-held record that is not known, and it is the conflation the outcome name was introduced to prevent.

unattributable is now its own bucket end to end: DaemonShutdownClaims.unattributable → the additive daemon-shutdown.json key (pinned by a tolerant-reader test) → DaemonStopResult.claimsUnattributable → its own daemon stop warning. That warning points at the default device status view, because deviceClaimRequiresStaleInspection is false for these classifications, so --stale hides them, and it states that no device-release route settles them rather than naming one that refuses. Ledger comments at the type, the ledger doc and the switch now describe that.

Both cases you asked for are pinned: the undecodable '{bad json' record and a real allocator-held record written through acquireAllocatorHeldDeviceClaim and re-inspected as allocator-held, both asserting the full four-bucket shape. Neither passes if the case is folded back into orphaned — I checked by mutating the switch.

Also corrected, each in its thread: the CLEAR_UNRECORDED sentinel test, the held-back repair test now running the second pass over the same live record, the vacuous in-window test driving a sweep that really runs, the real-clock sleeps replaced by a sweep-completion barrier (43 fixed sleeps → 16, every one of them now guarding an absence), and the sessionStore.delete comment — where the sentence about a budget releasing the lock was wrong and is fixed, but the suggested move of finalizeRepairTeardown below the guard is not safe, for the map-identity reason in that thread.

One thing I got wrong while working and corrected rather than shipped: I first justified the fs.unlinkSync spy over chmod by claiming chmod would burn the 30s claim-lock timeout. Measured, it reaches the sentinel in 4ms. The real reason is weaker and narrower — chmod reaches it through lock acquisition rather than the clear, so the assertion would pin the wrong invariant. That is what the thread says.

Gates at 3f1b9e7: tsc, format, lint, build, check:layering, gate fallow (clean over 33 changed files), gate production-exports, the test-file-size ratchet, and the affected lanes. Repeated runs of the two reaper files are stable; slowest test there is 111ms.

I did not run a fresh live device pass this round. The change is the shutdown-bucket reporting plus test timing, and the allocator-held and undecodable records cannot be produced by a normal simulator workflow — the unit tests cover them through the real claim store and the real decoder. Say the word if you want the simulator pass repeated anyway before merge.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 12 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/daemon/__tests__/daemon-shutdown-report.test.ts">

<violation number="1" location="src/daemon/__tests__/daemon-shutdown-report.test.ts:30">
P3: No test round-trips the new `unattributable` bucket with actual records: the write/read cases all use `unattributable: []`, and the non-empty values elsewhere (device-claims / daemon-shutdown-claims tests) only exercise the producer, not the report serialization. Put a claim in `unattributable` on one write+read pair so the new bucket's serialization (write spread + `readClaimSection` filter) is validated.</violation>
</file>

<file name="src/daemon/server/daemon-shutdown-claims.ts">

<violation number="1" location="src/daemon/server/daemon-shutdown-claims.ts:106">
P2: This reclassification removes the claim from shutdown diagnostics because the daemon emits no `unattributableDeviceKeys` field. Add the new bucket to that diagnostic so operators can identify claims whose ownership could not be determined.</violation>
</file>

<file name="src/daemon/__tests__/session-idle-expiry-harness.ts">

<violation number="1" location="src/daemon/__tests__/session-idle-expiry-harness.ts:163">
P3: `createSweepBarrier` returns a `sweeps()` counter that no test ever calls: the barrier's JSDoc advertises it as the proof that "a sweep really ran", but every test either awaits `swept()` (which already proves completion) or, where a sweep count is genuinely asserted, uses its own `withinDiagnosticsScope` closure (`sweeps++` in the long-window tests, `sweepsCompleted` in the re-stamp test). Remove the member (and its type/JSDoc mention) or use it in one of the `boundAbsence` tests.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

return;
default:
case 'unattributable':
claims.unattributable.push(record);

@cubic-dev-ai cubic-dev-ai Bot Sep 27, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This reclassification removes the claim from shutdown diagnostics because the daemon emits no unattributableDeviceKeys field. Add the new bucket to that diagnostic so operators can identify claims whose ownership could not be determined.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/server/daemon-shutdown-claims.ts, line 106:

<comment>This reclassification removes the claim from shutdown diagnostics because the daemon emits no `unattributableDeviceKeys` field. Add the new bucket to that diagnostic so operators can identify claims whose ownership could not be determined.</comment>

<file context>
@@ -90,12 +103,7 @@ export function createDaemonShutdownClaimLedger(): DaemonShutdownClaimLedger {
-          // and `--stale` would prove the claim live — the record means different things to a process
-          // that is leaving and one that is staying.)
-          claims.orphaned.push(record);
+          claims.unattributable.push(record);
           return;
         case CLEAR_UNRECORDED:
</file context>
Fix with cubic

writeDaemonShutdownReport(stateDir, {
providerReleases: { released: [lease], pending: [lease] },
claims: { released: [claim], orphaned: [], superseded: [claim] },
claims: { released: [claim], orphaned: [], superseded: [claim], unattributable: [] },

@cubic-dev-ai cubic-dev-ai Bot Sep 27, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: No test round-trips the new unattributable bucket with actual records: the write/read cases all use unattributable: [], and the non-empty values elsewhere (device-claims / daemon-shutdown-claims tests) only exercise the producer, not the report serialization. Put a claim in unattributable on one write+read pair so the new bucket's serialization (write spread + readClaimSection filter) is validated.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/__tests__/daemon-shutdown-report.test.ts, line 30:

<comment>No test round-trips the new `unattributable` bucket with actual records: the write/read cases all use `unattributable: []`, and the non-empty values elsewhere (device-claims / daemon-shutdown-claims tests) only exercise the producer, not the report serialization. Put a claim in `unattributable` on one write+read pair so the new bucket's serialization (write spread + `readClaimSection` filter) is validated.</comment>

<file context>
@@ -27,7 +27,7 @@ test('round-trips provider release and device claim records without lease creden
     writeDaemonShutdownReport(stateDir, {
       providerReleases: { released: [lease], pending: [lease] },
-      claims: { released: [claim], orphaned: [], superseded: [claim] },
+      claims: { released: [claim], orphaned: [], superseded: [claim], unattributable: [] },
     });
 
</file context>
Fix with cubic

});
}
},
sweeps: () => completed,

@cubic-dev-ai cubic-dev-ai Bot Sep 27, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: createSweepBarrier returns a sweeps() counter that no test ever calls: the barrier's JSDoc advertises it as the proof that "a sweep really ran", but every test either awaits swept() (which already proves completion) or, where a sweep count is genuinely asserted, uses its own withinDiagnosticsScope closure (sweeps++ in the long-window tests, sweepsCompleted in the re-stamp test). Remove the member (and its type/JSDoc mention) or use it in one of the boundAbsence tests.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/__tests__/session-idle-expiry-harness.ts, line 163:

<comment>`createSweepBarrier` returns a `sweeps()` counter that no test ever calls: the barrier's JSDoc advertises it as the proof that "a sweep really ran", but every test either awaits `swept()` (which already proves completion) or, where a sweep count is genuinely asserted, uses its own `withinDiagnosticsScope` closure (`sweeps++` in the long-window tests, `sweepsCompleted` in the re-stamp test). Remove the member (and its type/JSDoc mention) or use it in one of the `boundAbsence` tests.</comment>

<file context>
@@ -112,5 +144,35 @@ export function createIdleExpiryHarness(): Readonly<{
+            });
+          }
+        },
+        sweeps: () => completed,
+      };
+    },
</file context>
Fix with cubic

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 6eeab91. The earlier blocking finding is fixed: unattributable no longer lands in a daemon stop bucket that points at device release --stale, and the new tests pin that. I reviewed only the PR's own delta in src/daemon and src/cli, not the upstream files that came in with the rebase.

All 19 checks pass at this commit.

Two non-blocking notes. The "could not be read" warning text in src/cli/commands/daemon.ts#L110 also covers an allocator-held record with a readable principal, which --stale refuses as allocator-held-owner; should the wording say so? The reflowed doc comment in src/daemon/server/daemon-session-idle-expiry.ts#L446 leaves an orphan line after "That argument binds the".

I did not run the affected tests, tsc, or a mutation; I judged the regression tests from the 1e92484 code and the new assertions. Nothing else blocks a human review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opt-in inactivity expiry for local device claims

1 participant