Conversation
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
davidbudnick
added
type: bug
|
Thanks — the diagnosis is right, and the evidence here is what pointed at the real fix. I've taken a different route in #1323: instead of reconciling on a timer, the gate subscribes to powerd through Since your setup reproduces both the blink and the relaunch loop, could you run #1323 through a few lid closes? Look for @mayaanhafeez the same build should cover your Bolt setup too. The stale-snapshot point is a separate issue; I'll open it. |
|
Field failure on this branch's build (87cb4f31 = master + #1282 + this PR), 2026-09-11 20:16 PDT. The gate closed on a
Agent log (UTC): Nothing after that. The gate stayed shut until a manual kickstart at 03:20:26, four minutes in which Mechanism. Plugging the dock in dark-woke the machine; macOS promoted that to a full wake to show a notification and then posted The read-only probe that reads the same 922773e. A level read may now discharge Three details the relative proof needs to be worth anything:
2a75877. Fixes a pre-existing flake in the same file: the two tray tests that drive notifications both registered on, and posted to, the process-global Checks on macOS 26.6.2 / aarch64 with |
…ification The agent's hardware gate is driven by five NSWorkspace notifications: WillSleep / ScreensDidSleep / SessionDidResignActive close it, ScreensDidWake / SessionDidBecomeActive reopen it, and `CGDisplayIsAsleep(CGMainDisplayID())` seeds it at launch. Both halves of that are unsound, and a user's log shows each failing on the same machine within one evening. The notifications are edges, and the workspace center guarantees neither delivery nor pairing. On 2026-09-05 a one-second display blink at unlock logged "display/session suspended — pausing device I/O" with nothing after it, and the agent then ran 66 minutes with the gate shut while the machine was in use; `openlogi list` reported every paired receiver slot as "Unknown device" because the inventory probe was gated off. Restarting the agent was the only recovery. The line did not name the notification that suspended it, so the report could not say which edge went missing. The launch snapshot is worse. Across a 2.5 h display sleep the agent was relaunched 27 times, and every one logged "display/session resumed — enabling device I/O" — so CGDisplayIsAsleep answered false for the whole blank that NSWorkspaceScreensDidSleep had reported correctly a second earlier. Reading the full active list changes nothing: with the lid shut the external panel is both the main and the only online display. Nor is there a registry node to read instead — on Apple Silicon IODisplayWrangler carries no IOPowerManagement dictionary. CGDisplayIsAsleep has false negatives here; it is trustworthy only when it says asleep. So the gate now closes on the notifications and reopens on proof: - kCGSSessionOnConsoleKey is the level whose edges SessionDidBecomeActive / DidResignActive announce, and it is trustworthy in both directions, so it discharges the session source on its own. - Any HID input wakes a sleeping display, so CGEventSourceSecondsSinceLastEventType on the HID system state is what proves one is on. Which comparison is valid depends on what the owner knows. With no display-sleep report — a relaunched agent — only an idle timeout could have blanked the panel, and one minute is the shortest blank macOS can be configured for, so input newer than that rules it out. Once ScreensDidSleep *has* been recorded that argument is void, because a hot corner blanks the display a second after the last keystroke; there, only input newer than the suspension itself proves the display came back. - One reconciler thread parks on a condvar and wakes only while a discharge-able suspension has stood for two seconds. It performs no timed work and issues no CoreGraphics call while the gate is open or during a system sleep. - SYSTEM_SLEEP is never discharged by a level read. The process runs during a maintenance DarkWake, and opening HID there is what promoted an invisible wake into a full display wake (AprilNEA#656), so a system sleep is still cleared only by the notification that pairs with it — and the reconciler is parked, not polling, for the whole of it. - finish_startup fails closed: an unproven display keeps STARTUP held rather than resuming. ScreensDidWake discharges it directly when the display really returns. - Suspend and resume log lines now name their sources, so the next report of either shape is diagnosable from the agent log. Composes with AprilNEA#1164 rather than duplicating it: that PR ties the input hook to this same gate, so both a missed wake and an unproven launch would take the button remaps down with device I/O. The proof reopens the gate through the same DeviceIoSignal, so its lifecycle task reinstalls the tap with no extra wiring. Refs AprilNEA#952
…I/O gate The window-server levels the gate proves itself back from cannot tell a DarkWake from a full wake. Closing the lid on this machine puts the Mac into a DarkWake seconds after the user was last at it: the session is still on console, the HID idle timer still reads well under the one-minute display-sleep floor, and the external panel re-enumerates under a fresh CGDirectDisplayID that reports itself awake. Every level says "the user is here" while nothing is on screen. Read IOPMrootDomain's "System Capabilities" property alongside them. A readable value without kIOPMSystemCapabilityGraphics means a DarkWake, and then nothing may be discharged — not the startup hold, not a screen sleep, not an inactive session. An unreadable or missing property is a third state that proves nothing either way, so a macOS that renames the key degrades to the old behaviour instead of wedging the gate shut. The capability bits are public in <IOKit/pwr_mgt/IOPM.h> and generated by objc2-io-kit; only the registry key that carries the current set is undocumented, and reading it is an ordinary IORegistry lookup needing no entitlement. Also corrects the CGDisplayIsAsleep note: it is right about an ordinary idle blank and wrong after a display reconfiguration, which is a narrower claim than the one it carried.
…rives
The gate closed on NSWorkspaceWillSleepNotification and could only be
reopened again by ScreensDidWake or SessionDidBecomeActive. Those are not
guaranteed partners. On 2026-09-11 at 20:16 PDT this machine ran the
sequence pmset logged as
20:16:13 DarkWake from Deep Idle : due to USB-C_plug 1 secs
20:16:14 Wake DarkWake to FullWake : due to Notification
20:16:15 Sleep Entering DarkWake state due to
'Notification Wake Back to Sleep' 6 secs
20:16:21 Wake DarkWake to FullWake : due to HID Activity
— a dock hotplug, a notification that promoted the DarkWake to a full
wake, and the sleep back out of it. The agent logged
03:16:14.506 display/session resumed — enabling device I/O cleared=startup
03:16:15.560 display/session suspended — pausing device I/O source=system-sleep
and then nothing. The screens had never slept and the session had never
resigned, so neither wake notification had anything to announce and
neither fired. SYSTEM_SLEEP stayed held, `reconcilable` refused every
level read while it was, and the reconciler parked on the condvar for
good: `openlogi list` reported only the UVC camera — which is not behind
the gate — while WindowServer held a UserIsActive assertion from the
mouse the agent had stopped talking to. A manual kickstart four minutes
later was the only recovery. Any aborted or cancelled sleep strands the
gate the same way.
Excluding SYSTEM_SLEEP from reconciliation was the wrong shape of
defence against AprilNEA#656. What must not happen there is opening HID during a
maintenance DarkWake, and a DarkWake is a state the levels can now name:
`system_wake_is_proven` discharges a system sleep only on a positive
kIOPMSystemCapabilityGraphics read, with no display reporting itself
asleep, on console, and input newer than the suspension. A DarkWake
reads SystemGraphics::Down and discharges nothing, so the reconciler can
tick through one without doing anything but reading levels — the same
window-server and IORegistry reads it already performs through every
screen sleep, which is also a DarkWake. A real sleep is unaffected: the
process is frozen, so nothing runs at all until the machine is back.
Unlike every other source, this one refuses SystemGraphics::Unknown.
Elsewhere the capability read only vetoes levels that are themselves the
proof, so an unreadable key degrades to them; here it *is* the proof,
and the state it rules out is exactly the one every other level gets
wrong. A macOS that renames the key loses this recovery rather than
trading it for the regression.
Three things the relative proof needs to be worth anything:
- A five-second margin. Between WillSleep and the freeze — 1 s here, but
macOS gives every observer up to 30 s to return, so there is no bound
worth trusting — graphics are still up and a mouse brushed there does
not cancel a sleep already under way. Input that close to the
suspension is not accepted, or the gate opens on the way *into* a
sleep and puts a full HID enumeration on the wire as the machine goes
down. It is its own constant rather than a multiple of
RECONCILE_INTERVAL: the tick is a responsiveness knob, and shortening
it to make a startup hold recover faster must not shrink this.
- Every suspend edge restarts the clock, not only one that records a
source the set does not already hold. Otherwise a SYSTEM_SLEEP left
standing by an aborted sleep hands its whole age to the next sleep
attempt, and input a moment into that one clears an hours-old
held_for. Restarting unconditionally only ever makes a proof stricter;
SESSION_INACTIVE, the one source with no relative proof, does not care
either way. `suspend_from_at` opens the same explicit-`now` seam on
the recording side that `discharge_at` already opens on the proving
one, so which instant an edge writes is testable.
- The suspension is timed on a clock that keeps running while the
machine is asleep. std::time::Instant is CLOCK_UPTIME_RAW on Darwin
and stops at the freeze, and CGEventSourceSecondsSinceLastEventType
measures idle time against that same uptime clock, so a real sleep
would be invisible to held_for: whatever the sleep's length, the input
that woke the machine would sit a fixed pre-freeze gap *behind* the
suspension and could never prove the wake. Darwin's CLOCK_MONOTONIC is
the same monotonic clock with sleep counted in. Before any sleep the
two agree exactly, so the pre-sleep margin above is unaffected.
STARTUP, SCREEN_SLEEP and SESSION_INACTIVE keep their existing proofs
untouched. The reconciler's existing "a wake notification never arrived;
reconciled" warning now also covers system-sleep, so this recovery is
visible in a user's log.
Also drops a stale clause from the on-console comment: the input hook
does not follow this gate on master — that is AprilNEA#1164.
Refs AprilNEA#1281, AprilNEA#656
…e center `cargo test -p openlogi-agent` failed 6 runs in 10 on this machine, always on `startup_stays_suspended_when_the_display_is_already_asleep`, and never in a way either test could see. Both tests that drive notifications registered their `ActivityTarget` on the process-global `NSWorkspace` notification center and posted to it filtered by the one shared workspace object. That filter is not an isolation boundary: the two targets are equally valid observers of the same name/object pair, so whenever the two tests overlapped, the `WillSleep` / `ScreensDidSleep` / `SessionDidResignActive` posts of the overlapping-sources test also suspended the startup test's gate, and its own `ScreensDidWake` then had two sources to clear instead of one. Nothing in these tests needs the workspace's own center — the names are ordinary `NSNotificationName`s. So the registration moves into `observe_activity(center, object, signal)`, with `install_activity_observer` passing the workspace pair production actually uses, and each test running that same function on a private `NSNotificationCenter` with an `NSObject` sentinel of its own. No shared state left to collide over, and no serialization needed. Pre-existing on master; fixed here because the file is under change and the local gate has to be green. 12 consecutive runs of `cargo test -p openlogi-agent` pass after this, against 4 of 10 before. Replaces the serialization added in 86e9997: with each test on its own center there is no shared state left to take a lock over, so the WORKSPACE_NOTIFICATIONS mutex and its two lock sites go with it.
Two findings from review of the system-sleep reconciliation, and both are about a level reading being judged against a suspension it does not describe. The reconciler reads the levels outside the lock — they are window-server round trips, and nothing else may block on them — and the discharge step then re-locked and measured `held_for` from whatever `suspension.since` said at that moment. Now that every suspend edge resets `since`, those two halves can come from different suspensions: a reading taken while an old one stood reports that the user has been present for a second, and measured against an edge that landed during the read it becomes input newer than a suspension it actually predates. A hot corner blanking the display while a stranded suspension is being reconciled would therefore reopen the gate one second after a `ScreensDidSleep` that was delivered perfectly correctly. The same read was also timed from its end rather than its beginning. The idle timer is sampled somewhere inside `read_levels`, so taking the instant afterwards inflates `held_for` by however long the window server took to answer, and compares a too-small idle against a too-large suspension age — the lenient direction, for a call with no bound on it. That it happens to be harmless today rests on `idle` being the last field `read_levels` evaluates, which nothing states and nothing enforces. Multi-second stalls in that layer are what AprilNEA#952 was about. So both facts a reading has to be judged against are captured together, before it is taken, as a `ReadStart`: the suspend generation in force, and the instant `held_for` is measured from. A generation counter rather than `since` itself, because two reads of CLOCK_MONOTONIC can return the same value, so equal instants are not proof that no edge intervened. The reconciler and the startup path both open a `ReadStart` before calling `read_levels` and hand it to `discharge`, which clears nothing when the generation no longer matches, logging the mismatch at debug so a repeated discard is diagnosable from a user log. `held_for` is now a strict lower bound. The pure decision functions are untouched — none of this is a question of what the levels mean, only of which suspension they belong to. Second, the pre-sleep margin was five seconds, which was a guess about how fast the slowest sleep-transition client on this machine acknowledges. IOPMLib documents the bound: a client registered for kIOMessageSystemWillSleep has 30 s to acknowledge before power management proceeds without it, so the machine can still be fully awake half a minute after the WillSleep the gate closed on. The margin is now that documented 30 s. The cost is that a stranded system sleep recovers about half a minute after the stranding rather than a few seconds, and it is paid in the one state where the user is at the machine producing exactly the input that ends it. The 2026-09-11 failure this recovers from ran four minutes and stopped only because the agent was restarted by hand. Refs AprilNEA#1281, AprilNEA#656
4489953 to
8b587e1
Compare
|
Rebased onto master (e846e6f) to clear the conflict. 00e9eac's shutdown routing in |
This branch has not been deployed
3 participants
Summary
The macOS device-I/O gate trusts two things it should not. Both failed on my machine in one evening. Fixes #1281, refs #952 (the relaunch half).
The notifications are edges.
WillSleep/ScreensDidSleep/SessionDidResignActiveclose the gate,ScreensDidWake/SessionDidBecomeActivereopen it, and the workspace center guarantees neither delivery nor pairing. When a suspend edge arrives and its partner never does, the mask keeps that bit for the life of the process: device I/O stays paused,openlogi listreports paired slots as "Unknown device", and restarting the agent is the only recovery.The launch snapshot is unsound.
CGDisplayIsAsleep(CGMainDisplayID())seeds the gate at startup, and it cannot see the state a relaunched agent actually lands in.Evidence
macOS 26.6.2, aarch64. MacBook in clamshell on one external display behind a Thunderbolt dock; MX Master 3 on a Unifying receiver plugged into that dock.
The blink (#1281). A one-second display blink at unlock logged
display/session suspended — pausing device I/Oat 23:04:39Z with nothing after it. The agent then ran 66 minutes with the gate shut while the machine was in use.pmset -g log:Entering DarkWake state due to 'Clamshell Sleep'at 16:04:39 local,DarkWake to FullWake … due to HID Activityat 16:04:40.The relaunch loop (#952). 43 watchdog exits in one day, every one inside the same cycle:
pmset:Entering Sleep state due to 'Maintenance Sleep'HID CGEventTap lifecycle did not make progress before deadlinedisplay/session resumed — enabling device I/Opmset:DarkWake from Deep Idle [CDNP] : due to ATC0.PMGRCIOWakeup— the dock the receiver is onEntering Sleepagain, and round it goes43 of 43 exits fit that shape (16 of them back to back over 18:43–18:59). So the stall that trips the watchdog is at the system sleep transition, not merely "while the display is off" — my #1282 description had that wrong. I am not claiming the relaunched agent's HID open is what wakes the dock; I have not proved that. What is certain is that the relaunches keep reopening the gate, and that the Mac did not stay asleep with the lid shut.
What
CGDisplayIsAsleepactually gets wrong. A read-only probe logged the display list and the levels every 5 s. For a plain idle blank at 18:42:25 the display left the active list and reportedasleep=1, and the agent handled that sleep and its wake correctly. The false negative is specifically a reconfiguration: at 18:43:40 the lid closed, the external display re-enumerated under a newCGDirectDisplayID(3 → 13), and that new id reportedasleep=0through every DarkWake until the real wake at 19:54. That is why all 43 relaunches opened the gate. Widening the read pastCGMainDisplayIDdoes not help — with the lid shut that panel is both the main and the only online display — and there is nothing to read instead: on Apple SiliconIODisplayWranglercarries noIOPowerManagementdictionary.The fix
The gate still closes on the notifications; it now reopens on proof.
kCGSSessionOnConsoleKey(CGSession.h) is the level whose edgesSessionDidBecomeActive/DidResignActiveannounce, and it is trustworthy in both directions, so it discharges the session source on its own — saying nothing about the display, exactly like that notification.CGEventSourceSecondsSinceLastEventTypeon the HID system state is what proves one is on. Which comparison is valid depends on what the owner knows. With no display-sleep report (a relaunched agent has no history) only an idle timeout could have blanked the panel, andpmset displaysleep 1is the shortest blank macOS allows, so input newer than 60 s rules it out. OnceScreensDidSleephas been recorded that argument is void — a hot corner blanks the display a second after the last keystroke — so there, only input newer than the suspension proves the display came back.CGDisplayIsAsleepis kept as a positive-only short circuit:trueis a definite "the user can see nothing";falseproves nothing. An empty or failed display list reads the same way, so a headless or screen-shared Mac still works.IOPMrootDomain'sSystem Capabilitiesproperty: a readable value withoutkIOPMSystemCapabilityGraphicsmeans a DarkWake, and then nothing may be discharged — not the startup hold, not a screen sleep, not an inactive session. An additional necessary condition, not a replacement.SYSTEM_SLEEPincluded, is reconcilable, becauseWillSleepis no more guaranteed a partner than any other edge: a sleep the system aborts, or one it enters and leaves without the screens or the session ever moving, gets no wake notification at all and strands the gate (see the 2026-09-11 field failure in the comments). What keeps [Bug]: External display wakes from sleep #656 out is the proof rather than the parking: a system sleep is discharged only on a positivekIOPMSystemCapabilityGraphicsread, with no display reporting itself asleep, on console, and input that landed more than 30 s past the suspension — IOPMLib's acknowledgment bound forkIOMessageSystemWillSleep, the documented ceiling on how long the machine can stay fully awake after the notification the gate closed on; a margin of its own, not a multiple of the reconcile tick. A maintenance DarkWake reads graphics-down and discharges nothing, and an unreadable capability set, which degrades gracefully for every other source, is refused outright here because the positive read is the whole of this proof. Suspensions are timed on a clock that counts sleep (CLOCK_MONOTONIC, notstd::time::Instant, which isCLOCK_UPTIME_RAWon Darwin), and every suspend edge restarts that clock and bumps a generation counter. The reconciler and the startup path capture the generation and the instant before they sample the levels, and a sample read under an older generation discharges nothing, so neither a real freeze, a repeatedWillSleep, nor an edge landing mid-read can leaveheld_fordescribing a different sleep than the one being proved.finish_startupfails closed: an unproven display keeps the startup hold.ScreensDidWakedischarges it directly when the display really returns.source=screens-asleep,cleared=system-sleep+screens-asleep), and the unproven-launch and reconciler lines carry the levels they decided from.On the capability read
The bit values are public —
<IOKit/pwr_mgt/IOPM.h>declareskIOPMSystemCapabilityCPU/Graphics/Audio/Networkandobjc2-io-kitgenerates them, so no discriminant is hardcoded. The key that carries the current set,"System Capabilities", is not in any SDK header; I want to be plain about that. Reading it is still an ordinary IORegistry lookup through the documented API (IOServiceMatching+IOServiceGetMatchingService+IORegistryEntryCreateCFProperty, the pairioreguses), not a private SPI, and needs no entitlement. Because the key is undocumented, an unreadable or missing property is a third state that proves nothing either way, so a macOS that renames it degrades to the levels above rather than wedging the gate shut forever.It reads 15 in full wake here, and
pmset's own log draws the same line (DarkWake … [CDNP]versusFullWake … [CDNVA], theVbeing video). The probe has since caught the DarkWake sample: during the 2026-09-11 'Notification Wake Back to Sleep' DarkWake (see the comments) it read 9 (CPU + Network, no Graphics), and 15 again ten seconds later after the HID wake. The bit does separate the two states.Changes
All of it in
crates/openlogi-agent/src/tray.rs. Suspensions are stamped with a smallContinuousInstantnewtype overclock_gettime(CLOCK_MONOTONIC)(hencelibcin the macOS dependency block), andsuspend_from_at/discharge_attake an explicit now so the clock rules are unit-tested too; the notification registration isobserve_activity(center, object, signal), which production calls with the workspace center and the tests call with a privateNSNotificationCenterand a sentinel object.ActivityTarget's ivars become anArc<ActivitySources>holding the suspend mask and the instant it was recorded (the relative proof needs both) plus a condvar — still one transition authority for the gate.ActivityLevels(console, displays-report-asleep, system graphics, idle) and the puredischarged_by/display_is_proven_awakedecide from injected readings, so every rule above is unit-tested without a display; the graphics level is a three-stateSystemGraphics { Up, Down, Unknown }rather than anOption<bool>.finish_startuptakes a reading and returnsStartupDisplay,ScreensDidWakealso discharges the startup hold, andrun_app_loopstarts the reconciler only afterfinish_startupso it cannot race the launch sequence.Cargo.toml:objc2-core-foundationwithCFDictionary/CFNumber/CFString;objc2-core-graphicsgainsCGError,CGEventSource,CGEventTypes,CGSession; newobjc2-io-kitwithstd/libc/pwr_mgt. All three are already in the workspace table.Cargo.lockgains two lines; thegpuipins are unchanged..claude/rules/objc-ffi.md's inventory row andunsafelist fortray.rscover the new CoreGraphics and IOKit reads.Testing
macOS 26.6.2, aarch64,
RUSTFLAGS="-D warnings":cargo fmt --all -- --checkcargo clippy -p openlogi-agent --all-targets -- -D warningscargo test -p openlogi-agent: 44 passed, repeated runs green, no--test-threads=1(the observer tests no longer share the global workspace center). New or reworked tests also cover the stranded system sleep reconciled by later input, the DarkWake andUnknownrefusals for that source, the 30 s margin including the boundary, a repeatedWillSleeprestarting the clock, and a suspend edge landing during a level read discarding that read; the earlier set covers the missed screen wake discharged by later input; a forced display sleep not discharged by input that preceded it (the case an absolute idle floor gets wrong); the missed session activation; theSYSTEM_SLEEPnon-regression for [Bug]: External display wakes from sleep #656; the unproven launch hold and that it does not mask a second suspend source; the 60 s floor including an unreadable idle timer failing closed; the positive-onlyCGDisplayIsAsleepshort circuit; a DarkWake with 27 s of idle discharging nothing from any held source, and the same launch in full wake discharging the startup hold; an unreadable capability set neither proving nor blocking anything; fast-user-switching; thereconcilabletruth table; the log-source rendering.cargo clippy --workspace --all-targets --exclude openlogi-ui --exclude openlogi-desktop --exclude openlogi-overlay -- -D warningscargo test --workspace --exclude openlogi-ui --exclude openlogi-desktop --exclude openlogi-overlay: 1163 passedRUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --document-private-items --exclude openlogi-ui --exclude openlogi-desktop --exclude openlogi-overlay --exclude openlogi-agentkCGAnyInputEventTypeis0xffffffff(objc2 generates theCGEventTypenewtype but not that constant), and run back to back the Rust path read 0.800 s idle against the probe's 0.814 s.system_graphics()run here returnsUp, matchingioreg -n IOPMrootDomain's"System Capabilities" = 15.Not run:
clippy/tests (macos)jobs:gpui_macos's build script fails here withcannot execute tool 'metal' due to missing Metal Toolchain. Nothing depends onopenlogi-agent, and both the code and the manifest change are insidecfg(target_os = "macos")for that leaf binary.clippy (linux),clippy (windows),wasm, MSRV: no cross targets locally. Hand-audited:mod trayis macOS-gated, every new item is private totray.rs, and the manifest edit is inside[target.'cfg(target_os = "macos")'.dependencies].display state unproven at launchwithgraphics=Downin place ofdisplay/session resumed, and the Mac staying asleep with the lid shut.Composes with #1164 (the input hook joins this same gate, through the same
DeviceIoSignal) and does not overlap #1236: the tests register on a privateNSNotificationCenter, never on the process-globalNSWorkspacecenter.