bug(server): WatchSandbox can hand out a resume cursor past an event it never delivered or reported #4829

User Story

I use OpenShell's WatchSandbox to follow sandbox logs and platform events together, through the SDK watch helpers that reconnect from the highest cursor received. I implemented the loss-aware resume contract in #3209 and the TypeScript (#3778), Go (#3662) and Python (#3711) helpers. During review of #3209, @varshaprasad96 found two ways the replay can give a client a cursor that skips an event it never received. We agreed they predate #3209 but only become visible through its resume contract, and I said I would file this follow-up for them. The SDK helpers and the planned E2E coverage depend on them being fixed.

Problem Statement

When a watch follows both logs and platform events, its replay comes from two buffers that share one cursor space. A client that receives cursor N assumes every retained event up to N was either delivered or reported, and reconnects with resume_after_cursor = N. Two cases on main (39d95b42) break that assumption.

1. Tail depths differ between sources. This is deterministic and happens with the defaults.
The initial tail takes up to log_tail_lines logs (default 200) and event_tail platform events (default 0) independently. The batch can then contain logs on both sides of a platform event that the platform tail left out. For example, with history log 1, platform event 2, log 3 and the default depths, the client receives logs 1 and 3 but not event 2, with no warning. Its cursor is now past event 2, so resuming will never deliver it.

2. The two buffers are read at different moments. This is a race.
Both the initial tail and the resume replay read the log buffer and the platform buffer under separate locks (resume: crates/openshell-server/src/grpc/sandbox.rs:2128-2142). An event published between the two reads can be left out:

1. Read the log replay. It ends at cursor 9.
2. Log 10 is published, then platform event 11.
3. Read the platform replay. It includes 11.
4. The client receives 11 and disconnects before log 10 arrives live.
5. The client resumes after 11. Log 10 is still retained but is never delivered.

In both cases the client gets neither a SandboxStreamWarning nor OUT_OF_RANGE.

Impact / Why This Matters

• The resume contract promises that every gap is either replayed or reported. These cases lose events silently.
• Case 1 needs no race: any watch following both sources with the default event_tail can hit it once a sandbox has platform events between log lines.
• The SDK helpers reconnect automatically from the highest cursor received, so callers can't detect the gap or work around it.
• Workaround: none that keeps resume. A caller can avoid resume_after_cursor and restart from a full tail after every disconnect, but it still can't tell what it missed.

Acceptance Criteria

• When a watch follows both sources, every event between the oldest and newest cursor in the initial batch is delivered. If events must be held back to guarantee this, a SandboxStreamWarning reports how many, per source.
• An event published while the gateway builds the initial tail or the resume replay is either delivered in that batch or reported. It never arrives live after a higher cursor from the batch.
• A client that reconnects from the highest cursor it received never misses a retained event below that cursor without a warning or OUT_OF_RANGE.
• Single-source watches (for example openshell logs --tail N) return the same events as today.
• Regression tests cover differing tail depths (including equal depths over differently positioned histories) and a publish between the two buffer reads on both the initial and resume paths.
• The WatchSandbox field docs and docs/observability/accessing-logs.mdx explain when fewer events than requested are returned and what the warning means.

Reproduction Steps

Case 1:

1. Create a sandbox and let it produce a log line, then a platform event (for example a policy update or a lifecycle change), then another log line.
2. Call WatchSandbox for it with follow_logs and follow_events set and log_tail_lines and event_tail left at their defaults.
3. The initial batch contains both log lines but not the platform event between them, and no warning.

Case 2 is a race between gateway tasks, so it can't be triggered reliably from the CLI. Forcing a publish between the two buffer reads in a unit test shows the loss deterministically. #3613 adds tests for both cases.

Environment

• OpenShell: built from source, main at 39d95b42. The code path is unchanged since fix(api): make WatchSandbox loss-aware and resumable #3209 merged.
• OS: Omarchy (Arch Linux), kernel 7.2.5, x86_64
• Deployment: found through source review and unit tests. The behavior is in the gateway's WatchSandbox replay, so it applies to any gateway and compute driver.

Logs

None. Neither case produces a log line or a warning, which is the problem.