Fix recovery sidecar LIST-to-GET race

[aaltshuler]

Summary

• add a single-GET optional text read that maps only the object store's typed NotFound to absence
• use it for read-write recovery discovery, read-only coherence discovery, and the under-gate sidecar reread
• add deterministic LIST-to-GET disappearance regressions plus local, in-memory, and S3 storage contracts
• document the concurrent-completion rule and keep malformed sidecars, permission failures, and other storage errors loud

Root cause

Recovery first listed __recovery/ and then read each sidecar body. A live writer could publish successfully and delete its sidecar between those operations. The subsequent GET returned NotFound, which was treated as a storage failure and could reject an otherwise unrelated write or read-only open.

The new optional read performs one backend GET. A typed NotFound means the listed recovery operation completed concurrently and is skipped; every other error retains the existing fail-loud behavior.

Validation

• cargo test --workspace --locked
• cargo test -p omnigraph-engine --features failpoints --test failpoints --locked — 129 passed, 1 ignored helper
• scripts/check-agents-md.sh
• git diff --check

The S3 contract remains bucket-gated and compiles in the workspace suite.

Closes #330

Greptile Summary

This PR closes the recovery sidecar LIST-to-GET race. The main changes are:

• Adds a single-GET optional text read that maps typed NotFound to absence.
• Uses the new read in recovery and read-only sidecar discovery.
• Adds deterministic failpoint, local, in-memory, and S3 coverage.
• Documents concurrent sidecar completion and loud error handling.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
crates/omnigraph/src/storage.rs	Adds the optional text-read contract and preserves loud handling for all errors except typed NotFound.
crates/omnigraph/src/db/manifest/recovery.rs	Uses the optional read for discovery and under-gate rereads to remove the sidecar read races.
crates/omnigraph/src/instrumentation.rs	Adds separate counting and forwarding for optional text reads.
crates/omnigraph/tests/failpoints.rs	Adds deterministic coverage for sidecars deleted between listing and reading.
crates/omnigraph/tests/s3_storage.rs	Extends the S3 contract with present, missing, and deleted optional-read cases.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant R as Recovery
    participant S as Object storage
    participant W as Writer
    R->>S: LIST __recovery/
    S-->>R: Sidecar URI
    W->>S: Publish manifest
    W->>S: DELETE completed sidecar
    R->>S: GET sidecar URI
    S-->>R: NotFound
    R->>R: Skip completed operation

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant R as Recovery
    participant S as Object storage
    participant W as Writer
    R->>S: LIST __recovery/
    S-->>R: Sidecar URI
    W->>S: Publish manifest
    W->>S: DELETE completed sidecar
    R->>S: GET sidecar URI
    S-->>R: NotFound
    R->>R: Skip completed operation

Loading

_{Reviews (1): Last reviewed commit: "Fix recovery sidecar read race" | Re-trigger Greptile}

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)