Skip to content

fix(collab): cancel shutdown storage waits - #335

Merged
isac322 merged 3 commits into
mainfrom
fix/shutdown-context-deadline
Nov 3, 2026
Merged

isac322 merged 3 commits into
mainfrom
fix/shutdown-context-deadline

Conversation

@isac322

@isac322 isac322 commented Nov 3, 2026

Copy link
Copy Markdown
Member

TL;DR

  • Make local CAS waits cancelable during hub shutdown.
  • Preserve live co-editing snapshots, leases, and concurrent writers.
  • Publish recovery blobs atomically in the affected SQL fixture.

Root cause

Local file CAS used a blocking flock that ignored its context. Shutdown could therefore remain blocked after its deadline, and background lease or snapshot work could keep using uncancelable contexts. The SQL release test also exposed a non-atomic fixture publication path that could leave a durable invalid-data row.

Changes

  • Use a nonblocking local CAS wait with context cancellation and a private 5ms retry interval.
  • Tie background co-editing work to a process context canceled before shutdown snapshots, while using the shutdown caller context for explicit final snapshots and lease release.
  • Propagate selected contexts through room storage and upload paths, preserving normal lease and history behavior.
  • Capture park retry settings before launching background work.
  • Publish the SQL recovery blob through the existing atomic backend operation.
  • Document the new shutdown behavior and lifecycle ownership in the changelog and architecture diagram.

QA checklist

ID Behavior Coverage
SD-LOCK A canceled real file CAS wait returns without altering the holder or bytes TestLocalCASLockCancellationPreservesHolderAndBytes, TestLocalCASAlreadyCanceledContextDoesNotWriteOrHoldLock
SD-CLOSE A dirty co-editing room returns from shutdown while an external CAS lock remains held; an uncontended close still saves the tail TestCollabCloseCancelsContendedFileCAS, TestCollabCloseWritesLiveRooms
SD-BG / SD-ONLOAD / SD-PARK Background renewals, request-bound room loading, and upload-backed park retries stop without leaving writes after cancellation Focused webapp lifecycle tests
SD-HEALTHY Existing CAS, lease, room convergence, and Redis relay behavior remains intact Existing remote and webapp suites
SQL fixture Recovery blob publication is atomic while the original completion assertion remains unchanged TestDataQuerySQLReportsAPassThatMissedAFile

Verification

Final local head: 87f3d2bc32fb4d845c90ae790561793f312e8d59.

  • Focused remote and webapp shutdown/lifecycle tests: passed.
  • go vet ./...: passed.
  • go build ./...: passed.
  • Full go test ./... -count=1 -timeout=45m: passed on Go 1.25.8, Darwin arm64.
  • Remote and selected webapp race tests: passed after capturing park retry settings before background execution.
  • PostgreSQL/SQL conformance and the repaired fixture invocation: passed on the existing PostgreSQL 16.15 fixture.
  • Pre-fix counterfactuals: real CAS cancellation tests fail on f9fd8830; the healthy CAS and graceful-close controls pass. The pre-fix shutdown test remains blocked until the external lock guard, while the fixed test returns with the external holder still protecting its lock.

Behavior and limits

A canceled CAS wait now returns its context error without unlocking another writer or changing bytes. Normal graceful shutdown still writes an uncontended dirty room. The change does not add a public API, configuration setting, dependency, or new timeout policy.

The shutdown probe covered the real local file backend. GCS, S3, network filesystems, Cloud Run signal delivery, and ygo shutdown internals remain unverified. The original SQL CI incident's exact branch remains inconclusive, while the non-atomic fixture hazard is confirmed and corrected. Existing browser, Windows, and unrelated symlink limitations remain outside this PR.

Not changed

  • DataIndex runtime behavior, browser behavior, symlink behavior, CLI flags, public backend interfaces, and the existing v0.18.2 version plan.
  • PR334 remains a separate changelog-only release preparation PR and must be revalidated on the final main branch after this PR merges.

Release status

v0.18.2 remains unpublished until this fix is reviewed, merged, PR334 is rebased or otherwise revalidated, and all release gates pass.

@isac322 isac322 self-assigned this Nov 3, 2026
@isac322
isac322 requested a review from runbear-bot November 3, 2026 18:57

@runbear-bot runbear-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bobb's bot(Totoro) Review

Conclusion: ready-with-notes

Reviewed head: 87f3d2bc32fb4d845c90ae790561793f312e8d59

  • No merge-blocking defect found in the shutdown cancellation, local CAS, co-editing room cleanup, lease renewal, snapshot, or concurrent-write paths.
  • Independent OMX review also found no blocker after inspecting the changed paths and direct consumers.
  • Verification: git diff --check passed.
  • Repository tests, go vet ./..., and go build ./...: not run (0 / 0 / 0) because the Go toolchain is unavailable in this environment. GCS/S3/network filesystem, Cloud Run signal delivery, and ygo shutdown internals remain unverified.

@isac322
isac322 merged commit c764828 into main Nov 3, 2026
8 of 9 checks passed
@isac322
isac322 deleted the fix/shutdown-context-deadline branch November 3, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants