ref(change-stream): Make ChangeStream async, fallible, and cover before/after commit

[matt-codecov]

Closes FS-565

For a GC ChangeStream:

• Writes need to be .awaited and finished before returning, not fire-and-forget
• Errors need to be reported to the caller, not swallowed
• If the backend commits and the subsequent change stream message fails:
  • write: the object is invisible to GC and leaks forever
  • update: the GC inventory has the wrong expiration time for the object and can delete it prematurely
  • delete: this is actually okay

This PR addresses these issues:

• All trait methods are async fn now
• Relevant functions return Result<> so the caller can abort the request if an essential change stream fails
• write() and update() have been split into begin_{op} and commit_{op} so different streams can hook the right places

As a result, a GC ChangeStream would be "inconsistent but safe":

• if begin_write() succeeds but the backend commit fails, the GC will see a harmless ghost object and delete it later
• if begin_update() succeeds but the backend commit fails, the GC will see a later expiration time than is actually set on the object and delay cleanup
• if the backend delete succeeds but commit_delete() fails, the GC will see a harmless ghost object and delete it later
  • if the object has manual expiry then the ghost objects will leak in the GC catalog and need a reconciliation job to detect

As a drive-by fix: this PR adds a missing ChangeStream::commit_write in GCS's complete_multipart.

Note

A non-goal of this PR is safety amidst concurrent requests. That's a different discussion.

[linear-code]

FS-565

[codecov]

Codecov Report

❌ Patch coverage is 93.54839% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.33%. Comparing base (0abc6bf) to head (7fd1b64).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
objectstore-service/src/error.rs	0.00%	7 Missing ⚠️
objectstore-service/src/backend/in_memory.rs	94.73%	2 Missing ⚠️
objectstore-service/src/backend/gcs.rs	98.27%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #677      +/-   ##
==========================================
- Coverage   91.38%   91.33%   -0.05%     
==========================================
  Files         117      117              
  Lines       24056    24171     +115     
==========================================
+ Hits        21983    22077      +94     
- Misses       2073     2094      +21     

Components	Coverage Δ
Rust Backend	94.82% <93.54%> (-0.07%)	⬇️
Rust Client	81.70% <ø> (ø)
Python Client	93.81% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 7fd1b64. Configure here.}

[cursor]

Multipart commit skips expired objects

Medium Severity

After a successful multipart complete, size and expiry are taken from get_gcs_metadata, which returns none for expired objects and whose errors are discarded. report_object_write then skips commit_write, so a stored object is omitted from the change stream. Long uploads that outlive their TTL are the typical trigger; put_object does not have this gap because it reads size from the upload response.

 

^{Reviewed by Cursor Bugbot for commit 7fd1b64. Configure here.}