Skip to content

Expose remote cache read errors and fail on server disconnects - #2779

Open
seungpyoson wants to merge 8 commits into
mozilla:mainfrom
seungpyoson:agent/fail-closed-cache-errors
Open

Expose remote cache read errors and fail on server disconnects#2779
seungpyoson wants to merge 8 commits into
mozilla:mainfrom
seungpyoson:agent/fail-closed-cache-errors

Conversation

@seungpyoson

@seungpyoson seungpyoson commented Jul 23, 2026

Copy link
Copy Markdown

Summary

  • in the direct single-cache daemon path, surface non-NotFound remote read failures through the existing cache-error statistics path instead of counting them as ordinary misses
  • make post-CompileStarted response I/O failures follow the documented default-fatal policy while preserving SCCACHE_IGNORE_SERVER_IO_ERROR=1

Deliberately unchanged

  • startup rate-limit tolerance
  • automatic read-only downgrade when a configured backend cannot write
  • explicit read-only and public-cache configurations
  • client-side IPC and multi-level cache fallback policies

Scope

One commit changes src/cache/cache.rs and src/commands.rs only. There are no documentation, protocol, cache-format, dependency, disk-cache, multi-level-cache, or client-side-storage changes. Existing documentation already describes the server-I/O policy.

Validation

Focused regression tests cover unexpected direct remote read errors, default-fatal post-start disconnects, and the existing opt-in local fallback. Upstream CI and maintainer review remain required before merge.

@seungpyoson seungpyoson reopened this Jul 23, 2026
@seungpyoson
seungpyoson force-pushed the agent/fail-closed-cache-errors branch from e14c22c to 5abf492 Compare July 24, 2026 08:11
@seungpyoson seungpyoson changed the title Fix cache and server error propagation Fail closed on remote cache and server I/O errors Jul 24, 2026
@seungpyoson
seungpyoson marked this pull request as ready for review July 24, 2026 08:22
@seungpyoson
seungpyoson force-pushed the agent/fail-closed-cache-errors branch from 5abf492 to 83b66fd Compare July 24, 2026 08:38
@seungpyoson
seungpyoson force-pushed the agent/fail-closed-cache-errors branch from 83b66fd to a35fe90 Compare July 24, 2026 08:59
@seungpyoson seungpyoson changed the title Fail closed on remote cache and server I/O errors Expose remote cache read errors and fail on server disconnects Jul 24, 2026
@codecov-commenter

codecov-commenter commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.21%. Comparing base (ffac4a5) to head (d048ff9).

Files with missing lines Patch % Lines
src/commands.rs 94.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2779      +/-   ##
==========================================
+ Coverage   70.91%   73.21%   +2.30%     
==========================================
  Files          72       72              
  Lines       37684    37695      +11     
==========================================
+ Hits        26723    27600     +877     
+ Misses      10961    10095     -866     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented Jul 28, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 58 untouched benchmarks
⏩ 6 skipped benchmarks1


Comparing seungpyoson:agent/fail-closed-cache-errors (d048ff9) with main (2a40bc3)2

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on main (ffac4a5) during the generation of this report, so 2a40bc3 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@seungpyoson

Copy link
Copy Markdown
Author

Synced this PR with current main at 918b8ef. New CI, integration-test, and benchmark workflows are waiting for maintainer approval. Could a maintainer please approve the workflows and re-review the PR after they complete? The sync includes main's libc pin for the prior dist-test failure and allows CodSpeed to compare against the current runner generation.

@seungpyoson

Copy link
Copy Markdown
Author

@sylvestre Could you take a look at sccache PR #2779 (#2779) when you have a chance? I synced it with current main today. The underlying bugs are still present, the patch merges cleanly, and the fork workflow runs are waiting for maintainer approval.

If the approach looks good, could you approve the workflows and merge it?

@seungpyoson

Copy link
Copy Markdown
Author

Quick update: the sole non-successful check on d048ff9 was the CUDA 11.8 job, which was canceled after Ubuntu package mirror requests stalled during NVCC installation; it never reached tests. I pushed empty commit acfbdca to retrigger the workflows without source changes. The new CI, integration-test, and benchmark runs are waiting for maintainer approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants