Repository navigation
fix(npmjs): follower crashes on truncated registry responses - #1937
Merged
cdklabs-automation merged 3 commits intoSep 24, 2026
Merged
Conversation
Since 13:00 UTC the registry intermittently truncates gzip response bodies. Zlib raises Z_BUF_ERROR for these, which was not treated as retryable, so a single truncated packument crashed the whole run before the state was saved. On Gamma no state had been persisted for two hours, with 20-36 failed invocations per hour. Three changes: - Z_BUF_ERROR (truncated body) is retried like ECONNRESET. - A packument fetch that still fails no longer fails the batch. The entry is excluded from the receipts instead, so a later scan retries it. - The follower state is saved even when a scan fails part-way, so completed work (receipts, interim checkpoints, staged versions) is never lost to a crash.
cdklabs-automation
enabled auto-merge
September 23, 2026 15:47
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
9pace
reviewed
Sep 23, 2026
Removes the env-overridable request deadline again: tests now use HTTP 403 responses (client errors are not retried), so no test seam is needed. Adds a MetadataFetchFailures metric for entries whose packument fetch failed after retries, wired into the Feed Reliability widget.
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
rix0rrr
approved these changes
Sep 24, 2026
cdklabs-automation
deleted the
mrgrain/fix/npmjs/transient-fetch-errors
branch
September 24, 2026 08:55
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Sep 24, 2026
After deploying #1937 our environment is still stuck. The state file hadn't been saved for 24 hours because of the crashes, so the follower has a backlog of roughly 250k entries without receipts. That is more work than fits into one execution. Runs hit the Lambda timeout, and on a timeout Lambda kills the process without running the `finally` block. So the state was never saved, and the next run started over with the same backlog. We can fix this: The scan now processes entries in chunks of 1,000 and saves the state (receipts, checkpoints, laggy packuments, known versions) after every chunk. If a run is killed, only the current chunk is lost. Progress is kept between runs and the backlog shrinks until it's gone. The time budget is now also checked between chunks. Previously these checkpoints were only run after a full page, which can be up to 10k changes. During a catch-up a most entries will be without receipts, and processing them can take longer than the whole execution. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license* --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since about 13:00 UTC today, we observed responses from
registry.npmjs.orgintermittently returning truncated gzip bodies. No event is published on https://status.npmjs.org/.In our code zlib raises a
Z_BUF_ERRORfor these, and we didn't treat that as retryable: a truncated packument crashed the whole follower run. Even worse, the run died before saving its state.Three changes to fix this:
Z_BUF_ERRORis now retried likeECONNRESET. A truncated body is just as transient as a dropped connection.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license