fix: new versions silently missed due to non-monotonic changes feed by mrgrain · Pull Request #1933 · cdklabs/construct-hub

mrgrain · 2026-09-22T08:37:01Z

While investigating a Sources/NpmJs/Canary/SLA-Breached alarm, we found that the npm replica's _changes feed occasionally inserts rows behind the follower's cursor: a row's sequence number was assigned minutes before the follower read past that range, yet the row was never served in any batch. npm has confirmed this is "normal feed behavior" (community discussion, see also #128412). A follower that only ever reads forward silently misses these changes. In our incident, five consecutive construct-hub-probe versions (about 15 hours' worth) were never ingested until the next publish happened to be served on time.

Versions also get dropped a second way: when registry.npmjs.org serves a document revision older than the one announced by the feed, the follower used to wait up to 30 seconds and then process the stale document, which may simply not contain the new versions. This drop was completely silent: nothing was logged and no metric was emitted. The missed versions were only picked up when the package published again.

This PR fixes both:

Receipt log + overlap sweep. The follower now writes a receipt for every change it receives (the sequence numbers, stored in a new S3 object in a scannable, newline-delimited text format). Once an hour, a sweep re-reads the last 6 hours of the feed and processes any change it hasn't seen before (because there is no receipt for it); a daily deep sweep does the same over the last 24 hours, so we can also observe insertion delays larger than 6 hours. Receipts are checked before any metadata fetch, so a sweep costs about 10 feed pages and near-zero registry traffic. An interrupted sweep resumes from a persisted cursor on the next run.
Stale-metadata retry queue. Instead of processing a stale document, the change is deferred to a new SQS queue and re-checked on subsequent runs until the registry catches up; messages that never recover land in a DLQ that identifies exactly which package versions were missed.
Observability. New LateChangeCount, LateChangeLag, StaleMetadataDeferred, and StaleMetadataRecovered metrics feed a new Feed Reliability dashboard widget. LateChangeLag measures how far behind the cursor each late row was found; this is the data for tuning the sweep margin. Three new low-severity alarms (retry DLQ not empty, lag approaching the sweep margin, sweep not running), each with an operator runbook entry.

Rows that are lost from the feed entirely (never re-inserted at any sequence number) cannot be detected at all. They still self-heal when the package publishes its next version, and the canary alarm still covers the probe.

This PR also makes a small change to fillMetric: it now accepts an explicit discriminator, so two same-named metrics (queue and DLQ depth) can be rendered in one graph. Hashing dimensions instead was not an option, as they contain unresolved CDK tokens and would make synthesis non-deterministic.

Validated with 27 new unit tests (receipt format golden/round-trip/rejection cases, and follower scenarios for stale deferral, sweep discovery, and retry recovery), plus the full suite (283 tests), jsii-diff (additive API only), and unchanged integration test snapshots.

Internal reference: V2376798198

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

github-actions · 2026-09-22T08:37:13Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

… feed The npm replica's `_changes` feed occasionally inserts rows behind the follower's cursor, and the registry occasionally serves stale package metadata; both paths silently dropped package versions until the next publish of the affected package. The follower now keeps a receipt log of received sequence numbers and periodically re-reads a trailing window of the feed (hourly 6h overlap sweep, daily 24h deep sweep), processing any row it has no receipt for. Changes whose registry metadata is stale are deferred to a new SQS retry queue and re-checked on subsequent runs instead of being processed with a potentially incomplete document. New LateChangeCount / LateChangeLag / StaleMetadataDeferred / StaleMetadataRecovered metrics feed a new dashboard widget and three low-severity alarms (retry DLQ not empty, lag approaching sweep margin, sweep not running), all documented in the operator runbook. Internal reference: V2376798198

rix0rrr · 2026-09-22T11:53:09Z

+ */
+async function loadReceipts(
+  stagingBucket: string,
+  initialMarker: number


This parameter is only used to decide to not do work in this function. Could/should be outside the function.

Also, I don't think checking for exactly 0 makes sense. At the start, the file will not exist and we start with an empty collection anyway. Or if we are trying to protect against the sequence number going backwards and our current receipts file becoming nonsense, we should be checking for that.

rix0rrr · 2026-09-22T12:02:33Z

+          changes.staleResults.length,
+          Unit.Count
+        );
+        await deferStaleChanges(retryQueueUrl, changes.staleResults);


I'm looking at this now... do we even need separate mechanisms for the cases

some sequence numbers are missing and need to be retried later; and

some package versions can't be served right now and need to be retried later

?

Seems like those are basically different causes for the same effect -- might as well combine them?

I imagine a counterargument might be that we want to recover from them at different timescales... but I imagine there will be simple solution for that as well?

rix0rrr · 2026-09-22T12:07:15Z

+ * retry queue, for re-processing on subsequent runs once the registry has
+ * caught up with the revision announced by the changes feed.
+ */
+async function deferStaleChanges(


Suggested change

async function deferStaleChanges(

async function retryLaggyPackumentsLater(

?

rix0rrr · 2026-09-22T12:09:17Z

+      received < MAX_RETRY_MESSAGES_PER_RUN &&
+      context.getRemainingTimeInMillis() > RETRY_QUEUE_TIME_BUDGET_MS
+    ) {
+      const response = await SQS_CLIENT.send(


sqs.receive in a cron Lambda? I appreciate that this is a small architectural delta, but it feels like a weird choice to me.

rix0rrr · 2026-09-22T12:18:20Z

+ * w <lastSweepAt> <lastDeepSweepAt>   <- sweep state ("-" when unset)
+ * u <seq> <deep|regular>        <- sweep cursor (only when a sweep was interrupted)
+ * c <at> <seq>                  <- checkpoint (repeated, ascending by time)
+ * s <seq>                       <- received sequence number (repeated, ascending)


How large does this file get? A super easy way to make it smaller would be to include this:

Suggested change

* s <seq> <- received sequence number (repeated, ascending)

* s <seq> <- received sequence number (repeated, ascending)

* r <seq0> <seq1> <- range of received sequence numbers (repeated, ascending)

rix0rrr · 2026-09-22T12:43:16Z

+ *
+ * @returns the number of package versions that were sent for staging.
+ */
+async function runSweep(


I don't follow the logic in here. The contract between this file and the receipts file is a bit complex for my blood.

rix0rrr

I'll ship it because it's at least more reliable than what we have today.

But we should probably just rework this whole thing to simplify it way down.

cdklabs-automation enabled auto-merge September 22, 2026 08:37

mrgrain force-pushed the mrgrain/fix/npmjs/feed-late-insertions branch from e145695 to d836f42 Compare September 22, 2026 08:59

mrgrain changed the title ~~fix(npmjs): new versions silently missed due to non-monotonic changes feed~~ fix: new versions silently missed due to non-monotonic changes feed Sep 22, 2026