S3C-11127: add functional tests for the notification delivery pool

[anurag4DSB]

Intent: why does this change exist?

The three halves of the delivery pool were each unit tested on their own, which cannot catch them drifting apart from each other. This adds functional coverage that runs the real components against a real broker.

System impact: what's affected, including downstream?

Tests and one npm script. The suite has its own entry point, yarn ft_test:notification:deliverypool, and sits outside the notification glob so it does not share a mocha process with the other notification functional tests: the delivery pool builds several consumers per run, and the wedge described below gets likelier the more a single process builds. Two things a reviewer should know: the file is still inside the broad ft_test glob, and no CI job runs this suite today.

Preserved behavior: what explicitly stays the same?

No behavior changes. The one production file in the diff, NotificationConfigValidator.js, gains a comment on the spreadFactor field and nothing else, and the unit test file comes in through the stack rather than from this work.

Intended change: what's different after this PR?

Adds functional suites for delivery, worker restart, the drainer and the drop path. It also adds a seam test that drives the real populator into the real worker, rather than synthesizing the addressed record the way the other cases do, so the populator and worker halves cannot drift while every test stays green. Every topic of a run is created and confirmed cluster wide before any consumer is built, which mirrors the deployment, where the delivery topic is pre-created before the workers start, and is what turned the restart case from never passing into passing every time.

Verification: how do we know this worked, or how would we know if it didn't?

All 15 tests pass against a live broker. The delivery, restart and drainer suites have each gone green twice in a row; the drop and seam suites have gone green repeatedly but not yet twice consecutively. Every red run is the pre-existing consumer wedge, which reproduces with a bare BackbeatConsumer and no notification code in the process, so it is documented separately and is not evidence about this change. These runs were local against a kafka 3.9 broker built from .github/dockerfiles/kafka, not CI.

[codecov]

Codecov Report

❌ Patch coverage is 73.41390% with 88 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (poc/S3C-11127-integration-base@5f8f3db). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...otification/deliveryWorker/DeliveryTopicDrainer.js	75.19%	65 Missing ⚠️
bin/notificationDeliveryReplay.js	0.00%	23 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...nsions/notification/NotificationConfigValidator.js	100.00% <ø> (ø)
...ensions/notification/NotificationQueuePopulator.js	98.48% <100.00%> (ø)
extensions/notification/utils/deliveryKey.js	100.00% <100.00%> (ø)
bin/notificationDeliveryReplay.js	0.00% <0.00%> (ø)
...otification/deliveryWorker/DeliveryTopicDrainer.js	75.19% <75.19%> (ø)

Components	Coverage Δ
Bucket Notification	78.04% <0.00%> (?)
Core Library	80.58% <0.00%> (?)
Ingestion	70.63% <0.00%> (?)
Lifecycle	78.96% <0.00%> (?)
Oplog Populator	85.83% <0.00%> (?)
Replication	59.70% <0.00%> (?)
Bucket Scanner	85.76% <0.00%> (?)

@@                        Coverage Diff                        @@
##             poc/S3C-11127-integration-base    #2802   +/-   ##
=================================================================
  Coverage                                  ?   74.37%           
=================================================================
  Files                                     ?      207           
  Lines                                     ?    14250           
  Branches                                  ?        0           
=================================================================
  Hits                                      ?    10599           
  Misses                                    ?     3641           
  Partials                                  ?       10           

Flag	Coverage Δ
api:retry	8.75% <0.00%> (?)
api:routes	8.57% <0.00%> (?)
bucket-scanner	85.76% <ø> (?)
ft_test:queuepopulator	8.67% <0.00%> (?)
ingestion	12.00% <0.00%> (?)
lib	7.33% <0.00%> (?)
lifecycle	18.04% <0.00%> (?)
notification	0.99% <0.00%> (?)
oplogPopulator	0.13% <0.00%> (?)
replication	17.68% <0.00%> (?)
unit	52.09% <73.41%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.