ci: harden the test MongoDB for single-process coverage and dump diagnostics on failure - #1769
Merged
Merged
Conversation
…ump diagnostics on failure Coverage (Backend) and Coverage (Backend Base) lost mongod at the same cumulative point (~840-960 s, auth_device_service tests) on #1765/#1766 with ~7 GB host memory still free; mongod held ~8 GB when it vanished. Raise the descriptor limit far above the nextest peak (thousands of per-test databases' WiredTiger files stay open in one llvm-cov process) and cap the WiredTiger cache at 2 GB. On failure or cancellation, print the container state (exit code, OOMKilled), mongod descriptor count/limits, kernel OOM lines and the server log tail so any recurrence is diagnosable.
📊 Code coverage
Gate: line coverage must stay at or above the threshold. Ratchet plan (W21): Backend → 55%, CLI → 50%, Frontend → 30% by quarter end. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI-only hardening for the per-job test MongoDB. No application code changes and no version bump.
Since v0.60.0, the single-process
cargo llvm-cov --package nyxidjobs losemongodat the same cumulative point, about 840–960 s into the run around theauth_device_servicetests:Every DB-backed test then fails until the 90-minute cancel. The
[ci-resources]samples show available host memory still around 7 GB, whilemongodheld about 8 GB when it vanished. This is consistent with descriptor exhaustion: the start script already disables the snapshot window for the same reason, and the per-test ident count keeps growing with new collections and indexes. A host OOM looks unlikely.Changes
start-test-mongodb.sh: raisenofilefrom 65,536 to 1,048,576 and cap--wiredTigerCacheSizeGBat 2 (the test data set is tiny).dump-test-mongodb.sh: container state (exit_code,oom_killed, error), mongod descriptor count and limits, VmRSS/VmHWM, kernel OOM lines and the server log tail. Metadata and server log only, no environment.ci.yml: run the dump onfailure() || cancelled()after the backend test, billing smoke, head coverage and base coverage steps (continue-on-error).Validation
This PR's own Coverage (Backend) and Coverage (Backend Base) runs re-measure with the hardened container. The coverage identity fingerprint includes the start script, so the base report is re-measured rather than restored.