Skip to content

CI: Use the matrix CTK's compute-sanitizer instead of the latest release - #3397

Open
juenglin wants to merge 5 commits into
NVIDIA:mainfrom
juenglin:juenglin/drop-latest-sanitizer
Open

juenglin wants to merge 5 commits into
NVIDIA:mainfrom
juenglin:juenglin/drop-latest-sanitizer

Conversation

@juenglin

@juenglin juenglin commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #3347. This PR hardens and simplifies how CI fetches the mini CTK.

  • Hardening: fetch_ctk now fails closed when a public-server CUDA version has no pinned manifest hash, instead of warning and continuing. The per-archive hash checks from ci: harden fetch_ctk with manifest/archive verification and optional redist-base-url #3347 are only as trustworthy as that manifest, and since main moved to 13.5.0 it has gone unverified. This PR pins 13.5.0 and adds a test that requires a pin for every CUDA version in ci/versions.yml and every LOCAL_CTK: '1' row of ci/test-matrix.yml, so the next bump fails before merge. Manifests from an alternate redist server stay warn-only, since the public pins do not apply to them.
  • Simplification: the sanitizer step installs compute-sanitizer from the CTK under test instead of the latest release, which removes ci/tools/guess_latest.sh (a run-time scrape of the redist server's directory listing). Every fetch_ctk call now takes its CUDA version from ci/versions.yml or ci/test-matrix.yml, which is what makes failing closed practical. The "latest" sanitizer from CI: Install latest compute-sanitizer separately from CTK #594 no longer buys anything: the only matching matrix rows are the commented-out 12.9.1 rows, and 12.9.1 is the final 12.x release.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Fetch cuda_sanitizer_api from the same pinned redistrib manifest as the
rest of the mini CTK, rather than from whatever release
ci/tools/guess_latest.sh scrapes from the redist directory listing.
Remove guess_latest.sh, which has no other users.
@juenglin juenglin added this to the CI milestone Nov 5, 2026
@juenglin juenglin added CI/CD CI/CD infrastructure github_actions Pull requests that update GitHub Actions code labels Nov 5, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Nov 5, 2026
@juenglin

juenglin commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Chores
    • Updated automated Linux test runs to use the CUDA version selected for each run.
    • Removed automatic lookup of the latest CUDA release; changes to that lookup path now follow general CI path classification.
    • Strengthened validation of CUDA metadata used in automated testing. CI now rejects public metadata versions without a verified checksum.
    • These changes affect automated testing workflows and do not change end-user features or behavior.

The Linux wheel workflow now passes the matrix CUDA version to sanitizer setup. Test-mode environment setup no longer computes LATEST_CUDA_VERSION, and its helper script and Linux-specific CI path classification are removed. Public metadata verification now fails when a CUDA version has no pinned hash.

Changes: Sanitizer CUDA version

Layer / File(s) Summary
Sanitizer version selection
.github/workflows/test-wheel-linux.yml, ci/tools/env-vars, ci/tools/guess_latest.sh, ci/tools/compute_ci_plan.py, ci/tools/tests/test_compute_ci_plan.py
The workflow passes matrix.CUDA_VER to fetch_ctk. Test-mode setup no longer sets LATEST_CUDA_VERSION; the script that computed it and its Linux-only path classification and test case are removed.
Pinned metadata verification
ci/tools/fetch_ctk_redistrib.py, ci/tools/tests/test_fetch_ctk_redistrib.py, ci/versions.yml
CUDA 13.5.0 now has a pinned metadata SHA-256. Verification fails for public versions without a pinned hash. Tests check CI version coverage and digest format, and ci/versions.yml documents the pin requirement.

Change: Bug fix

Merge Risk: Minimal · up to 27f1b

No actionable issue is established; the change is mergeable after normal CI checks.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2b1b872 and 27f1b2f.

Files selected for processing: 5

  • ci/tools/compute_ci_plan.py
  • ci/tools/fetch_ctk_redistrib.py
  • ci/tools/tests/test_compute_ci_plan.py
  • ci/tools/tests/test_fetch_ctk_redistrib.py
  • ci/versions.yml
Files with no reviewable changes: 2

  • ci/tools/compute_ci_plan.py
  • ci/tools/tests/test_compute_ci_plan.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

verify-metadata used to warn and pass when a public-server CUDA version had
no pinned manifest hash, so the 13.4.2 -> 13.5.0 bump ran unpinned. Fail
closed instead, with an explicit fetch_ctk allow-unpinned input (and
--allow-unpinned flag) as the opt-out; it never overrides a mismatch against
an existing pin. Manifests from an alternate redist server stay warn-only.

Pin 13.5.0 and add a test that every CUDA version in ci/versions.yml and
every LOCAL_CTK row of ci/test-matrix.yml is pinned.
No caller sets it, and pinning a new CUDA version is a one-line addition to
KNOWN_REDISTRIB_SHA256 that the error message points to. Remove the
fetch_ctk input and the --allow-unpinned flag; fetch_ctk/action.yml is back
to its main-branch form.
Note in ci/versions.yml that each CUDA version there and in LOCAL_CTK
matrix rows needs a pin in KNOWN_REDISTRIB_SHA256, and make the coverage
test's failure message say where to add the pin and how to compute it.
@juenglin

juenglin commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test bc92709

@juenglin
juenglin requested a review from rwgk November 5, 2026 23:40
@juenglin juenglin assigned isVoid and unassigned isVoid Nov 5, 2026
@juenglin
juenglin requested a review from isVoid November 5, 2026 23:41
@juenglin

juenglin commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 27f1b2f

@juenglin
juenglin marked this pull request as ready for review November 6, 2026 00:12
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@juenglin
juenglin enabled auto-merge (squash) November 6, 2026 00:26
@rwgk

rwgk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor
  • Simplification: the sanitizer step installs compute-sanitizer from the CTK under test instead of the latest release

This reverts an earlier decision: PR 594 deliberately selected newer sanitizers to obtain bug fixes while testing older toolkits. A separately pinned sanitizer version would preserve that original intent without runtime scraping. Did you consider that already? It seems preferable to me.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants