[CI failure] CI: Nightly optional-deps scheduled runs #3211

Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   triageNeeds the team's attentionNeeds the team's attention

   CI/CDCI/CD infrastructureCI/CD infrastructure

   ci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.Assigned to rolling issues managed by the ci-workflow-health.yml workflow.

   on Oct 24, 2026
2. github-actions commented on Oct 25, 2026

   

   github-actionsBot

   on Oct 25, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   Another unhealthy run

   CI: Nightly optional-deps run #228, attempt 1 concluded with failure.

   • Trigger: schedule
   • Branch: main
   • Commit: a2fb804
   • Completed: 2026-10-25T03:21:30Z
3. github-actions commented on Oct 26, 2026

   

   github-actionsBot

   on Oct 26, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   Another unhealthy run

   CI: Nightly optional-deps run #229, attempt 1 concluded with failure.

   • Trigger: schedule
   • Branch: main
   • Commit: a2fb804
   • Completed: 2026-10-26T03:08:44Z
4. github-actions commented on Oct 27, 2026

   

   github-actionsBot

   on Oct 27, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   Another unhealthy run

   CI: Nightly optional-deps run #230, attempt 1 concluded with failure.

   • Trigger: schedule
   • Branch: main
   • Commit: 590a8bf
   • Completed: 2026-10-27T03:14:39Z
5. github-actions commented on Oct 28, 2026

   

   github-actionsBot

   on Oct 28, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   Another unhealthy run

   CI: Nightly optional-deps run #231, attempt 1 concluded with failure.

   • Trigger: schedule
   • Branch: main
   • Commit: 94625c1
   • Completed: 2026-10-28T03:27:15Z
6. github-actions commented on Oct 28, 2026

   

   github-actionsBot

   on Oct 28, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   Another unhealthy run

   CI: Nightly optional-deps run #231, attempt 2 concluded with failure.

   • Trigger: schedule
   • Branch: main
   • Commit: 94625c1
   • Completed: 2026-10-28T15:05:28Z
7. jpascucci-nv

   closed this as completedin #3263on Oct 28, 2026
8. leofang

   reopened this on Oct 28, 2026
9. leofang commented on Oct 28, 2026

   

   leofang

   on Oct 28, 2026

   Member

   More actions

   Nightly triage, Oct 24-28

   Looked at the last five runs. Three recurring real failures and one infra-side symptom. Full matrix is otherwise green.

   1. numba-cuda-mlir on Windows MCDM, rtxpro6000 — three nights running

   Oct 24 job, Oct 25 job, Oct 26 job. Each run opens with Write-Error "Switching to driver mode MCDM failed!" from the pre-test mode switch. Oct 24 and Oct 25 end with pytest exit 127 mid-run (no test summary written); Oct 26 completes and fails tests/test_dump_nvvm.py::test_compilation_dumps_exact_nvvm_input[ptx-sm_100] with assert 0 == 1. Oct 27-28 pass, so it is intermittent — but the common ancestor is the MCDM mode-switch warning, not a code change.

   2. numba-cuda-mlir on Linux rtxpro6000, CUDA 13.4.2 wheels — Oct 27

   Job log. Two tests in tests/test_cuda_array_interface_launch_cache.py both die with cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_INVALID_CONTEXT: invalid device context:

   • test_cupy_array_signature_cache_tracks_dtype_and_ndim
   • test_cupy_launch_does_not_reconstruct_device_arrays

   Looks like test-ordering sensitivity under pytest-randomly — an earlier test tears the primary context that cupy then inherits.

   3. standard linux-aarch64 Python 3.14t, CUDA 13.4.2 local — Oct 27

   Job log. cuda_bindings_13/tests/test_cuda.py::test_coredump_bool_uses_single_byte_storage fails with assert True is False, labeled PARALLEL FAILED. The test is @pytest.mark.thread_unsafe-marked but agent-authored; the process-global LIGHTWEIGHT coredump round-trip is not surviving concurrent test execution on free-threaded 3.14t.

   (Related: the Oct 22 aarch64 run saw the same four standard jobs all fail tests/test_load_nvidia_dynamic_lib.py::test_load_nvidia_dynamic_lib[cudss] with Child process failed for libname='cudss' with exit code 0. Has not recurred since.)

   4. Oct 28 — only failure is infra, not test

   Nightly numba-cuda linux-64 CUDA 13.4.2 wheels fails at host NVIDIA driver install (580.65.06 on kernel 6.8.0-147): ERROR: Building kernel modules → ERROR: The nvidia kernel module was not created. Job log.

   #3263 was merged as the fix and auto-closed this issue, but reopening because:

   • Both Oct 28 attempts (02:35 UTC, 15:02 UTC) ran before the PR merged at 18:29 UTC, so no nightly has exercised the new installer path yet.
   • Author's note on the PR: "Live GPU validation remains pending CI."
   • The patch only diverges from the old path when /proc/version's build-time GCC major differs from the host's default gcc -dumpversion. If 6.8.0-147 is built with gcc-13 (same as the Noble default), the patch is a no-op and the DKMS failure persists.
   • It assumes the .run installer picks up the exported CC/CXX for the kernel module build; passing --cc= to the installer explicitly would be safer.
   • /var/log/nvidia-installer.log is not uploaded as an artifact, so the actual compile error — and whether it really is "GCC 13 rejects GCC 14 flags" — has not been shown.

   Letting tomorrow's scheduled run speak before re-closing.

   5. Earlier context

   Oct 16-19 wiped the entire numba-cuda matrix (six jobs) with RuntimeError: NumPy only supports stream=None. from the numpy==2.4.6 + numba-cuda==0.30.4 combination. Resolved upstream by Oct 20.

   -- Leo's bot
10. github-actions commented on Oct 29, 2026

    

    github-actionsBot

    on Oct 29, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    Another unhealthy run

    CI: Nightly optional-deps run #232, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: d4c71ef
    • Completed: 2026-10-29T03:44:49Z
11. github-actions commented on Oct 30, 2026

    

    github-actionsBot

    on Oct 30, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    Another unhealthy run

    CI: Nightly optional-deps run #233, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 03b4153
    • Completed: 2026-10-30T02:51:44Z
12. jpascucci-nv

    added a commit that references this issue on Oct 30, 2026

    [no-ci] Use 580.126.18 for compatibility test on new ubuntu. (#3295)

    ...

    04fe681
13. github-actions commented on Oct 31, 2026

    

    github-actionsBot

    on Oct 31, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    Recovered

    CI: Nightly optional-deps run #234, attempt 1 concluded with success.

    • Trigger: schedule
    • Branch: main
    • Commit: 80b3359
    • Completed: 2026-10-31T02:55:32Z

    Closing this incident automatically.
14. github-actions

    closed this as completedon Oct 31, 2026