Skip to content

Test harness: hardening for the Intel runner, and coverage of the proteus package - #99

Open
cekees wants to merge 6 commits into
mainfrom
ci-harness-hardening
Open

cekees wants to merge 6 commits into
mainfrom
ci-harness-hardening

Conversation

@cekees

@cekees cekees commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Two pieces of test-harness work, one PR.

1. Hardening for the macOS Intel runner

The runner is shared and slow, and the near-term hardware and OS picture is likely to stay unstable; the decision is to chase its failures down and harden the harness rather than retire the platform. Three failures there today were the harness, not the code under test:

  • test_parallel_space_multigrid: the helper printed its complete RESULT line and then died on signal 11 at interpreter exit (PETSc/MPI teardown).
  • test_without_petsc4py: a 4-rank mpiexec hit its 300 s budget inside a job that took 2 h 15 min for a suite that takes 15 min on Linux.
  • test_launch_slots: its driver's own 120 s budget, same cause.

The wrapper that test/conftest.py already puts on every subprocess.run from a test (the relaunch of an mpiexec that never started, #95) now also:

  • relaunches a command that died by a signal, once, and reports it as a RuntimeWarning naming the command, the signal and the tail of stderr, so the count stays visible in the summary instead of vanishing into a green run; a repeat fails as before;
  • scales a test's timeout by PROTEUS_TEST_TIMEOUT_SCALE: a number, or auto for the one-minute load average per core clamped to [1, 4], read at launch;
  • runs every launched interpreter with PYTHONFAULTHANDLER=1, so a crash in Python prints the stack it happened in. That, with the warning's stderr tail, is the material for chasing the exit crash down.

The workflow sets the scale to 3 on the Intel leg and auto elsewhere, and PYTHONFAULTHANDLER=1 for the xdist workers. Six tests with a fake launcher that finishes its work and then kills itself. The full suite through the new wrapper: 1,806 passed, no relaunch warnings.

2. Coverage of the proteus package

[tool.coverage] in pyproject.toml measures proteus/ and nothing else: not test/, not scripts/, not the decks under proteus/tests. Branch coverage. The point is to make code nothing runs visible, so the measurement has to reach where the suite's work actually happens: most of it is in processes the tests start (parun, the helpers under mpiexec). coverage 7.10's patch = subprocess does that without the old .pth trick, and it reaches grandchildren through mpiexec: 40 of the 44 lines of Archiver.allGatherIncremental are reached by the rank processes of the parallel-metadata tests. pytest-cov combines the xdist workers' and the children's data files, and [tool.coverage.paths] reports the installed package that CI runs as the checkout's files.

The linux-64 / py3.12 leg runs under coverage (one leg; measuring slows the run and the answer is the same everywhere) and keeps coverage.xml and the HTML report as job artifacts. Locally:

python -m pytest --cov=proteus --cov-report=term-missing:skip-covered ../test

Environment files: pytest-cov >= 7.0.0, coverage >= 7.10.

Tool choice, briefly. coverage.py with pytest-cov remains the one that handles this suite's shape: xdist workers, subprocesses, MPI grandchildren, and a combine step with path mapping. slipcover is faster but has no subprocess or combine story; Codecov or a badge can sit on top of the coverage.xml artifact later if wanted. Not measured: the Cython extensions (cfemIntegrals, cmeshTools, the mprans kernels). That needs a build with linetrace on and CYTHON_TRACE defined, which is a separate, slower build and a Cython.Coverage plugin line in the config; worth doing once the Python-level numbers have settled, since C-level dead code is a different hunt.

🤖 Generated with Claude Code

cekees and others added 2 commits October 24, 2026 05:49
…s for the runner, faulthandler in every launched interpreter

Three failures on the macOS Intel runner today were the harness, not the
code under test: a helper that printed its complete result and then
died on signal 11 at interpreter exit; a 4-rank launch that hit its
300 s budget inside a job that took 2 h 15 min for a 15 min suite; and
the slot-semaphore test's own 120 s budget. The wrapper the root
conftest already puts on every subprocess.run from a test now:

* relaunches a command that died by a signal, once, with a
  RuntimeWarning naming the command and the signal, so the count stays
  visible in the summary; a repeat fails as before;
* scales a test's timeout by PROTEUS_TEST_TIMEOUT_SCALE, a number or
  "auto" (the load average per core, clamped to [1, 4]);
* runs the launched interpreter with PYTHONFAULTHANDLER=1, so a crash
  in Python prints the stack it happened in, which is what chasing the
  exit crash down needs.

The workflow sets the scale to 3 on the Intel leg and auto elsewhere,
and PYTHONFAULTHANDLER=1 for the workers. Six tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
[tool.coverage] in pyproject.toml measures the proteus package and
nothing else -- not the test or scripts directories, not the decks
under proteus/tests -- so that code nothing runs is visible. Branch
coverage; parallel data files combined at the end; and coverage's
patch = subprocess (7.10+), so every interpreter a test starts measures
itself: parun, and the helpers run under mpiexec, where most of the
archive's rank code runs. Checked: 40 of the 44 lines of
Archiver.allGatherIncremental are reached by the rank processes of the
parallel-metadata tests. [tool.coverage.paths] reports the installed
package CI runs as the checkout's files.

The linux-64 py3.12 leg runs under coverage and keeps coverage.xml and
the HTML report as artifacts. pytest-cov >= 7 and coverage >= 7.10 in
the environment files (pytest-cov 7 relies on coverage's own subprocess
patch rather than a .pth file). Cython extensions are not measured:
that needs a linetrace build, a separate and slower one.

Four tests pin the configuration and check, with a two-process script,
that a child measures itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cekees cekees changed the title test harness: relaunch a crash at exit once, scale subprocess timeouts for the runner, faulthandler in every launched interpreter Test harness: hardening for the Intel runner, and coverage of the proteus package Oct 24, 2026
cekees and others added 4 commits October 24, 2026 06:34
… conftest

A conftest is not importable as a module under --import-mode=importlib,
which is how CI runs: two tests failed to collect with No module named
conftest. The pieces they test come through a launch_policy fixture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each xdist worker runs in test_output/<worker id> (test/conftest.py), so
--cov-config=../pyproject.toml did not resolve in a worker and coverage
raised ConfigError before the first test. Reproduced locally with the
same worker layout and fixed: coverage.xml is written and the data files
combine.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r until it has enough points

It used the module's shared generator and 40 draws. Where that generator
stood depended on which tests the xdist worker had run first, and for n=3
the peak is narrow enough that one linux-aarch64 run got 4 points above
the cutoff and failed 'assert checked >= 5'. A generator of its own, and
draws until 8 points are above the cutoff, make it independent of order.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…st-cov's settings

On the coverage leg this process carries COV_CORE_* and COVERAGE_* variables;
the nested 'coverage run' sent its data to the outer measurement and the
nested 'coverage report' had nothing to report (exit 1, linux-64). Reproduced
locally under --cov and fixed by removing them from the nested environment.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant