Skip to content

fix(tests): make --skipall pass match the real run - #3610

Merged
mkoura merged 1 commit into
masterfrom
skipall_fixes
Aug 13, 2026
Merged

fix(tests): make --skipall pass match the real run#3610
mkoura merged 1 commit into
masterfrom
skipall_fixes

Conversation

@mkoura

@mkoura mkoura commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

The initial --skipall registration pass produced allure results that disagree with the real testrun, so the 'Skipped: collected, not run' marker count did not equal the number of collected tests:

  • pytest evaluates skipif markers before plain skip markers, so tests with their own skipif (SKIPIF_BUILD_EST_1199, SKIPIF_WRONG_ERA, ...) were registered under that marker's reason and then produced a second, identical result file in the real run. Use a prepended skipif(True) marker instead - it is evaluated before all of the test's own markers, so every collected test is registered with the 'collected, not run' reason.
  • The early return in pytest_collection_modifyitems skipped the dynamic marker additions (e.g. the dbsync marker), so a MARKEXPR relying on them selected different test sets in the skipall pass and the real run. Apply the skipall markers after the dynamic markers instead of returning early.

Also remove the dead xdist guard: 'slaveinput' was renamed to 'workerinput' in xdist 2.0, so the guard never fires - and it must not, since with xdist only the worker nodes collect tests and MARKEXPR filtering relies on the markers added by this hook. 'Fixing' the attribute name would have silently broken marker-based selection.

Update the count_test_results.py docstring that described the old registration behavior (pre-fix result directories are still handled correctly by the newest-wins rule).

Verified on a standalone reproduction: with the old behavior, 4 of 7 tests (class-level skipif, decorated skipif, param-level skipif, plain skip) were registered under their own reasons and '-m db' selected nothing during the skipall pass; with the new behavior all 7 register as 'collected, not run' and marker expression selection matches the real run exactly.

The initial --skipall registration pass produced allure results that
disagree with the real testrun, so the 'Skipped: collected, not run'
marker count did not equal the number of collected tests:

- pytest evaluates skipif markers before plain skip markers, so tests
  with their own skipif (SKIPIF_BUILD_EST_1199, SKIPIF_WRONG_ERA, ...)
  were registered under that marker's reason and then produced a
  second, identical result file in the real run. Use a prepended
  skipif(True) marker instead - it is evaluated before all of the
  test's own markers, so every collected test is registered with the
  'collected, not run' reason.
- The early return in pytest_collection_modifyitems skipped the
  dynamic marker additions (e.g. the dbsync marker), so a MARKEXPR
  relying on them selected different test sets in the skipall pass
  and the real run. Apply the skipall markers after the dynamic
  markers instead of returning early.

Also remove the dead xdist guard: 'slaveinput' was renamed to
'workerinput' in xdist 2.0, so the guard never fires - and it must
not, since with xdist only the worker nodes collect tests and MARKEXPR
filtering relies on the markers added by this hook. 'Fixing' the
attribute name would have silently broken marker-based selection.

Update the count_test_results.py docstring that described the old
registration behavior (pre-fix result directories are still handled
correctly by the newest-wins rule).

Verified on a standalone reproduction: with the old behavior, 4 of 7
tests (class-level skipif, decorated skipif, param-level skipif, plain
skip) were registered under their own reasons and '-m db' selected
nothing during the skipall pass; with the new behavior all 7 register
as 'collected, not run' and marker expression selection matches the
real run exactly.
@mkoura
mkoura requested a review from saratomaz as a code owner August 13, 2026 08:45
@mkoura
mkoura merged commit 373e4ca into master Aug 13, 2026
3 checks passed
@mkoura
mkoura deleted the skipall_fixes branch August 13, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant