fix(tests): make --skipall pass match the real run - #3610
Merged
Conversation
The initial --skipall registration pass produced allure results that disagree with the real testrun, so the 'Skipped: collected, not run' marker count did not equal the number of collected tests: - pytest evaluates skipif markers before plain skip markers, so tests with their own skipif (SKIPIF_BUILD_EST_1199, SKIPIF_WRONG_ERA, ...) were registered under that marker's reason and then produced a second, identical result file in the real run. Use a prepended skipif(True) marker instead - it is evaluated before all of the test's own markers, so every collected test is registered with the 'collected, not run' reason. - The early return in pytest_collection_modifyitems skipped the dynamic marker additions (e.g. the dbsync marker), so a MARKEXPR relying on them selected different test sets in the skipall pass and the real run. Apply the skipall markers after the dynamic markers instead of returning early. Also remove the dead xdist guard: 'slaveinput' was renamed to 'workerinput' in xdist 2.0, so the guard never fires - and it must not, since with xdist only the worker nodes collect tests and MARKEXPR filtering relies on the markers added by this hook. 'Fixing' the attribute name would have silently broken marker-based selection. Update the count_test_results.py docstring that described the old registration behavior (pre-fix result directories are still handled correctly by the newest-wins rule). Verified on a standalone reproduction: with the old behavior, 4 of 7 tests (class-level skipif, decorated skipif, param-level skipif, plain skip) were registered under their own reasons and '-m db' selected nothing during the skipall pass; with the new behavior all 7 register as 'collected, not run' and marker expression selection matches the real run exactly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The initial --skipall registration pass produced allure results that disagree with the real testrun, so the 'Skipped: collected, not run' marker count did not equal the number of collected tests:
Also remove the dead xdist guard: 'slaveinput' was renamed to 'workerinput' in xdist 2.0, so the guard never fires - and it must not, since with xdist only the worker nodes collect tests and MARKEXPR filtering relies on the markers added by this hook. 'Fixing' the attribute name would have silently broken marker-based selection.
Update the count_test_results.py docstring that described the old registration behavior (pre-fix result directories are still handled correctly by the newest-wins rule).
Verified on a standalone reproduction: with the old behavior, 4 of 7 tests (class-level skipif, decorated skipif, param-level skipif, plain skip) were registered under their own reasons and '-m db' selected nothing during the skipall pass; with the new behavior all 7 register as 'collected, not run' and marker expression selection matches the real run exactly.