pytestplugin/hooks: check for broken strategy state in pytest_runtest_setup()

Now that a strategy can be marked broken, check for that before
executing a test and its fixtures. If a broken strategy is found in one
of the targets, skip the test. It's not worth continuing with it. It
only leads to cascading errors.

The check for brokenness cannot be performed in the strategy fixture
itself because it's session scoped, meaning it only runs for the first
test requesting it (directly or indirectly). So wrap pytest's
pytest_runtest_setup hook running before fixture/test exexcution, Check
if the test requested the strategy fixture (directly or indirectly) and
perform the check only in that case.
Since we cannot know what target the strategy acts on (the target
fixture might be overridden by the test suite), simply check all targets
for broken strategies and skip the test if any of these have a broken
strategy.

This means a strategy exception leads to either a test error (when
triggered in a fixture) or a test fail (when triggered in the test
itself). All subsequent tests requesting the broken strategy (via the
strategy fixture) will be skipped.

Signed-off-by: Bastian Krause <bst@pengutronix.de>

Author: Bastian-Krause

labgrid/pytestplugin/__init__.py

@@ -1,2 +1,2 @@
 from .fixtures import pytest_addoption, env, target, strategy
-from .hooks import pytest_configure, pytest_collection_modifyitems, pytest_cmdline_main
+from .hooks import pytest_configure, pytest_collection_modifyitems, pytest_cmdline_main, pytest_runtest_setup

labgrid/pytestplugin/hooks.py

@@ -7,6 +7,7 @@
 from ..consoleloggingreporter import ConsoleLoggingReporter
 from ..util.helper import processwrapper
 from ..logging import StepFormatter, StepLogger
+from ..exceptions import NoStrategyFoundError
 
 LABGRID_ENV_KEY = pytest.StashKey[Environment]()
 
@@ -130,3 +131,23 @@ def pytest_collection_modifyitems(config, items):
                     reason=f'Skipping because features "{missing_feature}" are not supported'
                 )
             item.add_marker(skip)
+
+@pytest.hookimpl(tryfirst=True)
+def pytest_runtest_setup(item):
+    """
+    Skip test if one of the targets uses a strategy considered broken.
+    """
+    # Before any fixtures run for the test, check if the session-scoped strategy fixture was
+    # requested (might have been executed already for a prior test). If that's the case and the
+    # strategy is broken, skip the test.
+    if "strategy" in item.fixturenames:
+        env = item.config.stash[LABGRID_ENV_KEY]
+        # skip test even if only one of the targets in the env has a broken strategy
+        for target_name in env.config.get_targets():
+            target = env.get_target(target_name)
+            try:
+                strategy = target.get_strategy()
+                if strategy.broken:
+                    pytest.skip(f"{strategy.__class__.__name__} is in broken state")
+            except NoStrategyFoundError:
+                pass