Skip to content

Malformed registry, trace or covers() input crashes with exit 1 instead of a refusal #169

Open

Description

@inth3shadows

What happens

When the registry, the trace, or a test's covers(...) declaration is malformed, testgraph can crash with a traceback and exit 1. Exit 1 is the code for "tests failed", so CI reads a malformed input as a red test. In one case (the adapter, below) the crash comes only after the whole suite has run.

testgraph should refuse these inputs up front with exit 2 and a message that names the input at fault.

The final review of #168 found these items. Most behave the same on the commit before #168; the ones that don't are marked.

Checklist (main at fc3d000)

  • A failing test that declares a non-string journey id crashes verify. For example, a failing test with @covers("J1", 1), or a class-level covers(1) combined with a method-level covers("J1"). The result is AttributeError: 'int' object has no attribute 'rstrip' in journey_sort_key, reached from verify.py#L226, with exit 1. Main behaves the same way (verified by a reviewer). fix: one traced-run path for verify and the adapter; crash-safe trace writes #168 had briefly turned this into a refusal, and then relaxed the check so that covers(1) alone is not refused. Expected: validate the id and either drop it or report it as unvalidated without crediting it.
  • One unhashable id anywhere refuses the whole verify run. A deselected @covers(["J2"]) makes payload_shape_error refuse a run whose selected tests are fine, and the message blames "the trace file". Changed by fix: one traced-run path for verify and the adapter; crash-safe trace writes #168: before it, this case gave exit 0. Expected: name the test and its decorator, and refuse only when an affected test is selected.
  • A non-object JSON file in a journeys directory breaks registry auto-resolution for every repo. resolve_for_repo calls json.load(f).get("target") (L126) and catches only OSError and ValueError, so a stray [] raises AttributeError. Expected: load through registry.load inside the try, and skip that file.
  • The select and export CLIs don't catch registry.load's ValueError. A non-object registry gives a traceback with exit 1 at select.py#L353 and export.py#L364. The pre-push hook depends on select. Expected: a refusal with exit 2.
  • A registry that is an object but malformed ({}, {"journeys": []}, {"journeys": "x"}) makes pytest_adapter run the whole suite and then crash in journey_entries with KeyError or AttributeError, exit 1. verify crashes before the run. Expected: check the registry's shape at load time.
  • A path like symlink/.. resolves differently from the kernel. With link -> else/sub, --trace-out link/../t.json writes ./t.json, but the shell writes else/t.json. This happens in write_json_atomic and in tgtrace's _write, because abspath resolves .. without following the link. Expected: take realpath of the directory, then join the file name.
  • Wording: tgtrace and harness/trace.py call a directory target a stream. They refuse a directory-shaped target with the "streams and devices" message (tgtrace.py#L210). The adapter already says "is a directory".
  • Tidy: the check for a path that names a directory appears three times. Use one helper in pytest_adapter; tgtrace keeps its own copy by design.

Related: test_pre_push fails intermittently under load

_stray_processes scans all of /proc for its marker process (sleep 31337). When other sessions on the same host run the suite at the same time, they trip it, and 1 to 3 of the hung-hook tests fail. Run alone, the file passed 3 of 3 times on main and on #168's branch (verified by a reviewer). Expected: scope the scan to the test's own process group.

Related: #154, #168 (the review rounds that found these). Not #159, which is about Playwright flake policy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions