Skip to content

‼️ Split sphinx-test-reports: the Sphinx-free core becomes ub-test-reports 1.0.0 - #2009

Merged
chrisjsewell merged 9 commits into
masterfrom
split/ub-test-reports
Oct 1, 2026
Merged

‼️ Split sphinx-test-reports: the Sphinx-free core becomes ub-test-reports 1.0.0#2009
chrisjsewell merged 9 commits into
masterfrom
split/ub-test-reports

Conversation

@chrisjsewell

@chrisjsewell chrisjsewell commented Oct 1, 2026

Copy link
Copy Markdown
Member

sphinx-test-reports has carried two products since 2.0.0: a Sphinx extension, and a converter and pytest plugin that must run without Sphinx. This PR makes the second its own distribution, ub-test-reports (packages/ub-test-reports, import ub_test_reports, at 1.0.0.dev0), and leaves sphinx-test-reports as the Sphinx extension with hard dependencies on Sphinx, docutils, sphinx-needs and the core. By the sphinx-*/ub-* naming rule it is a tool, not a Sphinx extension: no Framework :: Sphinx, nothing to add to conf.py.

What moved (with git mv, so git log follows): cli, fields, identity, jsonparser, junitparser, needs_export, projectconfig, pytest_plugin, remote, results and schemas/JUnit.xsd. Inside them only imports, docstring references and the names the plugin prints changed (renames at 95–100 % similarity). The plugin's wire names sphinxcontrib.test_reports:file|line are unchanged and are now pinned by tests. lxml and ub-project go with the modules that import them.

What changes for users (the extension's changelog states each of these):

  • pip install sphinx-test-reports brings Sphinx, Sphinx-Needs and docutils again. This reverses 2.0.0's install-footprint change. A CI job or Bazel action that installed it only for test-reports or the plugin should install ub-test-reports instead.
  • The test-reports command belongs to ub-test-reports. pipx install sphinx-test-reports and uv tool install sphinx-test-reports now find no command (measured; the messages are quoted in the changelog): name ub-test-reports there, and in anything else that looks the script up in the installing package's own metadata. pip install sphinx-test-reports still puts test-reports on the path, through the dependency.
  • [sphinx] is accepted and ignored, and [pytest] passes through to ub-test-reports[pytest], until 4.0 — so 2.0.0's documented install lines keep working (dropping either makes them warn and silently skip pytest under both pip and uv, measured).
  • The plugin is -p ub_test_reports.pytest_plugin. Its registration name is ub_test_reports.xml_shape and its warning prefix ub_test_reports.pytest_plugin: (both pinned).
  • The old sphinxcontrib.test_reports.{junitparser,jsonparser,pytest_plugin} names still work until 4.0 (one FutureWarning, naming the core module), but only where the extension is installed — and so Sphinx. With only the core installed they are a plain ModuleNotFoundError. sphinx_test_reports.<module> for the moved modules was never released (PyPI's newest is 2.0.0), so it needs no alias.
  • The load-time toolchain check (toolchain.py) is gone: the dependencies are hard, so pip resolves the floors. A sphinx-needs downgraded in place after installing is no longer refused — it runs untested; pip check names the conflict. The extension calls app.require_sphinx((7, 4)). compat-requirements.txt is deleted (its two lines are Requires-Dist now).
  • Unchanged: extensions = ["sphinx_test_reports"] and its old alias; the tr_* values and tr_config_from_toml; the docs site (one site; the core's pages form a "Without Sphinx" section, its changelog is linked, not included — bump's release: labels would collide).

Tests: core 351 (324 moved, 10 new for the JSON parser — a core module whose only tests were Sphinx builds — 2 for the wire names, 2 for the shipped XSD, which turned out to have no reader under test, 1 for the printed names, 12 for a static ast walk refusing any toolchain import at any depth), extension 150, deleted 15 (test_toolchain.py and the two tests of the lazy check). The toolchain marker is gone. The core reads its own copies of 13 fixtures; the extension's doc_test/utils/ and the docs' :file: paths are untouched. test_aliases.py walks both packages.

CI: toolchain-free is now "(ub-test-reports, ub-project)": the core's artefact fence (flit; one top-level package, the XSD, py.typed, no tests/ in the sdist), the extension's existing hatchling two-package fence, then the whole core suite from the built wheel in a venv with no toolchain and with pytest-xdist — asserted present, so the three xdist tests that have been skipping silently in that job run. New plugin-floor job: the same shape on pytest 7.0.1 / Python 3.11 and pytest 7.3.2 / 3.12 (the [pytest] extra's floor; the lane the import retired), in check. Also: a ub-test-reports Codecov flag (reports will read about a point lower because statements moved — not a required status), the type-gate canary probe, the labelers and issue forms, and the extension's .readthedocs.yaml installing packages/ub-test-reports from the checkout before the extension (pip knows nothing of workspace sources; without the line an RTD build cannot resolve an unpublished core — measured) and rebuilding when the core changes.

Lock: master's lock plus the semantic entries, +38/−17; uv lock --check passes on 0.12.15 and 0.12.9 (the hook's rev) and a plain uv lock leaves it byte-identical.

Release order (tagging order): 1. this PR; 2. the core's release PR — poe bump ub-test-reports --to 1.0.0, which also moves the extension's two ub-test-reports specifiers to >=1.0.0,<2 through propagate_floors (ship the one-line uv.lock version change, not bump's 224-line relock — measured to pass both lock --checks); tag ub-test-reports-v1.0.0; 3. sphinx-needs' release (release-plan already requires it before sphinx-test-reports); 4. poe bump sphinx-test-reports --bump major, tag sphinx-test-reports-v3.0.0. The extension's floor is spelled ub-test-reports>=1.0.0.dev0,<2 in this PR because check_workspace check (4) refuses >=1.0.0 while the tree builds a dev version.

Red by design until the core is on PyPI: poe import-check-reports (ub-test-reports was not found in the package registry) and the extension's release gates (the plan job: "Release it first"; the compat cell's PyPI resolution). Every PR-time gate is green with the core unpublished — measured by replaying both CI jobs from the committed YAML, the cells (including sphinx 7.4 on 3.11), docs-reports under -nW, and the release build job in both orders with the core supplied as if published.

Review: recon, two adversarial reviewers, one fix round, one validation round, all recorded. One claim fell: the changelog's first draft said a below-floor sphinx-needs "fails with a traceback from a directive"; nine test projects built green on 8.4.0 and 6.3.0, so it now says what was measured.

Before merging / before the first tag:

  • pkg: ub-test-reports label created (applied by hand here; the labeler reads master's config).
  • Before ub-test-reports-v1.0.0: the GitHub environment pypi-ub-test-reports (tag rule ub-test-reports-v*) and the PyPI pending trusted publisher for ub-test-reports (useblocks/sphinx-needs, release.yaml, environment pypi-ub-test-reports).
  • Before ub-test-reports-v1.0.0: repoint the sphinx-test-reports Read the Docs project at useblocks/sphinx-needs (readthedocs_yaml_path = packages/sphinx-test-reports/.readthedocs.yaml) and confirm latest renders the "Without Sphinx" section — today it still builds the archived old repository, and the core's README and Documentation URL point at it.
  • Core release PR: link the README rows to PyPI.

Changelogs: packages/sphinx-test-reports/docs/changelog.rst (Unreleased, rewritten in place where the split made it false) and packages/ub-test-reports/docs/changelog.rst (Unreleased, one entry).

@chrisjsewell
♻️ Add ub-test-reports: the Sphinx-free core moved out of sphinx-test…
b60fb2d 
…-reports

A sixth publishable member, packages/ub-test-reports (distribution
`ub-test-reports`, import `ub_test_reports`, 1.0.0.dev0, flit). The ten
modules that never needed Sphinx -- cli, fields, identity, jsonparser,
junitparser, needs_export, projectconfig, pytest_plugin, remote, results --
and schemas/JUnit.xsd move with `git mv`; their intra-package imports and
docstring cross-references now name `ub_test_reports`, and so do the names
the plugin prints (its `-p` spelling, the `ub_test_reports.xml_shape`
registration name, the warning prefix). The wire names
`sphinxcontrib.test_reports:file|line` are unchanged.

sphinx-test-reports becomes the Sphinx extension only: hard dependencies on
sphinx, docutils, sphinx-needs and `ub-test-reports>=1.0.0.dev0,<2` (the
floor the core's release pull request moves to 1.0.0); lxml and ub-project
move to the core. `[sphinx]` stays as an empty extra and `[pytest]` as a
pass-through to `ub-test-reports[pytest]`, both until 4.0; the
`test-reports` script is the core's. The package root imports `setup`
eagerly, `setup` calls `app.require_sphinx((7, 4))`, and `toolchain.py`
and `compat-requirements.txt` are deleted. The three module aliases point
at `ub_test_reports`.

Root wiring: the member in `[project] dependencies` and the sources, ruff
`src`, ty `include` and `typecheck`, tasks `test-ub-test-reports`,
`build-ub-test-reports` and `import-check-ub-test-reports`; the `toolchain`
marker is gone. The lock is master's plus the semantic entries only.

The suite is NOT green at this commit: the tests still name the moved
modules and move in the next one.
@chrisjsewell
🧪 Divide the test suite between the core and the extension
455555e 
ub-test-reports gets its share of the suite, the extension keeps the
tests that need a Sphinx build. Six modules move whole with `git mv`
(cli_config, identity, junit_parser, junit_parser_gtest, needs_export,
pytest_plugin). test_cli_convert, test_project_config and
test_result_vocabulary are split by the old `toolchain` marker: the
unmarked tests go to the core's file of the same name, the marked ones
stay in the extension's, and the marker is gone from every site.

Deleted with the lazy toolchain check: tests/test_toolchain.py,
TestSphinxFree.test_a_missing_sphinx_needs_is_an_extension_error and
test_the_deprecation_comes_before_a_toolchain_error.
test_projectconfig_imports_without_sphinx moves to the core.

test_aliases.py: the three module aliases are checked against
`ub_test_reports`, and the walk of old names that must fail plainly
covers both packages, located with find_spec rather than an import of
the (now eager) extension root.

The core reads its own copies of the fixtures it needs, under
tests/fixtures/; the extension's tests/doc_test/utils/ is unchanged but
for the dead xml_data_2.xml, which nothing read. New in the core:
test_json_parser.py (the JSON parser called directly), and two tests
pinning the plugin's wire names `sphinxcontrib.test_reports:file|line`.
@chrisjsewell
🔧 CI for the core: toolchain-free rewritten, the plugin-floor job, co…
c665f8a 
…verage, labels, RTD

toolchain-free is now "Toolchain-free (ub-test-reports, ub-project)": the
property it fences belongs to a distribution, not an extra. It builds the
core's sdist and wheel as the release does and checks them (top level
exactly `ub_test_reports` plus dist-info, `schemas/JUnit.xsd` and
`py.typed` shipped, no `tests/` or `docs/` in the sdist), keeps the
extension's hatchling two-package fence unchanged (it only reads
archives), installs the core's wheel with `[pytest]`, pytest-xdist and
ub-project into a Python 3.11 environment with no toolchain, and runs the
core's whole suite there -- no marker, no file list.

plugin-floor: the same environment with pytest pinned to the oldest
each floor Python takes (7.0.1 on 3.11, 7.3.2 on 3.12), the pin
asserted, the suite run from the package directory. Added to `check`.

The Extensions cells run the core's suite with coverage under a new
`ub-test-reports` Codecov flag; the Lint type-gate canary probes the
core; the labelers and both issue forms know `ub-test-reports`; and the
extension's .readthedocs.yaml installs the core from the checkout before
the extension, so pip never looks for an unreleased core on PyPI.
@chrisjsewell
👌 ub-test-reports: test that validate() reads the shipped JUnit schema
d85cf9a 
`schemas/JUnit.xsd` is package data whose only reader is
`JUnitParser.validate()`, and nothing called it: a wheel that shipped
without the schema passed the whole suite (measured while proving the
toolchain-free artefact check). Two direct tests now load it -- a
conforming report validates, pytest's own report does not -- so the
suite run against a built wheel fails when the schema is missing.
@chrisjsewell
📚 Docs, changelogs, AGENTS and README for the split
3290073 
One documentation site for both distributions: the index groups the pages
into "The Sphinx extension", "Without Sphinx (ub-test-reports)" and
support, and says the version shown is the extension's. ub-test-reports'
changelog is included as a page of its own. `docs/conf.py` reads the
version from the installed distribution instead of a literal.

install.rst describes two distributions and three install lines, with a
3.0.0 note; its paragraph on Sphinx-Needs 6.0.1 and the load-time check,
false already, now states the real floors. cli.rst, pytest.rst and the
`[test_reports]` section of configuration.rst name ub-test-reports where
the code now lives.

The extension's Unreleased changelog is rewritten where the split makes
it false and gains the split's own entries: ub-test-reports exists;
`pip install sphinx-test-reports` brings Sphinx, Sphinx-Needs and docutils
again (and the fix for a CI or Bazel action that installed it for the
command: install ub-test-reports); `[sphinx]` and `[pytest]` kept until
4.0; the old module names need the extension; the plugin's printed names;
the load-time check and compat-requirements.txt gone.

AGENTS.md: the extension's describes the extension alone; ub-test-reports'
holds the core's rules; the root's table, naming, ub-project and command
paragraphs follow. README.md lists ub-test-reports and ub-project.
@chrisjsewell
👌 Read the Docs: build sphinx-test-reports' site when ub-test-reports…
88344b2 
… changes

The site documents ub-test-reports, includes its changelog and installs
it from the checkout, so a pull request that touches only
packages/ub-test-reports/ changes what the site builds. The skip filter
in post_checkout now looks at both packages.
@chrisjsewell
🧪 Review fixes: a static no-toolchain test, the printed names pinned,…
518c9f5 
… xdist fenced

ub-test-reports' tests/test_imports.py walks every module's source and
refuses an import statement naming sphinx, sphinx_needs, docutils,
sphinx_test_reports or sphinxcontrib at any depth -- function bodies,
`try` arms and `TYPE_CHECKING` included -- plus string-spelled
`importlib.import_module`/`__import__`. It runs in every environment and
does not depend on a test reaching the line.

The plugin's warning prefix and hook registration name, which the
changelog documents as changed from 2.0.0, are pinned by one pytester
run. The alias test now imports the old pytest_plugin name too, and is
renamed for what it proves: the alias chain does not load Sphinx.

CI: toolchain-free and plugin-floor fail when pytest-xdist is missing,
instead of skipping the plugin's three xdist tests green.
@chrisjsewell
📚 Review fixes: the changelog's downgrade and installer claims, a cha…
7e3ffb5 
…ngelog link, comments

The changelog no longer says a below-floor Sphinx-Needs fails with a
traceback: measured, it runs untested and nothing fails, which is what
it now says; `pip check` still names the conflict. A new entry says the
`test-reports` command belongs to ub-test-reports, so `pipx install` and
`uv tool install sphinx-test-reports` find no command and should name
ub-test-reports. The core's entry says its printed names changed.

ub-test-reports' changelog is linked from the site rather than included,
so its release labels can never collide with the extension's.

AGENTS: the core's no-toolchain rule names the static test and what the
toolchain-free job adds; the extension's Releasing paragraph says the
site must build this repository before the core's first tag. Comments
in ci.yaml, release.yaml and two tests corrected.
@chrisjsewell
📚 Say what the static import walk and the toolchain-free job each catch
c2d6354 
Validation round of the split review: the static test's docstring now names the
import spellings it reads (a positional string literal) and the ones it cannot; the
toolchain-free job's header and the core's AGENTS.md add the qualifier that the job
sees a dynamic import only on a line a test reaches; the three subprocess tests load
no Sphinx, not "none of the toolchain"; the quoted uv message keeps its backticks.
@chrisjsewell chrisjsewell added the pkg: ub-test-reports Concerns the ub-test-reports package (packages/ub-test-reports): the Sphinx-free test-reports core label Oct 1, 2026
@github-actions github-actions Bot added pkg: workspace The repository as a whole: workflows, CI, release, docker, tooling, the workspace root pkg: sphinx-test-reports Concerns the sphinx-test-reports package (packages/sphinx-test-reports) labels Oct 1, 2026
@codecov

codecov Bot commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.15385% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.86%. Comparing base (d68d10d) to head (c2d6354).
⚠️ Report is 31 commits behind head on master.

Files with missing lines Patch % Lines
...-test-reports/src/ub_test_reports/pytest_plugin.py 50.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #2009      +/-   ##
==========================================
+ Coverage   91.69%   91.86%   +0.16%     
==========================================
  Files         129      129              
  Lines       18155    18081      -74     
==========================================
- Hits        16648    16610      -38     
+ Misses       1507     1471      -36
Flag Coverage Δ
codelinks 93.69% <ø> (+0.26%) ⬆️
mounts 94.26% <ø> (+0.24%) ⬆️
pytests 91.48% <ø> (+0.07%) ⬆️
reports 88.01% <100.00%> (-0.74%) ⬇️
ub-test-reports 90.16% <93.75%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ubmarco
ubmarco approved these changes Oct 1, 2026
View reviewed changes
Comment thread .github/workflows/ci.yaml
cache-python: true
- name: Build the sdist and wheel as the release does, and check both
# The fence on the ARTEFACTS. Every other job installs this member editable, which
- name: Build ub-test-reports' sdist and wheel as the release does, and check both

@ubmarco ubmarco Oct 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker, but I'd appreciate if scripts of more than 3-4 lines are put into a dedicated file, so it can run locally as well.

@chrisjsewell chrisjsewell Oct 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — filed as #2013: one tools/src/sn_tools/check_artefacts.py <dist> run by path (like check_workspace.py), with a poe task per member so the same fence runs locally, and ci.yaml calling it; the two-line assertions stay inline.

@chrisjsewell
chrisjsewell merged commit 39b7315 into master Oct 1, 2026
39 of 40 checks passed
@chrisjsewell
chrisjsewell deleted the split/ub-test-reports branch October 1, 2026 07:27
This was referenced Oct 1, 2026
Open
Open
Open
Open
Open
Open
Merged
chrisjsewell added a commit that referenced this pull request Oct 1, 2026
@chrisjsewell
🚀 Release ub-test-reports 1.0.0 (#2019)
f91d8cf 
The first release of `ub-test-reports`, the Sphinx-free core carved out
of sphinx-test-reports in #2009 — stamped by `poe bump ub-test-reports
--to 1.0.0`:

- `packages/ub-test-reports/pyproject.toml` `version = "1.0.0"` and
`ub_test_reports.__version__`;
- the changelog entry (`release:1.0.0`, `:Released: 2026-10-01`, the
summary paragraph written by hand);
- the extension's two `ub-test-reports` specifiers → `>=1.0.0,<2`
(`propagate_floors`; the split PR had to say `>=1.0.0.dev0` because
`check_workspace` check (4) refuses a floor the tree does not build);
- `uv.lock`: the one version line, not `bump`'s relock (224 lines of
fork-marker churn; the one-line lock passes `uv lock --check` on 0.12.15
and 0.12.9 and a plain `uv lock` leaves it alone — measured in #2009's
review);
- the root README row links PyPI; two comments that named the dev
version updated.

Gates: `poe lint` ✓ (incl. `check-workspace`, `uv-lock`), `poe
test-ub-test-reports` 351 ✓, `poe test-reports` 150 ✓, `poe
import-check-ub-test-reports` 11 modules ✓, `release_plan.py --tag
ub-test-reports-v1.0.0` ✓ ("not on PyPI yet"; first release, no previous
tag → no generated notes, #1988).

**After the merge, before the tag** (`git tag ub-test-reports-v1.0.0 &&
git push origin ub-test-reports-v1.0.0` from master):
- [x] PyPI pending trusted publisher `ub-test-reports`
(`useblocks/sphinx-needs`, `release.yaml`, environment
`pypi-ub-test-reports`)
- [x] GitHub environment `pypi-ub-test-reports`, tag rule
`ub-test-reports-v*`
- [ ] Repoint the `sphinx-test-reports` Read the Docs project at
`useblocks/sphinx-needs` (`readthedocs_yaml_path =
packages/sphinx-test-reports/.readthedocs.yaml`) — the core's README and
`Documentation` URL point at that site's "Without Sphinx" section, which
the archived old repository it still builds does not have.

Then sphinx-needs' release (the planner wants it before
sphinx-test-reports), then `poe bump sphinx-test-reports --bump major` →
3.0.0.
chrisjsewell pushed a commit that referenced this pull request Oct 1, 2026
@claude
🧪 CI: upload coverage on pushes to master too
dadd8c4 
The Codecov upload steps required github.event.pull_request.head.repo
to equal the repository, a guard against fork pull requests that has no
value on a push event, so since #2009 every upload on master was skipped
(measured on the run for cc0ffe0: all five "upload to Codecov" steps
skipped). Codecov then compared every pull request against the newest
ancestor with a report, d68d10d, 35 commits back and from before that
PR split sphinx-test-reports, so the reports flag read -0.74% on pull
requests that never touched the package. The guard now applies to pull
request events only; a push to master uploads under its own secrets.
@chrisjsewell chrisjsewell mentioned this pull request Oct 1, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ubmarco ubmarco ubmarco approved these changes

Assignees

No one assigned

Labels

pkg: sphinx-test-reports Concerns the sphinx-test-reports package (packages/sphinx-test-reports) pkg: ub-test-reports Concerns the ub-test-reports package (packages/ub-test-reports): the Sphinx-free test-reports core pkg: workspace The repository as a whole: workflows, CI, release, docker, tooling, the workspace root

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chrisjsewell @ubmarco