✨ sphinx-codelinks: read ubproject.toml through ub-project, and let -D override the TOML

AGENTS.md

@@ -177,11 +177,10 @@ directives.)
 **sphinx-codelinks needs NEITHER renderer**: nothing in that package draws a diagram, its
 docs build installs no `apt_packages` and its CI cell asks for graphviz only because it
 shares a cell with sphinx-mounts. What it does need is `git` on `PATH` — its suite builds
-real repositories and a real `git worktree` — and, for the 56 tests behind the optional
+real repositories and a real `git worktree` — and, for the tests behind the optional
 preprocessor-aware C/C++ engine, the `libclang` wheel: a root dependency group,
 `codelinks-libclang`, which every `test-codelinks*` task adds for you. Without it those
-tests SKIP rather than fail, so a run that lacked it looks green
-(`303 passed, 26 skipped` instead of `359 passed`).
+tests SKIP rather than fail, so a run that lacked it looks green.
 `bazel` (or `bazelisk`) is the other optional binary — without it the `bazel`-marked tests
 skip, and `test-mounts` deselects them anyway. The browser tests (`-m jstest`, which
 `test-needs` excludes) additionally need a browser, and it is not a package: `uv run poe

CLAUDE.md

@@ -21,8 +21,8 @@ What a cloud environment for this repository needs, measured on the Ubuntu 24.04
   needflow tests fail rather than skip. Nothing else is needed: `prek` and `poe` come from
   `uv sync`. `libclang` is not a system package here and does not belong in this script:
   it is a 23 MiB wheel behind the root `codelinks-libclang` dependency group, which
-  `poe test-codelinks` adds for you. Without it 56 of sphinx-codelinks' 359 tests skip
-  rather than fail, so a run that lacked it would look green.
+  `poe test-codelinks` adds for you. Without it the libclang-gated tests of
+  sphinx-codelinks skip rather than fail, so a run that lacked it would look green.
 - **Environment variables**: none to set on the environment. `.claude/settings.json` sets
   `UV_HTTP_TIMEOUT=180` for every Claude Code session, cloud ones included (committed
   project settings are read there): uv's 30 s default is not enough for this lock through

packages/sphinx-codelinks/AGENTS.md

@@ -37,7 +37,7 @@ design/                 # import-commit-map.txt: old hash -> new hash for the 20
 src/sphinx_codelinks/   # Main source code
 ├── __init__.py         # `__version__` (public, in `__all__`) and the Sphinx `setup()`
 ├── cmd.py              # CLI commands using Typer
-├── config.py           # Configuration dataclasses + TypedDicts, and the TOML loader
+├── config.py           # Configuration dataclasses + TypedDicts, and the TOML loader, `load_codelinks_table`
 ├── logger.py           # Logging utilities
 ├── needextend_write.py # Write RST files with Sphinx-Needs directives
 ├── analyse/            # Code analysis module
@@ -60,7 +60,7 @@ src/sphinx_codelinks/   # Main source code
 tests/                  # Test suite -- `tests/__init__.py` is why this path is NOT in the
 ├── __init__.py         #   root `testpaths` (see the root AGENTS.md)
 ├── conftest.py         # Pytest fixtures and configuration
-├── test_*.py           # 16 test modules
+├── test_*.py           # 17 test modules
 ├── __snapshots__/      # Syrupy snapshot test fixtures
 ├── data/               # Test data and fixtures
 └── doc_test/           # minimal Sphinx projects for the integration tests
@@ -89,36 +89,32 @@ cell.
 **`test-codelinks` syncs the group into the DEFAULT `.venv`.** It has no
 `UV_PROJECT_ENVIRONMENT` of its own, unlike its three `-sphinx7/8/9` siblings, so the wheel
 lands in the environment every other command uses — and the next plain `uv sync --frozen`
-prunes it out again (`Uninstalled 1 package: - libclang==18.1.1`). So the two numbers only
+prunes it out again (`Uninstalled 1 package: - libclang==18.1.1`). So the two results only
 appear either side of that sync, and this is the sequence that shows both:
 
 ```bash
-uv run poe test-codelinks                                            # 359 passed
+uv run poe test-codelinks                                            # every test runs, none skipped
 uv sync --frozen                                                     # removes libclang again
-uv run --frozen --no-sync pytest packages/sphinx-codelinks/tests     # 303 passed, 26 skipped
+uv run --frozen --no-sync pytest packages/sphinx-codelinks/tests     # green, the libclang tests skipped
 ```
 
 **Both runs are green, and only the first tested the engine.** The four modules that need
 it carry `pytest.importorskip("clang.cindex")`, so a run without the group skips politely
 rather than failing — which means a task or a CI line that quietly lost the group would
 look like a pass. (CI is fenced: the Extensions cell asserts `import clang.cindex` right
-after its sync.) If you are changing anything under `analyse/preproc/`, check the number.
+after its sync.) If you are changing anything under `analyse/preproc/`, check that nothing
+skipped.
 
 The summary prints **26 skipped**, not 56: three of the four guards are module-level
 `pytest.importorskip`, which pytest reports as one skip per module and never collects the
 tests inside. 56 is how many test cases stop running.
 
-### This package caps `click` and `typer`, and nothing else in the lock does
+### It reads `ubproject.toml` through `ub-project`
 
-`click < 8.2` (8.2 produces empty errors when the CLI is given no arguments) and
-`typer >=0.16.0,<0.26.8` (0.26.8 removed `rich_utils.STYLE_METAVAR`, which
-`sphinxcontrib-typer` still imports for the docs build). Measured across every
-`requires-dist` in `uv.lock`: **no other workspace MEMBER names either**, and for `click`
-no package in the lock does at all. `typer`, `rich` and `shellingham` are named by third
-parties there — `sphinxcontrib-typer` (which is exactly what the `typer` cap exists for),
-`typer` itself, `memray` and `textual` — so the `typer` cap is the one that could bind on
-someone else. The direction to watch is the reverse one: the day a root, `test` or `dev`
-dependency wants `click>=8.2`, `uv lock` will fail and the reason will be here.
+`load_codelinks_table` in `config.py` is ub-project's `load_toml` + `select_table`, and both
+readers call it; `src/` imports no `tomllib`. It returns raw values: relative paths are
+anchored with ub-project's `anchor` where they are used, never in the loader. (`click` is no
+longer a dependency and `typer` is no longer capped — the changelog's `Unreleased` says why.)
 
 ## Documentation
 
@@ -182,7 +178,7 @@ def form_https_url(
 **No `--` before the pytest arguments.** poe appends trailing words to the task's command
 verbatim and forwards a `--` along with them, and pytest then reads `--snapshot-update` as
 a file path: `poe test-codelinks -- --collect-only -q` collects **0 items**, where
-`poe test-codelinks --collect-only -q` collects 359.
+`poe test-codelinks --collect-only -q` collects the whole suite.
 
 ### Test Structure
 
@@ -328,7 +324,7 @@ The extension connects to these Sphinx events (in execution order):
 
 1. **sphinx-needs Dependency**: The extension requires sphinx-needs and checks for its presence in `setup()`. It adds extra options (`project`, `file`, `directory`, URL fields) and a custom need type (`srctrace`).
 
-2. **TOML Configuration**: Configuration can be loaded from a TOML file specified in `conf.py` via `src_trace_config_from_toml`. The TOML is parsed and values are set on the Sphinx config object.
+2. **TOML Configuration**: Configuration can be loaded from a TOML file specified in `conf.py` via `src_trace_config_from_toml`. The TOML is parsed and values are set on the Sphinx config object, except a key given with `-D` — `src_trace_projects` and `src_trace_outdir` excepted (`NOT_OVERRIDABLE_FROM_D`): Sphinx refuses a `-D` for both yet keeps it in `config.overrides`.
 
 3. **Source Page Generation**: The `generate_code_page()` function yields tuples of `(pagename, context, template)` for each traced source file, allowing Sphinx to generate standalone HTML pages with syntax-highlighted source code and line-number anchors.
 
@@ -351,7 +347,9 @@ the shape a TOML file may carry, and a `@dataclass` holding the loaded, validate
 - validation is `jsonschema`'s `validate(instance=…, schema=…)` per field, against a
   schema each config class returns from its own `get_schema`, collected by its
   `check_schema` and `check_*` methods into a list of error strings — not raised
-- the TOML loader is `load_config_from_toml` in `cmd.py`
+- the TOML loader is `load_codelinks_table` (ub-project's `load_toml` + `select_table`),
+  which both `load_config_from_toml`s — the Sphinx hook in `sphinx_extension/source_tracing.py`
+  and the CLI's in `cmd.py` — call
 
 #### Source Discovery (`source_discover/`)
 

packages/sphinx-codelinks/compat-requirements.txt

@@ -14,14 +14,15 @@
 # without this line `clang.cindex` is absent -- and that failure is LOUD, not silent:
 # `release.yaml` runs the `import_check` walk BEFORE pytest, and the walk fails outright
 # on `sphinx_codelinks.analyse.preproc`, whose `__init__` imports the libclang loader
-# eagerly (`FAIL 1 of 21 modules failed to import`, exit 1, under the step's default
-# `bash -e`). Measured. So deleting this line does not give a green 301/26 run -- it turns
-# the release of sphinx-codelinks red, by name, before the suite starts.
+# eagerly (`FAIL 1 of ... modules failed to import`, exit 1, under the step's default
+# `bash -e`). Measured. So deleting this line does not give a green run with those
+# tests skipped -- it turns the release of sphinx-codelinks red, by name, before the
+# suite starts.
 #
-# What the line buys on top of that loud failure is the 56 tests behind
-# `pytest.importorskip("clang.cindex")`: with it the compat cell reports 359 passed, and
-# without it -- if the walk were ever removed or reordered -- it would report
-# `303 passed, 26 skipped`.
+# What the line buys on top of that loud failure is the tests behind
+# `pytest.importorskip("clang.cindex")`: with it the compat cell runs them, and without
+# it -- if the walk were ever removed or reordered -- they would skip rather than fail,
+# and the cell would look green.
 #
 # The floor is 18 -- what has actually been run here -- rather than the extra's 16, which
 # states the wider range the published package supports.

packages/sphinx-codelinks/docs/changelog.rst

@@ -151,9 +151,46 @@ New and Improved
 
   A default file that does not exist or contains no ``[codelinks]`` table is silently
   ignored, so existing projects without ``ubproject.toml`` keep building without new
-  warnings. Only a TOML file that was explicitly configured but cannot be loaded
-  triggers a Sphinx warning, as before. The documentation project itself now stores
-  its codelinks configuration in ``ubproject.toml``.
+  warnings. The default is the value ``ubproject.toml`` exactly -- left unset, or
+  written in :file:`conf.py` as that string -- so a 1.4.0 project that wrote
+  ``src_trace_config_from_toml = "ubproject.toml"`` no longer gets a warning for a
+  missing file or a missing table. Any other value, ``./ubproject.toml`` included, is an
+  explicit file and warns when missing or without the table. A file that exists but
+  cannot be read or parsed -- invalid TOML, not UTF-8, a directory, a ``codelinks`` key
+  that is not a table -- warns (``codelinks.config``) either way, as any configured file
+  did at 1.4.0. The
+  documentation project itself now stores its codelinks configuration in
+  ``ubproject.toml``.
+
+- 🐛 A value given on the command line with ``-D`` now overrides the TOML file.
+
+  ``sphinx-build -D src_trace_set_local_url=0`` was silently overwritten by a
+  ``set_local_url`` in the ``[codelinks]`` table. The order is now ``-D`` > TOML >
+  :file:`conf.py` > default, as in Sphinx-Needs. Only the full ``src_trace_<key>`` name
+  counts: a bare ``-D set_local_url=0``, which Sphinx rejects as an unknown setting,
+  leaves the TOML value alone. ``src_trace_projects`` always comes from :file:`conf.py`
+  or the TOML: Sphinx refuses to override a dictionary setting with ``-D``, and the
+  dotted ``-D src_trace_projects.<name>=...`` form is not supported. Sphinx refuses
+  ``-D src_trace_outdir`` too, whose default is a path, and the TOML value then stands.
+  A ``-D`` value Sphinx cannot convert (``-D src_trace_set_local_url=yes`` on
+  Sphinx 8.2 or newer) is now reported by Sphinx instead of being silently replaced by
+  the TOML, as it already was without a TOML.
+
+- 👌 ``ubproject.toml`` is read through `ub-project <https://pypi.org/project/ub-project/>`__,
+  the shared reader of the Sphinx-Needs family, which is now a dependency
+  (``ub-project>=1.1.0,<2``).
+
+  Both the Sphinx extension and ``codelinks analyse`` parse the file and select the
+  ``[codelinks]`` table through it, and relative paths are anchored through its
+  ``anchor``, at the same directories as before. What changes is what a broken file
+  says: every message names the file and the problem (``invalid TOML``,
+  ``not valid UTF-8``, ``[codelinks] must be a table, got str``), an explicitly
+  configured file without a ``[codelinks]`` table says so instead of printing
+  ``'codelinks'``, and ``codelinks analyse`` shows why a file could not be loaded rather
+  than only that it could not -- a ``codelinks`` key that is ``0``, ``false`` or ``[]``
+  is now reported as not a table instead of as a missing section. The extension's
+  warnings about its configuration file carry the type ``codelinks.config``, so
+  ``suppress_warnings = ["codelinks.config"]`` silences them.
 
 .. _`release:1.4.0`:
 

packages/sphinx-codelinks/docs/components/configuration.rst

@@ -24,7 +24,7 @@ src_trace_config_from_toml
 
 Specifies the path to a `TOML file <https://toml.io>`__ containing **Sphinx-CodeLinks** configuration options. This allows you to maintain configuration in a separate file for better organization.
 
-**Type:** ``str`` (relative path to the directory where conf.py is located)
+**Type:** ``str`` (relative to the directory where conf.py is located; an absolute path also works)
 **Default:** ``"ubproject.toml"``
 
 .. code-block:: python
@@ -37,8 +37,11 @@ When using a TOML configuration file:
 - Configuration options are placed under a ``[codelinks]`` section
 - The ``src_trace_`` prefix is omitted in the TOML file
 - TOML configuration overrides settings in :file:`conf.py`
+- A value given on the command line with ``-D`` (``sphinx-build -D src_trace_set_local_url=0``) overrides both.
+  ``src_trace_projects`` can only come from :file:`conf.py` or the TOML -- the dotted ``-D`` form is not supported --
+  and Sphinx refuses ``-D src_trace_outdir`` as well, so a TOML ``outdir`` stands
 
-.. note:: ``ubproject.toml`` is the shared ubCode project file, which other useblocks tools (e.g. Sphinx-Needs via ``needs_from_toml`` or the ubCode checker in VS Code) read as well. Keeping the ``[codelinks]`` configuration in this file makes all tools aware of the configured projects. If the default file does not exist or contains no ``[codelinks]`` section, it is silently ignored and the configuration from :file:`conf.py` is used. Only a TOML file that was explicitly configured but cannot be loaded triggers a Sphinx warning.
+.. note:: ``ubproject.toml`` is the shared ubCode project file, which other useblocks tools (e.g. Sphinx-Needs via ``needs_from_toml`` or the ubCode checker in VS Code) read as well. Keeping the ``[codelinks]`` configuration in this file makes all tools aware of the configured projects. The default is the value ``ubproject.toml`` exactly -- left unset, or written in :file:`conf.py` as that string: that file may be absent or have no ``[codelinks]`` section, and both are silent -- the configuration from :file:`conf.py` is then used. Any other value, ``./ubproject.toml`` or an absolute path to the same file included, is an explicit file: missing or without the ``[codelinks]`` section, it warns. A file that exists but cannot be read or parsed warns either way, as any configured file did at 1.4.0. These warnings are of type ``codelinks.config``, so ``suppress_warnings = ["codelinks.config"]`` silences them.
 
 .. caution:: Relative paths specified in the TOML file are resolved relative to the directory containing the TOML file, not the Sphinx project root.
 

packages/sphinx-codelinks/pyproject.toml

@@ -28,6 +28,7 @@ dependencies = [
   # so the floor is the only statement of what was actually tested, and the cap is what
   # stops a future major being co-installed with a wheel written against this one.
   "sphinx-needs>=8.5.0,<9",
+  "ub-project>=1.1.0,<2",
   "jinja2",
   "pygments",
   "docutils>=0.21",             # the `typing` group's series

packages/sphinx-codelinks/src/sphinx_codelinks/cmd.py

@@ -1,5 +1,4 @@
 import json
-import tomllib
 from collections import deque
 from os import linesep
 from pathlib import Path
@@ -14,6 +13,7 @@
     CodeLinksProjectConfigType,
     anchor_preproc_paths,
     generate_project_configs,
+    load_codelinks_table,
 )
 from sphinx_codelinks.logger import configure_cli, logger
 from sphinx_codelinks.needextend_write import MarkedObjType, convert_marked_content
@@ -23,6 +23,7 @@
     SourceDiscoverConfigType,
 )
 from sphinx_codelinks.source_discover.source_discover import SourceDiscover
+from ub_project import ProjectConfigError, anchor
 
 app = typer.Typer(
     no_args_is_help=True, context_settings={"help_option_names": ["-h", "--help"]}
@@ -134,8 +135,8 @@ def analyse(  # for CLI, so it needs the branches
             raise typer.BadParameter(f"{linesep.join(errors)}")
 
         # src dir shall be relevant to the config file's location
-        src_discover_config.src_dir = (
-            config.parent / src_discover_config.src_dir
+        src_discover_config.src_dir = anchor(
+            src_discover_config.src_dir, config.parent
         ).resolve()
 
         src_discover = SourceDiscover(src_discover_config)
@@ -147,8 +148,8 @@ def analyse(  # for CLI, so it needs the branches
 
         # git_root shall be relative to the config file's location (like src_dir)
         if analyse_config.git_root is not None:
-            analyse_config.git_root = (
-                config.parent / analyse_config.git_root
+            analyse_config.git_root = anchor(
+                analyse_config.git_root, config.parent
             ).resolve()
 
         # preprocessor compile_commands / include dirs are relative to the config
@@ -329,15 +330,16 @@ def write_rst(  # for CLI, so it takes as many as it requires
 
 def load_config_from_toml(toml_file: Path) -> CodeLinksConfigType:
     try:
-        with toml_file.open("rb") as f:
-            toml_data = tomllib.load(f)
-
-    except Exception as e:
+        codelink_dict = load_codelinks_table(toml_file)
+    except ProjectConfigError as error:
+        # ub-project's message already names the file and says what is wrong
+        raise typer.BadParameter(str(error)) from error
+    except Exception as error:
+        # the TOML parser can also fail with an exception ``load_toml`` does not wrap (a
+        # RecursionError on a pathologically nested file): still a usage error
         raise typer.BadParameter(
-            f"Failed to load CodeLinks configuration from {toml_file}"
-        ) from e
-
-    codelink_dict = toml_data.get("codelinks")
+            f"Failed to load CodeLinks configuration from {toml_file}: {error}"
+        ) from error
 
     if not codelink_dict:
         raise typer.BadParameter(f"No 'codelinks' section found in {toml_file}")