Skip to content

Speed up the docs build; refactor the board support matrix code - #11921

Merged
tannewt merged 14 commits into
adafruit:mainfrom
dhalbert:sphinx-parallel-build
Nov 19, 2026
Merged

tannewt merged 14 commits into
adafruit:mainfrom
dhalbert:sphinx-parallel-build

Conversation

@dhalbert

@dhalbert dhalbert commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Code by Claude, reviewed by @dhalbert.

A fresh make html goes from 57 seconds to 42 seconds on a 20-CPU Linux machine. The HTML output is unchanged. The docs build scripts were refactored to remove duplicate work and for clarity.

This also speeds up the GitHub CI docs build and the Read the Docs builds. This PR's Read the Docs build took 264 seconds instead of about 392 seconds for other recent PRs, because its Sphinx step went from about 315 to 186 seconds. Read the Docs doesn't use -j auto (and we can't make it do that), so that saving comes entirely from collecting the board data once.

Parallel Sphinx builds

  • Parallelized: make html, make latexpdf and the CI HTML build now use -j auto. Confirmed that the output is still identical after the change.

Board data collected twice per docs build

  • Redundant work: the docs build computed the board support matrix twice: once in board_stub_builder.py, run by make stubs, and again in conf.py. Each run invokes make once per board, about 15 seconds.
  • Fix: make stubs now saves the board data as circuitpython-stubs/board_data.json, and conf.py reuses it. If the file is missing, conf.py collects the data itself. The data file is not included in the stubs package.

Support matrix code

  • shared_bindings_matrix.py was moved to tools/, so other tools/ scripts don't have to add docs/ to their path.
  • collect_board_data() now gathers each board's data once. support_matrix_by_board() keeps its signature and formats that data. Its output for the existing callers is unchanged.
  • zephyr-cp boards now get empty MCU and flash values, instead of raising FileNotFoundError when add_chips is set. They don't record those yet.

Board stubs

  • Wrong NVM size: the NVM size in each board stub's docstring was wrong for many boards. SAMD51 boards showed the SAMD21 value (256 instead of 8192), and a board-level CIRCUITPY_INTERNAL_NVM_SIZE was ignored. The value is a C macro, which only the C preprocessor can compute.

  • Fix for now: removed the NVM size line. Board names now come from the same parsing the docs use. This fixes a glitch where lolin_c3_mini and lolin_c3_pico had a trailing // from Wemos MP comment in the name.

We can add back the (correct!) NVM size info later, maybe by moving the definition to .mk from .h.

Smaller fixes

  • tools/board_summaries.py iterated over a dict instead of .items() and raised ValueError before writing any CSV.
  • circuitpython_setboard -l showed hyphenated ports as their first word, such as atmel for atmel-samd, so filtering by atmel-samd found nothing.

CI

  • Problem: changes under tools/board_stubs/ matched no rule in ci_set_matrix.py, so they built every board and skipped the docs build. tools/shared_bindings_matrix.py needs the same docs-only treatment it had under docs/.
  • Fix: both paths now build the docs and skip the board builds, the Zephyr tests, the unix tests and mpy-cross. This would have been in CI: skip tests and mpy-cross jobs when changes can't affect them #11916 but was found too late.

PDF build

  • Too verbose: pdflatex's first pass prints a useless warning for every cross-reference, about 3,300 lines; later passes resolve the cross-references. The PDF prerequisites were listed only in the CI workflow.
  • Fix: make latexpdf runs latexmk with -silent. Warnings left after the last pass, and errors, are still reported. make latexpdf LATEXMKOPTS= shows the full output. docs/README.md was also updated: it now lists the packages the PDF build needs, and notes that setting SPHINXOPTS replaces the defaults.

dhalbert and others added 14 commits November 19, 2026 12:34
Parallel reading and writing gives identical HTML and LaTeX output
and saves a few seconds per build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six scripts in tools/ import it, each by adding docs/ to sys.path.
In tools/, they can import it directly. conf.py adds tools/ to sys.path.

ci_set_matrix.py: keep treating the file as docs-only: it triggers
the docs build, but not board builds or the Zephyr tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
collect_board_data() gathers everything about each board once, as JSON
types. support_matrix_by_board() keeps its signature but now only
formats that data, collecting it first if no board_data is passed.

zephyr-cp boards have no mpconfigboard.h or .mk, so they now get empty
MCU and flash values instead of raising FileNotFoundError when
add_chips is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NVM size in board stub docstrings has been wrong for many boards:

- SAMD51 boards showed 256, the SAMD21 value. The regex took the first
  define in atmel-samd's mpconfigport.h, ignoring #ifdefs.
- A board-level CIRCUITPY_INTERNAL_NVM_SIZE in mpconfigboard.h was never
  used: the code searched the port's file a second time instead.

The value is a C macro, so only the C preprocessor can compute it.
Stub builds don't have the port toolchains for that. The line was only
informational, so drop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
board_stub_builder.py collects the board data once, uses it for both the
support matrix and the board names, and saves it as
circuitpython-stubs/board_data.json for conf.py to reuse.

The names now come from the same parsing the docs use, which cuts at the
closing quote. lolin_c3_mini and lolin_c3_pico stubs no longer include
the trailing "// from Wemos MP" comment in the board name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before, the docs build collected the board data twice: once inside
make stubs and again here. Each collection runs make once per board
(about 15 seconds). A fresh HTML build drops from about 54 to 42 seconds.

If the saved file is missing, support_matrix_by_board() still collects
the data itself.

The old comment said the stubs must be built before the matrix. The
matrix no longer reads the stubs; now the matrix needs the saved data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
write_csvs() iterated over the dict itself, so unpacking each key into
(key, values) raised ValueError before any CSV was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The port was read by splitting the " - port: ..." line on every "-",
so atmel-samd became "atmel". `circuitpython_setboard -l` then showed
the wrong heading, and filtering by "atmel-samd" found nothing. Split
on the first ":" instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under docs/, shared_bindings_matrix.py matched ^docs/ in the tests and
mpy-cross ignore patterns. Its move to tools/ lost that. Add it to both
patterns so it is still treated as docs-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Changes under tools/board_stubs/ matched no rule, so CI built every
board and skipped the docs build. That code only builds the board
stubs, which the docs build makes.

Build the docs for it, and skip the board builds, the Zephyr tests,
the unix tests and mpy-cross.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The PDF build needs LaTeX and rsvg-convert, which were listed only in
the CI workflow. Document the Debian/Ubuntu packages and make latexpdf.

Setting SPHINXOPTS replaces the default -W --keep-going -j auto, so the
example now includes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pdflatex's first pass prints a warning for every cross-reference, about
3,300 lines, because nothing is resolved until a later pass. -silent
sends the pdflatex output to the .log file only. latexmk still reports
references left undefined after the last pass, and still fails on
LaTeX errors.

Use "make latexpdf LATEXMKOPTS=" to see the full output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dhalbert
dhalbert requested a review from a team November 19, 2026 19:42

@tannewt tannewt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@tannewt
tannewt merged commit c20fa17 into adafruit:main Nov 19, 2026
1430 of 1432 checks passed
@dhalbert
dhalbert deleted the sphinx-parallel-build branch November 19, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants