Repository navigation
The -W gate is blind in ci.yml, and a warning-only cache failure discards a completed execution #271
Description
Summary
Two independent weaknesses in cache.yml turned a one-line content defect into four days without a usable execution cache, and hid it from the check that should have caught it first.
What happened
Four cache.yml runs failed or were cancelled between 2026-08-20 and 2026-08-24, while the last success was 2026-08-17. The publish build for publish-2026aug24 consequently fell back on the eight-day-old artifact, which shipped a site with inconsistent chapter numbering (filed separately).
The causes were not a single recurring one:
| Run | Date | Cause |
|---|---|---|
| 32331172292 | 2026-08-20T04:14Z | FileNotFoundError: '_fonts/SourceHanSerifSC-SemiBold.otf' in learning_approximation.md, plus the hoist_failure warning below — fixed by #262 |
| — | 2026-08-20T22:56Z | hoist_failure warning alone |
| 32433407779 | 2026-08-21T00:38Z | cancelled by hand two minutes in, not a failure |
| 32686259809 | 2026-08-24T03:23Z | hoist_failure warning alone — fixed by #265 |
The hoist_failure cause was one Sphinx warning promoted to an error by -W: lectures/hoist_failure.md:34: WARNING: Document headings start at H2, not H1 [myst.header]. The stale duplicated frontmatter block left in the body by the #260 resync parsed as body text, and its trailing --- turned the block into a setext H2 ahead of the real # 故障树不确定性.
Both underlying defects were lecture-content defects introduced by translate seed/forward operations, and both are now fixed. What follows is why they cost four days.
Defect 1 — the -W gate never re-reads, so it fires late and in the wrong workflow
cache.yml runs the sphinx-tojupyter build and then the HTML build with -W. Because the doctrees survive between them, the -W step re-reads nothing, so the warning surfaces only on a cold run. The same shape exists in publish.yml, where the rm -r _build/.doctrees line is commented out.
The consequence is that a content defect that should fail a PR's CI instead fails the weekly cold cache build, days later and far from the change that caused it.
Suggested fix: clear _build/.doctrees between the two builds so the -W step re-reads every source. Smallest change with the strongest guarantee. Alternatively drop -W from the tojupyter step and rely on a re-reading HTML step.
Defect 2 — a warning-only failure discards 106 minutes of successful execution
On the 2026-08-24 run, all 142 notebooks executed successfully (142 Executed notebook in lines, zero CellExecutionError) over 1h46m, and the artifact was then thrown away because the upload step does not run on failure.
Suggested fix: add if: success() || failure() to the Upload "_build" folder (cache) step. jupyter-cache does not store failed executions, so the artifact stays correct even when a lecture genuinely errors, and the run's red status remains the signal. This change alone would have made the 2026-08-24 publish a cache hit despite the hoist_failure defect.
Also noted
cache.yml still carries bare apt-get install steps with no timeout-minutes and no retry — the weakness PR #263 would have hardened before it was closed. Not implicated in any of these four failures (apt was green throughout: graphviz 31s, texlive 3m28s), but it remains the open item behind the documented hang class.
Refs: QuantEcon/project-translation#48 (delivery integrity); #262, #265 (the two content fixes); #263 (closed apt hardening).
Activity
mmcky commentedon Aug 24, 2026
Correction — Defect 1 named the wrong workflow, and the real evidence is better than what I filed.
As written, Defect 1 said
cache.yml"runs the sphinx-tojupyter build and then the HTML build with-W" and that surviving doctrees make the-Wstep re-read nothing. That is wrong aboutcache.yml.cache.ymlhas exactly one build step (Build HTML, line 39) and downloads no artifact — it is a cold build from scratch. That is precisely why it did catch thehoist_failurewarning, correctly, and failed.cache.ymlwas doing its job.The two-build shape is in
ci.yml— the PR gate — and that is where the blindness lives. The evidence is the CI run for #260, the PR that introduced the duplicated frontmatter:rm -rf _build/.doctrees— ci.yml's clear step runsjb build lectures -n -W --keep-going --builder=custom --custom-builder=jupyter, underset -eo pipefailupdating environment: [new config] 143 added, 0 changed, 0 removed— it reads everythinglectures/hoist_failure.md:34: WARNING: Document headings start at H2, not H1 [myst.header]-Wand the warningupdating environment: 0 added, 0 changed, 0 removed— reads nothingbuild succeeded.— no##[error]anywhere;previewreported passSo two things compound, and both need naming:
-Wis not enforced on the custom/jupyter builder. The warning was emitted under-n -W --keep-goingwithset -eo pipefailin force, andjb build ... --builder=custom --custom-builder=jupyterstill exited 0. Whatever the mechanism inside jupyter-book, the practical effect is that-Won that step is decorative.-Wnever sees a warning. The HTML build reuses the doctrees the tojupyter build just wrote, so it re-reads nothing and there is nothing left to warn about. Clearing doctrees once after the download does not help — by the time the HTML build runs, the tojupyter build has repopulated them.The fix follows from (2) and is one line: clear
_build/.doctreesbetween the two builds inci.yml, not only after the download. Then the-WHTML build re-reads every source and the gate actually gates. Worth also confirming (1) independently, because if the custom builder silently drops-Wthen the same hole exists anywhere that builder runs.Note this applies to
publish.ymltoo — after #272 its HTML build will likewise re-read nothing, so its-Wis equally decorative. That does not affect what #272 fixes (the environment is fresh because the restored one is deleted, so the toctree and numbering are correct), but publish is not a warning gate and should not be mistaken for one.Defects 2 and 3 as filed are unaffected and stand: the upload step is not
if: always(), discarding a completed 106-minute execution; and the four failures had two distinct causes, not one.The body above has been amended. I am leaving the error visible here rather than editing it away, because "the weekly cache build is the thing that catches content defects" is exactly the wrong lesson to draw — the PR gate should have caught this four days earlier, and the reason it did not is now evidenced.
[-]cache.yml: the -W gate never re-reads, and a warning-only failure discards a completed execution[/-][+]The -W gate is blind in ci.yml, and a warning-only cache failure discards a completed execution[/+]mmcky commentedon Oct 18, 2026
Field evidence, 2026-10-18: the blind gate let three defects onto
mainin one day. #280 (four unresolvedprogramming:cross-references) and #290 (ak_aiyagariandtwo_computationraising onimport jax) both had the failures in their preview logs and both reported build succeeded; they were merged on the green check and repaired in #297. A second gap compounded it: when the HTML build is the step that fails,ci.ymluploads_build/html/reports, while myst-nb writes the traceback to_build/jupyter/reports, so there was nothing to download and the cause had to be found by executing the lecture locally. A one-line patch adding the second path to that upload step is in QuantEcon/project-translationresearch/2026-10-18-translate-status/zh-cn-ci-upload-reports.patch(private); it needs theworkflowtoken scope to push, which the session lacked (QuantEcon/infrastructure#47).