Add quarto call axe, a hidden experimental site accessibility scanner - #14815
Draft
cwickham wants to merge 45 commits into
Draft
Add quarto call axe, a hidden experimental site accessibility scanner#14815cwickham wants to merge 45 commits into
quarto call axe, a hidden experimental site accessibility scanner#14815cwickham wants to merge 45 commits into
Conversation
Registers `axe <site-dir>` under the already-hidden `dev-call` parent, with the six v1 flags (--pages, --max-pages, --viewports, --themes, --timeout, --settle) parsed into a typed AxeScanConfig and echoed. The scan, aggregate and report stages follow.
Ports the quarto-web harness's raw-CDP driver to typed TypeScript and replaces its render-time axe hook with scan-time injection: after load and settle, the vendored axe.min.js (4.10.3) is evaluated in the page and axe.run() awaited under the per-cell timeout. Any rendered site now scans as-is, offline, on one known axe version — so the CDN warm-up hack, the console-payload sniffing and the special render all come out. The orchestrator serves the site dir with httpFileRequestHandler, launches headless Chrome via getBrowserExecutablePath() on its own throwaway profile, walks the page x viewport x theme matrix, and writes raw per-cell JSON to _axe-checks/cells/. Cells fail closed: timeout, evaluation error or a missing payload is reported and exits 2, never a pass. Chrome rewrites its profile as it shuts down, so the temp profile dir is removed only after the process has really exited. On Ctrl-C an onCleanup handler kills Chrome and leaves the dir behind, since cleanup handlers can't await. Verified against a rendered website: all cells ok; identical results with Chrome's DNS blackholed (offline); prefers-color-scheme emulation catches a dark-only color-contrast failure and the mobile viewport a mobile-only one; a cell that blocks the main thread times out and the next cell still scans; SIGINT leaves no orphan Chrome; bogus, empty and non-directory site-dirs exit 2. Adds an optional onListen to handleHttpRequests so the scan's own progress output isn't preceded by Deno's "Listening on ..." line.
Neither exclusion did any work here. Tabster's `[data-tabster-dummy]` sentinels arrive with the preview client's Fluent UI bundle, and `.quarto-axe-report` is the in-page panel axe-check.js builds when a site is rendered with `axe:` metadata — so neither appears in the plain rendered sites this command scans. Both were carried over from axe-check.js's own axe.run() call, along with a "won't fix upstream" claim that tabster#288 does not support: that issue is still open with no maintainer response. If someone does scan an `axe:`-rendered site, the overlay's own violations are information rather than noise. Results on the test site are unchanged: 12 cells, 12 ok, same rule ids.
Ports the harness's aggregate and report stages, with the semantics the notes had already settled but the harness code had not caught up with. Signatures use the investigated "option 1" normalization: attribute values are kept with digit runs wildcarded, rather than stripped. Stripping turned `div[data-bs-target=".callout-4-contents"]` into the generic `div[data-bs-target]`, which every Bootstrap collapse, modal, tab and dropdown also matches — so one accepted callout defect silently suppressed unrelated conformance failures site-wide. The baseline is now the hand-written projection ledger: `signature` x `pages` x `impact`, empty `pages` meaning site-wide, a listed `pages` fail-closed at finding level, and escalation past the accepted impact re-alerting. `--update-baseline` is not ported; stale entries are reported and pruned by hand. Conformance labels come from `axe-check.js`'s own `axeConformanceLevel`, `impactRank` and `standardRank` rather than a second implementation, so a finding reads the same whether it came from a scan or the in-page report. That module guards its self-init on `typeof document`, which is what makes it importable here — the same property tests/unit/axe-*.test.ts rely on. `schemas.ts` holds Zod for both contracts. Baseline validation needed more than a schema: to Zod, a misspelled `impcat:` is an ignored unknown key plus a missing `impact`, so the report blamed `impact` while the culprit sat two characters away — and a `superRefine` never runs once a required field is missing. `parseBaseline` runs both checks and merges the issues, so the error names the typo and suggests the field it meant. `_axe-checks/` and `_axe-baseline.json` now anchor at the project root (nearest `_quarto.yml` at or above the site dir), falling back to the working directory for loose HTML. They sit beside the output dir, never inside it: a full render of a website or book deletes the output dir. Verified end-to-end on a rendered site: findings.json satisfies its own schema; all five baseline behaviours (site-wide accept, page-scoped accept, page-scoped re-alert on an unlisted page, impact-escalation re-alert, stale entry) land as expected; two hand-typo'd fields produce named errors with suggestions and exit 2. The report was driven in headless Chrome — row drill-down, column re-sort with row/detail pairing intact, the why-accepted column, the stale notice, and the copy-AI-briefing payloads.
The signature is the most consequential value the scanner computes: too broad and one accepted defect suppresses unrelated conformance failures site-wide, too narrow and a baseline entry stops matching when a counter changes. Two gaps around it. First, nothing recorded which normalizer produced a signature. `version` covers findings.json's field shape, so a normalizer change would leave every field intact while re-keying every signature — reported as "N new findings, M baseline entries not seen", which is indistinguishable from "you fixed everything and broke an equal amount". The ledger's notes, the record of why each finding was accepted, would quietly stop applying. `signatureScheme` makes that a named error instead, and points at occurrences[].target as the unchanged raw selector to re-annotate from. It is optional in the baseline, so the first hand-written ledger needs no ceremony. Second, the normalization had no tests. tests/unit/axe-signature.test.ts covers it as explicit should-collapse / must-stay-distinct pairs over selectors axe really emits on Quarto output, rather than as assertions about the regexes. The must-stay-distinct half is the important one: it pins that a callout, a modal, a carousel and a tabset link do not share a signature just because they all hang off data-bs-target, which is exactly what the pre-option-1 normalizer got wrong. A third group asserts exact output so a scheme change shows up as a diff rather than as a changed collapse count. 30 tests, all passing. Also covers a case with no fixture yet: a color-contrast payload missing its check data must fall back to the selector rather than key on "undefined on undefined" and collapse every contrast finding in the site into one.
The two-pass reader was built on a wrong premise. I had it that a schema can't report an unrecognized key and a missing required field together, because a `superRefine` never runs once a required field is missing. That is true of `superRefine`, but strict-key checking is part of the object parse itself, so `.strict()` reports both in one pass: findings.0.impact: Required findings.0: Unrecognized key(s) in object: 'impcat' Adjacent lines make the typo obvious, so the hand-rolled Levenshtein "did you mean" was polish sitting on top of ~60 lines that Zod already covers. Deleted, along with the tolerated-field set and the manual per-entry loop; schemas.ts drops from 341 to 275 lines. Keeping the paste-a-whole-finding affordance costs one expression rather than the machinery: declare every finding field as an ignorable optional and merge the entry schema over the top, so its required fields stay required. A finding pasted out of findings.json plus a note validates; an invented field is still rejected. `parseBaseline` survives only for the signature-scheme check, which needs to explain what a mismatch means rather than say "invalid literal". Behaviour change worth noting: the outer object is `.strict()` too, so an unknown top-level key is now an error. That catches a misspelled `findings` but also rejects a hand-added `"_comment"`, since JSON has nowhere else to put one. tests/unit/axe-baseline-parse.test.ts pins the deliberate choices: the paste shortcut validates, a typo names both halves, an invented field is rejected, `pages` and `note` are required so scope and rationale are never guessed, and a stale scheme is one named error rather than mass staleness. 42 axe unit tests passing.
A purpose-built inaccessible Quarto site under tests/docs/axe-scan/site, with every violation planted on purpose and a manifest README recording intent. Source only: a committed render would carry site_libs/ and go stale, so the smoke test renders it in setup and removes it after. Writing the manifest first paid off twice, because two planted cases did not behave as designed. The tier-2 links were meant to prove that three `#panel-N` components collapse to one signature while a `#settings-dialog` stays separate. They all collapsed. axe had keyed them on `href`, which the normalizer drops as volatile, so the attribute never reached the selector. Switching to `<button>` — no href to key on — makes axe select on the attribute and the case works as intended. That distinction is the whole justification for keeping attribute values, so it is now proven on real markup rather than on hand-written selectors. The dark-only contrast paragraph also produced a second, unplanned contrast failure: it contained inline code, and cosmo's code colour fails against the planted background. That would have made the fixture depend on a theme palette. The paragraphs are now kept free of any element carrying its own colour, and the README says why. One finding is left unplanned on purpose: Quarto's navbar-toggler, which is systemic and mobile-only and belongs to Quarto. It is documented, and no assertion depends on it — or on any total count — so a Quarto upgrade that fixes it cannot fail these tests. Tests, 77 passing: - axe-aggregate.test.ts (19) drives the aggregate stage from eight verbatim per-cell captures. Verbatim matters: keeping whole payloads means an axe upgrade that changes their shape surfaces in a fast unit test. - axe-baseline-reconcile.test.ts (15) uses hand-built cells, where control beats fidelity: site-wide and page-scoped accepts, the re-alert on an unlisted page, escalation, de-escalation staying accepted, stale detection in three variants, and duplicate-entry merging. - smoke/axe/axe-scan.test.ts runs the real thing — browser, static server, all three stages — in 15s, asserting projections looked up by signature. The fixture's committed _axe-baseline.json exercises all four reconcile outcomes in one file, including an entry accepted at a lower impact than planted (so escalation re-alerts) and one matching nothing (so staleness is reported). Its `.a11y-accepted-example` entry is a designed future failure: when exclude-in-source ships, that entry should turn stale.
`quarto-bld validate-bundle` runs eslint's no-undef over the bundled
quarto.js, and it caught two things this branch introduced.
The first was importing `axe-check.js` to reuse its three conformance
labellers. That looked like good reuse — the labels match the render-time
`axe:` report by construction — but `axe-check.js` is a browser module, so
esbuild inlined the entire overlay (reveal navigation, dashboard rescan,
report DOM) into the CLI bundle for the sake of three pure functions, and
its page globals surfaced as four no-undef errors:
'Reveal' is not defined
'getComputedStyle' is not defined
'bootstrap' is not defined
They are `typeof`-guarded and harmless at runtime, but whitelisting them in
the prelude would blunt a check whose whole job is catching real bundler
bugs. conformance.ts now mirrors the three functions in TypeScript, and
tests/unit/axe-conformance-parity.test.ts imports both and asserts they
agree across every tag shape axe emits — best-practice, each WCAG
version/level, multi-digit and multiple criteria, obsolete criteria, and no
conformance tags. Tests aren't bundled, so that comparison is free. The
bundle lost ~400 lines and all four errors.
A shared pure module would be better than either, and is verified viable —
`axe-check.js` is injected with `type="module"`, and a FormatDependency's
`resources` land in the same output dir as its `scripts`, so both sides
could import one `axe-conformance.js`. Deferred: it changes the render path
every `axe:` user hits, and adds a runtime 404 risk, for a hidden
prototype's benefit. Recorded in the design note's cut list for the public
command.
The second was `TextDecoderStream`, used to drain Chrome's stderr. It is a
genuine Deno global simply missing from the bug-finder prelude's list, so it
is added there rather than worked around in the scanner.
After both: 1 error, pre-existing, in puppeteer code — the
`@typescript-eslint/ban-ts-comment` rule-not-found that the prelude's own
comment already documents. Also confirmed axe.min.js is *not* inlined: it
stays a runtime resource, so the bundle doesn't carry 553KB of axe-core.
No changelog entry. `dev-call` has never appeared in a changelog across
eleven versions, and an entry would advertise a hidden command that carries
no API commitment while its semantics are in play.
Headers audited (twelve new files, current year), deno fmt and lint clean,
113 axe tests passing, full typechecked run green.
The theme axis was measuring nothing on a default Quarto site. Emulating `prefers-color-scheme: dark` leaves the dark stylesheet `rel="disabled-stylesheet"` and the body white, because `respect-user-color-scheme` defaults to false — so every dark cell was an exact duplicate of its light sibling, and one of the two axes justifying the 2x2 default earned nothing. The axis tests the two themes the *author* ships. The user's OS preference is only one possible route into a theme, and a page may offer either route: Quarto's `light:`/`dark:` themes are selected by its colour-scheme toggle, author CSS under `@media (prefers-color-scheme: dark)` by emulation. Both are now set, pointing at the same theme, and the result is read back rather than assumed — which makes `respect-user-color-scheme` invisible here: when it is true, emulation has already done the work and no click happens. Alignment runs per cell, not once per run. The toggle writes `localStorage` and every cell shares one origin, so a click leaks into later page loads. This is not hypothetical: on the fixture, the final light cell now records `colorScheme: "toggled"` because it *loaded in dark* and had to be clicked back. Aligning once would have scanned it dark while labelling it light. Each cell records how it got there — `emulated`, `toggled`, `assumed-identical`, or `unreachable`. The last matters most: a page that ships a dark theme but renders no toggle cannot be switched, so its dark cell is really a light cell and is reported rather than counted as coverage. Cells on pages with no Quarto dark theme reuse their sibling's payload instead of running axe twice on the same DOM. That is an assumption, not a proof, and it is wrong for a custom `.scss` rolling its own dark mode or for bslib's web components, whose `prefers-color-scheme` rules live in shadow roots. A sound version — enumerate every stylesheet including shadow roots, fail closed on anything unreadable — was built and removed as unjustified complexity for v1; both design notes record the gap as an M5 question. Reuse never omits a cell, so the accounting stays honest, and `--themes dark` alone scans such a page properly. Evidence this works: the new `theme.qmd` plants a contrast failure in darkly's own palette (`#6c757d on #222222`, 3.39:1). It appears in dark cells only, and it was **invisible before this change** — proof distinct from `media.qmd`, whose dark-only failure comes from an author media query and was already reachable by emulation. The fixture now covers both routes separately. 81 axe tests passing. Results are unchanged by the reuse optimisation: 24/24 cells, 12 findings, both before and after.
Found by scanning a real site with a `_brand.yml` that specifies no dark values: the "reused a sibling result" note never appeared, though the site has no dark mode. A light-only `_brand.yml` makes Quarto emit the colour-scheme link machinery with `data-mode="light"` links but no dark build and no toggle. The probe deciding whether a theme cell can reuse its sibling asked for `link.quarto-color-scheme[data-mode]`, which matches those light links, so it concluded there was a dark theme to select and scanned both themes for nothing. The alignment probe next to it already used `[data-mode="dark"]` correctly; the loose selector came in when the capability probe was simplified. Effect was wasted work rather than wrong results — the dark cells scanned the same DOM and produced the same findings — but on a brand.yml site that is half the matrix, which is exactly the cost this reuse exists to avoid. The fixture had no such case, which is why nothing caught it. `static/brand-light-only.html` reproduces the markup by hand rather than adding a second fixture project: the probe runs per page, not per site, so one static page pins it. Verified by reintroducing the bug and watching the new assertion fail, then restoring the fix. Its planted violation is a bare `<button>`, not an alt-less image. An image normalizes to `img` and would join the systemic `image-alt` finding from `_systemic.qmd`, changing that finding's page list — which is how the first attempt failed, coupling a colour-scheme regression fixture to an unrelated grouping assertion. Fixture is now 7 pages, 28 cells, 13 findings. 81 axe tests passing.
One site and one smoke test shared a page list, so adding a page moved
assertions in unrelated verifiers — twice. Now sites/{findings,matrix,
brand-light-only} each carry one concern with their own smoke test, the
shared plumbing lives in tests/smoke/axe/shared.ts, and the captured
unit-test cells are grouped per site so each aggregates as a scan that
really happened.
Page discovery now reads each rendered page once: the inline before-body script marker (quartoToggleColorScheme) means a light/dark pair; its absence means one cell, labelled 'default'. The matrix is known and printed before the browser launches. Each cell selects its mode by seeding localStorage["quarto-color-scheme"] before navigation and verifying from the body class after load — which deletes AxeColorSchemeMechanism and all four states, sibling-payload reuse, alignColorScheme, the toggle click, the post-click settle and both console warnings, and no longer misses a light-coloured theme in the dark: slot the way the [data-mode="dark"] probe did. A one-mode page whose sole bootstrap link measures dark (theme: darkly) is annotated dark-coloured — in the cell payload and on the console, never in the cell's name. --themes becomes a filter over the discovered modes: it prunes the light/dark pair, 'default' cells are always included, and a filter matching zero cells is an error. findings.json gains a per-page modes field; report.html is unchanged. New fixture site sites/darkly covers the annotation; the research behind the design is in llm-docs/html-dark-mode-architecture.md, with its discovery pointer in .claude/rules/formats/html-dark-mode.md.
_axe-checks/ is the scanner's report from a previous run — when the anchor is the site dir (output-dir: "." projects, or loose HTML scanned from inside it) a rerun would scan its own report. site_libs/ is vendored library code: reveal ships plugin/notes/speaker-view.html with every deck, and its findings belong upstream, not to the site's author. Both are skipped at any path depth. Found playtesting (M5): braid br-b2-self-scan, br-skip-site-libs.
scanCell's run body had no try/catch, so a transport failure — a crashed tab, a dropped WebSocket, a CDP protocol error — propagated out of runAxeScan uncaught: process dead, no findings.json, no report, even with hundreds of good cells on disk. Now it returns an error cell; if the connection is really dead, later cells fail fast the same way and the exit code says incomplete, with everything scanned so far still reported. The orchestrator also catches what scanCell can't (shared setup, a stage bug) and reports incomplete instead of dying on an uncaught error. Braid: br-b1-transport-fail-closed.
Redirect stubs (a zero-delay meta refresh generated from _redirects) load locally, then Chrome follows the refresh during the settle window — axe was auditing the external destination and attributing its violations to the local page path (found on positron-website: opensource.posit.co findings under blog/index.html). Also broke the offline story silently. After settle, the cell probes window.location.href and fails closed with a new 'redirected' status — origin and path decide; search and hash are ignored so reveal's history-API fragment rewrites still count as the same page. Belt and braces: axe's own result.url is checked after the run, so a redirect between probe and run is caught too. Braid: br-redirect-escape.
Every cell paid the worst case: 500ms of sleep whether the page needed it or not — an afternoon of pure settle at quarto-web scale. The cell now waits for the page to report ready (document.fonts.ready, then a double requestAnimationFrame so deferred-script layout has painted) and applies --settle on top as an additive floor, default 50ms. If the probe can't run (a redirect destroyed its context — the location guard names it next — or pathological page JS), the old fixed 500ms returns as the fallback. Braid: br-settle-probe.
Play on real sites produced the demand twice in one day: quarto-web's listing pages, and revealjs decks whose resting-DOM findings the author opts out of knowingly (decks are content, so a silent default skip would misrepresent coverage — this is the author's call, unlike site_libs). Applied after --pages and before --max-pages, so pruning frees cap room. findings.json's config echo gains an exclude field (additive, no version bump). Braid: br-exclude-globs.
Redirect stubs are expected furniture — aliases: front matter writes redirect-map.ejs pages, _redirects scripts write meta-refresh pages — and failing their cells as not-ok painted every healthy aliased site red and exited 2 (playtest feedback, positron-website: 30 ERROR lines). Discovery now classifies both quarto stub shapes (with a near-empty-body guard so docs pages quoting the markers don't match), prints one quiet note, and records them in findings.json as a top-level redirects[] (additive field). The scan-time redirect guard stays for redirects discovery can't see — those remain errors, correctly. positron-website: 198/198 ok, exit 0. Also applies the /code-review findings on the previous five commits: - the readiness probe is capped in-page (2s) so a hanging webfont fetch can't convert a scannable page into per-cell --timeout failures offline - the timeout path absorbs its own about:blank load event so it can't resolve the next cell's load waiter mid-navigation - redirectTarget resolves relative URLs against the request, and an empty location probe is an error, not a phantom redirect - --settle 0 is legal (the flag is an additive floor now) - findings.json config.exclude is nullish, so version-1 files written before the field keep validating - stale findings.json/report.html are removed at scan start, so an aborted scan can't leave a previous run's summary reading as current (abort now also logs the stack to debug) - a bare directory --exclude/--pages glob matches everything beneath it Braid: br-skip-redirect-stubs; review findings on br-b1/br-redirect-escape /br-settle-probe/br-exclude-globs commits.
The report is now GitHub-flavored markdown (decision 2026-08-25): it renders on GitHub, drops into a site (--report <path> chooses where), and is accessible by construction — the rich drill-down belongs to a future extension, not CLI code. The AI briefing goes with it; report.ts stays a dumb view (the only computation is partitioning on the baselined flag). Partial scans announce themselves: a --pages/--exclude/--max-pages run leads with a banner naming the filters, because a subset snapshot must not read as site health. _axe-checks/README.md is the new agent enabler, regenerated every scan by the build that wrote the artifacts: provenance versions, the exact regenerate command reconstructed from the config echo, what each artifact is, how to read findings.json, and the baseline how-to INLINE — an agent in a scanned site's repo can reach nothing else (observed failing exactly that way during play). Baseline hardening to match the how-to: notes must be non-empty (every acceptance records its why — and a future machine-drafted baseline stays honest), and an optional version field errors with an explanation on mismatch, like signatureScheme. Braid: br-md-report, br-artifacts-readme.
Play feedback: (1) findings-table ids now link to a #### <id> heading above each occurrence block — GitHub and Pandoc both auto-anchor headings, so the link works rendered anywhere. (2) The finding-level detail moves into the block as a Problem line stated once; the per-occurrence detail column only speaks when an occurrence diverges from it. Measured on the full quarto-web scan: 301 of 304 findings have exactly one distinct occurrence detail, but the 3 divergent ones carry real information (different contrast ratios, 'empty title' vs 'no title'), so the column stays rather than being dropped. Braid: br-md-report (post-close follow-up).
The id is <rule>-<hash>, so the rule column repeated the id's prefix while markdown's width-less tables squeezed the (linked, hyphen-wrappable) id column to a sliver in preview panes. Free the width instead.
The label did no mechanical work (display-only) and little informational work: a threshold over the pages/instances columns printed beside it. Measured on quarto-web's 304 findings, its honest core was just 'appears on 2+ pages' (87 of 97 systemic), the one-page-3-instances arm mislabels repeated authoring mistakes, it degrades on subset scans (the multi-page arm can't fire), and it inherits signature splinters — the same leaflet defect read both systemic and localized. The reports now say the honest thing in prose: many pages usually means a shared source. findings.json version bumps to 2 (breaking field removal, per the contract). The per-occurrence detail column is gone too: nearly always identical to the finding-level Problem line, and the rare divergent values remain in findings.json. Decisions by Charlotte, 2026-08-26.
baseline.file was absolute and config.siteDir was whatever was typed, so a committed or CI-uploaded findings.json diffed noisily across machines. Aggregate now relativizes both against the artifact anchor.
The scanner writes _axe-checks/.gitignore ('*') on first run: a --pages
subset scan overwrites findings.json with a subset snapshot, so a committed
copy diffs as if findings were fixed. The committed contract stays
_axe-baseline.json, beside the directory.
cellName mapped / to _, so docs/index.html and docs_index.html shared an artifact name and the second write silently replaced the first; colliding slugs now get a short path-hash suffix, everything else keeps its name. Page paths are percent-encoded per segment before Page.navigate: a # or ? in a file name truncated the URL, scanning the wrong page and failing on the 404 with a misleading message.
A page merely mentioning quartoToggleColorScheme (docs about the toggle) false-triggered two-mode classification: the dark cell then failed closed with a misleading mode-mismatch error, or passed silently as a duplicate cell where no body class exists. Match window.quartoToggleColorScheme = instead; highlighted samples split the name across spans and no longer match. The emitting template carries a pointer back at the sniffer.
Deno.exit skipped onCleanup handlers; safe today (the browser is closed in finally) but it sidesteps the quarto convention.
--fail-on minor|moderate|serious|critical exits 1 when a COMPLETE scan has new (non-baselined) findings at or above the threshold — the CI regression signal. Exit 2 (incomplete scan) takes precedence: an incomplete scan never reads as a pass. Baselined findings never trip the threshold. The flag is echoed into findings.json's config (additive, no version bump) and into the README's regenerate command.
Deno.serve shuts down asynchronously after its abort signal fires; exiting mid-teardown intermittently segfaults Deno 2.7.14 (SIGSEGV null-call on a tokio-runtime-worker — three identical macOS crash reports, one predating this week's changes; 'segfault at exit' is a known upstream bug family). Await server.finished on every exit path to close the widest race.
config.ts parsing gets unit coverage through axeScanConfig (defaults, list trimming, viewport/theme/integer/--fail-on validation, AxeOptionError). Exit codes get end-to-end coverage in a subprocess — the in-process harness can't survive exitWithCleanup — over a hand-written static fixture: findings alone exit 0, --fail-on exits 1, a timed-out scan exits 2 even under --fail-on, with fail-closed artifacts left behind.
The committed architecture doc replaces pointers into private design notes: scan.ts, cmd.ts and aggregate.ts referenced 'the design note' and the signature-breadth investigation, which reviewers cannot read. It records the decisions that superseded those notes (markdown report over HTML, generated README as the agent enabler, extension as the rich report's eventual home). cri.ts cross-references the scanner's second Chrome launcher; a path-scoped rule surfaces the doc when the axe source is edited.
The 'how do I start' tier for early adopters: when to reach for the scanner vs the render-time axe: option, flags, the baseline workflow end to end, exit codes, and the CI recipe (--fail-on gate + report.md as a PR comment, with the 65,536-char comment cap noted). Contributor-facing on purpose — it graduates to quarto.org when the command sheds dev-call.
Pandoc parses the text between <code> tags as markdown, so an excerpt from a page that documents fenced divs put literal ::: strings into the AST and Quarto's fenced-div check warned on every render of the report (found playtesting on quarto-web). Backtick spans parse as Code inlines, which the check ignores; fences widen past embedded backtick runs, and pipes inside table cells escape as \| — the one escape both GitHub and Pandoc honor at the cell boundary.
cderv's suggestion: axe is a user-facing subsystem function, which is call's namespace — dev-call is for internal dev tools. The module moves wholesale (same depth, imports unchanged) and stays .hidden() while experimental: invocable, not advertised in quarto call help. Every surface that said 'dev-call axe' — the generated README's regenerate command, help text, tests, fixture READMEs, docs, changelog — now says 'call axe'.
QUARTO_CHROMIUM pointing at a file that exists but is not a browser wins discovery and then fails to spawn — the closest forcible stand-in for 'no usable Chrome' on machines that have one. Pins exit 2, the 'Could not start headless Chrome' message, and that a pre-existing findings.json does not survive to read as current. (Discovery finding nothing at all is core/puppeteer.ts behavior, shared with mermaid, and not forcible here.)
core's 'Chrome not found' throws an empty Error after printing its own explanation and the chrome-headless-shell install suggestion; the wrapper line now ends cleanly instead of 'Could not start headless Chrome: '.
The how-to claimed a shared-include defect is one finding 'however many pages repeat it'. That holds when axe emits selectors that normalize identically (fixture-verified: the shared include yields bare img on both pages) and can break for anonymous elements whose surrounding DOM differs — axe then picks different minimal selectors and one cause splits. Say so.
Verified live: 'quarto call axe _site/' as a project post-render script scans after outputs are written, from the project dir (trailing slash fine); without --fail-on findings never fail the render, with it a threshold trip or incomplete scan fails quarto render with the scan's error line in the output. CI recipe now uses marocchino/sticky-pull-request-comment so each push updates one comment instead of stacking new ones.
The --help output is the first thing anyone who discovers the hidden command reads; align it with the docs' no-stability wording and point at dev-docs/axe-scan.md.
quarto preview of the report fails inside a project (_-prefixed dirs are not project inputs; verified: 'No output created by quarto render report.md'), while a plain quarto render works standalone. Document the hierarchy: editor/GitHub preview first, render-then-open for HTML, --report into the site source for the previewable-with-site flow. Deliberately no _quarto.yml inside _axe-checks: it would make renders there pick up the README and adds magic to a disposable directory.
320 CSS px is the one viewport width WCAG names (SC 1.4.10 Reflow — the 400%-zoom equivalent of a 1280 window). axe has no automated reflow rule, but every rule runs against the reflowed layout, and 320 sits in the same Bootstrap breakpoint regime as any phone width, so the mobile chrome still renders. 390x844 was the iPhone 12-14 logical size inherited, undocumented, from the quarto-web harness. Signatures never embed viewports, so committed baselines keep matching; only viewport-gated findings can churn.
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
# Conflicts: # news/changelog-1.11.md
# Conflicts: # news/changelog-1.11.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds
quarto call axe <site-dir>— a hidden, experimentalcommand that scans an already-rendered Quarto site for accessibility
violations with quarto-cli's vendored axe-core.
What it does:
as other Quarto features:
quarto install chrome-headless-shell, or asystem Chrome/Edge) over a page × viewport × color-mode matrix. Each
page's modes are discovered from its rendered HTML, so a light-only site
scans once per page and a
light:/dark:site scans both slots.as-is, offline, on the same axe version the render-time
axe:option uses.the color pair for
color-contrast), so one template defect on 40 pages isone finding with a count, and its id is stable across runs.
_axe-baseline.json):new findings alert, accepted ones are counted but out of the way,
escalations and out-of-scope recurrences re-alert.
_axe-checks/findings.json(machine contract),report.md(GitHub-flavored markdown), and a generated
README.mdthat documents theartifacts for whoever — or whatever agent — finds them in a repo.
infrastructure failure in the output and the exit code, never a pass.
Exit codes:
0complete,1new findings at/above--fail-on <impact>(opt-in, for CI),
2incomplete — and 2 takes precedence over 1.Why hidden under
call: the semantics (signature scheme, baselinegrain, flag surface) should be validated by real use before becoming public command. Nothing here is a public API commitment.
Docs in this PR:
dev-docs/axe-scan.md— how to use it (flags, baseline workflow, CIrecipe). Will eventually seed the
quarto-webdocs.llm-docs/axe-scan-architecture.md— how it works and why (pipeline, modediscovery, signatures, baseline semantics, the raw-CDP decision).
Deliberately not in scope (deferred to the public command):
_axe.ymlconfig, interaction states, ruleset scoping, source mapping (
.qmd→output), a rich report extension. The render-time
axe:option isuntouched; the scanner mirrors its conformance labellers, pinned by a parity
test.
Testing: 9 unit test files (115 tests) run the aggregate/baseline/discovery/
config logic browser-free against captured real axe payloads; 5 smoke test
files run the full command over fixture sites in
tests/docs/axe-scan/,including subprocess exit-code tests covering
--fail-on, fail-closedtimeouts, and a browser that cannot start. CI already installs
chrome-headless-shellunconditionally; a manualtest-smokes-paralleldispatch on this branch is green:
https://github.com/quarto-dev/quarto-cli/actions/runs/33106781443.
Checklist
I have (if applicable):
(not applicable yet: the command is hidden; docs live in
dev-docs/and graduate to quarto-web when it goes public)
AI-assisted PR