Repository navigation
test(e2e): scan editors for WCAG 2.2 AA issues with axe - #586
juliusknorr wants to merge 1 commit into
Conversation
Signed-off-by: Julius Knorr <jus@bitgrid.net> Assisted-by: ClaudeCode:claude-opus-5-5
bhumigopaniStarto
left a comment
There was a problem hiding this comment.
Nice start, the scan itself reads well. A few suggestions below, mostly around CI, test tags and repo conventions.
Nice to have: document the a11y spec in e2e/README.md
The README lists the specs in its folder tree and shows how to run them. A short note on accessibility.spec.ts would help: what it scans, which WCAG tags it uses, where the report goes, that it doesn't assert yet, and how to run only this spec (npm test -- tests/accessibility.spec.ts). The tree should also list the new file.
| targets: v.nodes.slice(0, 5).map((n) => n.target.join(' >> ')), | ||
| })); | ||
| mkdirSync('a11y-report', { recursive: true }); | ||
| writeFileSync(`a11y-report/${name}.json`, JSON.stringify(summary, null, 2)); |
There was a problem hiding this comment.
Improvement: the report isn't uploaded in CI
a11y-report/ is outside playwright-report/, so the e2e workflow never uploads it and the JSON is lost after the job. Until the baselines land, the findings are only visible locally. Attaching the summary with test.info().attach(name, { body, contentType: 'application/json' }) would put it in the existing report artifact without a workflow change.
Also, until the assertions exist this adds 4 editor sessions to every run (more on retry) without guarding anything. Is that acceptable for now, or would you rather keep it out of the default run until the baselines are in?
| test(`${editor} editor`, async ({ page }) => { | ||
| const { editorPage, frame } = await openNewEditor(page, `a.try-editor.${editor}`, ext); | ||
| await scan(editorPage, `${editor}-toolbar`); | ||
| await frame.locator('#fm-btn-info, .btn-file, [data-tab="file"]').first().click(); |
There was a problem hiding this comment.
Improvement: narrow the File-tab selector
'#fm-btn-info, .btn-file, [data-tab="file"]' is a union and .first() takes whichever matches first. If it matches something other than the File tab, the test still passes and silently scans the toolbar twice. Could you use the one selector that works in all three editors and assert it is visible before clicking?
| writeFileSync(`a11y-report/${name}.json`, JSON.stringify(summary, null, 2)); | ||
| } | ||
|
|
||
| test('example page', async ({ page }) => { |
There was a problem hiding this comment.
Improvement: tag the tests
The README's "Test tags" section says every test carries exactly one area tag plus @regression, so it can be selected with --grep. None of the tests here have tags, so --grep @regression and the per-area runs skip them. Could you add them, for example { tag: ['@document', '@regression'] } on the Word scan, @spreadsheet on the Cell scan and @presentation on the Slide scan? The example-page scan doesn't fit an editor area, so it's worth deciding which tag it should get, or whether the README needs a new tag for it.
| @@ -0,0 +1,38 @@ | |||
| import { test, type Page } from '@playwright/test'; | |||
There was a problem hiding this comment.
Suggestion: add the SPDX header and consider where the spec lives
-
New files carry the header used in
frontend/tests/document/delete-document.spec.ts:// SPDX-FileCopyrightText: 2026 Euro-Office contributors
// SPDX-License-Identifier: AGPL-3.0-only -
The other specs are grouped by editor type (
document/,spreadsheet/,presentation/,pdf-form/). Would it be worth keeping the example-page scan next toexample-page.spec.tsand moving the per-editor scans into those folders, or at least into anaccessibility/folder? The PDF editor isn't scanned, so is that intentional?
| nodes: v.nodes.length, | ||
| targets: v.nodes.slice(0, 5).map((n) => n.target.join(' >> ')), | ||
| })); | ||
| mkdirSync('a11y-report', { recursive: true }); |
There was a problem hiding this comment.
Suggestion: make the report path independent of the working directory
mkdirSync('a11y-report') resolves against the process cwd, so it lands in e2e/a11y-report/ only when tests start from e2e/ (as npm test does). From another directory the report is written elsewhere and the .gitignore entry doesn't cover it. Other outputs are config-relative (../test-results, ../playwright-report). Could you resolve the path from __dirname or write into Playwright's outputDir?
Adds an axe scan (
@axe-core/playwright, WCAG 2.0 to 2.2 A/AA tags) of the example page and the Document, Spreadsheet and Presentation editors, with the toolbar and with the File menu open. Each scene writes its violations toe2e/a11y-report/<scene>.json(git-ignored).Inspired by nextcloud/text#8081. Findings from a first run against
latestare tracked in web-apps#425, related to #69.Open before ready:
waitForTimeoutafter opening the File menu with a wait on a menu element