Skip to content

test(e2e): scan editors for WCAG 2.2 AA issues with axe - #586

Draft
juliusknorr wants to merge 1 commit into
mainfrom
test/a11y-axe
Draft

juliusknorr wants to merge 1 commit into
mainfrom
test/a11y-axe

Conversation

@juliusknorr

@juliusknorr juliusknorr commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Adds an axe scan (@axe-core/playwright, WCAG 2.0 to 2.2 A/AA tags) of the example page and the Document, Spreadsheet and Presentation editors, with the toolbar and with the File menu open. Each scene writes its violations to e2e/a11y-report/<scene>.json (git-ignored).

Inspired by nextcloud/text#8081. Findings from a first run against latest are tracked in web-apps#425, related to #69.

Open before ready:

  • Report only, no assertions yet. Next step: per-scene baseline snapshots so CI fails only on new violations
  • Replace the waitForTimeout after opening the File menu with a wait on a menu element

Signed-off-by: Julius Knorr <jus@bitgrid.net>
Assisted-by: ClaudeCode:claude-opus-5-5

@bhumigopaniStarto bhumigopaniStarto left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice start, the scan itself reads well. A few suggestions below, mostly around CI, test tags and repo conventions.

Nice to have: document the a11y spec in e2e/README.md

The README lists the specs in its folder tree and shows how to run them. A short note on accessibility.spec.ts would help: what it scans, which WCAG tags it uses, where the report goes, that it doesn't assert yet, and how to run only this spec (npm test -- tests/accessibility.spec.ts). The tree should also list the new file.

targets: v.nodes.slice(0, 5).map((n) => n.target.join(' >> ')),
}));
mkdirSync('a11y-report', { recursive: true });
writeFileSync(`a11y-report/${name}.json`, JSON.stringify(summary, null, 2));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement: the report isn't uploaded in CI

a11y-report/ is outside playwright-report/, so the e2e workflow never uploads it and the JSON is lost after the job. Until the baselines land, the findings are only visible locally. Attaching the summary with test.info().attach(name, { body, contentType: 'application/json' }) would put it in the existing report artifact without a workflow change.

Also, until the assertions exist this adds 4 editor sessions to every run (more on retry) without guarding anything. Is that acceptable for now, or would you rather keep it out of the default run until the baselines are in?

test(`${editor} editor`, async ({ page }) => {
const { editorPage, frame } = await openNewEditor(page, `a.try-editor.${editor}`, ext);
await scan(editorPage, `${editor}-toolbar`);
await frame.locator('#fm-btn-info, .btn-file, [data-tab="file"]').first().click();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement: narrow the File-tab selector

'#fm-btn-info, .btn-file, [data-tab="file"]' is a union and .first() takes whichever matches first. If it matches something other than the File tab, the test still passes and silently scans the toolbar twice. Could you use the one selector that works in all three editors and assert it is visible before clicking?

writeFileSync(`a11y-report/${name}.json`, JSON.stringify(summary, null, 2));
}

test('example page', async ({ page }) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement: tag the tests

The README's "Test tags" section says every test carries exactly one area tag plus @regression, so it can be selected with --grep. None of the tests here have tags, so --grep @regression and the per-area runs skip them. Could you add them, for example { tag: ['@document', '@regression'] } on the Word scan, @spreadsheet on the Cell scan and @presentation on the Slide scan? The example-page scan doesn't fit an editor area, so it's worth deciding which tag it should get, or whether the README needs a new tag for it.

@@ -0,0 +1,38 @@
import { test, type Page } from '@playwright/test';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: add the SPDX header and consider where the spec lives

  1. New files carry the header used in frontend/tests/document/delete-document.spec.ts:

    // SPDX-FileCopyrightText: 2026 Euro-Office contributors
    // SPDX-License-Identifier: AGPL-3.0-only

  2. The other specs are grouped by editor type (document/, spreadsheet/, presentation/, pdf-form/). Would it be worth keeping the example-page scan next to example-page.spec.ts and moving the per-editor scans into those folders, or at least into an accessibility/ folder? The PDF editor isn't scanned, so is that intentional?

nodes: v.nodes.length,
targets: v.nodes.slice(0, 5).map((n) => n.target.join(' >> ')),
}));
mkdirSync('a11y-report', { recursive: true });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: make the report path independent of the working directory

mkdirSync('a11y-report') resolves against the process cwd, so it lands in e2e/a11y-report/ only when tests start from e2e/ (as npm test does). From another directory the report is written elsewhere and the .gitignore entry doesn't cover it. Other outputs are config-relative (../test-results, ../playwright-report). Could you resolve the path from __dirname or write into Playwright's outputDir?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants