Skip to content

Make e2e failures loud; fix tooltip re-show race - #2028

Merged
benbalter merged 1 commit into
mainfrom
audit-test-reliability
Sep 27, 2026
Merged

benbalter merged 1 commit into
mainfrom
audit-test-reliability

Conversation

@benbalter

Copy link
Copy Markdown
Owner

Test reliability fixes from a general audit.

Silent skips

  • Build detection. detectBuildSystem / isAstroBuild date from the Jekyll era and defaulted to 'jekyll' whenever their markers were missing. Removed them from 36 call sites across 8 specs. Each site kept its Astro branch and dropped the dead Jekyll branch.
  • Status checks. seo-astro and accessibility-astro skipped when a route returned 404. They now assert a 200 status.
  • Waits. All 6 waitForTimeout calls became web-first waits. For tooltips, the tests now wait for data-tooltip-initialized; navigation-styling now polls with expect.poll.
  • endpoints.spec.ts. Added /humans.txt and /posts-meta.json, and asserts that an unknown path returns 404.

Real bug surfaced: tooltip re-show race

"Tooltip works after View Transitions navigation" had never run: it checked build detection on about:blank, got "jekyll", and skipped. Running it for real surfaced a bug in src/scripts/tooltip.ts:

  • showTooltip adds .show in a requestAnimationFrame.
  • If the tooltip is hidden before that frame runs (for example, hover and click in the same frame), the frame re-adds .show to a tooltip that's already untracked and on its way out.
  • The result is an orphan tooltip visible for up to 300 ms.

The frame callback now checks that the tooltip is still active.

Three click tests were passing only because of that bug. Playwright's .click() also hovers, which on desktop shows the tooltip, so the click toggled it back off. The tests now dispatch a bare click, as the file's existing toggle test already does.

stripMdxSyntax hardening

Neither issue hits a current post, but both are latent traps:

  • A ```` fence containing a ``` sample closed too early. Fences now close per CommonMark: same character, at least as long as the opener, no info string.
  • A prose line starting with "import" was dropped. Imports now need to look like real ESM.

I compared old vs. new output on all 197 posts: 0 differences.

Verification

  • Full chromium e2e run against a local preview: 389 passed, 6 skipped (fixme). The one failure is /resume.pdf returning 404, which is expected with SKIP_PDF; CI builds the PDF.
  • Tooltip specs pass on 3 repeated runs.
  • Vitest and npm run check are clean.

🤖 Generated with Claude Code

- Remove detectBuildSystem/isAstroBuild (Jekyll-era). It defaulted to
  "jekyll" when markers were missing, so 36 call sites could skip or take
  dead Jekyll branches silently; one tooltip test never ran at all.
- Assert 200 instead of skipping on 404 in seo-astro and accessibility-astro
- Replace all waitForTimeout calls with web-first waits
- endpoints: cover /humans.txt, /posts-meta.json, and a real 404 status
- tooltip.ts: the fade-in rAF re-added .show to a tooltip hidden in the same
  frame (hover+click), flashing an untracked tooltip for up to 300ms. Guard
  it; tooltip tests now dispatch a bare click like the existing toggle test.
- stripMdxSyntax: close fences per CommonMark (same char, >= length, no info
  string) and only treat real ESM as imports; output unchanged on all posts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@benbalter
benbalter merged commit 9b23da8 into main Sep 27, 2026
21 checks passed
@benbalter
benbalter deleted the audit-test-reliability branch September 27, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant