Skip to content

Fix inflated post-page reading time; dedupe post metadata logic - #2027

Merged
benbalter merged 1 commit into
mainfrom
audit-dedupe
Sep 27, 2026
Merged

benbalter merged 1 commit into
mainfrom
audit-dedupe

Conversation

@benbalter

Copy link
Copy Markdown
Owner

Stacked on #2024. Merge that one first; this PR's base will then retarget to main.

Reading time was about 2x too high on long post pages

The audit flagged that cards and post pages disagree on reading time. I measured this across the whole build: 96 of 165 carded posts disagree, by as much as 60 min on the page vs. 28 on the card (Towards a more agile government).

The page is the side that's wrong:

  • PostLayout counts the rendered HTML.
  • stripFootnotes only recognizes Markdown [^1]: syntax, so rendered footnote sections were counted in full.
  • Tag attributes (acronym tooltips, links) were counted as words too.

With footnotes excluded, which is what the util already intends, that post is about 30 min. That matches the card.

calculateReadingTime and calculateWordCount now also drop:

  • the rendered <section data-footnotes> and data-footnote-ref links
  • <script> and <style> contents
  • tag markup

After the fix, 17 posts differ, each by at most 2 min. The remainder comes from code blocks and URLs, which the Markdown and HTML paths count slightly differently. JSON-LD wordCount was inflated the same way and is fixed too.

Dedupe

  • KeepReading now uses getPostMetadata instead of reimplementing URL, date, and reading time.
  • feed.xml and llms.txt now use getPublishedPosts({ sorted: true }) instead of hand-rolled sorts.
    • llms.txt is byte-identical.
    • In the feed, only same-day posts change order: two 2026-04-27 posts swap, and the 20th item is a different 2023-05-19 post.
  • llms.txt:
    • Dropped the excerpt fallback. The schema requires description, so that branch could never run.
    • The bio now goes through stripHtmlTags.

Verification

  • npm run check: clean.
  • Vitest: 2016 passed. New tests show that rendered HTML (with footnotes, attributes, scripts) and the equivalent Markdown count the same.
  • Built the site and re-ran the page-vs-card comparison.

🤖 Generated with Claude Code

- calculateReadingTime/WordCount now drop rendered footnotes
  (<section data-footnotes>, data-footnote-ref links), script/style, and tag
  markup. Post pages count rendered HTML, so footnotes and attributes were
  counted there but not on cards: 96 of 165 carded posts disagreed, up to
  60 vs 28 min. Now 17 differ, by at most 2 min. Also fixes JSON-LD wordCount.
- KeepReading uses getPostMetadata instead of reimplementing URL/date/time
- feed.xml and llms.txt use getPublishedPosts({ sorted: true })
- llms.txt: drop the unreachable excerpt branch (description is required)
  and strip bio HTML with stripHtmlTags

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from audit-prod-bugs to main September 27, 2026 13:47
@benbalter
benbalter merged commit b4126b4 into main Sep 27, 2026
19 checks passed
@benbalter
benbalter deleted the audit-dedupe branch September 27, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant