Skip to content

Fix two stack overflows and an ignore token counted twice - #4116

Open
DmitrySharabin wants to merge 1 commit into
claude/inner-spec-detectionfrom
claude/token-stream-overflow
Open

DmitrySharabin wants to merge 1 commit into
claude/inner-spec-detectionfrom
claude/token-stream-overflow

Conversation

@DmitrySharabin

@DmitrySharabin DmitrySharabin commented Sep 17, 2026

Copy link
Copy Markdown
Member

Fix two stack overflows and a doubled ignore token in $inner embedding, found during a cold review of #4110/#4112/#4114 as one change.

  • embed and insertTokens both call splice(start, n, ...items) with an array as long as the inner token stream — any $inner language overflows the call stack past ~460KB of input, where v2 was fine. Both now go through replaceRange() (slice, truncate, push).
  • A selector naming both :text and a root-level ignore* token fed that token's text to the inner language twice, and the second copy changed how the first one parsed. Not reachable from a shipped grammar, but ignore* tokens are documented as dissolved into :text.
  • token-stream.js: split halves no longer share the original's alias array, insertTokens throws instead of silently dropping text on non-ascending offsets, and splitTokenStream's isolation claim now matches what it actually does.

10253 passing / 1 pre-existing pending / 0 failing. lint:ci, tsc, and tsc -p tests/tsconfig.json all clean. Each fix is mutation-pinned — reverting the guard fails exactly one test.

file +
tests/core/registry.js 35 0
tests/core/token-stream.js 28 1
src/util/token-stream.js 18 2
tests/core/embedding.js 10 0
src/core/tokenize/embed.js 3 3

🤖 Generated with Claude Code

@DmitrySharabin
DmitrySharabin added this pull request to stack #4115 September 17, 2026 11:20
@netlify

netlify Bot commented Sep 17, 2026

Copy link
Copy Markdown

Deploy Preview for dev-prismjs-com ready!

Name Link
🔨 Latest commit fa25cc2
🔍 Latest deploy log https://app.netlify.com/projects/dev-prismjs-com/deploys/6ab2453073d01b0008e4eda6
😎 Deploy Preview https://deploy-preview-4116--dev-prismjs-com.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@DmitrySharabin
DmitrySharabin removed this pull request from stack #4115 September 22, 2026 08:35
@DmitrySharabin
DmitrySharabin changed the base branch from claude/split-token-stream-perf to v2 September 22, 2026 08:48
@DmitrySharabin
DmitrySharabin force-pushed the claude/token-stream-overflow branch from 7e07d6c to 68439b4 Compare September 22, 2026 08:48
@DmitrySharabin
DmitrySharabin changed the base branch from v2 to claude/inner-spec-detection September 22, 2026 08:48
@DmitrySharabin
DmitrySharabin added this pull request to stack #4124 September 22, 2026 08:48
@DmitrySharabin
DmitrySharabin removed this pull request from stack #4124 September 22, 2026 08:49
@DmitrySharabin
DmitrySharabin force-pushed the claude/token-stream-overflow branch from 68439b4 to fa25cc2 Compare September 22, 2026 09:06
@DmitrySharabin
DmitrySharabin added this pull request to stack #4126 September 22, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant