Split token streams in one pass

[DmitrySharabin]

Follow-up to #4110, on top of its branch.

TL;DR

• splitTokenStream was quadratic in the number of selected runs: a 12k-line diff:javascript took 3197ms, now 307ms. One forward pass, no remainder copies.
• Two cases the one-pass form has to preserve, both found while reviewing this PR, both now tested: a zero-length item on a boundary (without it diff:markdown drops a token), and the no-offsets fast path (the common one — :text is the default selector).
• splitTokenStream / insertTokens are public API via shared.js and had no direct tests. Added.
• tests/languages/php!+css+css-extras/issue2008.test described a placeholder Declarative $inner selectors and a token overlay helper #4110 deletes, so it could no longer fail for the reason it exists. Note re-pointed at what it now guards.

The perf bug

splitTokenStream called splitAt once per offset, and splitAt copied the rest of the stream each time (stream.slice(i), [b, ...stream.slice(i + 1)]). embed passes one offset per run, i.e. one per diff line, so the copies dominate.

Repro — N reps of a three-line block, Prism.highlight(code, 'diff:javascript'):

for (let i = 0; i < N; i++) {
	code += ` const a${i} = ${i};\n-let b${i} = "x";\n+let b${i} = 'y';\n`;
}

lines	#4110	this PR
3,001	189 ms	103 ms
12,001	3197 ms	307 ms
48,001	—	1290 ms

The 3k row is within noise — at that size the quadratic term is small, and a re-measurement on another machine gave 113 ms before and 120 ms after. The 12k row is the real signal, and it reproduces: 2790 ms before, 313 ms after.

node --cpu-prof on the 12k case put 1829 of ~2300 samples in splitAt.

The two cases that are easy to lose

A zero-length item sitting exactly on an offset belongs to the segment that starts there, not the one that ends there. splitAt got this for free by testing pos >= offset before consuming the item; a strict offsets[next] < pos bound never fires for it. Markdown emits an empty code-block token for an empty fenced block, so the difference is visible:

' ```js\n-\n ```\n'  →  highlight(code, 'diff:markdown')
without:  <span class="token code">\n</span>
with:     <span class="token code"><span class="token code-block"></span>\n</span>

With no offsets there is nothing to split, and that is the path almost everything takes: only src/languages/diff.js sets select, so php, django, handlebars, ejs, erb, smarty, liquid, latte, ftl, etlua and tt2 all reach splitTokenStream with offsets === []. It must not walk the stream measuring items to get there.

200k random streams (nested tokens, aliases, empty items, offsets at 0 / total / past the end / duplicated) split by both implementations: no divergences.

Not fixed

splitItem builds the halves with new Token(type, content, alias), which leaves Token.length at 0. Inert today — split streams are never fed back to _matchGrammar, the only reader — and the field is documented @internal with no guaranteed meaning, so threading real lengths through felt like more machinery than the payoff. Worth revisiting if a consumer migrated to splitTokenStream ever re-tokenizes.

Full suite green (10244 passing, 1 pre-existing pending), lint clean, tsc and tsc -p tests/tsconfig.json clean with no flags.

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for dev-prismjs-com ready!

Name	Link
🔨 Latest commit	0b53870
🔍 Latest deploy log	https://app.netlify.com/projects/dev-prismjs-com/deploys/6ab240ebb088460008c97030
😎 Deploy Preview	https://deploy-preview-4112--dev-prismjs-com.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.