Declarative $inner selectors and a token overlay helper

[LeaVerou]

Follow-up to #4105 / #4107, per the review discussion there: meta-languages now declare which parts of their token tree are code in the inner language, instead of a custom tokenizer or placeholder tricks.

// diff.js — the before and after versions of the code, each highlighted as one whole
$inner: { select: ['unchanged, deleted, diff', 'unchanged, inserted, diff'] }

Selectors

$inner is either a grammar reference (as before) or { language?, select? }. A selector names containers, separated by commas: token names (matched by type or alias, since both become classes, so deleted covers deleted-sign and deleted-arrow) and :text for the unmatched text. The plain strings directly inside the selected containers are concatenated in document order, tokenized as one whole with the inner language, and the tokens are put back where the strings were.

• Default selector is :text, which is exactly what templating languages need, so they are unchanged (inner: markup and nothing else).
• 'foo, :text' highlights foo's content together with the unmatched text as one document; ['foo, :text', 'bar, :text'] does that twice, separately, later selectors winning where they overlap.
• Inner tokens that span several strings of the same container stay whole, with the container's other tokens (template tags, diff prefixes) inserted into them by position. Inner tokens that span different containers are split.
• :text is a pseudo-name because text is already a token in pug, haml, mermaid, plant-uml, textile and inform7; token names can never contain :.

diff is now fully declarative (66 lines, the original grammar plus the $inner line) and better than the old plugin: the two selectors are the before and after versions of the file, so a comment opened on an unchanged line and closed after a change is highlighted correctly on the deleted and the inserted lines. tests/core/embedding.js covers that; the five existing snapshots (diff:javascript, django:css, diff:django:css, php:none, …) are byte-identical to v2.

No placeholders

Non-selected tokens simply vanish from the highlighted document. That removes $placeholder, the collision handling, and the class of bugs #4107 found (the placeholder being markdown emphasis / a JS constant / split by false). Three expectations change, all the same case: a template expression that is an entire unquoted attribute value or tag name (<span data-has-name={{x}}>, <h{$level}>) is no longer parsed as a tag by markup. Quoted values (href="{{url}}") are unaffected.

The helper

src/util/token-stream.js (exported from shared.js) has the two primitives this is built on, for everything else in the repo that merges one tokenized structure with another:

• splitTokenStream(stream, offsets) — split at text offsets, cloning tokens that straddle a boundary.
• insertTokens(stream, [[offset, token], …]) — insert tokens by position, nesting into spanning tokens.

Consumers to migrate are listed in the follow-up issue (keep-markup's DOM Range juggling, command-line's line stashing on the HTML string, shell-session, the /// / * doc-comment prefixes in xml-doc/javadoclike/jsdoc, markdown blockquotes, the per-string tokenizeStrings plugins, jsx/xquery's plain-text flattening).

Removed

templating() (replaced by src/core/tokenize/embed.js), $placeholder, the inner grammar option (nothing needs it: php's <? check now just re-tokenizes with withoutTokenize(grammar)).

Full suite green, lint clean, tsc clean (with --moduleResolution bundler, since the checked-in config is broken by TS 6).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RbhmLXUybTbxCBSwGa2YZr

---

Generated by Claude Code

[netlify]

✅ Deploy Preview for dev-prismjs-com ready!

Name	Link
🔨 Latest commit	7e18205
🔍 Latest deploy log	https://app.netlify.com/projects/dev-prismjs-com/deploys/6ab2415b51cff3000860eb85
😎 Deploy Preview	https://deploy-preview-4110--dev-prismjs-com.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[DmitrySharabin]

Two corrections to the description, found while reviewing this PR together with #4112 and #4114. Both are about the text, not the code.

---

Three expectations change, all the same case: a template expression that is an entire unquoted attribute value or tag name (<span data-has-name={{x}}>, <h{$level}>) is no longer parsed as a tag by markup.

There is a third position, and two of the three changed fixtures in this PR are examples of it: an unquoted attribute name.

// django
'<div {{ x }}>a</div>'

v2   [tag [punctuation <][tag div] [attr-name [django …]][punctuation >]]
new  [tag [punctuation <][tag div] [django …][punctuation >]]

tests/languages/latte/html_feature.test and the second hunk of tests/languages/handlebars/handlebars_in_html_feature.test both move a template token out of attr-name, not out of attr-value.

The three positions differ in severity, which the one-line summary flattens:

position	example	effect
unquoted attribute value	<span data-x={{x}}>	the whole opening tag stops being a tag
tag name	<h{$level}>	tag survives, name is just h
unquoted attribute name	<div {{on "click"}}>	tag survives, loses the attr-name wrapper

Only the first loses a whole construct. Quoted values are unaffected, as the description says.

Measured across every markup-templating language — django, twig, handlebars, liquid, ejs, erb, etlua, tt2, smarty, ftl, latte, soy, php — all 13 change in all three positions. Filed as #4119 so it is tracked somewhere other than this description; it is recorded there as an accepted trade against the #4107 placeholder bugs, not as an argument to revert.

---

Full suite green, lint clean, tsc clean (with --moduleResolution bundler, since the checked-in config is broken by TS 6).

This does not reproduce. The repo pins TypeScript 5.8.3 (npx tsc --version), tsconfig.json:30 sets "moduleResolution": "node", and bare npx tsc exits clean both at a5e7eca and at the top of the stack — no flag needed, and npx tsc -p tests/tsconfig.json is clean too. Whatever produced the TS 6 error was probably a globally installed tsc rather than the repo's own. Worth dropping, since as written it sends a reader looking for a broken config.