Skip to content

fix(truncation): bound text on code points instead of UTF-16 units - #614

Open
MincongZhou wants to merge 1 commit into
openpi-dev:mainfrom
MincongZhou:fix/surrogate-safe-truncation
Open

MincongZhou wants to merge 1 commit into
openpi-dev:mainfrom
MincongZhou:fix/surrogate-safe-truncation

Conversation

@MincongZhou

Copy link
Copy Markdown

Problem

Four text bounds measure with String.prototype.length and cut with slice(), so they count UTF-16 code units rather than code points. When a bound lands between the two halves of a surrogate pair, the result ends on a lone high surrogate — a value that is not well-formed text:

Location Function Input Old result (tail)
extensions/sessions/sessions.ts:121 normalizeSnippet "a".repeat(18) + 🚀 "\uD83D…"
extensions/background-terminals/src/watch.ts:156 sanitizeLine "x".repeat(499) + 🚀 "\uD83D…"
extensions/subagents/navigation.ts:32 normalizeSubagentTitle "x".repeat(159) + 🚀 "x".repeat(159) + "\uD83D"
extensions/shared/web-observer-registry.ts:168 boundedActivityText "x".repeat(158) + 🚀 + "…" "\uD83D…"

Each of these values is displayed or serialized:

  • normalizeSnippet feeds the session list and session search entries shown in the TUI and Web UI.
  • sanitizeLine produces the reported line of a background-terminal watch match, which is stored in the match record.
  • normalizeSubagentTitle normalizes titles "before they enter snapshots, artifacts, or the TUI" (its own doc comment), so a broken title is persisted before it is rendered.
  • boundedActivityText projects activity text into the Web capability snapshot; a lone surrogate there cannot survive JSON representation intact and reaches the browser as a replacement character.

extensions/shared/terminal-text.ts does not repair lone surrogates, so nothing downstream heals the value.

Value

The value that crosses the bound can be an emoji or a supplementary-plane CJK character — common in titles and transcript lines for a coding agent that routinely carries user-supplied text. Today the bound corrupts the last visible character of such a string, and the corruption can be persisted into snapshots and artifacts rather than only being rendered once.

Fixing it keeps the bounded text well-formed and matches the behavior the repository already relies on elsewhere, including the two immediately preceding fixes in this area: #586 (user-input-fold) and #566 (web).

Approach

Count and cut on code points:

const characters = [...value];
if (characters.length <= MAX) return value;
return `${characters.slice(0, MAX - 1).join("")}…`;

This is the pattern already used by extensions/post-edit/index.ts, extensions/ask-user/index.ts, and extensions/subagents/src/by-the-way.ts, so the change aligns the four lagging call sites with the established convention rather than introducing a new one. No shared helper is added: AGENTS.md asks for the smallest scoped change, and these four sites live in four extensions.

Each site also gets a regression test asserting the exact bounded string, so a future slice() refactor fails loudly instead of silently splitting a pair again.

Validation

  • bun run check — pass (biome format check, biome lint with --error-on-warnings, tsc --noEmit, config/docs/discipline contracts, Web build).
  • bun run test — pass.
  • Targeted runs of the four touched suites: sessions 8/8, watch 16/16, navigation 9/9, observer-registry 13/13.
  • The four new tests use an emoji (U+1F680) as the boundary character and assert the complete emoji survives; each test fails against the pre-change implementation.

Impact

  • User-visible behavior: bounded text now ends on a complete character. For pure-BMP text (including all CJK in the BMP) nothing changes.
  • Model-visible context/tools: none. No prompt or tool schema changes.
  • Runtime/lifecycle: none.
  • Persisted config/data: none; the change affects the contents of already-persisted fields, not their shape.
  • Compatibility/risk: the bound's semantics change from UTF-16 code units to code points. For text containing astral characters, the emitted string can now be up to 2× longer in UTF-16 units than the nominal bound while holding at most MAX code points. The existing WATCH_LINE_MAX_CHARS comment and the boundedActivityText doc are updated to state the unit explicitly.

normalizeSnippet, sanitizeLine, normalizeSubagentTitle and
boundedActivityText measured with String#length and cut with slice(),
so a bound landing inside a surrogate pair emitted a lone surrogate
into session lists, background-terminal match records, subagent
snapshots and the Web capability snapshot.

Count and cut on code points instead, which is the pattern already used
by post-edit, ask-user and by-the-way. Each site gets a regression test
asserting the exact bounded string at an emoji boundary.
@github-actions github-actions Bot added area:subagents Subagent delegation, skills, or tests area:background-terminals Background terminal runtime, skill, or tests labels Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:background-terminals Background terminal runtime, skill, or tests area:subagents Subagent delegation, skills, or tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant