Skip to content

Tolerate invalid UTF-8 at the module string boundary - #778

Merged
emil-e merged 2 commits into
mainfrom
fix/invalid-utf8-module-strings
Nov 1, 2026
Merged

emil-e merged 2 commits into
mainfrom
fix/invalid-utf8-module-strings

Conversation

@dakra

@dakra dakra commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Closes #775.

An OSC 8 hyperlink whose URI is not valid UTF-8 (a CJK path cut mid-character, as emitted by the Antigravity CLI) made every redraw signal wrong-type-argument utf-8-string-p. The renderer handed the raw URI bytes to the module API's make_string, which requires valid UTF-8. The redraw aborted mid-row, so the buffer stopped updating while the link stayed on screen.

How this ended up as three parts

1. The reported bug. The root cause is Env.makeString, not the renderer: every terminal-supplied string reaches make_string through it, so the same crash was open for OSC 7 cwd URLs, OSC 52 payloads, OSC 9/777 notifications and the comint filter's links (libghostty validates titles and OSC 7501 upstream, but not these). makeString now validates with std.unicode.utf8ValidateSlice and, for invalid input, decodes through make_unibyte_string + decode-coding-string with utf-8-unix, the same way Emacs decodes process output. Bad bytes become raw-byte chars and the link still renders; nothing is lost.

2. The reverse direction. Review of part 1 found that raw-byte chars can now reach Elisp for the first time, and the module → Zig direction rejected them: copy_string_contents signals unicode-string-p on multibyte strings holding raw-byte chars, so extractString/extractStringAlloc failed. That made ghostel--comint-filter drop a whole chunk when the shell emitted one invalid byte, and made pasting such a URI back into the terminal error. Both helpers now retry once through rawByteFallback, which re-encodes the string with encode-coding-string + utf-8-unix only when the failure was a wrong-type-argument. A too-small buffer still fails without a retry. The success path is unchanged.

3. Performance. An A/B against the module built from main (same Emacs binary, 256 MB GC threshold, 8 interleaved rounds, medians) showed part 1 was free for ASCII but measurable on dense multi-byte text, because rendered row text and comint text also went through the validating makeString:

scenario (ms/iter) main after parts 1+2
unicode full redraw 0.333 0.347 (+4.8%)
unicode comint-filter 0.048 0.053 (+12.6%)
plain / styled / links within ±1%

That text is encoded from codepoints libghostty already decoded, so it is valid by construction. makeUtf8String is the plain make_string call, used at exactly those two sites (Renderer.zig span insert, comint_filter.zig text). Validation now runs only on OSC payload strings, a few bytes per event. Final numbers:

scenario (ms/iter) main this PR
unicode full redraw 0.333 0.331 (-0.8%)
unicode comint-filter 0.048 0.048 (-1.0%)
unicode write+redraw 0.187 0.188 (+0.5%)
write-vt only (extractString path), all inputs 0.0%
plain / styled / links, all scenarios within ±1.5%, ranges overlap

The "unicode" bench input is 100% CJK, the worst case for the validator.

Tests

  • ghostel-test-osc8-invalid-utf8-uri-renders: the reporter's byte sequence renders, help-echo keeps the raw byte.
  • ghostel-test-write-vt-accepts-raw-byte-chars: a string with raw-byte chars can be written to the terminal.
  • ghostel-comint-test-raw-byte-chars-pass-through: the comint filter decodes the byte to U+FFFD instead of erroring.

All three fail on main with the original signals. Also verified live with the reporter's printf in a sandboxed Emacs on the native PTY: link rendered, nothing in *Messages*.

Known caveat

The signal rawByteFallback recovers from is a debuggable catch, so with debug-on-error set a backtrace pops up for a call that then succeeds. This only happens on user-rate paths (paste or send of a string containing raw-byte chars), which were hard errors before. Pre-encoding every multibyte string would avoid it but adds a funcall to every extraction including the process-output write path, so the retry was kept and the caveat is noted in the doc comment.

An OSC 8 URI that is not valid UTF-8, for example a CJK path cut
mid-character, made every redraw signal wrong-type-argument
utf-8-string-p: the renderer handed the raw bytes to make_string, which
rejects them.  The same path was open for OSC 7 URLs, OSC 52 payloads,
notifications and the comint filter's links.

makeString now validates first and decodes invalid input through
make_unibyte_string and decode-coding-string, so bad bytes become
raw-byte chars instead of aborting the redraw.  extractString and
extractStringAlloc gain the reverse fallback: when copy_string_contents
rejects a multibyte string holding raw-byte chars, re-encode it and retry
once, so such strings can be written back to the terminal.

Closes #775.
@dakra
dakra requested a review from emil-e November 1, 2026 09:17
@emil-e
emil-e merged commit ff7c216 into main Nov 1, 2026
33 checks passed
@emil-e
emil-e deleted the fix/invalid-utf8-module-strings branch November 1, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Invalid UTF-8 in an OSC 8 hyperlink URI aborts every redraw (wrong-type-argument utf-8-string-p)

2 participants