Repository navigation
Tolerate invalid UTF-8 at the module string boundary - #778
Merged
Merged
Conversation
An OSC 8 URI that is not valid UTF-8, for example a CJK path cut mid-character, made every redraw signal wrong-type-argument utf-8-string-p: the renderer handed the raw bytes to make_string, which rejects them. The same path was open for OSC 7 URLs, OSC 52 payloads, notifications and the comint filter's links. makeString now validates first and decodes invalid input through make_unibyte_string and decode-coding-string, so bad bytes become raw-byte chars instead of aborting the redraw. extractString and extractStringAlloc gain the reverse fallback: when copy_string_contents rejects a multibyte string holding raw-byte chars, re-encode it and retry once, so such strings can be written back to the terminal. Closes #775.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #775.
An OSC 8 hyperlink whose URI is not valid UTF-8 (a CJK path cut mid-character, as emitted by the Antigravity CLI) made every redraw signal
wrong-type-argument utf-8-string-p. The renderer handed the raw URI bytes to the module API'smake_string, which requires valid UTF-8. The redraw aborted mid-row, so the buffer stopped updating while the link stayed on screen.How this ended up as three parts
1. The reported bug. The root cause is
Env.makeString, not the renderer: every terminal-supplied string reachesmake_stringthrough it, so the same crash was open for OSC 7 cwd URLs, OSC 52 payloads, OSC 9/777 notifications and the comint filter's links (libghostty validates titles and OSC 7501 upstream, but not these).makeStringnow validates withstd.unicode.utf8ValidateSliceand, for invalid input, decodes throughmake_unibyte_string+decode-coding-stringwithutf-8-unix, the same way Emacs decodes process output. Bad bytes become raw-byte chars and the link still renders; nothing is lost.2. The reverse direction. Review of part 1 found that raw-byte chars can now reach Elisp for the first time, and the module → Zig direction rejected them:
copy_string_contentssignalsunicode-string-pon multibyte strings holding raw-byte chars, soextractString/extractStringAllocfailed. That madeghostel--comint-filterdrop a whole chunk when the shell emitted one invalid byte, and made pasting such a URI back into the terminal error. Both helpers now retry once throughrawByteFallback, which re-encodes the string withencode-coding-string+utf-8-unixonly when the failure was awrong-type-argument. A too-small buffer still fails without a retry. The success path is unchanged.3. Performance. An A/B against the module built from
main(same Emacs binary, 256 MB GC threshold, 8 interleaved rounds, medians) showed part 1 was free for ASCII but measurable on dense multi-byte text, because rendered row text and comint text also went through the validatingmakeString:That text is encoded from codepoints libghostty already decoded, so it is valid by construction.
makeUtf8Stringis the plainmake_stringcall, used at exactly those two sites (Renderer.zigspan insert,comint_filter.zigtext). Validation now runs only on OSC payload strings, a few bytes per event. Final numbers:The "unicode" bench input is 100% CJK, the worst case for the validator.
Tests
ghostel-test-osc8-invalid-utf8-uri-renders: the reporter's byte sequence renders,help-echokeeps the raw byte.ghostel-test-write-vt-accepts-raw-byte-chars: a string with raw-byte chars can be written to the terminal.ghostel-comint-test-raw-byte-chars-pass-through: the comint filter decodes the byte to U+FFFD instead of erroring.All three fail on
mainwith the original signals. Also verified live with the reporter'sprintfin a sandboxed Emacs on the native PTY: link rendered, nothing in*Messages*.Known caveat
The signal
rawByteFallbackrecovers from is a debuggable catch, so withdebug-on-errorset a backtrace pops up for a call that then succeeds. This only happens on user-rate paths (paste or send of a string containing raw-byte chars), which were hard errors before. Pre-encoding every multibyte string would avoid it but adds a funcall to every extraction including the process-output write path, so the retry was kept and the caveat is noted in the doc comment.