Conversation
On Windows with a non-UTF-8 locale (e.g. cp936), the CLI crashes writing the generated doc file whose header contains '⚠️ ': src/transformers/cli/add_new_model_like.py:599 in add_new_model_like E UnicodeEncodeError: 'gbk' codec can't encode character '\u26a0' All generated/read repo files are UTF-8; pin the encoding on every open() in the command instead of relying on the locale. Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Contributor
CI recapDashboard: View test results in Grafana |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
transformers add-new-model-likereads and writes repository files with the locale's preferred encoding. On Windows with a non-UTF-8 locale (e.g. cp936) the command crashes when writing the generated doc file, whose header contains⚠️:(found by
tests/utils/test_add_new_model_like.py::TestAddNewModelLike::test_llama_without_tokenizerson a cp936 machine)Fix
Pin
encoding="utf-8"on all sevenopen()calls in the command — everything it reads (_toctree.yml, existing test files) and writes (modular file,__init__.py, test files, doc file) is UTF-8 by repository convention.Verification
Windows 11, cp936 locale, Python 3.14:
open(path, "w")(UnicodeEncodeError: 'gbk' ... '\u26a0') and succeeds withencoding="utf-8".test_llama_without_tokenizersprogresses past the previous crash point. Note: the test still fails later in this environment for unrelated reasons — its subprocesses resolve the installed site-packagestransformers(whosetokenizers<=0.23.0pin conflicts with the newer tokenizers needed by main), and the temp-repocheck_doc_toc.pystep errors on the partial doc copy. Both are local-environment artifacts, not affected by this change either way.🤖 Generated with Claude Code