Skip to content

Pin utf-8 in add-new-model-like file IO (Windows non-UTF-8 locale fix) - #48949

Open
dltsum wants to merge 2 commits into
huggingface:mainfrom
dltsum:fix-cli-addnewmodel-utf8
Open

dltsum wants to merge 2 commits into
huggingface:mainfrom
dltsum:fix-cli-addnewmodel-utf8

Conversation

@dltsum

@dltsum dltsum commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

CPU CI GPU run-slow

What

transformers add-new-model-like reads and writes repository files with the locale's preferred encoding. On Windows with a non-UTF-8 locale (e.g. cp936) the command crashes when writing the generated doc file, whose header contains ⚠️:

src/transformers/cli/add_new_model_like.py:599: in add_new_model_like
    f.write(doc_file)
E   UnicodeEncodeError: 'gbk' codec can't encode character '\u26a0' in position 604: illegal multibyte sequence

(found by tests/utils/test_add_new_model_like.py::TestAddNewModelLike::test_llama_without_tokenizers on a cp936 machine)

Fix

Pin encoding="utf-8" on all seven open() calls in the command — everything it reads (_toctree.yml, existing test files) and writes (modular file, __init__.py, test files, doc file) is UTF-8 by repository convention.

Verification

Windows 11, cp936 locale, Python 3.14:

  • The doc-file content fails to write with a bare open(path, "w") (UnicodeEncodeError: 'gbk' ... '\u26a0') and succeeds with encoding="utf-8".
  • With the fix, test_llama_without_tokenizers progresses past the previous crash point. Note: the test still fails later in this environment for unrelated reasons — its subprocesses resolve the installed site-packages transformers (whose tokenizers<=0.23.0 pin conflicts with the newer tokenizers needed by main), and the temp-repo check_doc_toc.py step errors on the partial doc copy. Both are local-environment artifacts, not affected by this change either way.

🤖 Generated with Claude Code

dltsum and others added 2 commits September 19, 2026 17:36
On Windows with a non-UTF-8 locale (e.g. cp936), the CLI crashes writing
the generated doc file whose header contains '⚠️':

  src/transformers/cli/add_new_model_like.py:599 in add_new_model_like
  E   UnicodeEncodeError: 'gbk' codec can't encode character '\u26a0'

All generated/read repo files are UTF-8; pin the encoding on every open()
in the command instead of relying on the locale.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 35441048549:2
Result: success | Jobs: 2 | Tests: 20 | Failures: 0 | Duration: 2m 20s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant