Skip to content

Pin utf-8 in test source reads (Windows non-UTF-8 locale fix) - #48948

Open
dltsum wants to merge 4 commits into
huggingface:mainfrom
dltsum:fix-export-test-utf8
Open

dltsum wants to merge 4 commits into
huggingface:mainfrom
dltsum:fix-export-test-utf8

Conversation

@dltsum

@dltsum dltsum commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

CPU CI GPU run-slow

What

Two test helpers read model source files with the locale's preferred encoding:

  • _skip_if_not_exportable (tests/exporters/test_export.py): open(inspect.getfile(...))
  • test_transformers_specific_model_import (tests/utils/test_import_structure.py): open(models_path / arch / f"{module}.py")

On Windows with a non-UTF-8 locale (e.g. cp936) these raise UnicodeDecodeError for any of the 91 modeling_*.py sources that contain non-ASCII characters:

tests\exporters\test_export.py:585: in _skip_if_not_exportable
    source_code = f.read()
E   UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 in position 9571: illegal multibyte sequence

tests\utils\test_import_structure.py:107
E   UnicodeDecodeError: 'gbk' codec can't decode byte 0x81 in position 1001: illegal multibyte sequence

The first failure aborts every test_torch_export / ONNX / ExecuTorch variant of the affected model test classes before any export is attempted; the second fails the import-structure consistency check wholesale.

Fix

Pin encoding="utf-8" on both reads — Transformers sources are UTF-8. Same class of fix as #48819.

Verification

Windows 11, cp936 locale, Python 3.14:

  • BeitModelTest::test_torch_export_{dynamic,static}: fail without the fix (above), pass with it (2 passed, 10 subtests passed).
  • test_transformers_specific_model_import: dozens of subtest failures without the fix, 1 passed + 516 subtests passed with it.

🤖 Generated with Claude Code

dltsum and others added 2 commits September 19, 2026 17:15
_skip_if_not_exportable reads the model source with the locale's preferred
encoding, which raises UnicodeDecodeError on Windows with a non-UTF-8 locale
(e.g. cp936) for the 91 modeling files that contain non-ASCII characters.

Same class of fix as huggingface#48819.

Co-Authored-By: Claude Code <noreply@anthropic.com>
test_transformers_specific_model_import reads every modeling source with
the locale's preferred encoding; on Windows cp936 it raises
UnicodeDecodeError for dozens of files (byte 0x81/0x80 in UTF-8 content).

Co-Authored-By: Claude Code <noreply@anthropic.com>
@dltsum dltsum changed the title Pin utf-8 in export test source read (Windows non-UTF-8 locale fix) Pin utf-8 in test source reads (Windows non-UTF-8 locale fix) Sep 19, 2026
@Rocketknight1

Copy link
Copy Markdown
Member

LGTM! Also cc @tarekziade I've seen a couple of these, maybe a Ruff rule to enforce open encodings where using the default might cause locale-dependent behaviour?

Comment thread tests/exporters/test_export.py Outdated
Comment thread tests/utils/test_import_structure.py Outdated
@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 35434920874:2
Result: cancelled | Jobs: 16 | Tests: 190,604 | Failures: 0 | Duration: 17h 37m

@tarekziade

Copy link
Copy Markdown
Collaborator

LGTM! Also cc @tarekziade I've seen a couple of these, maybe a Ruff rule to enforce open encodings where using the default might cause locale-dependent behaviour?

good idea that's PLW1514 I can add it

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@tarekziade

Copy link
Copy Markdown
Collaborator

LGTM! Also cc @tarekziade I've seen a couple of these, maybe a Ruff rule to enforce open encodings where using the default might cause locale-dependent behaviour?

here #48990

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants