Conversation
_skip_if_not_exportable reads the model source with the locale's preferred encoding, which raises UnicodeDecodeError on Windows with a non-UTF-8 locale (e.g. cp936) for the 91 modeling files that contain non-ASCII characters. Same class of fix as huggingface#48819. Co-Authored-By: Claude Code <noreply@anthropic.com>
test_transformers_specific_model_import reads every modeling source with the locale's preferred encoding; on Windows cp936 it raises UnicodeDecodeError for dozens of files (byte 0x81/0x80 in UTF-8 content). Co-Authored-By: Claude Code <noreply@anthropic.com>
|
LGTM! Also cc @tarekziade I've seen a couple of these, maybe a Ruff rule to enforce |
CI recapDashboard: View test results in Grafana |
good idea that's |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
here #48990 |
What
Two test helpers read model source files with the locale's preferred encoding:
_skip_if_not_exportable(tests/exporters/test_export.py):open(inspect.getfile(...))test_transformers_specific_model_import(tests/utils/test_import_structure.py):open(models_path / arch / f"{module}.py")On Windows with a non-UTF-8 locale (e.g. cp936) these raise
UnicodeDecodeErrorfor any of the 91modeling_*.pysources that contain non-ASCII characters:The first failure aborts every
test_torch_export/ ONNX / ExecuTorch variant of the affected model test classes before any export is attempted; the second fails the import-structure consistency check wholesale.Fix
Pin
encoding="utf-8"on both reads — Transformers sources are UTF-8. Same class of fix as #48819.Verification
Windows 11, cp936 locale, Python 3.14:
BeitModelTest::test_torch_export_{dynamic,static}: fail without the fix (above), pass with it (2 passed, 10 subtests passed).test_transformers_specific_model_import: dozens of subtest failures without the fix, 1 passed + 516 subtests passed with it.🤖 Generated with Claude Code