Repository navigation
fix: handle non-ASCII reports when stdout is redirected (cp1252 -> UTF-8) - #32
Conversation
AdityaWagh06
left a comment
There was a problem hiding this comment.
Thanks @hamid-digitalmarketer, this is a solid fix! The stream reconfiguration is exactly what #24 asked for, and the subprocess test with PYTHONIOENCODING=cp1252 really exercises the bug. Formatting and lint pass.
One thing before merging: this PR also changes the result schema title in cmd_schema from "ModelDiff run result" to "ModelDiff result file". That's unrelated to #24, and the title is part of the published schema output. Please revert that line so the PR only contains the fix.
Once that's done and CI is green on all platforms (including Windows), I'll merge it. Next time, please also comment on the issue before starting, so others know it's taken.
Reviewer feedback on OpenWave-India#32: the "result" title was changed without a connection to the stdout encoding fix. Restore the original string so the PR is scoped strictly to the redirected-stdout UTF-8 issue. Validation: - python -m pytest tests/ -q -> 158 passed - python -m ruff check . -> All checks passed
5032969 to
519ed16
Compare
|
Done — reverted the unrelated Branch updated to two commits: the original fix + the revert. Validation on the updated branch: Ready for re-review — will merge or adjust on your signal. |
|
Merged, thank you @hamid-digitalmarketer! This fixes a real crash for Windows users with non-English reports, and the subprocess test with cp1252 proves it. I resolved a small test-file conflict with #22 while merging. It’ll ship in v0.2.0 next week 🎉 If you enjoyed contributing, a ⭐ on the repo helps others find the project. Follow OpenWave India on GitHub and LinkedIn for new projects coming soon. |
OpenWave-India#45) Review on OpenWave-India#45: the README said the multilingual suite has "two sentiment classifications" where it has three -- `hindi-sentiment-positive`, `hindi-sentiment-neutral` and `tamil-sentiment-positive` -- and it kept a sentence saying a redirected run only works once the UTF-8 output fix lands, which OpenWave-India#32 already merged. Co-authored-by: Hermes Agent (DeepSeek V4) <noreply@nousresearch.com>
Fixes #24
Problem
When
modeldiff runis executed on Windows with stdout redirected (for examplemodeldiff run ... > out.txtor via CI), Python encodes the stream using the system code page (cp1252by default on Windows). Reports that contain non-ASCII text (e.g. Hindi, Tamil, or emoji) then raise:even when the suite has no regressions. The crash happens because only file output via
--outputis written with explicit UTF-8 encoding; the console/stdout path relies on the inherited stream encoding.Root cause
Python selects the encoding of
sys.stdout/sys.stderrwhen the interpreter starts. When the stream is redirected to a pipe or file, it falls back to the ANSI code page (e.g.cp1252), which cannot represent most non-ASCII characters. The CLI never reconfigures its text streams, so any report rendered to stdout crashes on non-ASCII output.Change
_reconfigure_streams()insrc/modeldiff/cli.pythat callsstream.reconfigure(encoding="utf-8", errors="replace")onsys.stdoutandsys.stderrat the start ofmain(). Non-file stream objects (e.g. pytest capture) that reject reconfiguration are left untouched.test_redirected_stdout_is_utf8_with_non_ascii_reportintests/test_cli.pythat runs the CLI as a subprocess withPYTHONIOENCODING=cp1252, redirected stdout, and a Hindi-language report, asserting exit code 0 and UTF-8 output.Validation
Notes
CONTRIBUTING.md.Contact: hamid-digitalmarketer (GitHub), open for review feedback.