Skip to content

fix: handle non-ASCII reports when stdout is redirected (cp1252 -> UTF-8) - #32

Merged
AdityaWagh06 merged 4 commits into
OpenWave-India:mainfrom
hamid-digitalmarketer:fix/windows-utf8-redirected-stdout
Oct 31, 2026
Merged

AdityaWagh06 merged 4 commits into
OpenWave-India:mainfrom
hamid-digitalmarketer:fix/windows-utf8-redirected-stdout

Conversation

@hamid-digitalmarketer

Copy link
Copy Markdown
Contributor

Fixes #24

Problem

When modeldiff run is executed on Windows with stdout redirected (for example modeldiff run ... > out.txt or via CI), Python encodes the stream using the system code page (cp1252 by default on Windows). Reports that contain non-ASCII text (e.g. Hindi, Tamil, or emoji) then raise:

UnicodeEncodeError: 'charmap' codec can't encode character '\u0928' in position ...: character maps to <undefined>

even when the suite has no regressions. The crash happens because only file output via --output is written with explicit UTF-8 encoding; the console/stdout path relies on the inherited stream encoding.

Root cause

Python selects the encoding of sys.stdout/sys.stderr when the interpreter starts. When the stream is redirected to a pipe or file, it falls back to the ANSI code page (e.g. cp1252), which cannot represent most non-ASCII characters. The CLI never reconfigures its text streams, so any report rendered to stdout crashes on non-ASCII output.

Change

  • Add _reconfigure_streams() in src/modeldiff/cli.py that calls stream.reconfigure(encoding="utf-8", errors="replace") on sys.stdout and sys.stderr at the start of main(). Non-file stream objects (e.g. pytest capture) that reject reconfiguration are left untouched.
  • Add regression test test_redirected_stdout_is_utf8_with_non_ascii_report in tests/test_cli.py that runs the CLI as a subprocess with PYTHONIOENCODING=cp1252, redirected stdout, and a Hindi-language report, asserting exit code 0 and UTF-8 output.

Validation

$ python -m pytest tests/ -q
158 passed in ...

$ python -m ruff check .
All checks passed!

$ python -m ruff format --check .
All checks passed!

# Manual reproduction before fix: exit 1 with UnicodeEncodeError under cp1252
# After fix: exit 0, report printed correctly

Notes

  • No behavior-breaking shortcuts: the fix only affects stream encoding; exit codes and output content are unchanged.
  • Follows the one-issue-per-PR convention in CONTRIBUTING.md.

Contact: hamid-digitalmarketer (GitHub), open for review feedback.

@AdityaWagh06 AdityaWagh06 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hamid-digitalmarketer, this is a solid fix! The stream reconfiguration is exactly what #24 asked for, and the subprocess test with PYTHONIOENCODING=cp1252 really exercises the bug. Formatting and lint pass.

One thing before merging: this PR also changes the result schema title in cmd_schema from "ModelDiff run result" to "ModelDiff result file". That's unrelated to #24, and the title is part of the published schema output. Please revert that line so the PR only contains the fix.

Once that's done and CI is green on all platforms (including Windows), I'll merge it. Next time, please also comment on the issue before starting, so others know it's taken.

Reviewer feedback on OpenWave-India#32: the "result" title was changed without a
connection to the stdout encoding fix. Restore the original string so
the PR is scoped strictly to the redirected-stdout UTF-8 issue.

Validation:
- python -m pytest tests/ -q -> 158 passed
- python -m ruff check . -> All checks passed
@hamid-digitalmarketer
hamid-digitalmarketer force-pushed the fix/windows-utf8-redirected-stdout branch from 5032969 to 519ed16 Compare October 31, 2026 12:46
@hamid-digitalmarketer

Copy link
Copy Markdown
Contributor Author

Done — reverted the unrelated result title change in cmd_schema (restored "ModelDiff run result"); the PR is now scoped strictly to the redirected-stdout UTF-8 fix.

Branch updated to two commits: the original fix + the revert. Validation on the updated branch: python -m pytest tests/ -q -> 158 passed; python -m ruff check . -> all checks passed.

Ready for re-review — will merge or adjust on your signal.

@AdityaWagh06
AdityaWagh06 merged commit f0c400c into OpenWave-India:main Oct 31, 2026
12 checks passed
@AdityaWagh06

Copy link
Copy Markdown
Member

Merged, thank you @hamid-digitalmarketer! This fixes a real crash for Windows users with non-English reports, and the subprocess test with cp1252 proves it. I resolved a small test-file conflict with #22 while merging. It’ll ship in v0.2.0 next week 🎉

If you enjoyed contributing, a ⭐ on the repo helps others find the project. Follow OpenWave India on GitHub and LinkedIn for new projects coming soon.

tnalbant added a commit to tnalbant/modeldiff that referenced this pull request Nov 3, 2026
OpenWave-India#45)

Review on OpenWave-India#45: the README said the multilingual suite has "two sentiment
classifications" where it has three -- `hindi-sentiment-positive`,
`hindi-sentiment-neutral` and `tamil-sentiment-positive` -- and it kept a
sentence saying a redirected run only works once the UTF-8 output fix lands,
which OpenWave-India#32 already merged.

Co-authored-by: Hermes Agent (DeepSeek V4) <noreply@nousresearch.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash on Windows when output contains non-English text and stdout is redirected

2 participants