Fix Windows test portability: line-ending translation, locale encoding, and /tmp assumptions - #212
Merged
Merged
Conversation
…tions Three root causes, all test-infrastructure (no library code changed), found by the Windows CI lanes and reproduced on a Windows 11 machine: 1. Missing .gitattributes: with git's core.autocrlf=true (the Windows default), checkout rewrites the CSV fixtures' LF to CRLF - including newlines EMBEDDED IN QUOTED FIELDS (git does not know CSV) - so the byte-level fastcsv reader sees corrupted input and the fast_roundtrip_newlines / fast_roundtrip_quotes_and_newlines round-trips fail. Fixed by marking *.csv / *.tsv -text. 2. Text-mode writes in test harnesses: Data.Text.IO writeFile/hPutStrLn honour the handle's text mode, which on Windows (a) translates \n to \r\n, corrupting quoted embedded newlines (typed_quote_spans_boundary, and the round-trip rewrite path in prettyPrintSeparated), and (b) encodes via the OS locale codepage, crashing fast_roundtrip_utf8 with 'cannot encode character' on non-UTF-8 codepages (e.g. CP936). Fixed by Data.Text.IO.Utf8 (same signatures, byte-mode UTF-8, no new dependency; text >= 2.1 is already required) and, for the streaming pretty-printer, hSetEncoding utf8 + hSetNewlineMode noNewlineTranslation. 3. Hardcoded /tmp paths: toCsv_roundTrip wrote to /tmp (does not exist on Windows); fixed with getTemporaryDirectory. The same pattern in Properties/Csv.hs (two sites) now uses the suite's existing tests/data/unstable_csv scratch dir. Verified on Windows 11 (GHC 9.12.4): dataframe suite 1097/1097, dataframe-fastcsv suite 61/61, zero errors/failures - previously toCsv_roundTrip, fast_roundtrip_newlines, fast_roundtrip_quotes_and_newlines and typed_quote_spans_boundary failed (https://github.com/skymanbp/dataframe/actions/runs/32095552668). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A repo-wide sweep for '/tmp' found one more test writing there: deriveSchemaReadsCsv in tests/Operations/Record.hs. It passed on a dev machine that happened to have C:\tmp (Windows resolves /tmp against the drive root) but fails on CI runners. Same fix as WriteCsv.hs: getTemporaryDirectory + the byte-mode UTF-8 writer. The two other grep hits are not test-filesystem uses (a benchmark default argument and a pure URI-predicate check) and are left untouched. Local re-run: dataframe 1097/1097, dataframe-fastcsv 61/61, zero errors/failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
skymanbp
force-pushed
the
fix/windows-test-portability
branch
from
August 18, 2026 18:01
402e473 to
ed1fcbd
Compare
skymanbp
marked this pull request as draft
August 18, 2026 18:47
skymanbp
marked this pull request as ready for review
August 18, 2026 18:47
We can leave them only where they are used.
We can add explanatory comments in the tests.
Removed unnecessary blank lines and comments in Record.hs
The comment read too much like AI
Member
|
Thanks. This is a great change! Reminds me that I need to split my dev time between unix and Windows. |
This was referenced Aug 19, 2026
skymanbp
added a commit
to skymanbp/dataframe
that referenced
this pull request
Aug 19, 2026
Same classes DataHaskell#212 fixed in tests. CSV and HTML writers and the lazy reader used locale handles; pin them to UTF-8. 'start' is a cmd builtin, so launch it through a shell. lazy-bench wrote to /tmp, which Windows lacks; use the system temp dir.
skymanbp
added a commit
to skymanbp/dataframe
that referenced
this pull request
Aug 19, 2026
Same classes DataHaskell#212 fixed in tests. CSV and HTML writers and the lazy reader used locale handles; pin them to UTF-8. 'start' is a cmd builtin, so launch it through a shell. lazy-bench wrote to /tmp, which Windows lacks; use the system temp dir.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Running the test suites on Windows currently fails in three independent ways. All three are test-infrastructure issues — this PR changes no library code (7 files, +53/−10: six test modules plus a new
.gitattributes).After these fixes, the full suites pass on Windows:
dataframe:test:tests—Cases: 1097 Tried: 1097 Errors: 0 Failures: 0dataframe-fastcsv:test:tests—Cases: 61 Tried: 61 Errors: 0 Failures: 0(local run: Windows 11, GHC 9.12; fork CI: windows-latest on GHC 9.6.7 and 9.12.2, links below)
Root causes and fixes
1. No
.gitattributes→ gitcore.autocrlfrewrites CSV fixtures on checkoutGitHub's Windows runners (and many Windows dev machines) default to
core.autocrlf=true, so every text-looking file — including CSV test fixtures — is checked out with CRLF line endings. Byte-level tests then see\rthat isn't in the committed fixture. This also silently corrupts fixtures with quoted embedded newlines (e.g. thequotes_and_newlinescase).Fix: add
.gitattributeswith*.csv -text/*.tsv -text.-textmeans "no translation of what's in the index", so fixtures that intentionally contain CRLF (e.g.crlf.csv) are preserved byte-for-byte as committed.2. Text-mode
Data.Text.IOin tests → newline translation + locale codepage crashesA default Haskell handle uses the OS locale encoding and native newline translation. On Windows this breaks roundtrip tests twice over:
\nis written as\r\n, corrupting output meant for the byte-level CSV reader (embedded quoted newlines come back different);fast_roundtrip_utf8dies withcommitAndReleaseBuffer: cannot encode character '\676'.Fix: in test writers,
Data.Text.IO→Data.Text.IO.Utf8(samewriteFilesignature, byte-mode UTF-8, no new dependencies; provided bytext >= 2.1). For the streaming handle inprettyPrintSeparated(fastcsv tests), set the handle explicitly:hSetEncoding handle utf8+hSetNewlineMode handle noNewlineTranslation.3. Hardcoded
/tmppaths (3 sites)/tmpdoesn't exist on Windows — the path resolves relative to the current drive's root, so the tests only pass by accident ifC:\tmphappens to exist, and fail withwithFile: does not existotherwise.Fix:
System.Directory.getTemporaryDirectory(sites:tests/Operations/WriteCsv.hs,tests/Operations/Record.hs; the two property-test paths indataframe-fastcsv/tests/Properties/Csv.hswere moved under./tests/data/unstable_csv/alongside the existing pattern there). Non-test/tmpuses (LazyBenchmarkdefault argument, the huggingface URI predicate) were deliberately left untouched.Verification
Upstream has no Windows CI lane, so the CI evidence below comes from my fork, where a temporary fork-only workflow (windows-latest + macos-14 × GHC 9.6.7/9.12.2) was stacked under these same fixes. That exact state is preserved on the
fix/windows-test-portability-with-cibranch; this PR branch is the identical fixes rebased onto currentmainwithout the workflow commit.Note on Haskell-CI: that workflow also fails on this branch, but it is red on upstream
mainas well (latest three runs onmainall concludefailure, most recent 2026-08-14), so it appears unrelated to this change.