Skip to content

Fix Windows test portability: line-ending translation, locale encoding, and /tmp assumptions - #212

Merged
mchav merged 10 commits into
DataHaskell:mainfrom
skymanbp:fix/windows-test-portability
Aug 19, 2026
Merged

Fix Windows test portability: line-ending translation, locale encoding, and /tmp assumptions#212
mchav merged 10 commits into
DataHaskell:mainfrom
skymanbp:fix/windows-test-portability

Conversation

@skymanbp

@skymanbp skymanbp commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Running the test suites on Windows currently fails in three independent ways. All three are test-infrastructure issues — this PR changes no library code (7 files, +53/−10: six test modules plus a new .gitattributes).

After these fixes, the full suites pass on Windows:

  • dataframe:test:testsCases: 1097 Tried: 1097 Errors: 0 Failures: 0
  • dataframe-fastcsv:test:testsCases: 61 Tried: 61 Errors: 0 Failures: 0

(local run: Windows 11, GHC 9.12; fork CI: windows-latest on GHC 9.6.7 and 9.12.2, links below)

Root causes and fixes

1. No .gitattributes → git core.autocrlf rewrites CSV fixtures on checkout

GitHub's Windows runners (and many Windows dev machines) default to core.autocrlf=true, so every text-looking file — including CSV test fixtures — is checked out with CRLF line endings. Byte-level tests then see \r that isn't in the committed fixture. This also silently corrupts fixtures with quoted embedded newlines (e.g. the quotes_and_newlines case).

Fix: add .gitattributes with *.csv -text / *.tsv -text. -text means "no translation of what's in the index", so fixtures that intentionally contain CRLF (e.g. crlf.csv) are preserved byte-for-byte as committed.

2. Text-mode Data.Text.IO in tests → newline translation + locale codepage crashes

A default Haskell handle uses the OS locale encoding and native newline translation. On Windows this breaks roundtrip tests twice over:

  • \n is written as \r\n, corrupting output meant for the byte-level CSV reader (embedded quoted newlines come back different);
  • non-ANSI text can't be encoded under a non-UTF-8 codepage at all — e.g. under a GBK locale, fast_roundtrip_utf8 dies with commitAndReleaseBuffer: cannot encode character '\676'.

Fix: in test writers, Data.Text.IOData.Text.IO.Utf8 (same writeFile signature, byte-mode UTF-8, no new dependencies; provided by text >= 2.1). For the streaming handle in prettyPrintSeparated (fastcsv tests), set the handle explicitly: hSetEncoding handle utf8 + hSetNewlineMode handle noNewlineTranslation.

3. Hardcoded /tmp paths (3 sites)

/tmp doesn't exist on Windows — the path resolves relative to the current drive's root, so the tests only pass by accident if C:\tmp happens to exist, and fail with withFile: does not exist otherwise.

Fix: System.Directory.getTemporaryDirectory (sites: tests/Operations/WriteCsv.hs, tests/Operations/Record.hs; the two property-test paths in dataframe-fastcsv/tests/Properties/Csv.hs were moved under ./tests/data/unstable_csv/ alongside the existing pattern there). Non-test /tmp uses (LazyBenchmark default argument, the huggingface URI predicate) were deliberately left untouched.

Verification

Upstream has no Windows CI lane, so the CI evidence below comes from my fork, where a temporary fork-only workflow (windows-latest + macos-14 × GHC 9.6.7/9.12.2) was stacked under these same fixes. That exact state is preserved on the fix/windows-test-portability-with-ci branch; this PR branch is the identical fixes rebased onto current main without the workflow commit.

Note on Haskell-CI: that workflow also fails on this branch, but it is red on upstream main as well (latest three runs on main all conclude failure, most recent 2026-08-14), so it appears unrelated to this change.

skymanbp and others added 4 commits August 18, 2026 14:00
…tions

Three root causes, all test-infrastructure (no library code changed),
found by the Windows CI lanes and reproduced on a Windows 11 machine:

1. Missing .gitattributes: with git's core.autocrlf=true (the Windows
   default), checkout rewrites the CSV fixtures' LF to CRLF - including
   newlines EMBEDDED IN QUOTED FIELDS (git does not know CSV) - so the
   byte-level fastcsv reader sees corrupted input and the
   fast_roundtrip_newlines / fast_roundtrip_quotes_and_newlines
   round-trips fail. Fixed by marking *.csv / *.tsv -text.

2. Text-mode writes in test harnesses: Data.Text.IO writeFile/hPutStrLn
   honour the handle's text mode, which on Windows (a) translates \n to
   \r\n, corrupting quoted embedded newlines (typed_quote_spans_boundary,
   and the round-trip rewrite path in prettyPrintSeparated), and (b)
   encodes via the OS locale codepage, crashing fast_roundtrip_utf8 with
   'cannot encode character' on non-UTF-8 codepages (e.g. CP936). Fixed
   by Data.Text.IO.Utf8 (same signatures, byte-mode UTF-8, no new
   dependency; text >= 2.1 is already required) and, for the streaming
   pretty-printer, hSetEncoding utf8 + hSetNewlineMode
   noNewlineTranslation.

3. Hardcoded /tmp paths: toCsv_roundTrip wrote to /tmp (does not exist
   on Windows); fixed with getTemporaryDirectory. The same pattern in
   Properties/Csv.hs (two sites) now uses the suite's existing
   tests/data/unstable_csv scratch dir.

Verified on Windows 11 (GHC 9.12.4): dataframe suite 1097/1097,
dataframe-fastcsv suite 61/61, zero errors/failures - previously
toCsv_roundTrip, fast_roundtrip_newlines,
fast_roundtrip_quotes_and_newlines and typed_quote_spans_boundary
failed (https://github.com/skymanbp/dataframe/actions/runs/32095552668).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A repo-wide sweep for '/tmp' found one more test writing there:
deriveSchemaReadsCsv in tests/Operations/Record.hs. It passed on a dev
machine that happened to have C:\tmp (Windows resolves /tmp against
the drive root) but fails on CI runners. Same fix as WriteCsv.hs:
getTemporaryDirectory + the byte-mode UTF-8 writer. The two other
grep hits are not test-filesystem uses (a benchmark default argument
and a pure URI-predicate check) and are left untouched.

Local re-run: dataframe 1097/1097, dataframe-fastcsv 61/61, zero
errors/failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@skymanbp
skymanbp force-pushed the fix/windows-test-portability branch from 402e473 to ed1fcbd Compare August 18, 2026 18:01
@skymanbp
skymanbp marked this pull request as draft August 18, 2026 18:47
@skymanbp
skymanbp marked this pull request as ready for review August 18, 2026 18:47
mchav added 6 commits August 18, 2026 18:58
We can leave them only where they are used.
We can add explanatory comments in the tests.
Removed unnecessary blank lines and comments in Record.hs
The comment read too much like AI
@mchav

mchav commented Aug 19, 2026

Copy link
Copy Markdown
Member

Thanks. This is a great change! Reminds me that I need to split my dev time between unix and Windows.

@mchav
mchav merged commit 771d2b3 into DataHaskell:main Aug 19, 2026
skymanbp added a commit to skymanbp/dataframe that referenced this pull request Aug 19, 2026
Same classes DataHaskell#212 fixed in tests. CSV and HTML writers and the lazy
reader used locale handles; pin them to UTF-8. 'start' is a cmd
builtin, so launch it through a shell. lazy-bench wrote to /tmp,
which Windows lacks; use the system temp dir.
skymanbp added a commit to skymanbp/dataframe that referenced this pull request Aug 19, 2026
Same classes DataHaskell#212 fixed in tests. CSV and HTML writers and the lazy
reader used locale handles; pin them to UTF-8. 'start' is a cmd
builtin, so launch it through a shell. lazy-bench wrote to /tmp,
which Windows lacks; use the system temp dir.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants