Skip to content

fix: parse multiline fields in CSVArray::toArray - #100

Merged
spawnia merged 7 commits into
masterfrom
AE-4872-csv-multiline
Nov 4, 2026
Merged

spawnia merged 7 commits into
masterfrom
AE-4872-csv-multiline

Conversation

@spawnia

@spawnia spawnia commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

CSVArray::toArray() split the CSV into lines before parsing.
A quoted field with a line break became several broken rows.
It now cuts records only at line breaks outside quoted fields, then parses each record with str_getcsv.

Line breaks, blank lines and empty input behave as before

Lines still split at \r\n, \n and \r, mixed in one file too.
Blank lines are skipped and keep their row number, missing columns default to '', and '' returns [].
Row keys count records, which differs from line numbers only for multiline fields.

fgetcsv over php://memory was tried and rejected

fgetcsv ends records only at \n, so \r-only and mixed line breaks merged rows.
It returns [null] for a blank line, the same as for a quoted "" in a one-column file, so such rows were dropped.
It also needed PHPStan ignores, since Safe\fgetcsv is missing in Safe 1.x.

For agents 🤖: Copilot threads addressed, CI running, then review
  • splitRecords() scans the input once and follows str_getcsv quoting: an enclosure opens only at the start of a field, after spaces, tabs, \v or \f. A doubled enclosure stays inside the field, and an escape character inside it skips the next character.
  • 100,000 random inputs per setting split into the same records as fgetcsv, for escape \, " and '', and for delimiter equal to enclosure. With escape \n, fgetcsv itself is no valid reference; a test covers that case.
  • 200,000 rows: 1.1 s and 122 MB peak, master 0.4 s and 122 MB.
  • The unreachable Missing column headers. throw stays as on master.
  • make rector fails locally after composer update: Undefined constant PHPUnitSetList::PHPUNIT_40. This comes from the newer Rector PHPUnit sets and is unrelated to this change.
  • After merge and release, update the consumer project to the new version.

🤖 Generated with Claude Code

Read records with fgetcsv instead of splitting lines first,
so quoted fields may contain line breaks.

🤖 Generated with Claude Code
@github-code-quality

github-code-quality Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Code coverage overview

Languages: PHP

PHP / code-coverage/phpunit

master 2949ba3 AE-4872-csv-multilin... 50bb216 +/-
89% 90% +1%
View coverage of the 6 most impacted files.
File master 2949ba3 AE-4872-csv-multilin... 50bb216 +/-
IlluminaSampleS...ingsSection.php 100% 100% 0%
InterOp/RunResult.php 100% 100% 0%
CSVArray.php 90% 96% +6%
PHPStan/Rules/M...ypehintRule.php 0% 88% +88%
PHPStan/Rules/M...ypehintRule.php 0% 100% +100%
PHPStan/Rules/C...ypehintRule.php 0% 100% +100%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated November 04, 2026 13:01 UTC

🤖 Generated with Claude Code
fgetcsv ends records only at \n, so input whose lines end in a lone \r,
such as classic Mac exports, parsed to no rows.
Input without any \n now has its \r normalized to \n before parsing.

🤖 Generated with Claude Code
fgetcsv ends records only at \n and returns [null] for blank lines,
which is indistinguishable from an enclosed empty value. That dropped
single-column rows with "", merged rows with mixed line breaks and
broke CR-only input with an enclosed \n.

Keep splitting at \r\n, \n and \r as before, but join lines while an
enclosed field is open, then parse each record with str_getcsv.

🤖 Generated with Claude Code
A stray enclosure inside an unenclosed field, such as an inch mark,
opened an enclosure that never closed and merged the following rows.
Track field state like str_getcsv does: an enclosure opens only at the
start of a field, after optional spaces and tabs.

🤖 Generated with Claude Code
@spawnia
spawnia requested a max review from Copilot November 4, 2026 12:36
@spawnia
spawnia marked this pull request as ready for review November 4, 2026 12:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved scanner edge cases split valid multiline CSV fields into separate rows, while retaining all line separators substantially increases peak memory.

4 open issues
What changed in this PR

Updates CSVArray::toArray() to parse quoted multiline fields as single records, improving round-trip conversion with toCSV().

Changes:

  • Adds quote-aware record splitting before parsing headers and values with str_getcsv().
  • Expands coverage for line endings, enclosure and escape behavior, and empty input; enables the existing multiline round-trip test.
File Description
tests/​CSVArrayTest.php Adds parsing regression cases and enables multiline round-trip coverage.
src/​CSVArray.php Introduces field-state tracking to join lines within enclosed CSV fields.

Review effort: Max

Comment thread src/CSVArray.php Resolved
Comment thread src/CSVArray.php Resolved
Comment thread src/CSVArray.php Resolved
Comment thread src/CSVArray.php Resolved
Splitting into lines first kept every captured line break in memory
and lost the escape state at line breaks. Scan the input once and cut
records at line breaks outside enclosed fields.

Also open an enclosure before checking for a delimiter, and skip
vertical tabs and form feeds before an enclosure, as str_getcsv does.

🤖 Generated with Claude Code
@spawnia
spawnia merged commit a211c1e into master Nov 4, 2026
29 checks passed
@spawnia
spawnia deleted the AE-4872-csv-multiline branch November 4, 2026 14:08
github-actions Bot pushed a commit that referenced this pull request Nov 4, 2026
## [7.0.1](v7.0.0...v7.0.1) (2026-11-04)

### Bug Fixes

* parse multiline fields in CSVArray::toArray ([#100](#100)) ([a211c1e](a211c1e))
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 7.0.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Development

Successfully merging this pull request may close these issues.

2 participants