Skip to content

fix(cpp): preserve leading backslashes in tsfile-cli CSV round trips - #994

Merged
ColinLeeo merged 4 commits into
apache:developfrom
ColinLeeo:colin/tsfile-143-csv-roundtrip
Oct 9, 2026
Merged

ColinLeeo merged 4 commits into
apache:developfrom
ColinLeeo:colin/tsfile-143-csv-roundtrip

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

CSV export emits the literal text \N as the same unquoted token used for NULL. Importing that output with write silently converts the text value to NULL.

Prefix non-null STRING/TEXT values that start with a backslash with one extra leading backslash when writing CSV, and remove exactly one prefix during import while preserving the distinction between literal \N and NULL. This applies to STRING TAG columns as well. Headers and backslashes inside values are unchanged. This addresses TsFile-155 and TsFile-159.

Document the shared encoding rule and examples in write --help and the tools README. The help block uses one raw string literal, and export reuses the formatted cell buffer when adding the prefix. The branch includes the latest develop UTF-8 output changes, with both sets of regression tests retained.

Validation:

  • Built TsFile_Test and tsfile_cli, with LZ4 enabled.
  • All 212 selected CLI, CSV, UTF-8 output, golden fixture, and tree/table model tests passed.
  • A separate 1,300-row check passed for STRING/TEXT/TAG values with 0–12 leading backslashes, NULL/literal distinctions, mixed quoting, Unicode, LF/CR, reordered headers, the 1,024-row batch boundary, and 128 KiB+ cells. Verified cat/export CSV output, file/stdin re-import, TAG filtering, duplicate-device timestamp rejection, and failed-import cleanup.
  • Verified the exact displayed backslash tokens in write --help.
  • Spotless formatting for the changed test files, clang-format 17.0.6 checks for the edited C++ files, and git diff --check passed.

@ColinLeeo
ColinLeeo requested a balanced review from Copilot October 8, 2026 07:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings
What changed in this PR

Fixes tsfile-cli CSV round-trips so that literal text beginning with \ (including \N) is preserved as a string value and not conflated with the unquoted CSV NULL marker (\N).

Changes:

  • CSV export: add an extra leading backslash for STRING/TEXT values that begin with \.
  • CSV import (write): undo the extra leading-backslash escape for STRING/TEXT and prevent decoded \N from being treated as NULL.
  • Add unit + end-to-end regression tests covering leading backslashes, literal \N vs NULL, commas/quoting, and embedded backslashes.
File Description
cpp/​tools/​format/​output_format.cc Escapes leading backslashes for STRING/TEXT when emitting CSV to preserve literal values like \N.
cpp/​tools/​commands/​cmd_write.cc Adds import-side unescape logic + tracking to distinguish literal \N from NULL.
cpp/​test/​tools/​output_format_test.cc Adds unit test validating header handling and selective escaping for STRING/TEXT.
cpp/​test/​tools/​command_e2e_test.cc Adds end-to-end CSV write→cat→write round-trip regression for leading backslashes and quoting cases.

🧠 Review effort: Lite


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread cpp/tools/commands/cmd_write.cc
Comment thread cpp/tools/commands/cmd_write.cc Outdated
Comment thread cpp/tools/commands/cmd_write.cc Outdated
@ColinLeeo
ColinLeeo requested a balanced review from Copilot October 8, 2026 08:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings
3 resolved since last review

🧠 Review effort: Lite


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread cpp/tools/cli/run_cli.cc Outdated
Comment thread cpp/tools/commands/cmd_write.cc
Comment thread cpp/tools/format/output_format.cc Outdated
Comment thread cpp/test/tools/output_format_test.cc Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings
4 resolved since last review

🧠 Review effort: Lite


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread cpp/tools/cli/run_cli.cc Outdated
Comment thread cpp/tools/format/output_format.cc Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation consistently preserves text values, NULL semantics, headers, and non-text columns with focused regression coverage.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

@ColinLeeo
ColinLeeo merged commit 9f35d76 into apache:develop Oct 9, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants