Skip to content

fix(dbt): quote exported YAML scalars so values keep their type - #372

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/integrations-dbt-exporter-yaml
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/integrations-dbt-exporter-yaml

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

export_dbt_tests hand-writes schema.yml, and _scalar left strings unquoted when they would still change type under YAML 1.1 loaders (PyYAML, which dbt uses): 2024-01-01 loaded as a date, 0x1F as 31, 0b101 as 5. The regex's $ anchor also matched before a trailing newline, so "a\n" was written unquoted and lost its newline. Newlines inside quoted scalars were not escaped, so they folded to spaces.

  • A string is written unquoted only if it fully matches a letter or underscore followed by [A-Za-z0-9_.-] and is not a bool/null/nan/inf word. Everything else is double-quoted.
  • Inside double quotes, backslash, quote, \n, \t, \r, other C0/C1 controls, DEL, U+2028/U+2029, lone surrogates and U+FFFE/U+FFFF are escaped.
  • Python floats now load back as floats: .nan / .inf / -.inf, and exponent forms get a mantissa dot (1e+16 becomes 1.0e+16).
  • Real ints, floats and bools are still written as plain YAML numbers and bools. No new runtime dependency.

Tests

New tests/test_integrations/test_dbt_tests_exporter_yaml.py. Every value is loaded back with yaml.safe_load, both as a mapping value and as a list item, and must come back as the same value and type:

  • Tricky strings: dates and timestamps, 12:30, 0x1F/0b101/0o17/017, 1e3, 1_000, .5, +1/-1, .inf/.NaN, yes/No/on/null/~/y/n, YAML indicators, leading/trailing spaces, quotes, backslashes, tabs, newlines, control characters, NEL, U+2028/U+2029, BOM, non-characters, non-ASCII.
  • Ints, bools, None, and floats including NaN, ±inf, 1e16, 1e-5 and numpy floats.
  • Every 1- and 2-character string over a tricky alphabet.
  • Ordinary identifiers stay unquoted.
  • The issue repro end-to-end through export_dbt_tests, plus a mixed accepted_values round trip.

Verification

  • ruff check . clean; mypy src/freshdata clean
  • pytest -m "not online and not large": py3.12 4374 passed; py3.9 4370 passed

Closes #342

export_dbt_tests hand-writes schema.yml. _scalar left a string unquoted
when it matched ^[A-Za-z0-9_][A-Za-z0-9_.-]*$ and float() rejected it,
but YAML 1.1 loaders (PyYAML, used by dbt) still re-type such text:
2024-01-01 loads as a date, 0x1F as 31, 0b101 as 5. The `$` anchor also
matched before a trailing newline, so "a\n" was written bare, and the
double-quoted branch left newlines raw, so they folded to spaces.

Strings are now emitted plain only when they fully match a letter or
underscore followed by [A-Za-z0-9_.-] and are not a bool/null/nan/inf
word; everything else is double-quoted. Inside quotes, backslash, quote,
\n, \t, \r, other C0/C1 controls, DEL, U+2028/U+2029, lone surrogates
and U+FFFE/U+FFFF are escaped. Python floats are written so PyYAML
reads floats back: .nan/.inf/-.inf, and 1e+16 as 1.0e+16. Genuine
ints, floats and bools stay unquoted YAML numbers/bools.

Closes #342
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5334ad9d-40b1-43c5-9381-33af9b81cd63


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit 503c16c into main Sep 15, 2026
21 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

export_dbt_tests writes unquoted YAML scalars that re-type ('2024-01-01', '0x1F', '0b101')

1 participant