Repository navigation
fix(spark): atomic renames, ordered keep-aware dedup, NaN/inf handling, exact numeric type names - #378
Merged
Conversation
…g, exact numeric type names - #330: _stage_rename applied the rename map one withColumnRenamed call at a time. withColumnRenamed renames every case-insensitive match, so a deduplicated map such as {"a b": "a_b", "a_b": "a_b_2"} or {"A": "a", "a": "a_2"} renamed both columns in the second call and left duplicate names that later F.col() references could not resolve. The map is now applied in one positional projection with sdf.toDF(...). - #331: full-row dedup used dropDuplicates(), a shuffle aggregate that ignores duplicate_keep and returns rows in arbitrary order. An ordinal (monotonically_increasing_id) is taken before any shuffle, row_number() over a window partitioned by every column keeps the first (ascending) or last (descending) occurrence, and sorting by the ordinal restores the input order. The ordinal follows the input DataFrame's partition order; docs say so. Detection-only runs still count with dropDuplicates().count(). - #332: Spark keeps float NaN as a value, so missing counts, empty column/row drops and impute missed it, and a column holding both nulls and NaN was "imputed" with a NaN mean. Float/double NaN is now read as null at ingestion, as the Polars and DuckDB engines do. Outlier fences are computed from finite values only (pandas drop_infinite); +-inf is still tested against them. Impute statistics are unchanged, since the pandas reference includes inf. - #333: numeric detection prefix-matched simpleString dtypes, so "interval day to second" counted as "int" and reached approxQuantile / float(timedelta). Type names are now matched exactly after stripping parameters (decimal(10,2) -> decimal), with the integer subset for _is_integer_column. pyspark is not available locally, so these were verified by inspection and by unit tests that drive SparkEngine through an evaluating fake pyspark.sql (DataFrame, functions, Window), including end-to-end execute() runs compared with the pandas engine. Real-SparkSession versions of each repro are included and skip without pyspark or a JVM. Closes #330 Closes #331 Closes #332 Closes #333
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This was referenced Sep 15, 2026
kevincostner17
added a commit
that referenced
this pull request
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes four Spark engine divergences from the pandas reference:
withColumnRenamedrenames every case-insensitive match, so maps like{"a b": "a_b", "a_b": "a_b_2"}produced duplicate column names. The map is now applied in one positionaltoDF(...)projection.dropDuplicates()ignoredduplicate_keepand shuffled rows. Dedup now numbers rows withmonotonically_increasing_id(), keepsrow_number() == 1per full row (ascending forfirst, descending forlast), and sorts by that number to restore input order. Order follows the input DataFrame's partition order (documented). Detection-only runs still usedropDuplicates().count().NaNis read as null on ingestion (as in the Polars/DuckDB engines), which fixes missing counts, empty column/row drops and impute. Outlier fences are computed from finite values only; ±inf is still tested against them. Impute statistics still include inf, as pandas does.interval day to secondis no longer taken asint.Docs:
docs/backends.mdanddocs/fallback-matrix.mdupdated for NaN handling, finite outlier fences and Spark dedup order.Tests
New
tests/test_execution/test_spark_engine_fixes.py:pyspark.sql(DataFrame, functions, Window), including end-to-endSparkEngine.execute()runs compared against the pandas engine (dedup keep first/last, NaN missing counts and drops, mean impute with null+NaN, zscore flags with ±inf, interval columns skipped, rename collisions).Verification
ruff check .clean;mypy src/freshdataclean.-m "not online and not large"): py3.12 4275 passed / 13 skipped; py3.9 4271 passed / 17 skipped.Closes #330
Closes #331
Closes #332
Closes #333