[kz930]

What changes were proposed in this PR?

Filter's exported Python compared the column to the value with pandas operators. The engine's FilterPredicate.evaluate compares by the column's type instead, so the two disagreed on numbers stored as text, booleans, NaN and -0.0.

Each comparison in the export now goes through _texera_filter_keep, a transcription of FilterPredicate.evaluate:

Column type	Comparison
STRING	as Java doubles when both sides parse as one, otherwise by UTF-16 text
BOOLEAN	the lower-case text "true"/"false"
INTEGER, DOUBLE	by Double.compareTo: NaN equals NaN and sorts above every number, -0.0 sorts below 0.0; a hex value past the largest double is infinity, as in Java
LONG	as integers, rejecting what Long.valueOf rejects
TIMESTAMP	by whole milliseconds; a value with an offset is read as the wall clock it shows in the machine's zone, with the offset that zone had at that instant

The column type comes from the input schema. Without one, it is inferred from the pandas dtype. is null and is not null are unchanged.

One approximation remains. The engine reads a TIMESTAMP value with DateParserUtils, which accepts more formats than pandas. A value pandas cannot read raises an error, so the export never gives a silently wrong answer.

A value with an offset is placed in the machine's zone by _texera_local_wall_clock from #8597. It uses the zone's IANA history, as the JVM does, and not Windows' localtime. Before 1900 it takes the zone's standard offset, as the engine does.

Any related issues, documentation, discussions?

Closes #8977

Depends on #8597, which adds _texera_local_wall_clock, and on #9035, which reads a column named in a different case. This PR stays a draft until both merge.

How was this PR tested?

A new test in SpecializedFilterOpDescSpec runs 33 predicates across the six column types through SpecializedFilterOpExec and through the export, and compares the kept rows by input position. A second test does the same for timestamp values with an offset, with both the JVM and Python set to Asia/Kolkata. Both fail on main and pass with this change. A third test sets both to America/Mexico_City and compares 2021-07-01T00:00:00Z (summer time) and 1931-01-01T00:00:00Z (an older offset). With #8597 merged in, the Filter specs pass (42 tests). A fourth test compares a LONG column Key through a predicate on KEY, past 2^53, and passes with #8597 and #9035 merged in.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 50.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.84%. Comparing base (6a29c59) to head (c21c9a2).
⚠️ Report is 51 commits behind head on main.

Files with missing lines	Patch %	Lines
...mber/operator/filter/SpecializedFilterOpDesc.scala	50.00%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8980      +/-   ##
============================================
- Coverage     92.61%   91.84%   -0.78%     
+ Complexity     5059     5047      -12     
============================================
  Files          1258     1257       -1     
  Lines         53780    54254     +474     
  Branches       6707     6775      +68     
============================================
+ Hits          49811    49829      +18     
- Misses         2313     2750     +437     
- Partials       1656     1675      +19     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.38% <ø> (ø)
agent-service	99.16% <ø> (ø)		Carriedforward from 2195568
amber	86.03% <50.00%> (-2.08%)	⬇️
computing-unit-managing-service	60.48% <ø> (ø)
config-service	87.37% <ø> (ø)
file-service	82.00% <ø> (ø)
frontend	96.59% <ø> (ø)		Carriedforward from 2195568
notebook-migration-service	83.73% <ø> (ø)
pyamber	98.58% <ø> (ø)		Carriedforward from 2195568
workflow-compiling-service	72.43% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 4 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main 6a29c59 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	539	0.329	17,510/25,826/25,826 us	🔴 -8.0% / 🔴 +81.4%
🔴	bs=100 sw=10 sl=64	1,242	0.758	79,663/102,629/102,629 us	🔴 +6.7% / 🟢 -10.5%
⚪	bs=1000 sw=10 sl=64	1,472	0.898	677,286/745,433/745,433 us	⚪ within ±5% / 🟢 +24.5%

Baseline details

Latest main 6a29c59 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	539 tuples/sec	586 tuples/sec	879.16 tuples/sec	-8.0%	-38.7%
bs=10 sw=10 sl=64	MB/s	0.329 MB/s	0.357 MB/s	0.537 MB/s	-7.8%	-38.7%
bs=10 sw=10 sl=64	p50	17,510 us	16,998 us	11,504 us	+3.0%	+52.2%
bs=10 sw=10 sl=64	p95	25,826 us	26,708 us	14,234 us	-3.3%	+81.4%
bs=10 sw=10 sl=64	p99	25,826 us	26,708 us	17,338 us	-3.3%	+49.0%
bs=100 sw=10 sl=64	throughput	1,242 tuples/sec	1,255 tuples/sec	1,147 tuples/sec	-1.0%	+8.2%
bs=100 sw=10 sl=64	MB/s	0.758 MB/s	0.766 MB/s	0.7 MB/s	-1.0%	+8.2%
bs=100 sw=10 sl=64	p50	79,663 us	78,949 us	88,987 us	+0.9%	-10.5%
bs=100 sw=10 sl=64	p95	102,629 us	96,209 us	96,473 us	+6.7%	+6.4%
bs=100 sw=10 sl=64	p99	102,629 us	96,209 us	102,791 us	+6.7%	-0.2%
bs=1000 sw=10 sl=64	throughput	1,472 tuples/sec	1,431 tuples/sec	1,182 tuples/sec	+2.9%	+24.5%
bs=1000 sw=10 sl=64	MB/s	0.898 MB/s	0.873 MB/s	0.721 MB/s	+2.9%	+24.5%
bs=1000 sw=10 sl=64	p50	677,286 us	708,207 us	872,751 us	-4.4%	-22.4%
bs=1000 sw=10 sl=64	p95	745,433 us	743,254 us	910,151 us	+0.3%	-18.1%
bs=1000 sw=10 sl=64	p99	745,433 us	743,254 us	941,479 us	+0.3%	-20.8%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,371.03,200,128000,539,0.329,17509.53,25826.40,25826.40
1,100,10,64,20,1610.75,2000,1280000,1242,0.758,79662.54,102628.54,102628.54
2,1000,10,64,20,13588.09,20000,12800000,1472,0.898,677285.99,745433.40,745433.40

[kz930]

@carloea2 Could you review this? Filter's export now compares each predicate by the column's type, as FilterPredicate.evaluate does, and a new spec test checks 26 predicates against the engine.

[carloea2]

The 40 existing Filter tests and scalafmtCheckAll pass against current upstream main on JDK 17. Two additional native versus export parity cases fail below. Finite hexadecimal, unzoned timestamp and null controls pass.

Commands: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec" scalafmtCheckAll; sbt -batch "WorkflowOperator/testOnly *FilterBoundaryReviewSpec"

[kz930]

@carloea2 Thanks, both findings are fixed in 55ce604 with regressions against the native executor. Could you take another look?

[carloea2]

Both original regressions now pass. A remaining Windows timezone problem is reproduced below. Of 47 focused tests, 44 pass and 3 fail: the two historical cases and the new Asia/Kolkata test, whose TZ setting is not honored as an IANA zone by Windows localtime. scalafmtCheckAll passes.

Command: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec *FilterBoundaryReviewSpec" scalafmtCheckAll

[carloea2]

The historical timezone finding is fixed. With #8597 at f0eedd9 merged locally into current upstream main, all 61 focused tests and scalafmtCheckAll pass on Windows. That includes my original hexadecimal, offset and historical regressions. The PR head alone lacks _texera_local_zone, so the CI dependency is real. Leaving the draft unchanged until #8597 lands and CI passes.

Command: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec *FilterBoundaryReviewSpec *TypeCastingOpDescSpec" scalafmtCheckAll

[carloea2]

Tested together with #8597 and #9035 after resolving only the import conflicts. Nine existing tests pass. The #9035 generated-code assertion needs updating for the new helper, and an added native-versus-export regression finds a LONG comparison mismatch.

sbt -batch 'WorkflowOperator/testOnly *SpecializedFilterOpDescSpec -- -oDF'

[carloea2]

The LONG Key and KEY regression is fixed. All 12 SpecializedFilterOpDescSpec tests pass with the #9035 lookup API and timestamp helpers supplied. The native filter and export now keep only the correct row beyond 2^53. scalafmtCheckAll passes. The combined #9035 generated-code assertion was updated mechanically for the new helper. CI is still red, so the dependencies and checks need clearing before merge.