[carloea2]

What changes were proposed in this PR?

Support naive local epoch timestamps in Python tuple hashing on Windows. Attach the local timezone before computing epoch seconds; aware timestamps keep their existing conversion.

Any related issues, documentation, discussions?

Closes #8211

Replaces #8212, which the stale bot closed. The original PR cannot be reopened by this account.

The loop CI failure is a separate PyIceberg signer race. Reproduced the failing S3 read locally and opened #9128 with the independent fix in #9129. This PR does not include that storage change.

How was this PR tested?

Fresh detached worktree from upstream main at 98c63374b7, Python 3.12 on Windows. All 366 focused tests pass, with one existing expected failure and no deselections. The epoch regression failed before the initial implementation.

The negative-epoch reference fixture uses UTC arithmetic so Windows can construct it. Its expected Java hash values are unchanged.

Generate protobufs with bash bin/python-proto-gen.sh, then run from amber:

python -m pytest src/test/python/core/models src/test/python/core/architecture/sendsemantics/test_partitioners.py -q --tb=line -p no:cacheprovider
python -m ruff check src/main/python src/test/python
python -m ruff format --check src/main/python src/test/python

Both Ruff checks pass for all 217 Python files. All CI checks pass, including Ubuntu and macOS integration and Python 3.11, 3.12, and 3.13. The separate signer race still needs #9129; a green run here does not resolve that intermittent storage problem.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex

[carloea2]

This replaces stale-closed #8212. The problem still reproduces on current main. The patch passes its regression, 349 broader tests and Ruff checks.

/request-review @aglinxinyuan

[carloea2]

/request-review @aglinxinyuan

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @eugenegujing, @aglinxinyuan
  You can notify them by mentioning @eugenegujing, @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.86%. Comparing base (98c6337) to head (5e34235).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8957      +/-   ##
============================================
- Coverage     91.87%   91.86%   -0.01%     
  Complexity     5049     5049              
============================================
  Files          1265     1265              
  Lines         54915    54916       +1     
  Branches       6940     6940              
============================================
- Hits          50453    50451       -2     
- Misses         2769     2771       +2     
- Partials       1693     1694       +1     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.38% <ø> (ø)		Carriedforward from 98c6337
agent-service	99.16% <ø> (ø)		Carriedforward from 98c6337
amber	86.06% <ø> (ø)		Carriedforward from 98c6337
computing-unit-managing-service	60.48% <ø> (ø)		Carriedforward from 98c6337
config-service	87.37% <ø> (ø)		Carriedforward from 98c6337
file-service	82.00% <ø> (ø)		Carriedforward from 98c6337
frontend	96.50% <ø> (ø)		Carriedforward from 98c6337
notebook-migration-service	83.73% <ø> (ø)		Carriedforward from 98c6337
pyamber	98.48% <100.00%> (-0.07%)	⬇️
workflow-compiling-service	72.43% <ø> (ø)		Carriedforward from 98c6337

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 8 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main 98c6337 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	367	0.224	24,721/40,710/40,710 us	🔴 +25.6% / 🔴 +186.0%
🔴	bs=100 sw=10 sl=64	771	0.471	127,276/156,461/156,461 us	🔴 +13.1% / 🔴 +62.2%
⚪	bs=1000 sw=10 sl=64	962	0.587	1,034,612/1,093,599/1,093,599 us	⚪ within ±5% / 🔴 +20.2%

Baseline details

Latest main 98c6337 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	367 tuples/sec	442 tuples/sec	879.16 tuples/sec	-17.0%	-58.3%
bs=10 sw=10 sl=64	MB/s	0.224 MB/s	0.27 MB/s	0.537 MB/s	-17.0%	-58.3%
bs=10 sw=10 sl=64	p50	24,721 us	20,319 us	11,504 us	+21.7%	+114.9%
bs=10 sw=10 sl=64	p95	40,710 us	32,409 us	14,234 us	+25.6%	+186.0%
bs=10 sw=10 sl=64	p99	40,710 us	32,409 us	17,338 us	+25.6%	+134.8%
bs=100 sw=10 sl=64	throughput	771 tuples/sec	831 tuples/sec	1,147 tuples/sec	-7.2%	-32.8%
bs=100 sw=10 sl=64	MB/s	0.471 MB/s	0.507 MB/s	0.7 MB/s	-7.1%	-32.7%
bs=100 sw=10 sl=64	p50	127,276 us	112,525 us	88,987 us	+13.1%	+43.0%
bs=100 sw=10 sl=64	p95	156,461 us	162,732 us	96,473 us	-3.9%	+62.2%
bs=100 sw=10 sl=64	p99	156,461 us	162,732 us	102,791 us	-3.9%	+52.2%
bs=1000 sw=10 sl=64	throughput	962 tuples/sec	966 tuples/sec	1,182 tuples/sec	-0.4%	-18.6%
bs=1000 sw=10 sl=64	MB/s	0.587 MB/s	0.59 MB/s	0.721 MB/s	-0.5%	-18.6%
bs=1000 sw=10 sl=64	p50	1,034,612 us	1,033,156 us	872,751 us	+0.1%	+18.5%
bs=1000 sw=10 sl=64	p95	1,093,599 us	1,076,333 us	910,151 us	+1.6%	+20.2%
bs=1000 sw=10 sl=64	p99	1,093,599 us	1,076,333 us	941,479 us	+1.6%	+16.2%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,545.48,200,128000,367,0.224,24721.15,40709.71,40709.71
1,100,10,64,20,2593.20,2000,1280000,771,0.471,127275.53,156460.57,156460.57
2,1000,10,64,20,20799.83,20000,12800000,962,0.587,1034612.33,1093599.27,1093599.27

[aglinxinyuan]

What might be the problem in the current main if we don't have this?

[carloea2]

On Windows, hashing a valid epoch timestamp can raise OSError: Invalid argument and stop hash routing. I reproduced it today: datetime.fromtimestamp(0) produces 1969-12-31 18:00 locally, and the naive timestamp() call fails. The new regression passes, along with all 356 broader tests. Python CI passes on 3.11, 3.12, and 3.13. I am checking the separate Ubuntu loop failure, which reports S3 GetObject AccessDenied.

[carloea2]

I reproduced the storage failure against RustFS rc.6 and Lakekeeper 0.11.0. A controlled read during PyIceberg's signer replacement gets 403; the same read succeeds once signing resumes. Separate filesystem instances also pass.

This matches the upstream signer race. Its proposed fix, #3783, is still open.

The full local REST-backed loop run finished with 7 passing tests and 1 canceled as flaky after a retry, not a clean eight-test pass. The JDBC-backed run passed all 8 without cancellation.

@aglinxinyuan, could you rerun the failed Ubuntu job? This account cannot rerun it. I have kept the timestamp change separate from this shared storage problem.

[carloea2]

Reproduced the loop failure locally: a concurrent PyIceberg signer replacement causes a real S3 read to fail. The separate fix is #9129. This branch stays timestamp-only; all 366 focused tests pass, with one existing expected failure. Fresh CI is running after the rebase.

[carloea2]

@aglinxinyuan, all CI checks are now green, including Ubuntu and macOS integration. The 366 focused tests also pass locally, with one existing expected failure. Could you take a final look and merge if satisfied?