[kz930]

What changes were proposed in this PR?

The JSON Lines and Arrow sources implement StandaloneCodeGenerator, so an exported script starts from the same data the workflow did.

Arrow is read into pandas' nullable dtypes rather than its numpy ones. Arrow says of every value whether it is there, and a numpy column has nowhere to put that: a missing double and a stored NaN both land on NaN, and a missing integer costs the column its integer type. The engine reads the file's own answer, so a set operation keeps a NaN row a null row does not match, a holed integer column is still made of integers, and a long past 2^53 is not rounded on its way through a float. The other sources state the types they need one at a time, their formats having no way to say it once.

A file also states the width and the sign of each of its numbers, and Texera has no column narrower than a double or a 32-bit integer. A single-precision or 16-bit column was arriving null, the parse having no case for the Float and Short those vectors hand back, and an unsigned one arrived as the -1 its storage counts down to. Each now reads as the Texera type that holds it, an unsigned 32-bit column as a long and an unsigned 64-bit one refused for want of a wider one, and the script normalizes the same widths so a later sum comes to the same number on both sides. This is the rule the Parquet source already follows. The end of an exported Arrow window is counted in Long, as the CSV readers count theirs.

A timestamp is read as the engine reads one. The engine hands each field to DateParserUtils, so a row states its own format, any year reads, an offset is moved to the local zone, and the reading is cut to the millisecond. pandas' date parsing does none of that. So JSONL reads the column as text and converts it with the helper #8597 adds.

This PR also changes the engine. ArrowUtils handed a TIMESTAMP column to a Python worker as an Arrow millisecond timestamp labelled UTC. A Python operator saw an aware datetime cut to the millisecond, and a year past 2262 overflowed. It is now a zoneless microsecond timestamp. That is what Texera's own Python side declares for a TIMESTAMP (pa.timestamp("us")), and what an exported script holds. Two things a user can see change. A Python UDF now receives a naive datetime instead of one in UTC. A result downloaded as an Arrow file now carries a zoneless microsecond timestamp column instead of a UTC millisecond one. A TIMESTAMP parameter a Python UDF declares is now parsed the same way, as a naive wall clock with any offset moved to the machine's zone, so it compares with a row's instead of raising.

Any related issues, documentation, discussions?

Part of #8325, step 12b; that issue lists the set in order. It needs #8341 for the shared source code and #8597 for the timestamp helpers, so both merge first.

Its JSON Lines export reads a JSON null as the text "null", as the engine does today (#8674). If #8740 merges first, that part changes to read a null before this one merges.

Closes #8702, the task this change is the whole of.

It also finishes #8602, where a scan window of no rows left the type inference nothing to read and JSONL declared a schema of no columns. #8341 fixes the CSV half.

Closes #8602.

How was this PR tested?

Each operator asserts the block it emits in its own spec, and the Arrow spec runs the engine's own executor and the generated script over the same file. The engine change is covered in both ArrowUtilsSpec files, which include a round trip on either side of the nanosecond range, and in the Python tests of the timestamp parser. Once the verification lands, every operator is also run both ways on every configuration its schema offers; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.

Arrow is the one worth calling out. Its column of timestamps only matches once #7672 is in: before that the engine read the file's numbers through the JVM's zone while pd.read_feather read the wall clock the file states, and the two differed by whatever the machine was set to. That change has since landed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @roshiiiz, @carloea2, @aglinxinyuan
  You can notify them by mentioning @roshiiiz, @carloea2, @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.08108% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.59%. Comparing base (535b97a) to head (5b7746a).
⚠️ Report is 81 commits behind head on main.

Files with missing lines	Patch %	Lines
...c/main/python/core/models/schema/attribute_type.py	81.08%	6 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main    #8704    +/-   ##
==========================================
  Coverage     92.59%   92.59%            
- Complexity     5040     5043     +3     
==========================================
  Files          1252     1254     +2     
  Lines         53587    53792   +205     
  Branches       6673     6703    +30     
==========================================
+ Hits          49619    49810   +191     
- Misses         2317     2328    +11     
- Partials       1651     1654     +3     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.44% <ø> (+0.06%)	⬆️
agent-service	99.16% <ø> (ø)		Carriedforward from 72dcf19
amber	88.06% <ø> (-0.01%)	⬇️	Carriedforward from 72dcf19
computing-unit-managing-service	61.84% <ø> (+1.35%)	⬆️
config-service	87.50% <ø> (+0.12%)	⬆️
file-service	81.92% <ø> (+0.38%)	⬆️
frontend	96.58% <ø> (ø)		Carriedforward from 72dcf19
notebook-migration-service	83.77% <ø> (+0.03%)	⬆️
pyamber	98.34% <81.08%> (-0.25%)	⬇️
workflow-compiling-service	74.09% <ø> (ø)		Carriedforward from 72dcf19

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

📊 Arrow Flight E2E bench

(no arrow-flight-e2e.csv in artifact)

Full dashboard · Workflow run

[kz930]

@carloea2 Split out of #8341 so each part reads in one sitting; the code is what you reviewed there, plus today's timestamp fixes.

[carloea2]

With the shared API from the declared #8341 dependency supplied locally, 32 source and Arrow tests pass but the added offset regression fails. The 42 core Arrow tests and 73 Linux Python tests also pass.

[kz930]

@carloea2 Fixed the offset-past-EOF case in d50d410: when the window sample is empty, the JSONL schema now samples from the top of the file, with your offset=10 repro as a test. Could you take another look?

[carloea2]

Retested the current head. The offset past EOF regression is fixed. All 75 focused tests and scalafmtCheckAll pass with the shared source API from #8341 supplied locally.

Command: sbt -batch "WorkflowCore/testOnly *ArrowUtilsSpec -- -oDF" "WorkflowOperator/testOnly *ArrowSourceOpDescSpec *JSONLScanSourceOpDescSpec *ArrowUtilsSpec -- -oDF" "scalafmtCheckAll".

Current CI still fails compilation on that missing API. This verifies the fix, not standalone merge readiness. #8341 and #8597 still need to land first, as the description states.

[carloea2]

The 75 focused Scala tests and scalafmtCheckAll pass with #8341 and #8597 supplied. The Python suites have 70 passes and three tests blocked by their unconditional time.tzset calls on Windows. A separate seven-case production-parser probe finds three pre-1970 offset inputs now raising OSError; ordinary, future, no-offset and invalid-input controls pass. The upstream parser accepts the same pre-1970 inputs.

sbt -batch 'WorkflowCore/testOnly *ArrowUtilsSpec -- -oDF' 'WorkflowOperator/testOnly *ArrowSourceOpDescSpec *JSONLScanSourceOpDescSpec *ArrowUtilsSpec -- -oDF' 'scalafmtCheckAll'

python -m pytest amber/src/test/python/core/models/schema/test_attribute_type.py amber/src/test/python/pytexera/udf/test_udf_operator.py -q

[carloea2]

The original Windows crash is fixed. The original timestamp probes now pass, along with 75 focused Scala checks and 81 Python checks, with seven platform skips. Formatting passes with the declared source prerequisites supplied. A separate value check still finds a mismatch before 1900, confirmed against JDK 17 using the engine's DateParserUtils. Its two historical cases fail while the 1960 and 2024 controls pass.

python -m pytest amber/src/test/python/core/models/schema/test_attribute_type.py amber/src/test/python/pytexera/udf/test_udf_operator.py -q

sbt -batch 'WorkflowCore/testOnly *ArrowUtilsSpec -- -oDF' 'WorkflowOperator/testOnly *ArrowSourceOpDescSpec *JSONLScanSourceOpDescSpec *ArrowUtilsSpec -- -oDF' 'scalafmtCheckAll'

[kz930]

@carloea2 Before 1900 an offset is now read at the zone's standard offset, the same rule as _texera_local_wall_clock, and your Mexico City and Kolkata cases are tests. The JSONL export also infers its schema once instead of once per column type. Could you take another look?