Skip to content

feat(workflow-operator): export the join and set operators as Python - #8506

Draft
kz930 wants to merge 35 commits into
apache:mainfrom
kz930:feat/standalone-relational-and-sampling
Draft

feat(workflow-operator): export the join and set operators as Python#8506
kz930 wants to merge 35 commits into
apache:mainfrom
kz930:feat/standalone-relational-and-sampling

Conversation

@kz930

@kz930 kz930 commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Six operators implement StandaloneCodeGenerator: the three joins and the three set operations. What they have in common is that they combine or compare tables rather than read what is inside a row.

#8340 keeps the operators that work on text. Aggregate and the sort family follow in #8709, and the samplers and control flow in #8710.

Review then found two places where the block answered a question the engine answers differently. The hash join asked pandas for a suffix, which appends once and then refuses the duplicate it just made, and let an outer join's holes cost an integer column its type.

Running the engine on those cases found the platform out of step with itself. HashJoin declared a right column's new name by checking only the left names, while the executor also avoids the other right names. So a right x beside a right x#@1 was declared as a second x#@1, and one value was lost. An unmatched right row in an outer join kept its raw names and landed in the left column. Both now go through one renaming rule in JoinUtils.

The shared table then found one more. In a right or full outer join on keys both sides name alike, the engine writes an unmatched right row's key into the left key column, and the export left it empty. The engine had its own half: an unmatched right key named like a left payload column landed in that payload. The export now fills the key, and the engine leaves it out of the payload.

The set operations matched rows the way pandas does. The engine's HashSet compares rows with Scala's ==, under which NaN equals nothing, and hashes them by each double's bits, which tell 0.0 from -0.0. The export now selects rows by a key that follows both rules.

A probe worker that received no build rows could not name an unmatched right row's columns, because it read the left names off the first build row. This happens when the left input is empty or no left key hashes to that worker. The runtime now records each input port's declared schema on the executor before the port finishes (OperatorExecutor.inputSchemas, filled in EndChannelHandler), and the probe reads the left names from it. This is the Scala counterpart of what #8766 does for Python operators.

Any related issues, documentation, discussions?

Part of #8325, step 11a; #8709 and #8710 follow. That issue lists the set in order.

The set operations and joins keyed on a null or a NaN need the nullable dtypes #8704 reads an Arrow file into. Without it a null and a NaN share a slot and both behave as they did before, so this PR is no worse on its own and only complete alongside that one.

Closes #8505, the task this change is the whole of.

How was this PR tested?

Each operator asserts the block it emits in its own spec. The interval join reads its keys off the merged frame rather than writing over an input column that shares a temporary's name.

The join cases the review named run in the verify tables of #8363, which put the same input through the engine and through the script and compare the two. What those tables cannot reach stays here as executed tests: a null kept apart from a NaN in the set operations and the join key. HashJoinProbeOpExecSpec pins the platform fixes.

Each set operation's spec runs its native executor and the export over NaN, signed-zero, finite and null rows. HashJoinProbeOpExecSpec covers an empty build side, and EndChannelHandlerSpec checks that the schema is in place before the operator finishes the port.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

🤖 Generated with Claude Code

@kz930 @claude
feat(workflow-operator): export the relational and sampling operators
85997d8 
Seventeen operators implement the trait: the three joins, the three set
operations, Aggregate, the sort family, Split, the two samplers, and If,
Dummy and Sleep. What they have in common is that they rearrange rows
rather than read what is inside one.

A sampler decides per row whether to keep it, so which rows survive is
fixed by the exact sequence java.util.Random produces. Seeding Python's
own generator selects a different set, and the script would then report a
different sample than the workflow it came from, so SamplingHelpers
transcribes the generator rather than approximating it.

Split out of apache#8340 on review, which had grown past what one reading can
hold. That change keeps the operators that work on text, and the cast
transcription they share.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 kz930 mentioned this pull request Sep 11, 2026
Merged
@github-actions

github-actions Bot commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @eugenegujing
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @eugenegujing in a comment.

@github-actions

github-actions Bot commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 6 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main 6a29c59 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 647 0.395 14,905/22,646/22,646 us 🔴 +8.5% / 🔴 +101.2%
🔴 bs=100 sw=10 sl=64 1,649 1.007 59,500/71,472/71,472 us 🟢 -7.4% / 🟢 -20.5%
🔴 bs=1000 sw=10 sl=64 2,021 1.233 486,583/582,941/582,941 us 🔴 +9.1% / 🟢 +28.8%
Baseline details

Latest main 6a29c59 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 647 tuples/sec 699 tuples/sec 1,119 tuples/sec -7.4% -42.2%
bs=10 sw=10 sl=64 MB/s 0.395 MB/s 0.427 MB/s 0.683 MB/s -7.5% -42.2%
bs=10 sw=10 sl=64 p50 14,905 us 13,735 us 8,992 us +8.5% +65.7%
bs=10 sw=10 sl=64 p95 22,646 us 24,657 us 11,255 us -8.2% +101.2%
bs=10 sw=10 sl=64 p99 22,646 us 24,657 us 13,808 us -8.2% +64.0%
bs=100 sw=10 sl=64 throughput 1,649 tuples/sec 1,731 tuples/sec 1,518 tuples/sec -4.7% +8.6%
bs=100 sw=10 sl=64 MB/s 1.007 MB/s 1.057 MB/s 0.927 MB/s -4.7% +8.7%
bs=100 sw=10 sl=64 p50 59,500 us 55,941 us 68,504 us +6.4% -13.1%
bs=100 sw=10 sl=64 p95 71,472 us 77,182 us 74,866 us -7.4% -4.5%
bs=100 sw=10 sl=64 p99 71,472 us 77,182 us 89,939 us -7.4% -20.5%
bs=1000 sw=10 sl=64 throughput 2,021 tuples/sec 2,009 tuples/sec 1,569 tuples/sec +0.6% +28.8%
bs=1000 sw=10 sl=64 MB/s 1.233 MB/s 1.226 MB/s 0.957 MB/s +0.6% +28.8%
bs=1000 sw=10 sl=64 p50 486,583 us 491,355 us 669,501 us -1.0% -27.3%
bs=1000 sw=10 sl=64 p95 582,941 us 534,074 us 706,487 us +9.1% -17.5%
bs=1000 sw=10 sl=64 p99 582,941 us 534,074 us 737,560 us +9.1% -21.0%
Raw CSV 
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,309.14,200,128000,647,0.395,14904.52,22645.69,22645.69
1,100,10,64,20,1212.66,2000,1280000,1649,1.007,59499.53,71472.29,71472.29
2,1000,10,64,20,9898.28,20000,12800000,2021,1.233,486583.11,582941.21,582941.21

@codecov-commenter

codecov-commenter commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown

❌ 13 Tests Failed:

Tests completed Failed Passed Skipped
36 13 23 0
View the top 3 failed test(s) by shortest run time 
org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec::HashJoinOpDesc.generateStandaloneCode should pair NaN, signed-zero and null keys as the engine does in a FULL_OUTER join
Stack Traces | 1.26s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/hashjoin-nan-zero-15549487215226597334.py", line 164, in <module>
    _left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
                                ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    return pd.arrays.FloatingArray(
        np.array([0.0 if v is None else v for v in vals]),
        np.array([v is None for v in vals]),
    )

in1df = pd.DataFrame({
    "k": col([np.nan, 0.0, 1.0, None]),
    "lv": ["L-nan", "L+0", "L1", "L-null"],
})
in2df = pd.DataFrame({
    "k2": col([np.nan, -0.0, 1.0, None]),
    "rv": ["R-nan", "R-0", "R1", "R-null"],
})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_left_cols = set(in1df.columns)
_right_cols = list(in2df.columns)
_payload = [_c for _c in _right_cols if _c != "k2"]
_payload_set = set(_payload)
_rename = {}
for _col in _payload:
    _new = _col
    _others = _payload_set - {_col}
    while _new in _left_cols or _new in _others:
        _new = _new + "#@1"
    _rename[_col] = _new
_probe_key = "k2"
_taken = _left_cols | set(_rename.values())
while _probe_key in _taken:
    _probe_key = _probe_key + "#@1"
if _probe_key != "k2":
    _rename["k2"] = _probe_key
_join_key = "_texera_join_key"
while _join_key in _taken or _join_key == _probe_key:
    _join_key = _join_key + "_"
_left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
out1df = in1df.assign(**{_join_key: _left_codes}).merge(
    in2df.rename(columns=_rename).assign(**{_join_key: _right_codes}),
    how="outer",
    on=_join_key,
).drop(columns=[_join_key])
if _probe_key != "k":
    out1df = out1df.drop(columns=[_probe_key])
out1df = out1df.reset_index(drop=True)
def show(v):
    if isinstance(v, float):
        return "nan" if v != v else repr(float(v))
    return str(v)
rows = []
for values, missing in zip(out1df.itertuples(index=False), out1df.isna().to_numpy()):
    rows.append(" ".join("None" if m else show(v) for v, m in zip(values, missing)))
print("\n".join(sorted(rows)))
1 was not equal to 0
	at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
	at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$21(HashJoinOpDescSpec.scala:221)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$10(HashJoinOpDescSpec.scala:220)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.difference.DifferenceOpDescSpec::DifferenceOpDesc.generateStandaloneCode should match rows by the engine's equality for NaN and signed zero
Stack Traces | 1.36s run time 
java.lang.AssertionError: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/difference-nan-zero-7822393439705543590.py", line 136, in <module>
    _texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
                                          ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    mask = np.array([v is None for v in vals])
    return pd.arrays.FloatingArray(np.array([0.0 if v is None else v for v in vals]), mask)

in1df = pd.DataFrame({"k": ["a"] * 6, "v": col([1.0, np.nan, 0.0, None, 2.0, np.nan])})
in2df = pd.DataFrame({"k": ["a"] * 5, "v": col([1.0, np.nan, -0.0, None, 3.0])})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
_texera_set_kept = ~_texera_set_left.duplicated() & ~_texera_set_left.isin(_texera_set_right)
out1df = in1df[_texera_set_kept.to_numpy()].reset_index(drop=True)
def render(v):
    return "None" if v is pd.NA else v if isinstance(v, str) else repr(float(v))
for row in sorted(" ".join(render(v) for v in r) for r in out1df.itertuples(index=False)):
    print(row)

      at org.apache.texera.amber.testsupport.setop.SetRows$.runExport(SetRows.scala:73)
      at org.apache.texera.amber.operator.difference.DifferenceOpDescSpec.$anonfun$new$16(DifferenceOpDescSpec.scala:205)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
      at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
      at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
      at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
      at sbt.TestRunner.runTest$1(TestFramework.scala:153)
      at sbt.TestRunner.run(TestFramework.scala:168)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFunction.apply(TestFramework.scala:348)
      at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
      at sbt.std.Transform$$anon$4.work(Transform.scala:69)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
      at sbt.Execute.work(Execute.scala:292)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.symmetricDifference.SymmetricDifferenceOpDescSpec::SymmetricDifferenceOpDesc.generateStandaloneCode should match rows by the engine's equality for NaN and signed zero
Stack Traces | 1.45s run time 
java.lang.AssertionError: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/symmetric-difference-nan-zero-8672247049477945033.py", line 136, in <module>
    _texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
                                          ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    mask = np.array([v is None for v in vals])
    return pd.arrays.FloatingArray(np.array([0.0 if v is None else v for v in vals]), mask)

in1df = pd.DataFrame({"k": ["a"] * 6, "v": col([1.0, np.nan, 0.0, None, 2.0, np.nan])})
in2df = pd.DataFrame({"k": ["a"] * 5, "v": col([1.0, np.nan, -0.0, None, 3.0])})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
_texera_set_left_kept = ~_texera_set_left.duplicated() & ~_texera_set_left.isin(_texera_set_right)
_texera_set_right_kept = ~_texera_set_right.duplicated() & ~_texera_set_right.isin(_texera_set_left)
out1df = pd.concat(
    [in1df[_texera_set_left_kept.to_numpy()], in2df[_texera_set_right_kept.to_numpy()]],
    ignore_index=True,
)
def render(v):
    return "None" if v is pd.NA else v if isinstance(v, str) else repr(float(v))
for row in sorted(" ".join(render(v) for v in r) for r in out1df.itertuples(index=False)):
    print(row)

	at org.apache.texera.amber.testsupport.setop.SetRows$.runExport(SetRows.scala:73)
	at org.apache.texera.amber.operator.symmetricDifference.SymmetricDifferenceOpDescSpec.$anonfun$new$14(SymmetricDifferenceOpDescSpec.scala:151)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec::HashJoinOpDesc.generateStandaloneCode should pair NaN, signed-zero and null keys as the engine does in a LEFT_OUTER join
Stack Traces | 1.49s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/hashjoin-nan-zero-7173550960554514640.py", line 164, in <module>
    _left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
                                ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    return pd.arrays.FloatingArray(
        np.array([0.0 if v is None else v for v in vals]),
        np.array([v is None for v in vals]),
    )

in1df = pd.DataFrame({
    "k": col([np.nan, 0.0, 1.0, None]),
    "lv": ["L-nan", "L+0", "L1", "L-null"],
})
in2df = pd.DataFrame({
    "k2": col([np.nan, -0.0, 1.0, None]),
    "rv": ["R-nan", "R-0", "R1", "R-null"],
})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_left_cols = set(in1df.columns)
_right_cols = list(in2df.columns)
_payload = [_c for _c in _right_cols if _c != "k2"]
_payload_set = set(_payload)
_rename = {}
for _col in _payload:
    _new = _col
    _others = _payload_set - {_col}
    while _new in _left_cols or _new in _others:
        _new = _new + "#@1"
    _rename[_col] = _new
_probe_key = "k2"
_taken = _left_cols | set(_rename.values())
while _probe_key in _taken:
    _probe_key = _probe_key + "#@1"
if _probe_key != "k2":
    _rename["k2"] = _probe_key
_join_key = "_texera_join_key"
while _join_key in _taken or _join_key == _probe_key:
    _join_key = _join_key + "_"
_left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
out1df = in1df.assign(**{_join_key: _left_codes}).merge(
    in2df.rename(columns=_rename).assign(**{_join_key: _right_codes}),
    how="left",
    on=_join_key,
).drop(columns=[_join_key])
if _probe_key != "k":
    out1df = out1df.drop(columns=[_probe_key])
out1df = out1df.reset_index(drop=True)
def show(v):
    if isinstance(v, float):
        return "nan" if v != v else repr(float(v))
    return str(v)
rows = []
for values, missing in zip(out1df.itertuples(index=False), out1df.isna().to_numpy()):
    rows.append(" ".join("None" if m else show(v) for v, m in zip(values, missing)))
print("\n".join(sorted(rows)))
1 was not equal to 0
	at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
	at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$21(HashJoinOpDescSpec.scala:221)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$10(HashJoinOpDescSpec.scala:220)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.intersect.IntersectOpDescSpec::IntersectOpDesc.generateStandaloneCode should keep the null row a null row matches and not the NaN row
Stack Traces | 1.53s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/intersect-null-nan-11075088920652304560.py", line 135, in <module>
    _texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
                                          ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals, mask):
    return pd.arrays.FloatingArray(np.array(vals), np.array(mask))

in1df = pd.DataFrame({"v": col([0.0, np.nan], [True, False])})
in2df = pd.DataFrame({"v": col([0.0], [True])})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
_texera_set_kept = ~_texera_set_left.duplicated() & _texera_set_left.isin(_texera_set_right)
out1df = in1df[_texera_set_kept.to_numpy()].reset_index(drop=True)
print(len(out1df), [("NULL" if x is pd.NA else repr(float(x))) for x in out1df["v"]])
1 was not equal to 0
      at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
      at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
      at org.apache.texera.amber.operator.intersect.IntersectOpDescSpec.$anonfun$new$9(IntersectOpDescSpec.scala:124)
      at org.scalatest.Assertions.withClue(Assertions.scala:1065)
      at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
      at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
      at org.apache.texera.amber.operator.intersect.IntersectOpDescSpec.$anonfun$new$7(IntersectOpDescSpec.scala:123)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
      at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
      at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
      at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
      at sbt.TestRunner.runTest$1(TestFramework.scala:153)
      at sbt.TestRunner.run(TestFramework.scala:168)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFunction.apply(TestFramework.scala:348)
      at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
      at sbt.std.Transform$$anon$4.work(Transform.scala:69)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
      at sbt.Execute.work(Execute.scala:292)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.symmetricDifference.SymmetricDifferenceOpDescSpec::SymmetricDifferenceOpDesc.generateStandaloneCode should keep the NaN row that a null row does not cancel
Stack Traces | 1.54s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/symmetric-difference-null-nan-4017035808484972343.py", line 135, in <module>
    _texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
                                          ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals, mask):
    return pd.arrays.FloatingArray(np.array(vals), np.array(mask))

in1df = pd.DataFrame({"v": col([0.0, np.nan], [True, False])})
in2df = pd.DataFrame({"v": col([0.0], [True])})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
_texera_set_left_kept = ~_texera_set_left.duplicated() & ~_texera_set_left.isin(_texera_set_right)
_texera_set_right_kept = ~_texera_set_right.duplicated() & ~_texera_set_right.isin(_texera_set_left)
out1df = pd.concat(
    [in1df[_texera_set_left_kept.to_numpy()], in2df[_texera_set_right_kept.to_numpy()]],
    ignore_index=True,
)
print(len(out1df), [("NULL" if x is pd.NA else repr(float(x))) for x in out1df["v"]])
1 was not equal to 0
	at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
	at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
	at org.apache.texera.amber.operator.symmetricDifference.SymmetricDifferenceOpDescSpec.$anonfun$new$13(SymmetricDifferenceOpDescSpec.scala:131)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
	at org.apache.texera.amber.operator.symmetricDifference.SymmetricDifferenceOpDescSpec.$anonfun$new$11(SymmetricDifferenceOpDescSpec.scala:130)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec::IntervalJoinOpDesc.generateStandaloneCode should add a time interval on the machine's wall clock as the engine does
Stack Traces | 1.56s run time 
org.scalatest.exceptions.TestFailedException: day from 2024-06-01T12:00, python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/intervaljoin-zone-728923621420955474.py", line 137, in <module>
    _iv_hi = _texera_wall_clock_instants(_iv_r + pd.DateOffset(days=1))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_texera_wall_clock_instants' is not defined

script:
import numpy as np
import pandas as pd

in1df = pd.DataFrame({"l": np.array(["2024-06-02T12:00", "2024-06-02T12:01"], dtype="datetime64[us]")})
in2df = pd.DataFrame({"r": np.array(["2024-06-01T12:00"], dtype="datetime64[us]")})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_pairs = in1df.merge(in2df, how="cross", suffixes=("", "#@1"))
_iv_l = _pairs["l"]
_iv_r = _pairs["r" if "r" not in in1df.columns else "r" + "#@1"]
if pd.api.types.is_datetime64_any_dtype(_iv_r):
    _iv_hi = _texera_wall_clock_instants(_iv_r + pd.DateOffset(days=1))
    _iv_l = _texera_wall_clock_instants(_iv_l)
    _iv_r = _texera_wall_clock_instants(_iv_r)
else:
    _iv_hi = _iv_r + 1
_iv_match = (_iv_l >= _iv_r) & (_iv_l <= _iv_hi)
out1df = _pairs[_iv_match].reset_index(drop=True)
for row in out1df.itertuples(index=False):
    print(" ".join(pd.Timestamp(v).isoformat() for v in row))
1 was not equal to 0
      at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
      at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
      at org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec.$anonfun$new$28(IntervalJoinOpDescSpec.scala:319)
      at org.scalatest.Assertions.withClue(Assertions.scala:1065)
      at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
      at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
      at org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec.$anonfun$new$23(IntervalJoinOpDescSpec.scala:318)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec.$anonfun$new$21(IntervalJoinOpDescSpec.scala:280)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
      at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
      at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
      at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
      at sbt.TestRunner.runTest$1(TestFramework.scala:153)
      at sbt.TestRunner.run(TestFramework.scala:168)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFunction.apply(TestFramework.scala:348)
      at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
      at sbt.std.Transform$$anon$4.work(Transform.scala:69)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
      at sbt.Execute.work(Execute.scala:292)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.difference.DifferenceOpDescSpec::DifferenceOpDesc.generateStandaloneCode should keep the NaN row that a null row does not cancel
Stack Traces | 1.71s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/difference-null-nan-13912539386287533212.py", line 135, in <module>
    _texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
                                          ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals, mask):
    return pd.arrays.FloatingArray(np.array(vals), np.array(mask))

in1df = pd.DataFrame({"v": col([0.0, np.nan], [True, False])})
in2df = pd.DataFrame({"v": col([0.0], [True])})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
_texera_set_kept = ~_texera_set_left.duplicated() & ~_texera_set_left.isin(_texera_set_right)
out1df = in1df[_texera_set_kept.to_numpy()].reset_index(drop=True)
print(len(out1df), [("NULL" if x is pd.NA else repr(float(x))) for x in out1df["v"]])
1 was not equal to 0
      at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
      at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
      at org.apache.texera.amber.operator.difference.DifferenceOpDescSpec.$anonfun$new$15(DifferenceOpDescSpec.scala:185)
      at org.scalatest.Assertions.withClue(Assertions.scala:1065)
      at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
      at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
      at org.apache.texera.amber.operator.difference.DifferenceOpDescSpec.$anonfun$new$13(DifferenceOpDescSpec.scala:184)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
      at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
      at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
      at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
      at sbt.TestRunner.runTest$1(TestFramework.scala:153)
      at sbt.TestRunner.run(TestFramework.scala:168)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFunction.apply(TestFramework.scala:348)
      at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
      at sbt.std.Transform$$anon$4.work(Transform.scala:69)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
      at sbt.Execute.work(Execute.scala:292)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec::IntervalJoinOpDesc.generateStandaloneCode should compare timestamps by the millisecond as the engine does
Stack Traces | 1.75s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/intervaljoin-millis-15826208414653733851.py", line 137, in <module>
    _iv_hi = _texera_wall_clock_instants(_iv_r + pd.DateOffset(seconds=3))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_texera_wall_clock_instants' is not defined

script:
import numpy as np
import pandas as pd

in1df = pd.DataFrame({"l": np.array(["2024-01-01T12:00:00.0005", "2024-01-01T12:00:02", "2024-01-01T12:00:03.00095"], dtype="datetime64[us]")})
in2df = pd.DataFrame({"r": np.array(["2024-01-01T12:00:00.0009"], dtype="datetime64[us]")})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_pairs = in1df.merge(in2df, how="cross", suffixes=("", "#@1"))
_iv_l = _pairs["l"]
_iv_r = _pairs["r" if "r" not in in1df.columns else "r" + "#@1"]
if pd.api.types.is_datetime64_any_dtype(_iv_r):
    _iv_hi = _texera_wall_clock_instants(_iv_r + pd.DateOffset(seconds=3))
    _iv_l = _texera_wall_clock_instants(_iv_l)
    _iv_r = _texera_wall_clock_instants(_iv_r)
else:
    _iv_hi = _iv_r + 3
_iv_match = (_iv_l >= _iv_r) & (_iv_l <= _iv_hi)
out1df = _pairs[_iv_match].reset_index(drop=True)
for row in out1df.itertuples(index=False):
    print(" ".join(pd.Timestamp(v).isoformat() for v in row))
1 was not equal to 0
      at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
      at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
      at org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec.$anonfun$new$20(IntervalJoinOpDescSpec.scala:237)
      at org.scalatest.Assertions.withClue(Assertions.scala:1065)
      at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
      at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
      at org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec.check$1(IntervalJoinOpDescSpec.scala:236)
      at org.apache.texera.amber.operator.intervalJoin.IntervalJoinOpDescSpec.$anonfun$new$13(IntervalJoinOpDescSpec.scala:243)
      at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
      at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
      at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
      at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
      at sbt.TestRunner.runTest$1(TestFramework.scala:153)
      at sbt.TestRunner.run(TestFramework.scala:168)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFunction.apply(TestFramework.scala:348)
      at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
      at sbt.std.Transform$$anon$4.work(Transform.scala:69)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
      at sbt.Execute.work(Execute.scala:292)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec::HashJoinOpDesc.generateStandaloneCode should pair NaN, signed-zero and null keys as the engine does in a RIGHT_OUTER join
Stack Traces | 2.02s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/hashjoin-nan-zero-568542041378277658.py", line 164, in <module>
    _left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
                                ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    return pd.arrays.FloatingArray(
        np.array([0.0 if v is None else v for v in vals]),
        np.array([v is None for v in vals]),
    )

in1df = pd.DataFrame({
    "k": col([np.nan, 0.0, 1.0, None]),
    "lv": ["L-nan", "L+0", "L1", "L-null"],
})
in2df = pd.DataFrame({
    "k2": col([np.nan, -0.0, 1.0, None]),
    "rv": ["R-nan", "R-0", "R1", "R-null"],
})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_left_cols = set(in1df.columns)
_right_cols = list(in2df.columns)
_payload = [_c for _c in _right_cols if _c != "k2"]
_payload_set = set(_payload)
_rename = {}
for _col in _payload:
    _new = _col
    _others = _payload_set - {_col}
    while _new in _left_cols or _new in _others:
        _new = _new + "#@1"
    _rename[_col] = _new
_probe_key = "k2"
_taken = _left_cols | set(_rename.values())
while _probe_key in _taken:
    _probe_key = _probe_key + "#@1"
if _probe_key != "k2":
    _rename["k2"] = _probe_key
_join_key = "_texera_join_key"
while _join_key in _taken or _join_key == _probe_key:
    _join_key = _join_key + "_"
_left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
out1df = in1df.assign(**{_join_key: _left_codes}).merge(
    in2df.rename(columns=_rename).assign(**{_join_key: _right_codes}),
    how="right",
    on=_join_key,
).drop(columns=[_join_key])
if _probe_key != "k":
    out1df = out1df.drop(columns=[_probe_key])
out1df = out1df.reset_index(drop=True)
def show(v):
    if isinstance(v, float):
        return "nan" if v != v else repr(float(v))
    return str(v)
rows = []
for values, missing in zip(out1df.itertuples(index=False), out1df.isna().to_numpy()):
    rows.append(" ".join("None" if m else show(v) for v, m in zip(values, missing)))
print("\n".join(sorted(rows)))
1 was not equal to 0
	at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
	at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$21(HashJoinOpDescSpec.scala:221)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$10(HashJoinOpDescSpec.scala:220)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec::HashJoinOpDesc.generateStandaloneCode should match a missing key but not a NaN one
Stack Traces | 2.43s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/hashjoin-null-nan-13017413586341233089.py", line 155, in <module>
    _left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k"]])
                                ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals, mask):
    return pd.arrays.FloatingArray(np.array(vals), np.array(mask))

in1df = pd.DataFrame({"k": col([0.0, np.nan], [True, False]), "lv": [1, 2]})
in2df = pd.DataFrame({"k": col([0.0], [True]), "rv": [9]})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_left_cols = set(in1df.columns)
_right_cols = list(in2df.columns)
_payload = [_c for _c in _right_cols if _c != "k"]
_payload_set = set(_payload)
_rename = {}
for _col in _payload:
    _new = _col
    _others = _payload_set - {_col}
    while _new in _left_cols or _new in _others:
        _new = _new + "#@1"
    _rename[_col] = _new
_probe_key = "k"
_taken = _left_cols | set(_rename.values())
while _probe_key in _taken:
    _probe_key = _probe_key + "#@1"
if _probe_key != "k":
    _rename["k"] = _probe_key
_join_key = "_texera_join_key"
while _join_key in _taken or _join_key == _probe_key:
    _join_key = _join_key + "_"
_left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k"]])
out1df = in1df.assign(**{_join_key: _left_codes}).merge(
    in2df.rename(columns=_rename).assign(**{_join_key: _right_codes}),
    how="left",
    on=_join_key,
).drop(columns=[_join_key])
if _probe_key != "k":
    out1df = out1df.drop(columns=[_probe_key])
out1df = out1df.reset_index(drop=True)
print(len(out1df), int(out1df["rv"].notna().sum()))
1 was not equal to 0
	at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
	at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$8(HashJoinOpDescSpec.scala:127)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$6(HashJoinOpDescSpec.scala:126)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec::HashJoinOpDesc.generateStandaloneCode should pair NaN, signed-zero and null keys as the engine does in a INNER join
Stack Traces | 2.64s run time 
org.scalatest.exceptions.TestFailedException: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/hashjoin-nan-zero-10062042568342011640.py", line 164, in <module>
    _left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
                                ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    return pd.arrays.FloatingArray(
        np.array([0.0 if v is None else v for v in vals]),
        np.array([v is None for v in vals]),
    )

in1df = pd.DataFrame({
    "k": col([np.nan, 0.0, 1.0, None]),
    "lv": ["L-nan", "L+0", "L1", "L-null"],
})
in2df = pd.DataFrame({
    "k2": col([np.nan, -0.0, 1.0, None]),
    "rv": ["R-nan", "R-0", "R1", "R-null"],
})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_left_cols = set(in1df.columns)
_right_cols = list(in2df.columns)
_payload = [_c for _c in _right_cols if _c != "k2"]
_payload_set = set(_payload)
_rename = {}
for _col in _payload:
    _new = _col
    _others = _payload_set - {_col}
    while _new in _left_cols or _new in _others:
        _new = _new + "#@1"
    _rename[_col] = _new
_probe_key = "k2"
_taken = _left_cols | set(_rename.values())
while _probe_key in _taken:
    _probe_key = _probe_key + "#@1"
if _probe_key != "k2":
    _rename["k2"] = _probe_key
_join_key = "_texera_join_key"
while _join_key in _taken or _join_key == _probe_key:
    _join_key = _join_key + "_"
_left_codes, _right_codes = _texera_key_codes(in1df[["k"]], in2df[["k2"]])
out1df = in1df.assign(**{_join_key: _left_codes}).merge(
    in2df.rename(columns=_rename).assign(**{_join_key: _right_codes}),
    how="inner",
    on=_join_key,
).drop(columns=[_join_key])
if _probe_key != "k":
    out1df = out1df.drop(columns=[_probe_key])
out1df = out1df.reset_index(drop=True)
def show(v):
    if isinstance(v, float):
        return "nan" if v != v else repr(float(v))
    return str(v)
rows = []
for values, missing in zip(out1df.itertuples(index=False), out1df.isna().to_numpy()):
    rows.append(" ".join("None" if m else show(v) for v, m in zip(values, missing)))
print("\n".join(sorted(rows)))
1 was not equal to 0
	at org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:392)
	at org.scalatest.matchers.should.Matchers$AnyShouldWrapper.shouldBe(Matchers.scala:7539)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$21(HashJoinOpDescSpec.scala:221)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.flatspec.AnyFlatSpec.withClue(AnyFlatSpec.scala:1686)
	at org.apache.texera.amber.operator.hashJoin.HashJoinOpDescSpec.$anonfun$new$10(HashJoinOpDescSpec.scala:220)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
	at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
	at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
	at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
	at scala.collection.immutable.List.foreach(List.scala:323)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
	at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
	at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
	at org.scalatest.Suite.run(Suite.scala:1114)
	at org.scalatest.Suite.run$(Suite.scala:1096)
	at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
	at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
	at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
	at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
	at sbt.TestRunner.runTest$1(TestFramework.scala:153)
	at sbt.TestRunner.run(TestFramework.scala:168)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
	at sbt.TestFunction.apply(TestFramework.scala:348)
	at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
	at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
	at sbt.Execute.work(Execute.scala:292)
	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.apache.texera.amber.operator.intersect.IntersectOpDescSpec::IntersectOpDesc.generateStandaloneCode should match rows by the engine's equality for NaN and signed zero
Stack Traces | 3.05s run time 
java.lang.AssertionError: python said:
Traceback (most recent call last):
  File ".../hn37q6pj7sj_9dmqqc2113800000gn/T/intersect-nan-zero-11450140896313473668.py", line 136, in <module>
    _texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
                                          ^^^^^^^^^^^^^^^^^
NameError: name '_texera_key_codes' is not defined

script:
import pandas as pd, numpy as np

def col(vals):
    mask = np.array([v is None for v in vals])
    return pd.arrays.FloatingArray(np.array([0.0 if v is None else v for v in vals]), mask)

in1df = pd.DataFrame({"k": ["a"] * 6, "v": col([1.0, np.nan, 0.0, None, 2.0, np.nan])})
in2df = pd.DataFrame({"k": ["a"] * 5, "v": col([1.0, np.nan, -0.0, None, 3.0])})
# AttributeTypeUtils, transcribed so a cast answers as the engine does.
def _texera_cast_boolean(x):
    # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
    if isinstance(x, str):
        text = x.strip()
        lowered = text.lower()
        if lowered == "true":
            return True
        if lowered == "false":
            return False
        return int(text) == 1
    return x != 0


def _texera_parse_number(text):
    # java.text.NumberFormat for Locale.US, which a cast reaches through
    # `parseField(force = true)`. Lenient where Python's int is not: it stops
    # at the first character that cannot continue a number ("12abc" is 12),
    # drops "," without checking group sizes ("1,23" is 123), and reads "."
    # as a decimal point. It refuses a leading "+" and text with no digits.
    #
    # Also returns whether it came back as a Double, which decides between
    # the two narrowings below.
    import re

    match = re.match(r"\s*(-?)([0-9,]*)(?:\.([0-9]*))?", text)
    sign = -1 if match.group(1) == "-" else 1
    digits = (match.group(2) or "").replace(",", "")
    fraction = match.group(3) or ""
    if not digits and not fraction:
        raise ValueError("Unparseable number: " + repr(text))
    if fraction:
        return sign * float((digits or "0") + "." + fraction), True
    value = sign * int(digits)
    # A whole number past long's range comes back as a Double.
    if -(2 ** 63) <= value <= 2 ** 63 - 1:
        return value, False
    return float(value), True


def _texera_long_value(value, is_double):
    # Number.longValue(): a Double truncates toward zero and saturates at
    # long's bounds, where a Long is already itself.
    if is_double:
        return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
    return value


def _texera_cast_integral(x):
    # The LONG target. Scala's toLong takes no decimal point of its own, so
    # a fraction only ever arrives already parsed.
    if isinstance(x, str):
        return _texera_long_value(*_texera_parse_number(x))
    if isinstance(x, bool):
        return 1 if x else 0
    if isinstance(x, float):
        return _texera_long_value(x, True)
    return int(x)


def _texera_cast_int32(x, wrap):
    # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
    # 2147483648 comes back as -2147483648, while Double.toInt saturates.
    #
    # Text decides by what the parse returned; everything else is told by the
    # SOURCE column's declared type, because `Series.apply` hands the cells of
    # a nullable integer column over as floats.
    if isinstance(x, str):
        value, is_double = _texera_parse_number(x)
        wrap = not is_double
    elif isinstance(x, bool):
        value = 1 if x else 0
    else:
        value = int(x)
    if wrap:
        return ((value + 2147483648) % 4294967296) - 2147483648
    # int() first: Double.intValue truncates toward zero before it clamps.
    return max(-2147483648, min(2147483647, int(value)))


def _texera_epoch_millis_to_timestamp(s):
    # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
    # to nanoseconds, and renders in the JVM's default zone, so leaving the
    # result in UTC would put it a whole offset away. Text needs neither: a
    # parsed wall clock is already the wall clock.
    # tzlocal() and not the current offset: the zone carries its daylight
    # rules, and each instant needs the offset in force when it happened.
    from dateutil.tz import tzlocal

    return (
        pd.to_datetime(s, unit="ms", errors="coerce", utc=True)
        .dt.tz_convert(tzlocal())
        .dt.tz_localize(None)
    )


def _texera_cast_double(x):
    if isinstance(x, str):
        return float(x.strip())
    return float(x)


def _texera_cast_string(s):
    # `toString` on the field, so the COLUMN's type decides the text and
    # not the shape of the value: a double keeps its point, whether or not
    # it lands on a whole number, and an integer never grows one. A column
    # of no single type is read value by value.
    if pd.api.types.is_bool_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else ("true" if x else "false"))
    if pd.api.types.is_integer_dtype(s):
        return s.map(lambda x: None if pd.isna(x) else str(int(x)))
    if pd.api.types.is_datetime64_any_dtype(s):
        # java.sql.Timestamp.toString: trailing zeros dropped from the
        # fraction, but never all of them. Python's own str() writes no
        # fraction at all on a whole second and six digits otherwise.
        def _ts(x):
            if pd.isna(x):
                return None
            text = x.strftime("%Y-%m-%d %H:%M:%S.%f").rstrip("0")
            return text + "0" if text.endswith(".") else text

        return s.map(_ts)
    return s.map(
        lambda x: None
        if pd.isna(x)
        else ("true" if x else "false") if pd.api.types.is_bool(x) else str(x)
    )
_texera_set_left, _texera_set_right = _texera_key_codes(in1df, in2df, signed=True)
_texera_set_kept = ~_texera_set_left.duplicated() & _texera_set_left.isin(_texera_set_right)
out1df = in1df[_texera_set_kept.to_numpy()].reset_index(drop=True)
def render(v):
    return "None" if v is pd.NA else v if isinstance(v, str) else repr(float(v))
for row in sorted(" ".join(render(v) for v in r) for r in out1df.itertuples(index=False)):
    print(row)

      at org.apache.texera.amber.testsupport.setop.SetRows$.runExport(SetRows.scala:73)
      at org.apache.texera.amber.operator.intersect.IntersectOpDescSpec.$anonfun$new$10(IntersectOpDescSpec.scala:143)
      at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
      at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
      at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
      at org.scalatest.Transformer.apply(Transformer.scala:22)
      at org.scalatest.Transformer.apply(Transformer.scala:20)
      at org.scalatest.flatspec.AnyFlatSpecLike$$anon$5.apply(AnyFlatSpecLike.scala:1832)
      at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
      at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
      at org.scalatest.flatspec.AnyFlatSpec.withFixture(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.invokeWithFixture$1(AnyFlatSpecLike.scala:1830)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTest$1(AnyFlatSpecLike.scala:1842)
      at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest(AnyFlatSpecLike.scala:1842)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTest$(AnyFlatSpecLike.scala:1824)
      at org.scalatest.flatspec.AnyFlatSpec.runTest(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$runTests$1(AnyFlatSpecLike.scala:1900)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:390)
      at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:427)
      at scala.collection.immutable.List.foreach(List.scala:323)
      at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
      at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
      at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests(AnyFlatSpecLike.scala:1900)
      at org.scalatest.flatspec.AnyFlatSpecLike.runTests$(AnyFlatSpecLike.scala:1899)
      at org.scalatest.flatspec.AnyFlatSpec.runTests(AnyFlatSpec.scala:1686)
      at org.scalatest.Suite.run(Suite.scala:1114)
      at org.scalatest.Suite.run$(Suite.scala:1096)
      at org.scalatest.flatspec.AnyFlatSpec.org$scalatest$flatspec$AnyFlatSpecLike$$super$run(AnyFlatSpec.scala:1686)
      at org.scalatest.flatspec.AnyFlatSpecLike.$anonfun$run$1(AnyFlatSpecLike.scala:1945)
      at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
      at org.scalatest.flatspec.AnyFlatSpecLike.run(AnyFlatSpecLike.scala:1945)
      at org.scalatest.flatspec.AnyFlatSpecLike.run$(AnyFlatSpecLike.scala:1943)
      at org.scalatest.flatspec.AnyFlatSpec.run(AnyFlatSpec.scala:1686)
      at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
      at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
      at sbt.TestRunner.runTest$1(TestFramework.scala:153)
      at sbt.TestRunner.run(TestFramework.scala:168)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
      at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
      at sbt.TestFunction.apply(TestFramework.scala:348)
      at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
      at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
      at sbt.std.Transform$$anon$4.work(Transform.scala:69)
      at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
      at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
      at sbt.Execute.work(Execute.scala:292)
      at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
      at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
      at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:840)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@kz930 kz930 mentioned this pull request Sep 11, 2026
Open
13 of 31 tasks
@kz930
kz930 marked this pull request as draft September 11, 2026 05:55
renovate-bot pushed a commit to renovate-bot/apache-_-texera that referenced this pull request Sep 17, 2026
@kz930 @claude @aglinxinyuan
feat(workflow-operator): export the operators that work on text (apac…
ddb5305 
…he#8340)

### What changes were proposed in this PR?

Six operators that read or rewrite a text column implement
`StandaloneCodeGenerator`: Type Casting, Keyword Search, Substring
Search, Unnest String, Regex and Dictionary Matcher.

The sixteen that rearrange rows rather than read inside one are in
apache#8506.

Each is written against what its executor does rather than against what
its name suggests, and the verification that came with the export runs
both and compares them. Some of what that turned up is visible in the
code: a cast reads the column's declared type rather than the value in
front of it, because a hole widens an integer column to float and "6"
would otherwise come back "6.0"; the 32-bit narrowing wraps for an
integral source and saturates for a double, as `Long.toInt` and
`Double.toInt` do; Dictionary Matcher's substring branch asks whether
the cell sits inside a dictionary entry, which is the direction the
executor tests and not the one the name suggests.

`StandaloneHelpers` holds what these six share: a transcription of
`AttributeTypeUtils`, emitted once near the top of a script rather than
inlined per operator. A cast goes through `parseField(force = true)`,
whose numeric branch is `java.text.NumberFormat`, so the engine reads
"12abc" as 12 and "false" as false where Python's own conversions answer
differently.

Four of these operators change what they do, not only how they export.
Regex, Substring Search and Unnest String answered a null cell by
raising, which a generated script has no way to reproduce and no reason
to: nothing in a column matches nothing, and unnests to no rows. They
answer it that way now, and their specs say so. Keyword Search states
which keywords its query parser will take, which a user had no way to
know from the field alone.

### Any related issues, documentation, discussions?

Part of apache#8325, 10 of 27; that issue lists the set in order. apache#8327 has
since merged, so the trait is in place and this branch compiles on its
own. The rows these operators add to the verification runner follow with
the harness rather than as whole new files here.

Closes apache#8074, closes apache#7548.

Closes apache#8414, the task this change is the whole of.

### How was this PR tested?

Each operator asserts the block it emits in its own spec. Type Casting's
spec goes further: it runs the block it emits through a real Python and
compares every answer against `AttributeTypeUtils`, so the two findings
the review raised are held there, and that part does run on this diff's
CI. Once the verification lands each operator is also run through the
engine and through its generated script, on every configuration its
schema offers, and the two answers compared; this branch is cut from
main and does not carry that machinery, so those runs are not on this
diff's CI.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xinyuan Lin <xinyual3@uci.edu>
carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Java random copy does not match Java integer overflow in next_int. Java rejects some values after the addition overflows, but this Python code always accepts them. This can select different rows for Split and Reservoir Sampling. Please copy the overflow check and add a seed that reaches the rejection case.

carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are more behavior gaps. If always defaults to the true branch because the exported script never receives the state. Sampling only matches one worker, while native runs with one random generator per worker. Aggregate CONCAT also writes Python boolean names instead of lowercase Java names. Reservoir size zero fails natively but silently returns empty in standalone. These cases need matching behavior or a clear unsupported error.

carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More operators still differ. Sleep removes the delay. Timestamp sum fails and timestamp average returns the wrong type. Outer joins can turn integers into floats and fail chained name collisions. Dummy loses extra port routing. Sort order differs for NaN and some Unicode strings. Please add executed parity tests for these cases.

carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integer sum can return different data in the two paths. With 2147483647 and 1, native integer aggregation wraps to negative 2147483648, while pandas returns positive 2147483648 as a long. Please match the native result and add a boundary test.

carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set operations merge a null double and a real NaN. For example, if the left input has null and NaN and the right input has null, native Difference keeps the NaN row, but the exported code returns no rows. Please preserve this distinction and add a set operation test with both values.

carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same null and NaN problem also changes Join and Aggregate results. Left join keys null and NaN against a right null produce one native match but two exported matches. Grouping null and NaN with values 1 and 2 produces two native groups but one exported group totaling 3. Please cover keyed operators too.

kz930 and others added 2 commits September 18, 2026 11:31
@kz930
Merge remote-tracking branch 'upstream/main' into feat/standalone-rel…
ab530db 
…ational-and-sampling
@kz930 @claude
test(operator): run both routes of the exported If
ef38771 
The engine picks If's route from a State message on the Condition port, and the
verification harness writes rows per port with no State channel, so a fixture
can only ever reach the default route, True. The exported block reads the same
decision from a global instead, which the spec can set, so both routes are run
there: the rows leave by one output and the other comes back empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 kz930 mentioned this pull request Sep 18, 2026
Draft
kz930 and others added 7 commits September 18, 2026 12:26
@kz930 @claude
docs(operator): say where a sorted string column parts from the engine
64600d7 
The engine compares with String.compareTo, which reads UTF-16 code units, while
pandas compares by code point. They agree across the basic plane and differ only
where a character above U+FFFF meets one in U+E000..U+FFFF, because the
surrogate pair encoding the first holds units below that range. U+1F600 against
U+FFFD is the shape of it: pandas reads the first as greater, the engine as
smaller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 @claude
fix(operator): reject the draws Java rejects, and a reservoir of zero
76b4690 
The transcribed nextInt carried its rejection sum as a Python integer, where
Java lets an int overflow: a draw within bound of 2**31 wraps negative there and
is taken again. Carrying it keeps the value Java throws away and shifts every
later draw, which seed 12345 at bound 1000 reaches on draw 2126624.

It also never refused a bound of zero, which Reservoir Sampling hands it when
the reservoir holds nothing. The engine ends the run there; the script answered
with an empty table instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 @claude
fix(operator): answer SUM, AVERAGE and CONCAT the way the engine answ…
8522786 
…ers them

Four places where pandas is asked the same question and answers differently.

SUM keeps the column's declared type, so an INTEGER sum is Java int arithmetic
and wraps at 2**31 where pandas widens, and a TIMESTAMP sum is a timestamp built
from the added epoch milliseconds where pandas refuses to reduce datetime64 at
all. AVERAGE is declared DOUBLE whatever it reads, so a timestamp average is the
mean of those milliseconds and not a timestamp. CONCAT folds with Java's
toString, which spells a boolean in lower case.

The first three need the declared type rather than the dtype pandas inferred: a
holed INTEGER column arrives as a float, and an INTEGER and a LONG arrive alike.
The schema-free form keeps pandas' own answers rather than guessing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 @claude
fix(operator): rename a twice-colliding join column, and keep an inte…
f4ba91d 
…ger integral

The engine appends "#@1" to a colliding right column until the name is free, so
one that meets an already-suffixed column lands on "#@1#@1". The export asked
pandas for `suffixes` instead, which appends once and then refuses the duplicate
it just made, ending the run. Compute the rename the way HashJoinProbeOpExec
computes it, the way the Cartesian product export already did, and hand the
merge columns that no longer collide.

An outer join also leaves holes, and pandas pays for a hole in an integer column
by reading the whole column as float. The engine writes a null and the column
stays INTEGER, so the declared integer columns are read back as the nullable
integer that says the same thing. Only the declared type can say which those
are, and without a schema the widening stands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 @claude
docs(operator): cut today's comments back to what the code cannot say
a240f01 
The reasoning behind each fix was written twice, once in the commit that made it
and again beside the line. Keep the half a reader needs to follow the code and
drop the worked examples and the derivations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 @claude
fix(operator): sort a null and a NaN where the engine sorts them
42404db 
The engine puts a null last whichever way a key points, and compares a NaN above
every number, so a NaN goes last ascending and first descending. The export
asked for na_position="last", which is right for the null and wrong for the NaN
on the way down.

Sort each key in three tiers instead: whether the value is missing, always
ascending; whether it is a NaN, following the key; then the value. A column read
into a numpy dtype holds a null and a NaN in the same slot, and there both land
in the missing tier, which is where they were before. Telling them apart needs
the nullable dtypes the Arrow source now reads into.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930 @claude
test(operator): keep a null and a NaN apart in the set, join and keye…
0ac4039 
…d paths

The engine reads a missing value and a stored NaN as two different values, so a
difference keeps the NaN row a null row does not cancel, a join matches the
missing key and not the NaN one, and a group keyed on each stays two groups.
Each is run on the nullable dtype an Arrow file is read into, which is the only
place the two are distinct.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kz930

kz930 commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Fixed in this PR: next_int's overflow rejection, a reservoir bound of zero, the INTEGER sum wrap, CONCAT's lower-case boolean, the join's rename and its integer column, and the NaN sort. If now has a spec that runs both routes. The null and NaN cases were the Arrow read rather than the operators, so #8341 reads the nullable dtypes instead, and Difference, the hash join and Aggregate each carry your case as a run test.

Four I would push back on. Dummy never overrides getPhysicalOp, so it cannot execute at all and has no routing to lose. Sleep's delay never reaches the output. Sampling is seeded with the worker count by the engine, and one process can only state one seed. A sorted string parts only where a character above U+FFFF meets one in U+E000..U+FFFF, since the engine reads UTF-16 code units and pandas reads code points.

Your two newest both reproduce here. Fixing them now.

carloea2
carloea2 suggested changes Sep 18, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new timestamp aggregation helpers assume the integer values are nanoseconds. With a microsecond column containing 2024-01-01, Sum returns 1970-01-20 17:21:07.200 instead. The nanosecond sum also overflows before conversion: six copies of 2024-01-01 return a date in 1709 instead of 2293. I reproduced both with pandas 2.2.3. Please convert each value to epoch milliseconds before aggregating and test both resolutions.

@kz930 @claude
fix(workflow-operator): match set-operator rows by the engine's equality
77c64a3 
The engine's HashSet[Tuple] compares rows with Scala's ==, under which NaN
equals nothing, and hashes them by each double's bits, which tell 0.0 from
-0.0. pandas' duplicate checks count both pairs as equal, so Intersect,
Difference and Symmetric Difference now select rows by a key that follows
the engine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
carloea2 suggested changes Oct 13, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The set equality fixes pass the original NaN and signed-zero reproductions. The empty full-outer-join finding remains: with output columns [k, x, x#@1], the native result is [2, right, null] instead of [2, null, right]. The differently named key case still puts the right key into a left payload column. No duplicate inline comment added; these are the existing join finding.

The expanded focused run gives 83 passed and 2 failed, with formatting passing. Please address that remaining case before merging.

sbt -batch 'WorkflowOperator/testOnly *HashJoin*Spec *JoinUtilsSpec *DifferenceOpDescSpec *IntersectOpDescSpec *SymmetricDifferenceOpDescSpec *Union*Spec *SetExportReviewSpec -- -oDF'

@kz930 @claude
fix(workflow-operator): name a HashJoin probe's right columns from th…
ce2b893 
…e declared build schema

A probe worker read the left column names off its first build row, so one
that received no build rows put an unmatched right row's columns into the
left ones. The runtime now records each input port's declared schema on the
executor before the port finishes, and the probe reads the left names from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930

kz930 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 Both findings are fixed, the set operations in 77c64a3 and the empty build side in ce2b893. The second adds OperatorExecutor.inputSchemas, an engine change described in the PR body. Could you take another look?

@kz930 @claude
Merge upstream/main into feat/standalone-relational-and-sampling
72380df 
OperatorExecutor: upstream now keeps executor state beside the executors
rather than in trait fields, which a Java class implementing the trait would
have to declare. inputSchemas follows suit and becomes a final def over a
weak-keyed map in the companion object.

HashJoinOpDesc: keep both imports, NoLoopVariable from upstream and
AttributeType for the export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kz930 added a commit to Nicoleee1108/texera_workflow_to_py that referenced this pull request Oct 13, 2026
@kz930 @claude
test(verify): give the executor each finished port's declared schema
fd2de23 
As the worker's EndChannelHandler now does (apache#8506). Without it the native
side of the HashJoin right-outer variant put the right columns into the left
ones. Moves to apache#8357 once apache#8506 is merged upstream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kz930 added a commit to kz930/texera that referenced this pull request Oct 13, 2026
@kz930 @claude
test(verify): give the executor each finished port's declared schema
a739b78 
As the worker's EndChannelHandler does since apache#8506: a HashJoin probe that
received no build rows names its right columns from the build port's
declared schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 13, 2026
Draft
Draft
@kz930 @claude
fix(workflow-operator): let a NaN key match nothing in Hash Join, as …
fa07950 
…the engine does

The probe looks each key up in a HashMap under Scala's ==, where a NaN
finds nothing, while merge paired NaN with NaN. Merge on codes that follow
the engine's equality instead, each NaN taking one of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930

kz930 commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 One more case from your keyed-operator finding: two NaN keys were still treated as equal, while the engine never matches them. Hash Join now lets a NaN key match nothing (fa07950), and Aggregate in #8709 gives each NaN key its own group (14d7de2).

carloea2
carloea2 approved these changes Oct 14, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The empty-build join fix passes, including colliding payload names and differently named keys. All 101 focused operator tests and 13 EndChannelHandler tests pass, with formatting clean. The original NaN and signed-zero set reproductions also pass. The remaining CI failure is LoopIntegrationSpec, covered by #8938. Code looks good. Leaving the draft status unchanged.

@kz930 @claude
fix(operator): compare interval join timestamps by the millisecond as…
c59c02f 
… the engine does

IntervalJoinOpExec compares a timestamp by getTime, so two moments in the
same millisecond are one point to it, and a left key just past the upper
bound but inside its millisecond still joins. The export compared the full
resolution and dropped those rows. Floor the two keys and the bound to the
millisecond for the comparison, adding the bound before it is cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930
kz930 marked this pull request as ready for review October 15, 2026 01:12
@kz930

kz930 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 One more commit since your approval: Interval Join now compares timestamps by the millisecond, as the engine does (c59c02f).

@kz930 @claude
fix(operator): add Interval Join's constant in the key's declared typ…
f4004e8 
…e, as the engine does

An INTEGER bound is an Int sum: the engine cuts the constant to 32 bits and
the sum wraps past Int.MaxValue, where pandas held the column as int64 and
did neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930

kz930 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 One more: Interval Join now adds an INTEGER constant as an Int sum, cut to 32 bits and wrapping as the engine does (f4004e8).

@kz930 @claude
fix(operator): add Interval Join's time interval on the machine's wal…
2036f15 
…l clock, as the engine does

The executor adds the constant to toLocalDateTime and reads the sum back
with Timestamp.valueOf, so a day across a daylight-saving change is 23 or
25 hours and a month counts from the local date. The export added it to the
UTC wall clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930

kz930 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 One more: Interval Join now adds a time interval on the machine's wall clock and reads it back in the machine's zone, as the engine does, so a day across a daylight-saving change is 23 or 25 hours (2036f15). It calls the zone helpers from #8597, so its Python tests pass once #8597 lands.

@kz930
kz930 marked this pull request as draft October 15, 2026 22:57
carloea2
carloea2 suggested changes Oct 23, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 76 operator tests, 14 EndChannelHandlerSpec tests and formatting pass with #8597 and #9003 supplied. A cross-operator timestamp regression still fails: two controls pass and both clock-change cases fail.

sbt -batch 'WorkflowOperator/testOnly *ReviewIntervalWallClockSpec -- -oDF'

This compares the native IntervalJoinOpExec with emitted code fed through the shared source timestamp reader, not a full connected workflow.

Comment thread ...erator/src/main/scala/org/apache/texera/amber/operator/intervalJoin/IntervalJoinOpDesc.scala Outdated
kz930 and others added 3 commits October 23, 2026 14:43
@kz930 @claude
fix(operator): add Interval Join's interval to the wall clock the scr…
b9620be 
…ipt is handed

A source already hands the script a timestamp as the machine's wall clock, so
labelling it UTC and converting it again moved the bound by the local offset.
The interval is now added to that wall clock as it stands, and the keys and
the bound are read back into instants as Timestamp.valueOf reads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930 @claude
feat(workflow-operator): keep the set and join helpers' notes out of …
16d13e1 
…the exported script

_texera_set_row_keys, _texera_set_select and _texera_join_codes explained
themselves in Python comments, which every exported script with a set
operator or a Hash Join printed. The notes now sit in Scala, one line
per function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930 @claude
feat(workflow-operator): keep the sampling helper's notes out of the …
d183c31 
…exported script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
carloea2 approved these changes Oct 24, 2026
View reviewed changes

@carloea2 carloea2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The daylight-saving finding is fixed. The 64 focused join and set tests pass, including the original four source-reader wall-clock probes. All 14 EndChannelHandlerSpec tests and scalafmtCheckAll pass too. Tested with the shared source and timestamp prerequisites supplied. CI remains red and must clear before merge.

kz930 and others added 4 commits October 25, 2026 03:14
@kz930 @claude
feat(workflow-operator): read Interval Join's wall clocks with the sh…
6856d50 
…ared helper

The shared helper also reads a wall clock in the hours around 1900-01-01
at the offset the engine does, where the old one was a whole offset off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930 @claude
feat(workflow-operator): match Hash Join's keys with the shared key c…
770bbdc 
…odes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930 @claude
feat(workflow-operator): keep Interval Join's exported Python free of…
24d5d9b 
… comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kz930 @claude
feat(workflow-operator): select the set operators' rows with the shar…
9d7468d 
…ed key codes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@carloea2 carloea2 carloea2 approved these changes
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

@kz930 kz930

Labels

common engine feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Export the join and set operators as Python

3 participants

@kz930 @codecov-commenter @carloea2