[ConorWorthington]

Relax dependency bounds to pyspark>=3.4.0,<5.0.0 and numpy>=1.22.0 (no upper bound), fixing installs on Databricks runtimes that ship numpy 2.

Spark 4 enables ANSI mode by default, under which division/modulo by zero and out of bounds element_at raise instead of returning null. Guard these explicitly so results match Spark 3 and the Keras layers:

• Divide, CosineSimilarity, PairwiseCosineSimilarity: zero divisor -> 0.0
• Modulo: zero divisor -> null
• SubStringDelimAtIndex: out of bounds index -> defaultValue
• ListMedian: empty list -> nanFillValue

Tests:

• Add test_ansi_mode.py covering the above with ANSI on and off.
• Fix test helper scalar conversion for numpy 2.5.
• Patch the concrete DataFrame class in the checkpoint test, since pyspark.sql.DataFrame is abstract in Spark 4.
• Check JIT-incompatible layers via experimental_get_compiler_ir, avoiding a TF 2.21 abort in the XLA compilation cache on repeated failed compiles of string ops.

CI: add pyspark 4.0.1 lane, set up Java 17, keep a numpy 1.26 lane.

Description

Provide a short description of the PR changes.

The below checklists come from the docs page on adding new transformers here

Keras Layer Checklist

Verify that:

• The new Keras layer extends BaseLayer
• The _call method has been implemented in the new layer.
• The compatible_dtypes property is defined in the new layer.
• The new layer is decorated with @tf.keras.utils.register_keras_serializable(package=kamae.__name__).
• The new layer takes a name, input_dtype, and output_dtype as arguments to the constructor and that this is passed to the super constructor.
• The Keras layer is serializable. I have implemented the get_config method.
• There are unit tests of the new layer.
• There is a specific test of layer serialisation added here.
• The new layer is imported in the init.py file in the layers directory.

Spark Transformer/Estimator Checklist

Verify that:

• The new Spark Transformer extends BaseTransformer.
• If the new transform needs a fit method, a Spark Estimator has been implemented that extends BaseEstimator.
• The instructions in the above docs page have been followed for the __init__ and setParams methods.
• The transformer uses one of the input/output mixin classes from base.py.
• If the new transformer requires more parameters that would need to be serialised to the Spark ML pipeline, there is a implemented parameter class by extending the Params class here.
• The compatible_dtypes property has been implemented to specify the input/output data types that my transformer/estimator supports.
• A Keras subclassed layer is returned in the transformer's get_tf_layer method.
• There are unit tests of the new transform. In particular, there are parity tests between the Spark and Keras implementations.
• The new transformer/estimator is imported in the init.py file in the transformers/estimators directory.

Finally, please verify that:

• There is a new entry (alphabetical order) in the README table describing the new layer/transformer