:spark carries its own SCALAR_SIGS in spark/src/main/scala/io/substrait/spark/expression/FunctionMappings.scala, entirely separate from the isthmus list, and nothing checks the two against each other. So a function isthmus can produce may be unconvertible on the Spark side, and the only way to find out is to hit it.
Current examples: spark has upper and lower but not reverse, initcap, char_length, replace, left, right or strpos. #1251 widens the gap by two.
#1012's scope note already records "unifying the parallel :spark Scala mapping implementation" as out of scope / a separate follow-up, naming an AbstractFunctionInvocation<T, I> seam in core as the intended direction. This issue is that follow-up, plus the smaller thing that can be done independently of it: a test that reports which Substrait function names one binding maps and the other does not, so the divergence is visible rather than discovered.
One caveat for anyone adding entries rather than building the seam: Spark's Catalyst Reverse covers reverse(array) as well as strings, so a plain s[Reverse]("reverse") would mis-map array reverse onto the string function. It needs the same special-casing the ss[...] entries get.
Found while reviewing #1251.
:sparkcarries its ownSCALAR_SIGSinspark/src/main/scala/io/substrait/spark/expression/FunctionMappings.scala, entirely separate from the isthmus list, and nothing checks the two against each other. So a function isthmus can produce may be unconvertible on the Spark side, and the only way to find out is to hit it.Current examples: spark has
upperandlowerbut notreverse,initcap,char_length,replace,left,rightorstrpos. #1251 widens the gap by two.#1012's scope note already records "unifying the parallel
:sparkScala mapping implementation" as out of scope / a separate follow-up, naming anAbstractFunctionInvocation<T, I>seam in core as the intended direction. This issue is that follow-up, plus the smaller thing that can be done independently of it: a test that reports which Substrait function names one binding maps and the other does not, so the divergence is visible rather than discovered.One caveat for anyone adding entries rather than building the seam: Spark's Catalyst
Reversecoversreverse(array)as well as strings, so a plains[Reverse]("reverse")would mis-map array reverse onto the string function. It needs the same special-casing thess[...]entries get.Found while reviewing #1251.