Managing R support in main and deployments when the R runtime is installed separately

[zyratlo]

R UDFs run through texera-rudf, a package installed separately because it depends on rpy2 (GPLv2), which is not compatible with the Apache license (see #4155). The rest of R support still lives in main, and each piece is handled on its own. I ran into this while adding R to the notebook migration tool (#8009) and wanted to raise it.

R in main today

• RUDF and RUDFSource operators, listed under an R group in the operator menu
• r.path in udf.conf, passed to Python workers as R_HOME
• R mode in the code editor
• An R UDF guide in the agent service prompt
• Example R UDFs in pytexera
• Soon, an R option in the migration tool (feat(notebook-migration): add r prompts and build rudf operators for r sources #8911, feat(notebook-migration): add r notebooks to the import modal #8912)

How each part offers or restricts R

Every part makes its own decision, and none of them knows whether texera-rudf is installed:

• Operator menu: always shows the R group.
• Agent service: adds the R UDF guide to an agent's system prompt unless that agent's allowed operator list is set and leaves out RUDF. An empty list means all operators, so by default every agent is taught to write R UDFs.
• Code editor: always opens R operators in R mode.
• Migration tool: will have its own flag, rNotebookMigrationEnabled, default off.
• Execution: the Python worker imports texera-rudf only when an R operator runs. If it is missing, the run fails with an install hint.

So a user can be offered R by the menu or the agent, build a whole workflow, and only learn at execution time that R is not available. None of our images install texera-rudf, so out of the box no deployment can run R.

Questions

1. Should one R setting control all of this, hiding the operators, the agent guide and the migration option together?
2. Should it be a config flag, or detected from whether texera-rudf is installed?
3. Should there be an image option that installs R and texera-rudf?
4. How can we streamline managing these types of situations for active deployments?

[parshimers]

I think this is a very interesting question. I have been thinking about it a bit since this discussion was posted.

It is my understanding that today, pretty much everything you would need to run R workflows is possible to do without further editing the source. All that is needed is an R runtime in the CU, and the aforementioned texera-rudf package along with it's dependency rpy2. It might even be the case that having the CU image itself be built on top of another image that contains R could be permissable, but it's not something I am sure about because I don't fully understand how the licensing works out in this manner. It would be hard to find an image with both R and rpy2 as well. It's also not particularly scalable. The more languages and things we want to support, if this trick works, the harder it gets to use other images without altering them.

Based on that I think for option 2, having a simple flag that users can turn on if they have enabled the requisite dependencies themselves (understanding the license implications), seems quite doable, as a first step. Autodetection could come later.

One could also imagine taking this whole feature (R support) and trying to use it as an example of what parts of the system need to be turned into an interface for extension support. That way users can add their own operators separately in a controlled fashion.

[chenlica]

@kunwp1 Can you chime in based on your experiences of using R UDF?

[kunwp1]

In #4155, the plan was for main to turn R on only when texera-rudf is installed and for our Dockerfiles to have an R option, but #4385 took R out of our images because R itself is GPLv2, so today only the check at execution time is left. I agree with @parshimers that the right first step is one flag, off by default, that hides the R operators, the agent's R guide and the migration option together. Detection can come later from the computing unit, since that is where texera-rudf is installed. For images, I'd keep R out of apache/texera and add a Dockerfile to Texera/texera-rudf that builds on top of our computing-unit image, so a deployment that wants R builds that image and turns the flag on.