[BUG][cudf_polars] distributed cluster creates a new `Context` for every query run

[nirandaperera]

Describe the bug

https://github.com/rapidsai/cudf/blob/main/python/cudf_polars/cudf_polars/experimental/rapidsmpf/dask.py#L173-L183

Dask cluster mode creates a new context for every query run. This also creates a new Options object which is different to the Options object used to create BufferResource.

Steps/Code to reproduce bug

Expected behavior
Since we dont use tasks anymore, we should create a rapidsmpf Context and stash it in the Dask worker context similar to BufferResource and reuse it for every query

Environment overview (please complete the following information)

• Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
• Method of cuDF install: [conda, Docker, or from source]
  • If method of install is [Docker], provide docker pull & docker run commands used

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Additional context
Add any other context about the problem here.

[TomAugspurger]

I suspect #21812 will fix this.

[nirandaperera]

@TomAugspurger I'm reviewing #21812. Looks like we'll still have to take care of Single cluster separately.
https://github.com/rapidsai/cudf/pull/21812/changes#diff-79cdc50dd7b58133c3c0482429e0018ffd8f7ef9e671110d3363b5643aa21496R180-R194

[TomAugspurger]

Ah, I didn't realize the issue here was related to Single clusters.

[nirandaperera]

@TomAugspurger Sorry I meant to tag @rjzamora 🤦. #21812 will indeed fix the distributed context reuse issue. But we need a separate PR to fix single cluster.

[rjzamora]

If I understand correctly, the long-term plan that @madsbk proposed is for "single" to go away (or redirect to spmd) and for single-rank "spmd" to become the new default?

[nirandaperera]

If spmd can work without rrun for single execution, then yes. But IINM, last time I ran spmd cluster, it required the processes to be spawned by rrun.
Let's merge #21812 first. Then we'll have a better idea.

[nirandaperera]

But IINM, last time I ran spmd cluster, it required the processes to be spawned by rrun.

@rjzamora I was mistaken and looks like we dont need rrun for 1-worker spmd cluster with the latest changes. So, yes, we can default to 1worker spmd cluster in single now, and remove custom code for single cluster.