Allow Evaluation Using Multiple Python Versions

[HassanAbouelela]

Description

Install multiple python versions within the container, and add an option in the API to select between them.

Rationale

Currently, the bot supports evaluation of both 3.10 and 3.11 code, something I think should be kept and expanded in the future (to include major/important python versions). This is achieved by using the latest 3.10 image published here. This is fine for now, but it makes it harder to provide feature parity and avoid issues when users switch between modes, and it makes it difficult to patch and fix security issues.

Implementation

The way I see us doing this is by building all the versions of python we want to support directly into the image, then aiming the jail at the right executable. This would simplify implementation by having one server, with one codebase, while allowing us to evaluate user code on any version.

Specifics

Some more specific implementation details that can be up for debate:

1. The docker image will simply install whatever versions of python you pass to it in the build arg, to well defined locations
2. We'll build two images in CI, one with all versions we want supported, and one lightweight for dev
3. We can expose an endpoint to get all supported versions
4. We should investigate the best way to install all the versions to the docker image. I imagine the best solution is to just start from an image with all the required build-tools, install all the python versions, and use that as the base for the image. I found this docker image which has all the versions from 2.3-3.9 installed, and it comes out to a whooping... 600 MBs! I don't think this specific project would work for us since it hasn't been updated since 3.9, but it does make the idea seem feasiable.
5. Are the exposed packages going to be an issue, or can we simply install all the same versions to all python interpreters.

[onerandomusername]

Are the exposed packages going to be an issue, or can we simply install all the same versions to all python interpreters.

I think it would be a nice idea to be able to have different packages installed. Any c-extension could compile differently based on the targeted python version, so it would be nice to have different versions.

[ChrisLovering]

The snekbox Dockerfile now installs both 3.12.0 and 3.13-dev to /lang/python/<version>.
There is also /lang/python/default which is a symlink to /lang/python/3.12/ which is what snekbox currently uses for eval.

Now we need to update snekbox's API to support multiple versions.
The following is my opinion on how to do that.

A new endpoint GET /versions is added, which returns all versions snekbox supports. The response body would include the name, a human friendly name, and whether it is a default. this would allow clients to easily give their users a list of versions that they can use.

POST /eval will be updated to have an optional version key, which is the version to eval the code under. This value of this key would be the name field from GET /versions. If it is not given, then the version marked as default in GET /versions is used instead.

[MarkKoz]

@ChrisLovering This was essentially the approach taken in #175, which has some discussion.

I think it is better to accept an arbitrary path to a binary to execute. I'm hesitant to make design decisions that pigeon-hole snekbox into just a Python evaluator or just a code evaluator in general. Technically there is not much difference between that and sandboxing arbitrary binaries.

On the other hand, that is the common use case, so we can compromise here. I don't see a strong need for exposing versions via an API, but if we were to do it, I think it should return the paths to the binaries, so they can be passed to the eval endpoint.

[onerandomusername]

Noticing there's not further progress on this, but it seems that snekbox supports this, just undocumented. What do we need to do to move forward on this?

[jb3]

Noticing there's not further progress on this, but it seems that snekbox supports this, just undocumented. What do we need to do to move forward on this?

There are ongoing internal discussions about this that probably need condensing into a well-formed public issue at some point.

Step 1 is complete, we do build additional interpreters in the snekbox image. We do this in a slightly different order to the initially proposed order (we build interpreters first and then build snekbox on top of that). I believe the reason we do this is probably for caching (building new interpreters is expensive, and if we can avoid changing those layers it's better).

This is partially documented, though not in the best places. We do document how to add new interpreters here: https://github.com/python-discord/snekbox/blob/main/.github/CONTRIBUTING.md#adding-and-updating-python-interpreters

Tasks still needed are:

• Document /snekbin/ and the executable_path body parameter to the /eval endpoint
• Decide on how we want to separate snekbox from Python executable building (basically where the line gets drawn between snekbox as an app and pydis as a snekbox consumer)
• We need to decide how/if we are exposing an executable list to end users. This is also something we've discussed at length internally and the consensus seems to be returning a list of the /snekbin/ files whilst supporting symlinks from there to prevent accidental restriction of functionality.

The idea with that last point is that the bot no longer needs updating when we change Python versions in snekbox but that's potentially something we can close this issue without solving.

The initial comment about exposed packages has proved to be mostly a non-issue as we just use version specifiers and update which packages are available in which interpreter environment as and when they become compatible.

The TL;DR is that we do support this issue 90% but I'm not sure the way we currently do this is the best, it certainly makes snekbox a lot less attractive to external users now that we bundle a bunch of other interpreter versions in our distributed images.