Debug on hosts without ptrace (Raspberry Pi workers, Rosetta), and pe…

Dockerfile

@@ -12,6 +12,8 @@ RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
 COPY ./install_prerequisite.sh /opt/openrepl/
 COPY ./bin/gdb /usr/bin/
+# the debug helpers (ptrace probe, openrepl-gdb, the rappel wrapper), installed by the script
+COPY ./scripts/ptrace-probe.c ./scripts/openrepl-gdb ./scripts/openrepl-rappel /opt/openrepl/scripts/
 RUN ./install_prerequisite.sh --cleanup-tools --run-tests
 
 FROM builder as build-image

README.md

@@ -185,6 +185,7 @@ Then do a quick manual check at `localhost:8080`:
 ### Notes
 
 - **No per-REPL sandboxing locally.** Colima's VM uses cgroup v2, so the log shows `Unable to create Container` and REPLs run without their own namespaces or memory limits. Test sandboxing on a cgroup v1 host.
+- **Debug runs under QEMU locally.** Rosetta cannot trace programs, so plain gdb fails with `Couldn't get registers`. Debug detects it and runs your program under QEMU, which gdb connects to (see [LLD 04](docs/lld/04-ide-run-and-files.md)). The program starts paused: set breakpoints, then type `c`. The assembly REPL (rappel) needs real ptrace and does not start here.
 - **Don't commit `bin/gotty`.** The dev container rebuilds this tracked file. Restore it before committing with `git checkout -- bin/gotty`, and don't commit `node_modules` or `dist` folders.
 - **Genie and Practice question generation need an OpenAI key.** Set `OPENREPL_OPENAI_API_KEY` (see [Settings and secrets](#settings-and-secrets)), then restart the server.
 - **Stopping and restarting the VM:** `colima stop openrepl` and `colima start openrepl`. `colima delete openrepl` removes the VM and its images.

docs/distributed-mode.md

@@ -189,6 +189,10 @@ worker_token     = "<the token from step 1>"  // or GOTTY_WORKER_TOKEN
 
 A worker reconnects by itself, with a delay that grows from 1 to 30 seconds, when the gateway restarts or the network drops.
 
+### Workers that cannot trace programs (Raspberry Pi)
+
+An arm64 worker runs the amd64 image under `qemu-user`, which has no `ptrace`. Debug (gdb) still works there: the image's `openrepl-gdb` notices the emulator and lets it serve gdb (`QEMU_GDB`), so the program starts paused and you type `c` to start it. Programs that use the 32-bit `int 0x80` system call do not run under qemu-user at all. The assembly REPL (`rappel`) needs real `ptrace` and prints a short message instead of starting; the dashboard shows "no ptrace" for such a worker, and you can switch `rappel` off for it there (next section). See LLD 04 §2.
+
 ### The worker's own port
 
 A worker opens no port by default. Give it `--port` (or `--address`) and it also serves that address, like a standalone server, so people on the same network can use the worker directly at `http://<worker-address>:<port>/`:
@@ -282,6 +286,15 @@ Things to do:
 
 Limits: files over 50 MB are not synchronized; sockets, pipes and device files are skipped; names that start with `.wsync-` are reserved; ownership and set-user-id bits are not copied; Linux only.
 
+## Languages per worker
+
+`--worker-languages` says what a worker has when it starts. In the dashboard (*Workers*, then a node) the Languages card lets you change that without restarting anything, for each language: **Default** (what the worker declared), **Off** (refuse new terminals of it on this node), or **On** (take it although the worker did not declare it, for a language you installed on the worker afterwards). It works the same for the gateway's own node. Visitors on that node no longer see a refused language in the language picker, and open terminals keep running. The card also says whether the node's host can trace programs, which the assembly REPL needs.
+
+| Route | What it does |
+|---|---|
+| `GET /admin/workers/<id>/languages` | The languages with what the node declares and what you decided. `<id>` is a worker id, or `local`. |
+| `POST /admin/workers/<id>/languages` | Body `{"language": "rappel", "rule": "default"}` (or `"off"`, `"on"`). Saved with the site settings. |
+
 ## Operating a fleet
 
 The easiest way is the dashboard at `/admin` (admin sign-in): *Workers* lists the nodes with their load, state and how many sessions each has been given, opens a node's details (address, version, languages, sync state, clock difference) and drains, undrains or reconnects it. *Sessions* lists who is on which node, and can end a session or move it to another node. *Add a worker* shows the command line for a new one, with the gateway's tunnel host key fingerprint. The same things are available as routes on the gateway. They need the same admin sign-in; to change something from `curl`, pass the browser's session cookie and the header `X-Requested-With: openrepl-admin` (the dashboard sends it; without it a change is refused):
@@ -353,7 +366,8 @@ A few things to know when reading it:
 | Browser: `execution node unavailable` (503) | The worker that holds this session is offline. It works again when the worker reconnects. |
 | Browser: `workspace node unavailable` (503) | A signed-in user's worker is offline or draining. |
 | Browser: `no execution node available` (503) | No online node has a weight: no worker is connected and the gateway runs with `--local-weight 0`. |
-| Browser: `this language is not available on your execution node` | The session's worker was started with `--worker-languages` and lacks that REPL. |
+| Browser: `this language is not available on your execution node` | The session's worker was started with `--worker-languages` and lacks that REPL. Install it and switch the language **On** for that node (Languages card), or restart the worker with the list. |
+| Browser: `This language is switched off on your execution node` | An admin switched it off for that node in the Languages card. Set it back to Default. |
 | Terminal: `exceeding max number of connections` | That node's memory budget is used up. |
 
 All messages go to `/gottyTraces/gotty.log` on the machine concerned.

docs/lld/04-ide-run-and-files.md

@@ -60,12 +60,27 @@ A Run request executes `/bin/bash -c "$SCRIPT" "$ARG0" "$FLAGS"` (the `Prefix` i
 | `$0` | Path of the saved editor file (`IdeFileName`), or the base64 editor content when no file is selected. Scripts decode it with `echo $0 \| base64 --decode`. |
 | `$1` | The *Compiler/Repl Args* text box, passed as one string. |
 | `$IdeLang` | The UI option value (`c`, `cpp`, `go`, `python`, …), so one backend (for example `cling`) can pick `gcc` or `g++`. |
-| `$CompilerOption` | `debug` when **debug** was pressed. Scripts then add `-g` and launch `gdb`, `rust-gdb` and similar. |
+| `$CompilerOption` | `debug` when **debug** was pressed. Scripts then add `-g` and launch `openrepl-gdb` (below) instead of a bare `gdb`. |
 | `$IdeFileName`, `$HOME` | The same file path, and the workspace directory. |
 | client `EnvFlags` | Extra variables from the *Env Vars/Paths* box. |
 
 Scripts usually write `test.<ext>` into `$HOME` when there is no file, compile it, run it, and `printf "\n"` at the end.
 
+### Debug on a host without ptrace (`openrepl-gdb`)
+
+gdb traces the program with `ptrace`, and not every host has it: a Raspberry Pi worker runs the amd64 image under `qemu-user` (`ptrace` answers "Function not implemented"), and x86 code under Rosetta can start a traced child but not read its registers (`Couldn't get registers: Input/output error`). Scripts therefore start `openrepl-gdb [--gdb rust-gdb] PROGRAM [gdb arguments]` (`scripts/openrepl-gdb`, installed to `/usr/local/bin`), which asks `openrepl-ptrace-probe` (`scripts/ptrace-probe.c`, built once by the install script) and takes one route:
+
+| Route | When | What it does |
+|---|---|---|
+| `ptrace` | The probe passes (it forks a child, traces it and reads its registers). | `exec gdb` as before. If only the address randomization cannot be switched off (the probe's exit 2), gdb gets `set disable-randomization off`, so it does not warn. |
+| `emulator` | The probe fails and `QEMU_VERSION=1 /bin/true` prints a `qemu-` version: the container itself runs under qemu-user (the Pi workers). | Starts the program with `QEMU_GDB=<port>` in its environment, which makes the emulator that runs it wait for gdb on that port. No second emulator, no extra slowdown. |
+| `qemu` | The probe fails, there is no emulator around, and the host is x86-64 (Rosetta, a sandbox). | Runs the program as `QEMU_GDB=<port> QEMU_LD_PREFIX=/ qemu-x86_64 PROGRAM` (`qemu-user` from apt). |
+| none | Neither. | Prints "Debugging is not available on this server" and "Run still works", exit 1. |
+
+On the last three routes gdb is started with `set sysroot /` and `target remote 127.0.0.1:<port>`, on a free port chosen per session. It only connects: the user sets breakpoints and continues. qemu describes its registers to gdb in XML, so these routes need a gdb built with XML support (`gdb --configuration` shows `--with-expat`). The helper takes the first `gdb` on the `PATH` that has it (for `rust-gdb`, it sets `RUST_GDB`). The gdb 8.1.1 the repo bundles as `bin/gdb` has none and, when it is first on the `PATH`, fails with `Remote 'g' packet reply is too long` and lets the program run away; Ubuntu's gdb (installed by the install script) is used instead, and if there is none the helper says so before it starts the program. The program starts paused at its first instruction, because a remote target cannot "run": `run` and `r` are redefined to `continue`, and the helper prints a two line banner saying so. The program keeps the terminal for its input, and Ctrl-C reaches gdb only (the shell ignores it for the background job), which interrupts the program. When gdb ends, or the terminal closes (a small watcher notices that `/dev/tty` is gone, because a gdb that waits for a running remote program ignores the hangup), the helper stops the program.
+
+`OPENREPL_GDB_ROUTE=ptrace|emulator|qemu` forces a route; the install script's test uses `qemu`. Known limits of the QEMU routes: x86-64 only, slower than native, and a program that uses the 32-bit `int 0x80` system call gate does not work under qemu-user (the assembly sample uses `syscall`, which does). The assembly REPL (rappel) cannot work without `ptrace` at all: `/usr/local/bin/rappel` is a wrapper (`scripts/openrepl-rappel`) that runs the probe first and, if `ptrace` is missing, tells the user to use the editor's Run or Debug; otherwise it starts the real rappel (`OPENREPL_RAPPEL_BIN`, default `/opt/gotty/rappel/bin/rappel`).
+
 ### Language routing on the client
 
 `handleTerminalOptions` (`gotty.ts`):

docs/lld/08-build-and-deploy.md

@@ -53,11 +53,13 @@ For a local build and test loop on macOS (Colima with Rosetta, plus a dev contai
 
 It has three stages, all based on `ubuntu:22.04`:
 
-1. **`builder`:** copies `install_prerequisite.sh` and `bin/gdb`, then runs `./install_prerequisite.sh --cleanup-tools --run-tests`. This installs every REPL runtime:
+1. **`builder`:** copies `install_prerequisite.sh`, `bin/gdb` and the debug helpers (`scripts/ptrace-probe.c`, `scripts/openrepl-gdb`, `scripts/openrepl-rappel`), then runs `./install_prerequisite.sh --cleanup-tools --run-tests`. This installs every REPL runtime:
    - from apt: gcc/g++, default-jdk, python2.7/3, ipython/ipython3, golang, yaegi, npm/nvm/node, ruby, perl, tcl, sqlite3, jq, rustc/cargo, rust-gdb, nasm, rlwrap, net-tools, libcap2-bin;
    - prebuilt: cling (`repls/cling-Ubuntu-22.04-x86_64-*.tar.bz2`) and evcxr;
    - from source: gointerpreter, jq-repl, perli, rappel;
-   - from npm: `typescript@4.9.5` and `ts-node`.
+   - from npm: `typescript@4.9.5` and `ts-node`;
+   - for Debug on hosts without `ptrace` (LLD 04 §2): `qemu-user` from apt, `openrepl-ptrace-probe` (built from `scripts/ptrace-probe.c` once, here, so Debug never compiles anything), `openrepl-gdb`, and `/usr/local/bin/rappel` as a wrapper in front of the real rappel. The bundled `bin/gdb` is copied but Ubuntu's gdb (pulled in by `rust-gdb`) is the one that ends up in use.
+   - `--run-tests` runs each command through `bash -c`, so the `tclsh` check is a real pipe. It also debugs a small program through the QEMU route (this one must pass), through `ptrace` and the rappel REPL (reported, not fatal, and skipped when the build host has no `ptrace`).
 2. **`build-image`:** adds make, git and npm (pinned to 8.5.1), copies the repo, and runs `make all`.
 3. **Final:** `builder` plus `/usr/local/bin/gotty` and `/opt/scripts/run_app.sh`. It creates `/gottyTraces` and `/opt/gotty`, sets `ENV TERM=xterm GODEBUG=cgocheck=1 GOPATH=/opt/gotty/`, `EXPOSE 80`, `ENTRYPOINT run_app.sh`, `CMD ["-p","80"]`.
 

docs/lld/09-adding-a-repl.md

@@ -49,7 +49,7 @@ printf "\n";
 </Demo>
 ```
 
-See LLD 04 §2 for the Compiler script contract (`$0`, `$1`, `$IdeLang`, `$CompilerOption`). Escape `<`, `>` and `&` in XML. `<Prefix>` can wrap the REPL (for example with `rlwrap`) in interactive mode only.
+See LLD 04 §2 for the Compiler script contract (`$0`, `$1`, `$IdeLang`, `$CompilerOption`). A script that starts gdb for `debug` should call `openrepl-gdb PROGRAM` (or `openrepl-gdb --gdb rust-gdb PROGRAM`) rather than `gdb`, so Debug also works on hosts without ptrace. Escape `<`, `>` and `&` in XML. `<Prefix>` can wrap the REPL (for example with `rlwrap`) in interactive mode only.
 
 ## 4. Expose it in the UI (`src/resources/index.html`)
 

docs/lld/11-distributed-execution.md

@@ -189,6 +189,7 @@ The gateway applies the site rules (maintenance mode, languages that are switche
 - *Use:* `applyWorkerConfig` (`server/worker_config.go`) makes the config the worker's `SiteSettings`, in memory only; the worker reads no `settings.json` or database for them. `/settings.js` of the worker then shows the gateway's banner, and `wrapWorkerControls` closes a terminal of a visitor of the worker's own port with `site notice: ...` for maintenance and for a switched-off language. A request the gateway forwarded (trusted) is not checked again: the gateway knows who is an admin and the worker does not. On the worker's own port nobody is exempt, admins included; an admin who needs a terminal during maintenance uses the gateway.
 - *Before the first config* (or if the gateway never sends one, an older gateway) the worker has the default settings: no rules.
 - *Dashboard:* the worker's drawer shows "Site rules: up to date" or the revision it follows (`WorkerInfo.ConfigRev`, `ConfigCurrent`).
+- *Per worker:* the list of languages that are off is the worker's own (`ServerConfig.WorkerConfig`, `server.gatewayWorkerConfigFor`): the site's, the ones an admin switched off on that worker, and the ones it did not declare (`--worker-languages`) unless an admin switched them on (LLD 13, "Languages per node"). The revision therefore differs from worker to worker; `ConfigRevisionFor` gives the one a worker should follow.
 - *Not sent on purpose:* the MongoDB or Firestore settings (a worker keeps files, LLD 05), keys, the admin list, and limits such as capacity, which are the worker's own flags. Together with the secret (section 6.1a), this is everything a worker takes from the gateway.
 
 A failure to listen ends `Run` before the worker connects to the gateway. A listener that stops later cancels the worker's context, like a standalone server. On shutdown `runWorker` closes both servers, then waits for the live WebSockets.
@@ -232,7 +233,7 @@ Randomized weighted selection, the method of `ServerPool.Select` in `sish-lb/lb.
 
 Differences from sish-lb: the candidate list is rebuilt on every pick because eligibility changes with state and load; there are no global flags; the pool has its own `*rand.Rand` under a mutex instead of calling `rand.Seed` on the shared generator; the key is the session, not a hostname.
 
-Placement happens once per session, on the first page load, before the language is known. So the language is not part of selection: a worker that lacks the requested REPL (`--worker-languages`) answers that terminal with 503 "this language is not available on your execution node". With no `--worker-languages` a worker is taken to have every REPL.
+Placement happens once per session, on the first page load, before the language is known. So the language is not part of selection: a worker that lacks the requested REPL (`--worker-languages`) has that terminal refused with "this language is not available on your execution node" (a notice the page shows; a plain 503 for a request that is not a WebSocket). With no `--worker-languages` a worker is taken to have every REPL. An admin can override this per node and language (LLD 13, "Languages per node"): Off refuses a language the node can run, On takes one it did not declare. The picker hides what the visitor's node refuses (`Router.NodeOf`).
 
 ## 9. Tunnel (`src/tunnel`)