diff --git a/README.md b/README.md index fdfda24e..c9076575 100755 --- a/README.md +++ b/README.md @@ -195,6 +195,58 @@ Then do a quick manual check at `localhost:8080`: One public server, the **gateway**, serves the site and hands REPL sessions to **workers**. A worker needs no public port: it connects out to the gateway. Same binary, same URLs for the browser. +The picture below is a small fleet: one gateway and two workers. + +```mermaid +flowchart TB + B["Browser
openrepl.example.com"] + + subgraph GW["Gateway (the only public server)"] + direction TB + SITE["Site
pages, sign-in, blog, admin,
Genie proxy"] + RT["Router
picks a node by weight,
keeps a visitor on it"] + TS["Tunnel server
/api/tunnel"] + LR["Own REPLs
(weight 10, or 0 to only route)"] + GD[("Databases and settings
files, MongoDB or Firestore")] + GH[("Copy of every home
with --workspace-sync")] + end + + subgraph W1["worker-01 (no public port)"] + direction TB + T1["Tunnel client"] + R1["REPL sandboxes
namespaces + cgroup"] + H1[("/tmp/home
users' files")] + end + + subgraph W2["worker-02 (no public port)"] + direction TB + T2["Tunnel client"] + R2["REPL sandboxes
namespaces + cgroup"] + H2[("/tmp/home
users' files")] + end + + EXT["Firebase, OpenAI, OpenRouter"] + + B -- "HTTPS and WSS" --> SITE + SITE --> RT + SITE --> GD + SITE -- "API keys stay here" --> EXT + RT --> LR + RT --> TS + T1 -- "connects out: wss or ssh,
shared token" --> TS + T2 -- "connects out: wss or ssh,
shared token" --> TS + T1 --> R1 --> H1 + T2 --> R2 --> H2 + H1 <-. "workspace sync" .-> GH + H2 <-. "workspace sync" .-> GH +``` + +- **The browser talks to the gateway only.** Pages, sign-in, Genie and the admin dashboard are served there. A terminal or a file request is passed through the tunnel to the node that holds the visitor's session. +- **Workers connect out** to `/api/tunnel` with the shared token, so they can sit behind NAT or a home router. Each tells the gateway its name, weight and languages; the gateway sends back the site's rules (maintenance, switched-off languages). +- **The router** places a new visitor on a node at random, in proportion to the weights, and keeps them there. With two workers of weight 10 and a gateway of weight 10, each takes about a third. +- **Data:** accounts, settings and the other databases live on the gateway (see [Data and storage](#data-and-storage)). Users' files live on the node that runs their session, and on the gateway too with `--workspace-sync`. +- **If a worker stops**, its visitors see a countdown; with `--workspace-sync` their session moves to another node with its files. + **1. Create a shared token** ```bash @@ -425,10 +477,11 @@ OpenREPL is one Go binary (`bin/gotty`, a fork of GoTTY). The same binary does t 2. **Starts a REPL for each browser terminal.** Each REPL runs as a child process on a pseudo-terminal (PTY), inside a lightweight container made of Linux namespaces and a cgroup v1 memory limit. 3. **Streams the terminal over a WebSocket.** It relays PTY output to the browser (xterm.js) and keystrokes back to the REPL. -Two external services sit around the core: +These services sit around the core: - **Firebase:** Authentication (sign-in) and the Realtime Database (live sharing of a REPL, and Genie chat history). -- **OpenAI:** reached only through a server-side proxy, for the *Genie* assistant and *Practice* question generation. +- **OpenAI and OpenRouter:** reached only through a server-side proxy, for the *Genie* assistant and *Practice* question generation. +- **A database, optional:** the server keeps its own data (accounts and sessions, feedback, blog, shared code, practice progress, admin settings) in files on its disk by default, or in MongoDB or Firestore when one is configured. See [Data and storage](#data-and-storage). ### System context @@ -452,12 +505,14 @@ flowchart LR CG["containers
namespaces + cgroup v1"] REPL["REPL processes
cling, python, node, ..."] FS[("/tmp/home/*
user workspaces")] - DB[("/opt/gotty/*.db
UnQLite: sessions,
feedback, blogs")] + PS["persist
one key-value interface"] + DB[("/opt/gotty/*.db
UnQLite files (default)")] end + RDB[("MongoDB or Firestore
(optional, instead of the files)")] FAUTH["Firebase Auth"] FRTDB["Firebase Realtime DB"] - OAI["OpenAI API"] + OAI["OpenAI / OpenRouter API"] UI -- "HTTPS" --> HTTP TERM -- "WSS (webtty protocol)" --> WS @@ -465,7 +520,9 @@ flowchart LR LC -. "cwd / HOME" .-> FS FB -. "watch" .-> FS FB -- "file events" --> WT - HTTP --> DB + HTTP --> PS + PS --> DB + PS -. "when configured" .-> RDB CHAT -- "HTTPS" --> CP -- "API key added server-side" --> OAI UI -- "sign-in" --> FAUTH TERM -- "share / mirror" --> FRTDB @@ -482,7 +539,8 @@ flowchart LR | Local command backend | `src/backend/localcommand/`, `src/github.com/kr/pty/` (patched) | Builds the REPL command line and environment, starts it on a PTY, handles resize and close. | | Containers | `src/containers/` | One parent cgroup per REPL type and one child cgroup per process with a memory limit. Also sets up the namespaces (UTS, PID, mount, net, user) and joins forked sessions through `nsenter`. | | File browser | `src/filebrowser/` | Workspace tree, a 50 MB quota, and fsnotify events pushed to the browser. | -| Users and sessions | `src/user/`, `src/cookie/`, `src/cachedb/` | Firebase-backed login sessions and the signed session cookie. Maps each user to a home directory. Storage is UnQLite with a freecache read cache. | +| Users and sessions | `src/user/`, `src/cookie/` | Firebase-backed login sessions and the signed session cookie. Maps each user to a home directory. Accounts can be blocked by an admin. | +| Persistence | `src/persist/`, `src/cachedb/` | One key-value interface (`persist.Store`) under every database of the site, with three backends: UnQLite files, MongoDB and Firestore. `cachedb` adds an in-memory read cache for the user database. See [Data and storage](#data-and-storage). | | Utilities | `src/utils/`, `src/encoder/` | Constants, the job scheduler (removes guest workspaces), `demos.xml` types, AES-GCM helpers, and the process-id encoding used for fork links. | | REPL catalog | `src/resources/meta/demos.xml` | One `` per REPL: the demo animation, usage, docs link, starter code, and the `` script used by **Run**. | | Web frontend | `src/resources/`, `src/js/` | Landing page and IDE (`index.html`, and `scribbler.js` built from `js/src/page/`), terminal engine (`js/src/*.ts` → `gotty-bundle.js`), Genie chat widget, Practice pages, and the JavaScript console (`jsconsole`). | @@ -493,8 +551,63 @@ flowchart LR 2. **Run or debug editor code.** **Run** reconnects the terminal with the editor content in the init payload (`IdeLang`, `IdeContent`, `IdeFileName`, flags). The server writes the content to the selected file, then runs `/bin/bash -c ` in the same sandbox, with 3× the memory limit. 3. **Fork a REPL and add terminal tabs.** The window title carries a `jid`, an encoded PID. **Fork REPL** and every extra tab open `?jid=`, and the server `nsenter`s the new shell into the parent's namespaces and working directory. This lets two terminals talk to each other, for example for socket programming. 4. **Share a REPL.** The owner's browser (the *master*) mirrors terminal output, language changes and file events to Firebase RTDB under `openrepl/`. A viewer who opens `…/#` renders that stream, and their keystrokes are relayed to the master's WebSocket. The viewer never starts a REPL of their own. -5. **Sign in.** FirebaseUI, in a dialog over the home page (Google, GitHub or email, with email verification), signs the user in. The browser then posts the user to `/login`. The server stores the session in `user_sessions.db` and sets the `user-session` cookie. Signed-in users get a stable home directory. Guest directories are deleted 60 minutes after last use. -6. **Ask Genie or generate a practice question.** The browser calls `/chat/completions` with a per-session access token. The server checks the origin, the token and a cookie-based rate limit, then forwards the request to OpenAI with the server's API key. +5. **Sign in.** FirebaseUI, in a dialog over the home page (Google, GitHub or email, with email verification), signs the user in. The browser then posts the user to `/login`. The server stores the session in the user database (`user_sessions`) and sets the `user-session` cookie. Signed-in users get a stable home directory. Guest directories are deleted 60 minutes after last use. +6. **Ask Genie or generate a practice question.** The browser calls `/chat/completions` with a per-session access token. The server checks the origin, the token and a cookie-based rate limit, then forwards the request to OpenAI or OpenRouter with the server's API key. + +### Data and storage + +The server's own data goes through one small interface, `persist.Store` (`Store`, `Fetch`, `Delete`, `Each`, `Commit`, `Rollback`, `Close`). Every database of the site is a key-value store behind it, so the rest of the code does not know where the data lives. + +```mermaid +flowchart TB + subgraph APP["gotty (gateway or standalone server)"] + U["user, cookie
accounts, sessions"] + H["server
feedback, blog, shared code,
practice progress"] + ST["server
admin settings"] + C["cachedb
15 MB read cache"] + P["persist.Store
chosen once at start-up"] + SS["settings store
one versioned document"] + end + U --> C --> P + H --> P + ST --> SS + P --> F[("MongoDB, first choice
collections kv_*")] + P --> G[("Firestore, second choice
collections kv_*")] + P --> L[("UnQLite files, the default
/opt/gotty/*.db")] + SS --> F + SS --> G + SS --> J[("settings.json")] +``` + +**Which backend.** `persist.Init` chooses once, when the server starts, and the choice holds until it stops: + +1. **MongoDB**, if `OPENREPL_MONGODB_URI` is set and answers (for example an Atlas cluster; the database is `OPENREPL_MONGODB_DB`, default `openrepl`). +2. Otherwise **Firestore**, if `OPENREPL_FIRESTORE_CREDENTIALS` holds a service account key (or the emulator is set) and the project's Firestore answers. The server uses Firestore's REST API directly, with no client library. +3. Otherwise **UnQLite files** under `/opt/gotty`, as the server has always done. + +Each backend is tried three times. A database that was asked for and did not answer is written to the log with the reason, the server goes on with the next choice, and the admin Health page shows a "Database choice" warning. Use MongoDB or Firestore on a host whose disk is wiped on every deploy (Render, for example): with files only, accounts, blog posts and settings are lost on each deploy. + +**The databases.** With files, each is a file; with MongoDB or Firestore, each is a collection named after the file (`user_sessions.db` becomes `kv_user_sessions`). + +| Database | Key → value | Used by | +|---|---|---| +| `user_sessions` | `` → the user's profile and login sessions (JSON); `SESSION_KEY` → the secret that signs the session cookie, when `OPENREPL_SECRET` is not set; `blocked:` → set by an admin; `worker-pin:` → the execution node that holds the user's workspace (distributed mode) | sign-in, the session cookie, admin accounts page | +| `feedback` | `` → `{Name, Email, Message, Read}` | `/feedback`, the admin inbox | +| `blog` | `` → the post (JSON) | `/blog`, and Genie's knowledge of the site | +| `snippets` | `<8-character id>` → the shared code (JSON) | `/snippet`, `/s/` | +| `practice` | `u:` → the user's practice progress (JSON) | `/practice/progress` | + +A record is `{_id: , v: , t: