diff --git a/.github/workflows/build-linux.yml b/.github/workflows/build-linux.yml index cbace07619..8ddd2ba9c9 100644 --- a/.github/workflows/build-linux.yml +++ b/.github/workflows/build-linux.yml @@ -98,6 +98,14 @@ jobs: run: meson setup _build -Dprofile=development - name: Format check run: cargo fmt --all -- --check + - name: No thread-local state in app services + # Trackers belong to a workspace; a thread_local! registry shares + # one window's unsaved edits with every other window. + run: | + if grep -rn 'thread_local!' crates/app/src/services; then + echo "::error::crates/app/src/services must not hold thread_local! state" + exit 1 + fi - name: Clippy run: meson devenv -C _build cargo clippy --locked --workspace --all-targets -- -D warnings - name: Clippy without default features diff --git a/linux/ARCHITECTURE.md b/linux/ARCHITECTURE.md index 5c7bd5797d..4bbfff2a81 100644 --- a/linux/ARCHITECTURE.md +++ b/linux/ARCHITECTURE.md @@ -13,59 +13,58 @@ linux/ ├── po/ translation catalogue └── crates/ ├── app/ binary, GTK4 entry point, Relm4 components - ├── core/ domain types and traits, no GUI deps - ├── net/ TCP dialer and rustls client configs + ├── core/ domain types and the driver contract, no GUI deps + ├── net/ direct TCP transport and rustls client configs ├── ssh/ system OpenSSH transport and the askpass helper + ├── session/ connect, sessions, health monitor, operation runner ├── storage/ secrets, settings, file persistence ├── test-fixtures/ dev-only container fixtures with a generated PKI + ├── test-support/ dev-only fakes and the conformance suites └── drivers/ - ├── postgres/ sqlx-postgres impl - ├── mysql/ sqlx-mysql impl - ├── sqlite/ sqlx-sqlite impl - └── ... one crate per database engine + ├── postgres/ tokio-postgres + ├── mysql/ sqlx-mysql, MySQL and MariaDB + ├── sqlite/ sqlx-sqlite + ├── mssql/ tiberius + └── clickhouse/ clickhouse over hyper ``` ## Dependency graph -``` - ┌─────────┐ - │ app │ binary, all GUI code - └────┬────┘ - ┌─────────┼──────────────────┐ - ▼ ▼ ▼ - ┌────────┐ ┌─────────┐ ┌──────────────────┐ - │ core │ │ storage │ │ drivers/* │ - └────────┘ └────┬────┘ └─────────┬────────┘ - │ │ - └────► ┌────────┐ ◄ - │ core │ - └────────┘ -``` +| Crate | Depends on | +|---|---| +| `core` | no workspace crate | +| `net`, `ssh`, `storage` | `core` | +| `drivers/postgres`, `drivers/mysql`, `drivers/mssql`, `drivers/clickhouse` | `core`, `net` | +| `drivers/sqlite` | `core` | +| `session` | `core`, `net`, `ssh`, `storage` | +| `app` | every crate above | Rules, enforced by review: -- `core` depends on **no other workspace crate**. Only the standard library and small utility crates (`async-trait`, `serde`, `thiserror`). -- `storage` depends on `core` only. -- Each `drivers/` crate depends on `core` only. **Drivers never depend on each other.** -- `app` depends on `core`, `storage`, and every `drivers/*`. It is the only crate that knows about every driver. -- No reverse dependencies. `core` never imports anything from `drivers/*` or `app`. +- `core` depends on **no other workspace crate**. It holds the domain types and the driver contract, and no I/O beyond the traits. +- `net`, `ssh` and `storage` depend on `core` only. +- Each `drivers/` crate depends on `core`, and on `net` when it speaks TLS or dials TCP. **Drivers never depend on each other, on `session` or on `ssh`.** +- `session` depends on `core`, `net`, `ssh` and `storage`, and on no driver: it reaches drivers through `DriverRegistry`. +- `app` depends on all of them. It is the only crate that names every driver. +- `test-support` and `test-fixtures` are dev-dependencies, with one exception: `session` takes `test-support` as an optional dependency behind its `fakes` feature, which only `app`'s dev-dependencies turn on. Consequences: -- Adding a driver does not touch `core`, `storage`, or any other driver. +- Adding a driver does not touch `core`, `session`, `storage` or any other driver. - Replacing the GUI framework would require rewriting only `app`. -- Drivers can be unit-tested against `core` traits without pulling GTK. -- The build graph is shallow — incremental rebuilds stay fast. +- Drivers are tested against `core` traits without pulling GTK. ## Composition root -The driver registry is built once in `app::main` before the GTK application starts running: +The driver registry is built once, by `build_registry` in `crates/app/src/lib.rs`, before the GTK application starts running: ```rust fn build_registry() -> DriverRegistry { let mut r = DriverRegistry::new(); - r.register(Arc::new(drivers_postgres::PgDriver)); + r.register(Arc::new(drivers_clickhouse::ClickhouseDriver)); + r.register(Arc::new(drivers_mssql::MssqlDriver)); r.register(Arc::new(drivers_mysql::MysqlDriver)); + r.register(Arc::new(drivers_postgres::PgDriver)); r.register(Arc::new(drivers_sqlite::SqliteDriver)); r } @@ -82,8 +81,8 @@ Two runtimes coexist: Bridging uses Relm4's built-in primitives instead of hand-rolled channels: -- `sender.command(move |out, shutdown| shutdown.register(async move { ... out.send(...) }).drop_on_shutdown())` — a component-scoped tokio task that cancels when the component drops. The `out` sender feeds `CmdOutput` back into the component's update loop on the GTK thread. Used for every per-tab fetch (browse rows, schema introspection, save transaction). -- `relm4::spawn(async move { ... })` — fire-and-forget tokio task with no component lifetime tie. Used for storage writes (`touch_last_opened`, `query_history::record`, column-width persistence) and other side-effect work. +- `sender.command(move |out, shutdown| shutdown.register(async move { ... out.send(...) }).drop_on_shutdown())`: a component-scoped tokio task that cancels when the component drops. The `out` sender feeds `CmdOutput` back into the component's update loop on the GTK thread. Used for every per-tab fetch (browse rows, schema introspection, save transaction). +- `relm4::spawn(async move { ... })`: a fire-and-forget tokio task with no component lifetime tie. Used for storage writes (`touch_last_opened`, `query_history::record`, column-width persistence) and other side-effect work. - `sender.input(AppMsg::...)` from inside an async block routes back into the component's `update` on the GTK thread. Combined with `sender.command`, this is how a "fetch-then-render" round trip lands its result on the right widget. `main.rs` builds a tiny `tokio::runtime::Builder::new_multi_thread().worker_threads(1)` runtime exclusively to `block_on` the history-DB init / prune at startup, then `shutdown_timeout`s it before `RelmApp::run` takes over. The query-history sqlx pool is stored in a `OnceLock` and reused from Relm4's runtime afterward. @@ -99,23 +98,100 @@ The `app` crate uses [Relm4](https://relm4.org) for component-based UI structure See [docs/state-management.md](docs/state-management.md) for the patterns and naming we use. -## Driver contract +## Driver layer -Every driver crate exports a single zero-sized struct that implements `core::DatabaseDriver`. The trait is async (via `async_trait`), small, and stable. +How the app reaches a database: the contract in `tablepro-core`, a transport that carries the bytes, TLS each driver builds from the connection's trust settings, and a session that watches the link. -```rust -#[async_trait::async_trait] -pub trait DatabaseDriver: Send + Sync { - fn id(&self) -> &'static str; - fn display_name(&self) -> &'static str; - fn default_port(&self) -> u16; - async fn connect(&self, opts: ConnectOptions) -> Result, DriverError>; -} -``` +### The contract + +`DatabaseDriver` describes an engine before any connection exists, through `DriverCapabilities`, and opens a `Connection`. The `Connection` serves the workspace from a pool; every call takes `CallOptions` (a deadline and a cancel token) and runs on a task the connection owns (`OwnedCalls`), so a stop reaches the server instead of leaving the statement running. Saves report what they left in the database through a `WriteLedger`. Each editor tab gets an `EditorSession` of its own that streams results into a `ResultSink`. [docs/adding-drivers.md](docs/adding-drivers.md) is the full contract, [ADR 0016](docs/decisions/0016-driver-contract-v2.md) the reasons for it, and the engine pages say how each driver meets it: [PostgreSQL](docs/drivers/postgresql.md), [MySQL and MariaDB](docs/drivers/mysql.md), [SQLite](docs/drivers/sqlite.md), [SQL Server](docs/drivers/sql-server.md), [ClickHouse](docs/drivers/clickhouse.md). + +### Connecting + +`DriverConnector` in `tablepro-session` turns a saved connection into a connected one: + +1. It checks the request against the driver's capabilities: the auth mode, the TLS settings for the route, a required database name. +2. It builds the transport: `SshSession::connect` and then `SshSession::forward` for an SSH connection, or a `DirectTcpTransport`. +3. It calls `DatabaseDriver::connect` with `ConnectOptions`: the `NetworkTarget` (transport, credentials, database, TLS settings) or the `FileEndpoint`, the `WorkspaceSettings`, the `LivenessPolicy` and the app's version. +4. It reads the first catalogue, the schemas and the default schema's objects, under the metadata deadline. + +The cancel token is checked between stages, and a stage that finishes after a cancel closes what it built, because a driver blocked in a login cannot always be interrupted. + +### Transports + +`Transport` (`crates/core/src/transport.rs`) is how a driver reaches a server without knowing whether the route is direct or through SSH: + +- `service_endpoint()`: the host and port the user typed, which TLS verifies and Kerberos names. +- `kind()` and `route()`: `DirectTcp` with a TCP endpoint, or `SshForward` with the Unix socket the forward listens on. +- `open()`: a new byte stream to the server. +- `reroute(endpoint)`: the same kind of route to another endpoint, for SQL Server's login routing. +- `explain_closed_stream()`: why a stream closed, when the transport knows better than "connection lost", as when the SSH master exited. + +| | `DirectTcpTransport` (`tablepro-net`) | `SshForwardTransport` (`tablepro-ssh`) | +|---|---|---| +| `open` | Resolves the name, then tries each address in turn under the connect timeout | Connects to the forward's Unix socket under the connect timeout | +| Socket options | `TCP_NODELAY`; keepalive after 30 s idle, every 10 s, 3 probes; `TCP_USER_TIMEOUT` 45 s | OpenSSH's own: `ServerAliveInterval=15`, `ServerAliveCountMax=3` | +| `reroute` | A new direct transport | A new forward on the same master | +| `explain_closed_stream` | Nothing to add | The master's exit, or the "open failed" line ssh printed | + +PostgreSQL, SQL Server and ClickHouse open every stream through `open()`. sqlx dials its own sockets, so the MySQL driver passes it the route instead: host and port, or the forward's socket path. Nothing relays bytes between the two. The socket options above apply only to streams `open()` makes, so a direct MySQL connection has no keepalive and finds a dead peer through the probe and its deadlines. -A `Connection` exposes the operations that `app` needs: list tables, fetch rows, run a query, etc. The full surface is defined in `core::connection`. +SSH runs the system OpenSSH client ([ADR 0013](docs/decisions/0013-openssh-client-transport.md)): one ControlMaster (`ssh -M -N -T`) per connection, with the user's `~/.ssh/config` applied, and one `ssh -O forward` Unix socket per server it reaches. The sockets live under `$XDG_RUNTIME_DIR/app//ssh/` in directories only the user can open, with mode 0600. Every prompt (host key, password, passphrase, keyboard-interactive) reaches the app through the `tablepro-askpass` helper, and the 20 s handshake timeout pauses while a prompt is open. Under Flatpak, `ssh` comes from the GNOME runtime, and a `ProxyCommand` or `Match exec` that runs a host program, the host's `/etc/ssh` and FIDO keys kept in files do not work. -The full step-by-step guide for adding a driver lives in [docs/adding-drivers.md](docs/adding-drivers.md). +### Trust + +A connection's `TlsConfig` holds the `TlsMode`, an optional CA file, an optional client certificate and key, and an optional server name: + +- `Disable`: no TLS. +- `Require`: encrypted, and the certificate is not checked. +- `VerifyFull`: the certificate must chain to a trusted root and carry the server name. + +The name checked is `NetworkTarget::tls_server_name()`: the override when one is set, otherwise the host the user typed, never the address of a local SSH forward. `TlsConfig::validate` refuses a mode, file or override the driver cannot use on that route, and the connect dialog reads the chosen files up front, so a file with no certificate is named before anything connects. + +Each engine's TLS stack takes trust in its own form, so the driver builds it: + +| Engine | `VerifyFull` trusts | Client certificate | A failure is classified from | +|---|---|---|---| +| PostgreSQL | The system store (rustls-platform-verifier), plus the CA file | Yes | The rustls error | +| ClickHouse | The system store (rustls-platform-verifier), plus the CA file | Yes, as the login, with no password sent | The rustls error | +| MySQL, MariaDB | The Mozilla roots compiled into sqlx, plus the CA file. Not the system store | Yes | The rustls error | +| SQL Server | The system store (loaded by tiberius), plus the CA file | No | rustls's message text, because tiberius keeps nothing else | +| SQLite | No TLS | | | + +PostgreSQL and ClickHouse get their rustls `ClientConfig` from `tablepro_net::tls::build_client_config`. Every engine uses the aws-lc-rs provider ([ADR 0009](docs/decisions/0009-crypto-provider-and-sources.md)). A CA file is added to the engine's roots, never in place of them. `docs/error-handling.md` lists [what each TLS failure means and what the app offers](docs/error-handling.md#tlsfailure-and-its-remedies). + +On Linux the system store is the distribution's CA bundle. Under Flatpak it is the runtime's bundle: a CA added on the host with `update-ca-certificates` or `update-ca-trust` is not visible in the sandbox (freedesktop-sdk issue 1905). + +#### Choosing a CA file under Flatpak + +- For a server whose certificate comes from a private CA, choose the CA's PEM file in the connection's TLS settings, whatever the host trusts. +- The development manifest grants `--filesystem=home`, so a file under your home folder is read at its real path. +- A file chosen from outside the home folder comes back from the file chooser portal as a path under `/run/user//doc/`. It works for as long as the portal keeps the grant. If it goes, the connect fails naming the CA file, and choosing the file again fixes it. Keeping a copy of the CA in your home folder avoids this. +- The file is read at every connect, so a replaced CA applies on the next connect. + +### Liveness and deadlines + +`LivenessPolicy::DESKTOP` bounds every phase of a connection: + +| Bound | Value | +|---|---| +| TCP connect, per address | 10 s | +| Login, including TLS | 15 s | +| TCP keepalive | after 30 s idle, every 10 s, 3 probes | +| `TCP_USER_TIMEOUT` | 45 s | +| Probe | 5 s | +| Wait for a pool slot | 5 s | +| Cancel grace, after a stop | 5 s | +| SSH handshake, paused while a prompt is open | 20 s | +| SSH forward request | 10 s | +| SSH keepalive | every 15 s, 3 missed | +| Close | 10 s | + +`DefaultDeadlines` in `tablepro-session` sets each operation's deadline: metadata 15 s, a page or a count 60 s, row changes 120 s, schema changes none. A call returns within its deadline plus the cancel grace. + +### Sessions and health + +A window holds one `Session`. Every use of its connection goes through the `OperationRunner`, which adds no timer of its own: the deadline travels in `CallOptions` so the driver stops the statement on the server. The session's monitor probes every 30 s and calls the connection lost after three failed probes in a row, or as soon as `Connection::lost()` fires, which a driver resolves only when the link is gone, never for one dead backend in its pool. It then reconnects on a backoff from 5 s to 60 s. A failure that a retry cannot fix (authentication, TLS, configuration, a host key to review) parks the session until the user acts. A read that failed because its connection was lost runs once more on the replacement link; a save never does. ## Workspace tab system @@ -138,18 +214,16 @@ Reopening a closed tab uses a 10-deep `VecDeque` snapshot t ## Per-tab pending-change registries -Two parallel thread-local registries hold the in-flight edit state for each tab, keyed by the same UUID as the workspace slot: - -| Registry | What it tracks | Materialised by | -|---|---|---| -| `services::change_tracker` | Row-level INSERT / UPDATE / DELETE for browse tabs | `BrowseTab::commit_save` → `Vec<(String, Vec)>` | -| `services::structure_tracker` | Column / index / FK / table-rename DDL for structure tabs | `sql_ddl::materialize_ops` → `Vec` | +Each window owns two registries holding the in-flight edit state for its tabs, keyed by the same UUID as the workspace slot. App creates them, registers a tracker when it opens a tab, passes the handle into the tab's init, and unregisters it when the tab closes or the connection goes away: -Both registries live in `thread_local!` `RefCell>` because relm4 + GTK is single-threaded on the UI side, and a single map keyed by tab UUID is simpler than passing trackers through every component handler. Helpers (`with_tab`, `with_tab_ref`, `open_tab`, `close_tab`, `any_pending_globally`) are the only public surface. +| Registry | Handle | What it tracks | Materialised by | +|---|---|---|---| +| `ChangeTrackerRegistry` | `TabTrackerHandle` | Row-level INSERT / UPDATE / DELETE for Table tabs | `TabTrackerHandle::change_set` → `TrackedChanges` | +| `StructureTrackerRegistry` | `StructureTrackerHandle` | Column / index / FK / table-rename DDL for Structure tabs | `sql_ddl::materialize_ops` → `Vec` | -A `Table` tab owns BOTH a row tracker and a DDL tracker against the same UUID. `close_workspace_tab_by_id` closes both registries; the close-with-pending dialog ORs both `has_pending()` flags and may dispatch up to two save transactions, gated through a `close_after_save: HashMap` counter. +A handle is a newtype over `Rc>`. Every method borrows only for its own body, calls no GTK and no callbacks, and returns owned data (`CellOverlay`, `RowState`, `TrackerDelta`), so a grid bind that runs while a tab is applying a change never meets a held borrow. A row edit goes in as a `TrackerEdit` and comes back as a `TrackerDelta`: the rows to rebind, the draft rows the list store gains, loses or updates, and the new pending count. The tab applies it to its `gio::ListStore` after the call returns. `App::dirty_tabs` reads both registries for the close prompts, the teardown guard and the `•` title prefix. CI fails on a `thread_local!` under `crates/app/src/services`, since a process-wide registry would share one window's unsaved edits with every other window. -The Structure tab is **snapshot + diff**, not per-op log. `original_*` snapshots capture the load-time schema; the diff against the live model produces ops via `sql_ddl::diff_to_ops`. Discard restores the snapshot. There is no per-op undo — Discard is the only restore point. The tracker just caches the most recent diff so out-of-band callers (close prompt, save dispatcher) read the same op list without re-deriving. +The Structure tab is **snapshot + diff**, not per-op log. `original_*` snapshots capture the load-time schema; the diff against the live model produces ops via `sql_ddl::diff_to_ops`. Discard restores the snapshot. There is no per-op undo: Discard is the only restore point. The tracker just caches the most recent diff so out-of-band callers (close prompt, save dispatcher) read the same op list without re-deriving. ## Persistence @@ -174,7 +248,7 @@ Meson is the build system and Cargo is the Rust compiler; see [ADR 0006](docs/de The floor is GNOME 50 (GTK 4.22, libadwaita 1.9, GtkSourceView 5.18, GLib 2.88); see [ADR 0015](docs/decisions/0015-gnome-50-platform-floor.md). The host runner image (`ubuntu-24.04`) ships GLib 2.80, well below that, so the system-deps check fails there. -`.github/workflows/build-linux.yml` runs the **Fast checks** job inside a `container: ubuntu:26.04`, which ships GLib 2.88 and the meson, appstream and desktop-file-utils packages the data suite needs. The container is minimal, so the install step pulls `ca-certificates`, `curl` and `git` before rust-toolchain and Swatinem can run. The **integration** job stays on the host runner because the driver crates depend only on `tablepro-core` and don't pull libadwaita. +`.github/workflows/build-linux.yml` runs the **Fast checks** job inside a `container: ubuntu:26.04`, which ships GLib 2.88 and the meson, appstream and desktop-file-utils packages the data suite needs. The container is minimal, so the install step pulls `ca-certificates`, `curl` and `git` before rust-toolchain and Swatinem can run. The **integration** job stays on the host runner because the driver crates depend only on `tablepro-core` and `tablepro-net` and don't pull libadwaita. ## Out of scope diff --git a/linux/CHANGELOG.md b/linux/CHANGELOG.md index 5aa8e0614c..7f68f55227 100644 --- a/linux/CHANGELOG.md +++ b/linux/CHANGELOG.md @@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - Password prompt on connect when the keyring holds no password, with a choice to remember it. -- SSH tunnels through the system OpenSSH client, using your SSH agent, host aliases and jump hosts. +- SSH tunnels through the system OpenSSH client, with your agent, host aliases and jump hosts, confirming new host keys. - Security group in the connect dialog: encryption mode, CA certificate, client certificate and TLS server name. - SQLite file chooser in the connect dialog, with New Database to create a file. - Results list in the SQL editor with one entry per statement, its rows affected or error, and the open transaction. @@ -18,3 +18,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed - Format, a query opened from history and a loaded SQL file could not be undone with Ctrl+Z. +- PostgreSQL statement timeouts showed as a plain server error when the server's messages were not in English. +- SQL Server money and smallmoney in the SQL editor and query results showed as approximate floats. +- Saving a SQL Server row with a char or varchar key locked the whole table until the save ended. +- Stop could not end a SQLite statement waiting for another connection's lock. +- A SQLite save that failed in a way SQLite rolls back itself closed the connection and its temporary tables. +- A SQLite script ending in an unclosed comment showed later results under the first statement's columns. +- SQLite text containing a NUL character exported as SQL that SQLite cannot read. +- ClickHouse query or export with a FORMAT clause filled memory instead of failing. +- ClickHouse editor tab silently lost its temporary tables and settings when the server expired its session. +- ClickHouse DateTime values showed as UTC text when the server profile or session set ISO output. +- ClickHouse save with an unknown outcome could read as applied because of another table's mutation. +- A new row removed with Delete stayed in the grid while the save left it out. +- A save that failed on a new row did not flash that row. +- The window title showed no unsaved-changes bullet for pending structure changes. +- `real` values showed extra digits in the grid, copies and exports, such as 0.10000000149011612 for 0.1. +- Excel export rounded integers past 2^53 and wrote decimals as text when their column padded the scale. +- Exported arrays wrote each row of a multi-dimensional array as a quoted string. +- Markdown export broke its table when a column name held a pipe. + +### Security + +- CSV export did not neutralise formulas in values of a type TablePro does not recognise. +- Format could put a space inside `+=`, `b'01'` or `@@version`, or between a MySQL function such as `GROUP_CONCAT` and its parenthesis, and change what the statement does. +- Opening a pipe or a device as a SQL file never finished, or read past the 8 MB limit. diff --git a/linux/crates/app/src/services/cell_overlay.rs b/linux/crates/app/src/services/cell_overlay.rs new file mode 100644 index 0000000000..0bc7a4def8 --- /dev/null +++ b/linux/crates/app/src/services/cell_overlay.rs @@ -0,0 +1,14 @@ +use tablepro_core::Value; + +use super::change_tracker::CellState; + +/// One cell as the grid draws it, copied out of the tracker so a bind +/// holds no borrow while it updates widgets. +#[derive(Debug, Clone, PartialEq)] +pub struct CellOverlay { + /// The pending edit when there is one, else the fetched value. + pub value: Value, + pub state: CellState, + /// The row produced the statement a save just failed on. + pub error_flash: bool, +} diff --git a/linux/crates/app/src/services/change_tracker.rs b/linux/crates/app/src/services/change_tracker.rs index 8c569e14b3..f2a8218646 100644 --- a/linux/crates/app/src/services/change_tracker.rs +++ b/linux/crates/app/src/services/change_tracker.rs @@ -1,34 +1,8 @@ -//! Per-tab pending-changeset tracker for inline spreadsheet edits. -//! -//! The tracker is the single source of truth for "what has the user -//! changed but not yet saved" in a Browse tab. It owns three buckets: -//! -//! - `inserts`: draft rows the user added (not yet persisted; PK -//! will be assigned by the database on Save for auto-increment -//! columns). -//! - `deletes`: row keys marked for deletion. Original cell values -//! are kept so the row can re-render with strikethrough styling. -//! - `updates`: per-cell modifications keyed by `(RowKey, col)`. -//! -//! `RowKey` is PK-based and stable across sort, filter, and page -//! navigation. This is the entire reason the tracker is owned by a -//! services-layer registry rather than the GTK widget tree: a -//! position-keyed identity dies on every reorder, while PK-keyed -//! identity survives. -//! -//! Threading: the tracker is **single-thread, main-thread only**. -//! All UI events run on the GTK main thread (relm4 contract), so -//! interior mutability uses `RefCell` rather than `Mutex`. The -//! registry uses `thread_local!` to avoid lock overhead in the hot -//! `connect_bind` path. -//! -//! Tests should construct a fresh `ChangeTrackerRegistry::new()` to -//! avoid contaminating each other through the thread-local global. - -use std::cell::RefCell; -use std::collections::{HashMap, VecDeque}; - -use uuid::Uuid; +//! Pending row edits for one browse tab. Rows are keyed by primary key +//! so an edit survives sort, filter and paging; the tracker is reached +//! only through `TabTrackerHandle`. + +use std::collections::{HashMap, HashSet, VecDeque}; use tablepro_core::dml::BuildSqlError; use tablepro_core::edit::CellInput; @@ -36,6 +10,10 @@ use tablepro_core::meta::RowIdentity; use tablepro_core::save::{ChangeOrigin, ChangeSet, RowInsert, RowUpdate}; use tablepro_core::{ColumnInfo, Value}; +use super::cell_overlay::CellOverlay; +use super::tracker_delta::{TrackerDelta, VisualChange}; +use super::tracker_edit::TrackerEdit; + const UNDO_LIMIT: usize = 50; /// Stable identity for a row across sort / filter / page navigation. @@ -74,7 +52,7 @@ pub type KeyValue = tablepro_core::value::ValueIdentity; /// A draft row collected by the tracker. `values` is the full column /// vector (length = `columns.len()` at the time of creation). Empty /// cells start as `Value::Null`. -#[derive(Debug, Clone)] +#[derive(Debug, Clone, PartialEq)] pub struct DraftRow { pub draft_id: u64, pub values: Vec, @@ -88,7 +66,6 @@ pub struct CellEdit { pub new_value: Value, } -/// What the grid factory asks the tracker per cell at bind time. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum CellState { Clean, @@ -96,7 +73,6 @@ pub enum CellState { InsertDraft, } -/// What the grid factory asks the tracker per row at bind time. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum RowState { Clean, @@ -133,18 +109,6 @@ pub enum UndoOp { }, } -/// Event emitted to subscribers when the tracker mutates. Subscribers -/// (BrowseTab) use the row keys to call `store.items_changed(...)` for -/// affected rows so `connect_bind` re-fires with updated CSS classes. -#[derive(Debug, Clone)] -pub enum TrackerEvent { - ChangedRows(Vec), - PendingCountChanged(usize), - Cleared, -} - -/// Per-tab tracker. Created on `BrowseTab::init`, dropped when the -/// tab is closed (via `ChangeTrackerRegistry::close_tab`). #[derive(Debug, Default)] pub struct TabChangeTracker { inserts: Vec, @@ -153,47 +117,42 @@ pub struct TabChangeTracker { undo: VecDeque, redo: VecDeque, next_draft_id: u64, - subscribers: Vec>, - /// Row that produced a failing statement during the most recent - /// save. The grid bind callback consults this to apply the - /// `tp-row-leftmost-error-flash` class. Cleared by a timeout on - /// the BrowseTab side ~1.8s after the flash starts. + /// Row behind the statement the last save failed on. error_row: Option, - /// Generation counter incremented every time `set_error_row` - /// records a new failing row. The clear-timeout closure captures - /// the gen at scheduling and only clears `error_row` if the - /// counter still matches — protects against a first-flash timeout - /// blanking out a second flash that started inside its 1.8s window. + // A flash that starts inside an older flash's timeout must not be + // cleared by that older timer. error_row_gen: u64, } impl TabChangeTracker { - #[cfg(test)] - pub fn new() -> Self { - Self::default() - } - pub fn pending_count(&self) -> usize { self.inserts.len() + self.deletes.len() + self.updates.len() } - pub fn has_pending(&self) -> bool { - self.pending_count() > 0 - } - - pub fn subscribe(&mut self, sender: relm4::Sender) { - self.subscribers.push(sender); - } - - fn emit(&self, event: TrackerEvent) { - for s in &self.subscribers { - let _ = s.send(event.clone()); + pub fn apply(&mut self, edit: TrackerEdit) -> TrackerDelta { + match edit { + TrackerEdit::Cell { + row_key, + col, + original, + value, + } => self.track_cell_edit(row_key, col, original, value), + TrackerEdit::DraftCell { draft_id, col, value } => self.track_draft_cell_edit(draft_id, col, value), + TrackerEdit::Insert { values } => self.track_insert(values), + TrackerEdit::Delete { + row_key, + original_values, + } => self.track_delete(row_key, original_values), + TrackerEdit::DiscardDraft { draft_id } => self.discard_draft(draft_id), } } - fn emit_changed(&self, keys: Vec) { - self.emit(TrackerEvent::ChangedRows(keys)); - self.emit(TrackerEvent::PendingCountChanged(self.pending_count())); + fn delta(&self, rows: Vec, visual: Vec) -> TrackerDelta { + TrackerDelta { + rows, + visual, + pending_count: self.pending_count(), + } } fn push_undo(&mut self, op: UndoOp) { @@ -208,7 +167,7 @@ impl TabChangeTracker { /// updates the new value but keeps the original `prev_value` (so /// undoing restores the very-first untouched value, not the last /// intermediate state). - pub fn track_cell_edit(&mut self, row_key: RowKey, col: usize, original: Value, new: Value) { + fn track_cell_edit(&mut self, row_key: RowKey, col: usize, original: Value, new: Value) -> TrackerDelta { let key = (row_key.clone(), col); let prev_value = self .updates @@ -216,7 +175,6 @@ impl TabChangeTracker { .map(|e| e.prev_value.clone()) .unwrap_or(original.clone()); if prev_value == new { - // User reverted to the original value — drop the edit. self.updates.remove(&key); } else { self.updates.insert( @@ -233,180 +191,175 @@ impl TabChangeTracker { prev_value: original, new_value: new, }); - self.emit_changed(vec![row_key]); + self.delta(vec![row_key], Vec::new()) } - /// Append a draft row. Returns the assigned RowKey for UI to use - /// when prepending the GObject to the grid's ListStore. - pub fn track_insert(&mut self, default_values: Vec) -> RowKey { + fn track_insert(&mut self, values: Vec) -> TrackerDelta { let draft_id = self.next_draft_id; self.next_draft_id += 1; self.inserts.push(DraftRow { draft_id, - values: default_values.clone(), + values: values.clone(), }); - let key = RowKey::Draft(draft_id); self.push_undo(UndoOp::Insert { draft_id, - values: default_values, + values: values.clone(), }); - self.emit_changed(vec![key.clone()]); - key + self.delta( + vec![RowKey::Draft(draft_id)], + vec![VisualChange::InsertDraft { draft_id, values }], + ) } - /// Mark a persisted row for deletion. `original_values` is the - /// full row at the time of mark — needed both for the strikethrough - /// render and the undo path. - pub fn track_delete(&mut self, row_key: RowKey, original_values: Vec) { + /// `original_values` is the full row at the time of the mark, for + /// the strikethrough render and for undo. + fn track_delete(&mut self, row_key: RowKey, original_values: Vec) -> TrackerDelta { self.deletes.insert(row_key.clone(), original_values.clone()); self.push_undo(UndoOp::Delete { row_key: row_key.clone(), original_values, }); - self.emit_changed(vec![row_key]); + self.delta(vec![row_key], Vec::new()) } - /// Drop a draft row entirely. Mirrors what `undo` does for an - /// `UndoOp::Insert`, but reachable from the bulk-delete path - /// (Ctrl+A → Delete) so a mixed selection of drafts + persisted - /// rows behaves uniformly: persisted rows get strikethrough - /// pending-delete, drafts disappear. - /// - /// Returns `true` when a draft with `draft_id` was found and - /// removed; `false` otherwise (already discarded, or never - /// existed). Also clears any draft-cell edits and undo / redo - /// entries scoped to this draft so a subsequent Ctrl+Z doesn't - /// resurrect the draft into a half-edited state. - pub fn discard_draft(&mut self, draft_id: u64) -> bool { + // The draft's undo and redo entries go with it, so Ctrl+Z cannot + // bring it back half-edited. + fn discard_draft(&mut self, draft_id: u64) -> TrackerDelta { let before = self.inserts.len(); self.inserts.retain(|d| d.draft_id != draft_id); if self.inserts.len() == before { - return false; + return self.delta(Vec::new(), Vec::new()); } - let key = RowKey::Draft(draft_id); self.updates .retain(|(k, _), _| !matches!(k, RowKey::Draft(id) if *id == draft_id)); self.deletes .retain(|k, _| !matches!(k, RowKey::Draft(id) if *id == draft_id)); self.undo.retain(|op| !undo_op_matches_draft(op, draft_id)); self.redo.retain(|op| !undo_op_matches_draft(op, draft_id)); - self.emit_changed(vec![key]); - true + self.delta( + vec![RowKey::Draft(draft_id)], + vec![VisualChange::RemoveDraft { draft_id }], + ) } - /// Update a draft row's cell value (only valid for `RowKey::Draft`). - pub fn track_draft_cell_edit(&mut self, draft_id: u64, col: usize, new: Value) -> bool { - if let Some(draft) = self.inserts.iter_mut().find(|d| d.draft_id == draft_id) - && col < draft.values.len() - { - let prev = draft.values[col].clone(); - draft.values[col] = new.clone(); - self.push_undo(UndoOp::CellEdit { - row_key: RowKey::Draft(draft_id), - col, - prev_value: prev, - new_value: new, - }); - self.emit_changed(vec![RowKey::Draft(draft_id)]); - return true; - } - false + fn track_draft_cell_edit(&mut self, draft_id: u64, col: usize, new: Value) -> TrackerDelta { + let Some(prev) = self.draft_cell(draft_id, col) else { + return self.delta(Vec::new(), Vec::new()); + }; + let visual = self.set_draft_cell(draft_id, col, new.clone()); + self.push_undo(UndoOp::CellEdit { + row_key: RowKey::Draft(draft_id), + col, + prev_value: prev, + new_value: new, + }); + self.delta(vec![RowKey::Draft(draft_id)], visual.into_iter().collect()) } - /// Pop one entry off the undo stack, revert the tracker's - /// internal state, push it onto the redo stack, and return the - /// `UndoOp` so the UI layer knows which visual change to apply - /// (revert a RowObject's cell, remove a draft from the - /// ListStore, or just re-bind a row to drop its strikethrough). - /// - /// Returning the full op (instead of only the `RowKey`) is what - /// lets the visual revert happen: the tracker holds the - /// `prev_value` needed to restore the RowObject's cell, and the - /// caller would otherwise have to peek `self.redo.back()` after - /// the call — fragile and bypasses the encapsulation. - pub fn undo(&mut self) -> Option { + fn draft_cell(&self, draft_id: u64, col: usize) -> Option { + self.inserts + .iter() + .find(|d| d.draft_id == draft_id) + .and_then(|d| d.values.get(col)) + .cloned() + } + + fn set_draft_cell(&mut self, draft_id: u64, col: usize, value: Value) -> Option { + let cell = self + .inserts + .iter_mut() + .find(|d| d.draft_id == draft_id) + .and_then(|d| d.values.get_mut(col))?; + *cell = value.clone(); + Some(VisualChange::SetDraftCell { draft_id, col, value }) + } + + pub fn undo(&mut self) -> Option { let op = self.undo.pop_back()?; - let row_key = match &op { + let (row_key, visual) = match &op { UndoOp::CellEdit { - row_key, + row_key: RowKey::Draft(draft_id), col, prev_value, .. - } => { - if let RowKey::Draft(draft_id) = row_key { - if let Some(draft) = self.inserts.iter_mut().find(|d| d.draft_id == *draft_id) - && *col < draft.values.len() - { - draft.values[*col] = prev_value.clone(); - } - } else { - self.updates.remove(&(row_key.clone(), *col)); - } - row_key.clone() + } => ( + RowKey::Draft(*draft_id), + self.set_draft_cell(*draft_id, *col, prev_value.clone()) + .into_iter() + .collect(), + ), + UndoOp::CellEdit { row_key, col, .. } => { + self.updates.remove(&(row_key.clone(), *col)); + (row_key.clone(), Vec::new()) } UndoOp::Insert { draft_id, .. } => { self.inserts.retain(|d| d.draft_id != *draft_id); - RowKey::Draft(*draft_id) + ( + RowKey::Draft(*draft_id), + vec![VisualChange::RemoveDraft { draft_id: *draft_id }], + ) } UndoOp::Delete { row_key, .. } => { self.deletes.remove(row_key); - row_key.clone() + (row_key.clone(), Vec::new()) } }; - self.redo.push_back(op.clone()); - self.emit_changed(vec![row_key]); - Some(op) + self.redo.push_back(op); + Some(self.delta(vec![row_key], visual)) } - /// Pop one entry off the redo stack, re-apply the tracker's - /// internal state, push back onto the undo stack, and return - /// the `UndoOp` so the UI layer knows which visual change to - /// re-apply (set a RowObject's cell to `new_value`, re-add a - /// draft to the ListStore, or re-bind a row's strikethrough). - pub fn redo(&mut self) -> Option { + pub fn redo(&mut self) -> Option { let op = self.redo.pop_back()?; - let row_key = match &op { + let (row_key, visual) = match &op { + UndoOp::CellEdit { + row_key: RowKey::Draft(draft_id), + col, + new_value, + .. + } => ( + RowKey::Draft(*draft_id), + self.set_draft_cell(*draft_id, *col, new_value.clone()) + .into_iter() + .collect(), + ), UndoOp::CellEdit { row_key, col, prev_value, new_value, } => { - if let RowKey::Draft(draft_id) = row_key { - if let Some(draft) = self.inserts.iter_mut().find(|d| d.draft_id == *draft_id) - && *col < draft.values.len() - { - draft.values[*col] = new_value.clone(); - } - } else { - self.updates.insert( - (row_key.clone(), *col), - CellEdit { - prev_value: prev_value.clone(), - new_value: new_value.clone(), - }, - ); - } - row_key.clone() + self.updates.insert( + (row_key.clone(), *col), + CellEdit { + prev_value: prev_value.clone(), + new_value: new_value.clone(), + }, + ); + (row_key.clone(), Vec::new()) } UndoOp::Insert { draft_id, values } => { self.inserts.push(DraftRow { draft_id: *draft_id, values: values.clone(), }); - RowKey::Draft(*draft_id) + ( + RowKey::Draft(*draft_id), + vec![VisualChange::InsertDraft { + draft_id: *draft_id, + values: values.clone(), + }], + ) } UndoOp::Delete { row_key, original_values, } => { self.deletes.insert(row_key.clone(), original_values.clone()); - row_key.clone() + (row_key.clone(), Vec::new()) } }; - self.undo.push_back(op.clone()); - self.emit_changed(vec![row_key]); - Some(op) + self.undo.push_back(op); + Some(self.delta(vec![row_key], visual)) } #[cfg(test)] @@ -419,20 +372,42 @@ impl TabChangeTracker { !self.redo.is_empty() } - /// Drop all pending changes, undo, redo. Called by Discard and by - /// Save (after successful commit). - pub fn clear(&mut self) { + /// Drop every pending change and the undo history. + pub fn discard(&mut self) -> TrackerDelta { + let mut seen = HashSet::new(); + let rows: Vec = self + .updates + .keys() + .map(|(row_key, _)| row_key) + .chain(self.deletes.keys()) + .cloned() + .chain(self.inserts.iter().map(|draft| RowKey::Draft(draft.draft_id))) + .filter(|row_key| seen.insert(row_key.clone())) + .collect(); + let visual = self + .inserts + .iter() + .map(|draft| VisualChange::RemoveDraft { + draft_id: draft.draft_id, + }) + .collect(); self.inserts.clear(); self.deletes.clear(); self.updates.clear(); self.undo.clear(); self.redo.clear(); - self.emit(TrackerEvent::Cleared); - self.emit(TrackerEvent::PendingCountChanged(0)); + self.delta(rows, visual) + } + + pub fn cell_overlay(&self, row_key: &RowKey, col: usize, original: &Value) -> CellOverlay { + CellOverlay { + value: self.current_cell_value(row_key, col, original).clone(), + state: self.cell_state(row_key, col), + error_flash: self.error_row.as_ref() == Some(row_key), + } } - /// State for a given (row, col) cell. Used by grid `connect_bind`. - pub fn cell_state(&self, row_key: &RowKey, col: usize) -> CellState { + fn cell_state(&self, row_key: &RowKey, col: usize) -> CellState { if matches!(row_key, RowKey::Draft(_)) { return CellState::InsertDraft; } @@ -462,47 +437,28 @@ impl TabChangeTracker { &self.inserts } - /// Mark a row as the "currently failing" row for the flash animation. - /// Returns the generation counter the caller should pass to - /// `clear_error_row_if_gen` after the flash timeout, so a new flash - /// during the timeout window doesn't get blanked out by the older - /// timer firing. - pub fn set_error_row(&mut self, key: RowKey) -> u64 { + /// Mark the row a failed save points at. The returned generation + /// goes back to `clear_error_row_if` when the flash ends. + pub fn flash_error_row(&mut self, key: RowKey) -> u64 { self.error_row_gen = self.error_row_gen.wrapping_add(1); self.error_row = Some(key); self.error_row_gen } - /// Clear the error row only if the generation hasn't been bumped by - /// a newer `set_error_row` call. Used by the flash timeout to avoid - /// clearing a fresh flash that started after this timer was scheduled. - pub fn clear_error_row_if_gen(&mut self, generation: u64) { - if self.error_row_gen == generation { - self.error_row = None; + /// Clear the flash unless a newer one has started. Returns whether + /// it cleared, so the caller knows the row needs a rebind. + pub fn clear_error_row_if(&mut self, generation: u64) -> bool { + if self.error_row_gen != generation { + return false; } + self.error_row = None; + true } - /// True when the given generation matches the most recent - /// `set_error_row`. Used by the flash-clear timeout to detect - /// whether its scheduled clear is still authoritative. - pub fn is_error_row_gen(&self, generation: u64) -> bool { - self.error_row_gen == generation - } - - /// True when `key` matches the row currently flagged as the error - /// row. Used by the grid bind callback to apply the flash class. - pub fn is_error_row(&self, key: &RowKey) -> bool { - self.error_row.as_ref() == Some(key) - } - - /// Display value for a cell — pending edit if present, else - /// `original`. Used by the grid bind to render the user's pending - /// edit instead of the stale DB value. - pub fn current_cell_value<'a>(&'a self, row_key: &RowKey, col: usize, original: &'a Value) -> &'a Value { - if let Some(edit) = self.updates.get(&(row_key.clone(), col)) { - &edit.new_value - } else { - original + fn current_cell_value<'a>(&'a self, row_key: &RowKey, col: usize, original: &'a Value) -> &'a Value { + match self.updates.get(&(row_key.clone(), col)) { + Some(edit) => &edit.new_value, + None => original, } } @@ -634,82 +590,6 @@ fn undo_op_matches_draft(op: &UndoOp, draft_id: u64) -> bool { } } -/// Per-tab registry of trackers. Singleton via `thread_local!` because -/// all UI work runs on the GTK main thread. -#[derive(Debug, Default)] -pub struct ChangeTrackerRegistry { - trackers: HashMap, -} - -impl ChangeTrackerRegistry { - pub fn new() -> Self { - Self::default() - } - - pub fn open_tab(&mut self, tab_id: Uuid) { - self.trackers.entry(tab_id).or_default(); - } - - pub fn close_tab(&mut self, tab_id: Uuid) { - self.trackers.remove(&tab_id); - } - - pub fn any_pending(&self) -> bool { - self.trackers.values().any(|t| t.has_pending()) - } - - pub fn pending_tabs(&self) -> Vec { - self.trackers - .iter() - .filter(|(_, t)| t.has_pending()) - .map(|(id, _)| *id) - .collect() - } -} - -thread_local! { - static REGISTRY: RefCell = RefCell::new(ChangeTrackerRegistry::new()); -} - -/// Run a closure with mutable access to a specific tab's tracker. -/// Returns `None` if the tab has not been registered yet (e.g., -/// during early init). -pub fn with_tab(tab_id: Uuid, f: F) -> Option -where - F: FnOnce(&mut TabChangeTracker) -> R, -{ - REGISTRY.with(|reg| reg.borrow_mut().trackers.get_mut(&tab_id).map(f)) -} - -/// Run a closure with read-only access to a specific tab's tracker. -pub fn with_tab_ref(tab_id: Uuid, f: F) -> Option -where - F: FnOnce(&TabChangeTracker) -> R, -{ - REGISTRY.with(|reg| reg.borrow().trackers.get(&tab_id).map(f)) -} - -/// Open a tracker for a tab. Idempotent. -pub fn open_tab(tab_id: Uuid) { - REGISTRY.with(|reg| reg.borrow_mut().open_tab(tab_id)); -} - -/// Drop a tab's tracker. Called when the BrowseTab is closed. -pub fn close_tab(tab_id: Uuid) { - REGISTRY.with(|reg| reg.borrow_mut().close_tab(tab_id)); -} - -/// True if any open tab has pending changes — used by the app-level -/// quit guard to decide whether to show the "Unsaved changes" dialog. -pub fn any_pending_globally() -> bool { - REGISTRY.with(|reg| reg.borrow().any_pending()) -} - -/// Tabs with pending changes — ordered arbitrary (HashMap iteration). -pub fn pending_tabs() -> Vec { - REGISTRY.with(|reg| reg.borrow().pending_tabs()) -} - #[cfg(test)] mod tests { /// A column carrying the server's spelling and the kind it means, @@ -774,9 +654,16 @@ mod tests { RowKey::from_pk_values(values).unwrap() } + fn insert(t: &mut TabChangeTracker, values: Vec) -> u64 { + match t.track_insert(values).visual.as_slice() { + [VisualChange::InsertDraft { draft_id, .. }] => *draft_id, + other => panic!("an insert adds one draft row: {other:?}"), + } + } + #[test] fn cell_edit_creates_modified_state() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(1)]); t.track_cell_edit(key.clone(), 1, Value::Text("a".into()), Value::Text("b".into())); assert_eq!(t.cell_state(&key, 1), CellState::Modified); @@ -786,7 +673,7 @@ mod tests { #[test] fn cell_edit_back_to_original_drops_modification() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(1)]); t.track_cell_edit(key.clone(), 1, Value::Text("a".into()), Value::Text("b".into())); // User edits again, this time back to "a" @@ -799,18 +686,17 @@ mod tests { #[test] fn insert_assigns_unique_draft_ids() { - let mut t = TabChangeTracker::new(); - let k1 = t.track_insert(vec![Value::Null, Value::Null]); - let k2 = t.track_insert(vec![Value::Null, Value::Null]); + let mut t = TabChangeTracker::default(); + let k1 = insert(&mut t, vec![Value::Null, Value::Null]); + let k2 = insert(&mut t, vec![Value::Null, Value::Null]); assert_ne!(k1, k2); - assert!(matches!(k1, RowKey::Draft(_))); assert_eq!(t.drafts().len(), 2); assert_eq!(t.pending_count(), 2); } #[test] fn delete_marks_row_pending() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(7)]); t.track_delete(key.clone(), vec![Value::Int(7), Value::Text("dave".into())]); assert_eq!(t.row_state(&key), RowState::PendingDelete); @@ -819,21 +705,18 @@ mod tests { #[test] fn undo_reverts_cell_edit() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(1)]); t.track_cell_edit(key.clone(), 1, Value::Text("a".into()), Value::Text("b".into())); let undone = t.undo(); - assert!(matches!( - undone, - Some(UndoOp::CellEdit { ref row_key, col: 1, .. }) if row_key == &key - )); + assert_eq!(undone.map(|delta| delta.rows), Some(vec![key.clone()])); assert_eq!(t.cell_state(&key, 1), CellState::Clean); assert!(t.can_redo()); } #[test] fn redo_reapplies_cell_edit() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(1)]); t.track_cell_edit(key.clone(), 1, Value::Text("a".into()), Value::Text("b".into())); t.undo(); @@ -843,7 +726,7 @@ mod tests { #[test] fn undo_stack_caps_at_50() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(1)]); for i in 0..100 { t.track_cell_edit(key.clone(), 1, Value::Int(i), Value::Int(i + 1)); @@ -853,12 +736,12 @@ mod tests { #[test] fn clear_removes_all_state() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let key = rk(&[Value::Int(1)]); t.track_cell_edit(key.clone(), 1, Value::Text("a".into()), Value::Text("b".into())); t.track_insert(vec![Value::Null]); t.track_delete(rk(&[Value::Int(2)]), vec![Value::Int(2)]); - t.clear(); + t.discard(); assert_eq!(t.pending_count(), 0); assert!(!t.can_undo()); } @@ -878,7 +761,7 @@ mod tests { #[test] fn change_set_orders_inserts_then_updates_then_deletes() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let columns = vec![pk_col("id"), data_col("name")]; let updated = rk(&[Value::Int(5)]); t.track_cell_edit(updated.clone(), 1, Value::Text("old".into()), Value::Text("new".into())); @@ -902,7 +785,7 @@ mod tests { #[test] fn a_failing_step_maps_back_to_the_row_that_made_it() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let columns = vec![pk_col("id"), data_col("name")]; let doomed = rk(&[Value::Int(9)]); t.track_insert(vec![Value::Null, Value::Text("draft".into())]); @@ -917,7 +800,7 @@ mod tests { #[test] fn an_update_sets_only_the_moved_cell_and_keys_on_the_whole_primary_key() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let columns = vec![pk_col("a"), pk_col("b"), data_col("name")]; let key = RowKey::from_pk_values(&[Value::Int(1), Value::Int(2)]).unwrap(); t.track_cell_edit(key, 2, Value::Text("old".into()), Value::Text("new".into())); @@ -944,7 +827,7 @@ mod tests { reason = "a test fixture stands in for the catalogue, which is where a default expression comes from" )] fn a_draft_leaves_server_filled_columns_to_the_server() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let mut stamped = data_col("created_at"); stamped.default = tablepro_core::column::ColumnDefault::Expression( tablepro_core::column::SqlExpression::from_catalog_text("now()"), @@ -966,7 +849,7 @@ mod tests { #[test] fn a_table_without_a_primary_key_cannot_save_an_update() { - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let columns = vec![data_col("name")]; t.track_cell_edit( rk(&[Value::Int(1)]), @@ -978,39 +861,10 @@ mod tests { assert!(matches!(t.change_set(&columns), Err(BuildSqlError::NoRowIdentity))); } - #[test] - fn registry_open_close_idempotent() { - let mut reg = ChangeTrackerRegistry::new(); - let id = Uuid::new_v4(); - reg.open_tab(id); - reg.open_tab(id); - assert!(reg.trackers.contains_key(&id)); - reg.close_tab(id); - assert!(!reg.trackers.contains_key(&id)); - } - - #[test] - fn registry_any_pending_reports_correctly() { - let mut reg = ChangeTrackerRegistry::new(); - let id = Uuid::new_v4(); - reg.open_tab(id); - assert!(!reg.any_pending()); - let key = rk(&[Value::Int(1)]); - reg.trackers - .get_mut(&id) - .unwrap() - .track_cell_edit(key, 0, Value::Int(1), Value::Int(2)); - assert!(reg.any_pending()); - assert_eq!(reg.pending_tabs(), vec![id]); - } - #[test] fn discard_draft_removes_insert_and_clears_undo() { - let mut t = TabChangeTracker::new(); - let key = t.track_insert(vec![Value::Null, Value::Text("hi".into())]); - let RowKey::Draft(draft_id) = key else { - panic!("track_insert must return Draft key"); - }; + let mut t = TabChangeTracker::default(); + let draft_id = insert(&mut t, vec![Value::Null, Value::Text("hi".into())]); // Edit the draft so undo stack has a CellEdit on top of Insert. t.track_draft_cell_edit(draft_id, 1, Value::Text("edited".into())); assert_eq!(t.drafts().len(), 1); @@ -1018,30 +872,26 @@ mod tests { let removed = t.discard_draft(draft_id); - assert!(removed, "discard_draft must report removal"); + assert_eq!(removed.visual, vec![VisualChange::RemoveDraft { draft_id }]); assert!(t.drafts().is_empty(), "draft must be gone"); assert!(!t.can_undo(), "undo entries scoped to the draft must be cleared"); - assert!(!t.has_pending(), "no pending state after discard"); + assert_eq!(t.pending_count(), 0, "no pending state after discard"); } #[test] fn discard_draft_no_op_for_unknown_id() { - let mut t = TabChangeTracker::new(); - assert!(!t.discard_draft(99_999)); - assert!(!t.has_pending()); + let mut t = TabChangeTracker::default(); + assert_eq!(t.discard_draft(99_999), TrackerDelta::default()); + assert_eq!(t.pending_count(), 0); } #[test] fn discard_draft_leaves_other_drafts_intact() { - let mut t = TabChangeTracker::new(); - let RowKey::Draft(a) = t.track_insert(vec![Value::Int(1)]) else { - panic!() - }; - let RowKey::Draft(b) = t.track_insert(vec![Value::Int(2)]) else { - panic!() - }; + let mut t = TabChangeTracker::default(); + let a = insert(&mut t, vec![Value::Int(1)]); + let b = insert(&mut t, vec![Value::Int(2)]); assert_eq!(t.drafts().len(), 2); - assert!(t.discard_draft(a)); + assert_eq!(t.discard_draft(a).pending_count, 1); assert_eq!(t.drafts().len(), 1); assert_eq!(t.drafts()[0].draft_id, b); // The remaining draft is still undoable on its own. @@ -1054,7 +904,7 @@ mod tests { // for persisted rows): track_delete N times. Each push is its // own undoable op so Ctrl+Z restores rows one at a time — // matches the per-op undo contract elsewhere. - let mut t = TabChangeTracker::new(); + let mut t = TabChangeTracker::default(); let k1 = rk(&[Value::Int(1)]); let k2 = rk(&[Value::Int(2)]); let k3 = rk(&[Value::Int(3)]); @@ -1070,4 +920,153 @@ mod tests { assert!(matches!(t.row_state(&k1), RowState::PendingDelete)); assert!(matches!(t.row_state(&k2), RowState::PendingDelete)); } + + #[test] + fn apply_routes_each_edit_and_reports_the_rows_it_touched() { + let mut t = TabChangeTracker::default(); + let key = rk(&[Value::Int(1)]); + + let edited = t.apply(TrackerEdit::Cell { + row_key: key.clone(), + col: 1, + original: Value::Text("a".into()), + value: Value::Text("b".into()), + }); + let deleted = t.apply(TrackerEdit::Delete { + row_key: rk(&[Value::Int(2)]), + original_values: vec![Value::Int(2)], + }); + + assert_eq!(edited.rows, vec![key]); + assert!(edited.visual.is_empty()); + assert_eq!(edited.pending_count, 1); + assert_eq!(deleted.rows, vec![rk(&[Value::Int(2)])]); + assert_eq!(deleted.pending_count, 2); + } + + #[test] + fn a_draft_cell_edit_updates_the_draft_row_and_undo_restores_it() { + let mut t = TabChangeTracker::default(); + let draft_id = insert(&mut t, vec![Value::Null, Value::Text("a".into())]); + + let edited = t.apply(TrackerEdit::DraftCell { + draft_id, + col: 1, + value: Value::Text("b".into()), + }); + let undone = t.undo(); + + assert_eq!( + edited.visual, + vec![VisualChange::SetDraftCell { + draft_id, + col: 1, + value: Value::Text("b".into()) + }] + ); + assert_eq!( + undone.map(|delta| delta.visual), + Some(vec![VisualChange::SetDraftCell { + draft_id, + col: 1, + value: Value::Text("a".into()) + }]) + ); + assert_eq!(t.drafts()[0].values[1], Value::Text("a".into())); + } + + #[test] + fn a_draft_cell_outside_the_row_changes_nothing() { + let mut t = TabChangeTracker::default(); + let draft_id = insert(&mut t, vec![Value::Null]); + + let delta = t.apply(TrackerEdit::DraftCell { + draft_id, + col: 4, + value: Value::Int(1), + }); + + assert!(delta.rows.is_empty() && delta.visual.is_empty()); + assert_eq!(delta.pending_count, 1); + } + + #[test] + fn undo_and_redo_of_an_insert_remove_and_restore_the_draft_row() { + let mut t = TabChangeTracker::default(); + let draft_id = insert(&mut t, vec![Value::Int(7)]); + + let undone = t.undo().map(|delta| delta.visual); + let redone = t.redo().map(|delta| delta.visual); + + assert_eq!(undone, Some(vec![VisualChange::RemoveDraft { draft_id }])); + assert_eq!( + redone, + Some(vec![VisualChange::InsertDraft { + draft_id, + values: vec![Value::Int(7)] + }]) + ); + } + + #[test] + fn redo_of_a_persisted_cell_edit_leaves_the_fetched_row_alone() { + let mut t = TabChangeTracker::default(); + let key = rk(&[Value::Int(1)]); + t.track_cell_edit(key.clone(), 1, Value::Text("a".into()), Value::Text("b".into())); + let _ = t.undo(); + + let redone = t.redo(); + + assert_eq!( + redone.map(|delta| (delta.rows, delta.visual)), + Some((vec![key], Vec::new())) + ); + } + + #[test] + fn discard_drops_every_draft_row_and_reports_each_touched_row_once() { + let mut t = TabChangeTracker::default(); + let key = rk(&[Value::Int(1)]); + t.track_cell_edit(key.clone(), 1, Value::Int(1), Value::Int(2)); + t.track_cell_edit(key.clone(), 2, Value::Int(3), Value::Int(4)); + let draft_id = insert(&mut t, vec![Value::Null]); + + let delta = t.discard(); + + assert_eq!(delta.pending_count, 0); + assert_eq!(delta.visual, vec![VisualChange::RemoveDraft { draft_id }]); + assert_eq!(delta.rows.len(), 2, "{:?}", delta.rows); + assert!(delta.rows.contains(&key) && delta.rows.contains(&RowKey::Draft(draft_id))); + } + + #[test] + fn cell_overlay_shows_the_pending_value_over_the_fetched_one() { + let mut t = TabChangeTracker::default(); + let key = rk(&[Value::Int(1)]); + let fetched = Value::Text("a".into()); + t.track_cell_edit(key.clone(), 1, fetched.clone(), Value::Text("b".into())); + + let edited = t.cell_overlay(&key, 1, &fetched); + let untouched = t.cell_overlay(&key, 0, &Value::Int(1)); + + assert_eq!(edited.value, Value::Text("b".into())); + assert_eq!(edited.state, CellState::Modified); + assert_eq!(untouched.value, Value::Int(1)); + assert_eq!(untouched.state, CellState::Clean); + } + + #[test] + fn an_older_flash_timer_does_not_clear_a_newer_flash() { + let mut t = TabChangeTracker::default(); + let first = rk(&[Value::Int(1)]); + let second = RowKey::Draft(3); + let old = t.flash_error_row(first.clone()); + let new = t.flash_error_row(second.clone()); + + assert!(!t.clear_error_row_if(old)); + assert!(t.cell_overlay(&second, 0, &Value::Null).error_flash); + assert!(!t.cell_overlay(&first, 0, &Value::Null).error_flash); + assert!(t.clear_error_row_if(new)); + assert!(!t.cell_overlay(&second, 0, &Value::Null).error_flash); + } } diff --git a/linux/crates/app/src/services/change_tracker_registry.rs b/linux/crates/app/src/services/change_tracker_registry.rs new file mode 100644 index 0000000000..5784f2d96d --- /dev/null +++ b/linux/crates/app/src/services/change_tracker_registry.rs @@ -0,0 +1,71 @@ +use super::tab_registry::{TabRegistry, TrackerHandle}; +use super::tab_tracker_handle::TabTrackerHandle; + +pub type ChangeTrackerRegistry = TabRegistry; + +impl TrackerHandle for TabTrackerHandle { + fn has_pending(&self) -> bool { + self.pending_count() > 0 + } +} + +#[cfg(test)] +mod tests { + use tablepro_core::Value; + use uuid::Uuid; + + use super::*; + use crate::services::change_tracker::RowKey; + use crate::services::tracker_edit::TrackerEdit; + + fn edit(tracker: &TabTrackerHandle) { + let _ = tracker.apply(TrackerEdit::Cell { + row_key: RowKey::from_pk_values(&[Value::Int(1)]).expect("a key"), + col: 1, + original: Value::Int(1), + value: Value::Int(2), + }); + } + + #[test] + fn registries_isolated_per_workspace() { + let first = ChangeTrackerRegistry::default(); + let second = ChangeTrackerRegistry::default(); + let tab = Uuid::new_v4(); + + edit(&first.register(tab)); + let other = second.register(tab); + + assert_eq!(first.dirty(), vec![tab]); + assert!(second.dirty().is_empty()); + assert_eq!(other.pending_count(), 0); + } + + #[test] + fn dirty_lists_only_pending_tabs() { + let registry = ChangeTrackerRegistry::default(); + let edited = Uuid::new_v4(); + let clean = Uuid::new_v4(); + edit(®istry.register(edited)); + registry.register(clean); + + assert_eq!(registry.dirty(), vec![edited]); + + let _ = registry.get(edited).expect("registered").discard(); + assert!(registry.dirty().is_empty()); + } + + #[test] + fn register_returns_the_same_tracker_until_unregistered() { + let registry = ChangeTrackerRegistry::default(); + let tab = Uuid::new_v4(); + edit(®istry.register(tab)); + + assert_eq!(registry.register(tab).pending_count(), 1); + + registry.unregister(tab); + assert!(registry.get(tab).is_none()); + assert!(registry.dirty().is_empty()); + assert_eq!(registry.register(tab).pending_count(), 0); + } +} diff --git a/linux/crates/app/src/services/mod.rs b/linux/crates/app/src/services/mod.rs index 1f02a3f8aa..ff92e24c3f 100644 --- a/linux/crates/app/src/services/mod.rs +++ b/linux/crates/app/src/services/mod.rs @@ -3,7 +3,9 @@ use std::sync::OnceLock; use tablepro_storage::StoragePaths; pub mod active_session; +pub mod cell_overlay; pub mod change_tracker; +pub mod change_tracker_registry; pub mod history_availability; pub mod history_service; pub mod secret_labels; @@ -11,6 +13,12 @@ pub mod secret_save_report; pub mod session_context; pub mod single_instance; pub mod structure_tracker; +pub mod structure_tracker_handle; +pub mod structure_tracker_registry; +pub mod tab_registry; +pub mod tab_tracker_handle; +pub mod tracker_delta; +pub mod tracker_edit; pub mod workspace_state; static PATHS: OnceLock = OnceLock::new(); diff --git a/linux/crates/app/src/services/structure_tracker.rs b/linux/crates/app/src/services/structure_tracker.rs index c6d23b5ddb..e537729c13 100644 --- a/linux/crates/app/src/services/structure_tracker.rs +++ b/linux/crates/app/src/services/structure_tracker.rs @@ -1,32 +1,10 @@ -//! Passive per-tab DDL ops cache for the Structure workspace tab. -//! -//! StructureTab now owns the canonical source of truth for pending -//! changes — it computes them on every edit by diffing the loaded -//! snapshot against the live model (`sql_ddl::diff_to_ops`). On each -//! recompute the tab calls [`with_tab`] to overwrite this cache so -//! out-of-band callers (close-with-pending dialog, save-by-id, the -//! window-close prompt) can ask whether a tab is dirty and emit its -//! current SQL without coupling to the tab's UI internals. -//! -//! There is no undo / redo machinery here — DDL undo is a session -//! concept (Discard reverts the model to the snapshot). Per-keystroke -//! op history is the wrong granularity for schema editing and was -//! removed in the Structure-Refactor batch. -//! -//! Threading: GTK main thread only. `RefCell` interior mutability. +//! The DDL a Structure tab would run, cached so the close prompt and +//! the save dispatcher can read it without reaching into the tab. The +//! tab recomputes the ops from its snapshot diff on every edit; DDL has +//! no per-op undo, only Discard back to the snapshot. -use std::cell::RefCell; -use std::collections::HashMap; +use tablepro_core::sql_ddl::StructureOp; -use uuid::Uuid; - -use tablepro_core::dialect::SqlDialect; -use tablepro_core::sql_ddl::{BuildDdlError, StructureOp, materialize_ops}; - -/// Per-tab cache. Holds the current pending-op list emitted by the -/// tab's diff. `materialize` re-runs SQL emission on demand; the -/// cache means out-of-band callers don't have to re-derive ops from -/// the tab's model. #[derive(Debug, Default)] pub struct StructureChangeTracker { ops: Vec, @@ -41,9 +19,6 @@ impl StructureChangeTracker { &self.ops } - /// Replace the cached op list. Called by StructureTab on every - /// model mutation; the tab is the source of truth for what ops - /// the diff currently produces. pub fn set_ops(&mut self, ops: Vec) { self.ops = ops; } @@ -51,87 +26,6 @@ impl StructureChangeTracker { pub fn clear(&mut self) { self.ops.clear(); } - - /// The SQL for the cached ops in `dialect`, which has to be the - /// connection's own: renames and default drops are spelled per - /// server. - pub fn materialize(&self, driver_id: &str, dialect: &dyn SqlDialect) -> Result, BuildDdlError> { - materialize_ops(&self.ops, driver_id, dialect) - } -} - -#[derive(Debug, Default)] -pub struct StructureTrackerRegistry { - trackers: HashMap, -} - -impl StructureTrackerRegistry { - pub fn open_tab(&mut self, tab_id: Uuid) { - self.trackers.entry(tab_id).or_default(); - } - - pub fn close_tab(&mut self, tab_id: Uuid) { - self.trackers.remove(&tab_id); - } - - pub fn with_tab(&mut self, tab_id: Uuid, f: F) -> R - where - F: FnOnce(&mut StructureChangeTracker) -> R, - { - f(self.trackers.entry(tab_id).or_default()) - } - - pub fn with_tab_ref(&self, tab_id: Uuid, f: F) -> Option - where - F: FnOnce(&StructureChangeTracker) -> R, - { - self.trackers.get(&tab_id).map(f) - } - - pub fn any_pending_globally(&self) -> bool { - self.trackers.values().any(|t| t.has_pending()) - } - - pub fn pending_tabs(&self) -> Vec { - self.trackers - .iter() - .filter_map(|(id, t)| if t.has_pending() { Some(*id) } else { None }) - .collect() - } -} - -thread_local! { - static REGISTRY: RefCell = RefCell::new(StructureTrackerRegistry::default()); -} - -pub fn open_tab(tab_id: Uuid) { - REGISTRY.with(|r| r.borrow_mut().open_tab(tab_id)); -} - -pub fn close_tab(tab_id: Uuid) { - REGISTRY.with(|r| r.borrow_mut().close_tab(tab_id)); -} - -pub fn with_tab(tab_id: Uuid, f: F) -> R -where - F: FnOnce(&mut StructureChangeTracker) -> R, -{ - REGISTRY.with(|r| r.borrow_mut().with_tab(tab_id, f)) -} - -pub fn with_tab_ref(tab_id: Uuid, f: F) -> Option -where - F: FnOnce(&StructureChangeTracker) -> R, -{ - REGISTRY.with(|r| r.borrow().with_tab_ref(tab_id, f)) -} - -pub fn any_pending_globally() -> bool { - REGISTRY.with(|r| r.borrow().any_pending_globally()) -} - -pub fn pending_tabs() -> Vec { - REGISTRY.with(|r| r.borrow().pending_tabs()) } #[cfg(test)] @@ -172,41 +66,4 @@ mod tests { t.set_ops(vec![]); assert!(!t.has_pending()); } - - #[test] - fn materialize_emits_sql() { - let mut t = StructureChangeTracker::default(); - t.set_ops(vec![StructureOp::AddColumn { - schema: None, - table: "t".into(), - column: dc("a", "int"), - }]); - let sql = t - .materialize( - "postgres", - &drivers_postgres::PgDialect::new(&tablepro_core::ServerInfo::new("PostgreSQL", "18.0", 18, 0, 0)), - ) - .unwrap(); - assert_eq!(sql.len(), 1); - assert!(sql[0].contains("ALTER TABLE")); - } - - #[test] - fn registry_isolates_per_tab() { - let mut r = StructureTrackerRegistry::default(); - let a = Uuid::new_v4(); - let b = Uuid::new_v4(); - r.with_tab(a, |t| { - t.set_ops(vec![StructureOp::AddColumn { - schema: None, - table: "t".into(), - column: dc("a", "int"), - }]) - }); - assert!(r.with_tab_ref(a, |t| t.has_pending()).unwrap()); - assert!(!r.with_tab_ref(b, |t| t.has_pending()).unwrap_or(false)); - assert_eq!(r.pending_tabs(), vec![a]); - r.close_tab(a); - assert!(!r.any_pending_globally()); - } } diff --git a/linux/crates/app/src/services/structure_tracker_handle.rs b/linux/crates/app/src/services/structure_tracker_handle.rs new file mode 100644 index 0000000000..eab37083f9 --- /dev/null +++ b/linux/crates/app/src/services/structure_tracker_handle.rs @@ -0,0 +1,29 @@ +use std::cell::RefCell; +use std::rc::Rc; + +use tablepro_core::sql_ddl::StructureOp; + +use super::structure_tracker::StructureChangeTracker; + +/// A Structure tab's cached DDL. Every method borrows only for its own +/// body and returns owned data. +#[derive(Debug, Clone, Default)] +pub struct StructureTrackerHandle(Rc>); + +impl StructureTrackerHandle { + pub fn set_ops(&self, ops: Vec) { + self.0.borrow_mut().set_ops(ops); + } + + pub fn ops(&self) -> Vec { + self.0.borrow().ops().to_vec() + } + + pub fn has_pending(&self) -> bool { + self.0.borrow().has_pending() + } + + pub fn clear(&self) { + self.0.borrow_mut().clear(); + } +} diff --git a/linux/crates/app/src/services/structure_tracker_registry.rs b/linux/crates/app/src/services/structure_tracker_registry.rs new file mode 100644 index 0000000000..45b9122ab4 --- /dev/null +++ b/linux/crates/app/src/services/structure_tracker_registry.rs @@ -0,0 +1,61 @@ +use super::structure_tracker_handle::StructureTrackerHandle; +use super::tab_registry::{TabRegistry, TrackerHandle}; + +pub type StructureTrackerRegistry = TabRegistry; + +impl TrackerHandle for StructureTrackerHandle { + fn has_pending(&self) -> bool { + StructureTrackerHandle::has_pending(self) + } +} + +#[cfg(test)] +mod tests { + use tablepro_core::sql_ddl::{DraftColumn, StructureOp}; + use uuid::Uuid; + + use super::*; + + fn add_column() -> StructureOp { + StructureOp::AddColumn { + schema: None, + table: "t".into(), + column: DraftColumn { + original: None, + name: "a".into(), + data_type: "int".into(), + nullable: true, + primary_key: false, + auto_increment: false, + default_value: None, + comment: None, + }, + } + } + + #[test] + fn registries_isolated_per_workspace() { + let first = StructureTrackerRegistry::default(); + let second = StructureTrackerRegistry::default(); + let tab = Uuid::new_v4(); + + first.register(tab).set_ops(vec![add_column()]); + second.register(tab); + + assert_eq!(first.dirty(), vec![tab]); + assert!(second.dirty().is_empty()); + } + + #[test] + fn dirty_lists_only_pending_tabs() { + let registry = StructureTrackerRegistry::default(); + let edited = Uuid::new_v4(); + registry.register(edited).set_ops(vec![add_column()]); + registry.register(Uuid::new_v4()); + + assert_eq!(registry.dirty(), vec![edited]); + + registry.get(edited).expect("registered").clear(); + assert!(registry.dirty().is_empty()); + } +} diff --git a/linux/crates/app/src/services/tab_registry.rs b/linux/crates/app/src/services/tab_registry.rs new file mode 100644 index 0000000000..26c2642b75 --- /dev/null +++ b/linux/crates/app/src/services/tab_registry.rs @@ -0,0 +1,52 @@ +use std::cell::RefCell; +use std::collections::HashMap; + +use uuid::Uuid; + +/// A tab's tracker as a workspace registry holds it. +pub trait TrackerHandle: Clone + Default { + fn has_pending(&self) -> bool; +} + +/// One workspace's trackers, keyed by tab. Each window owns its own, so +/// edits in one window never count as unsaved in another. +#[derive(Debug)] +pub struct TabRegistry { + trackers: RefCell>, +} + +impl Default for TabRegistry { + fn default() -> Self { + Self { + trackers: RefCell::new(HashMap::new()), + } + } +} + +impl TabRegistry { + /// The tab's tracker, created on first use. + pub fn register(&self, tab_id: Uuid) -> H { + self.trackers.borrow_mut().entry(tab_id).or_default().clone() + } + + pub fn unregister(&self, tab_id: Uuid) { + self.trackers.borrow_mut().remove(&tab_id); + } + + pub fn get(&self, tab_id: Uuid) -> Option { + self.trackers.borrow().get(&tab_id).cloned() + } + + /// Tabs with unsaved changes, in a stable order. + pub fn dirty(&self) -> Vec { + let mut dirty: Vec = self + .trackers + .borrow() + .iter() + .filter(|(_, tracker)| tracker.has_pending()) + .map(|(tab_id, _)| *tab_id) + .collect(); + dirty.sort_unstable(); + dirty + } +} diff --git a/linux/crates/app/src/services/tab_tracker_handle.rs b/linux/crates/app/src/services/tab_tracker_handle.rs new file mode 100644 index 0000000000..b30b133d5d --- /dev/null +++ b/linux/crates/app/src/services/tab_tracker_handle.rs @@ -0,0 +1,132 @@ +use std::cell::RefCell; +use std::rc::Rc; + +use tablepro_core::dml::BuildSqlError; +use tablepro_core::{ColumnInfo, Value}; + +use super::cell_overlay::CellOverlay; +use super::change_tracker::{DraftRow, RowKey, RowState, TabChangeTracker, TrackedChanges}; +use super::tracker_delta::TrackerDelta; +use super::tracker_edit::TrackerEdit; + +/// A browse tab's tracker. Each method borrows only for its own body +/// and hands back owned data, so a grid bind that runs while the +/// caller is still applying a delta never meets a held borrow. +#[derive(Debug, Clone, Default)] +pub struct TabTrackerHandle(Rc>); + +impl TabTrackerHandle { + pub fn cell_overlay(&self, row_key: &RowKey, col: usize, original: &Value) -> CellOverlay { + self.0.borrow().cell_overlay(row_key, col, original) + } + + pub fn row_state(&self, row_key: &RowKey) -> RowState { + self.0.borrow().row_state(row_key) + } + + #[must_use = "the grid shows stale rows until the delta is applied"] + pub fn apply(&self, edit: TrackerEdit) -> TrackerDelta { + self.0.borrow_mut().apply(edit) + } + + #[must_use = "the grid shows stale rows until the delta is applied"] + pub fn undo(&self) -> Option { + self.0.borrow_mut().undo() + } + + #[must_use = "the grid shows stale rows until the delta is applied"] + pub fn redo(&self) -> Option { + self.0.borrow_mut().redo() + } + + pub fn pending_count(&self) -> usize { + self.0.borrow().pending_count() + } + + pub fn drafts(&self) -> Vec { + self.0.borrow().drafts().to_vec() + } + + pub fn change_set(&self, columns: &[ColumnInfo]) -> Result { + self.0.borrow().change_set(columns) + } + + #[must_use = "the grid shows stale rows until the delta is applied"] + pub fn discard(&self) -> TrackerDelta { + self.0.borrow_mut().discard() + } + + pub fn flash_error_row(&self, row_key: RowKey) -> u64 { + self.0.borrow_mut().flash_error_row(row_key) + } + + pub fn clear_error_row_if(&self, generation: u64) -> bool { + self.0.borrow_mut().clear_error_row_if(generation) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::services::tracker_delta::VisualChange; + + fn row(id: i64) -> RowKey { + RowKey::from_pk_values(&[Value::Int(id)]).expect("a key") + } + + #[test] + fn apply_returns_delta_with_pending_count() { + let tracker = TabTrackerHandle::default(); + + let edited = tracker.apply(TrackerEdit::Cell { + row_key: row(1), + col: 1, + original: Value::Text("a".into()), + value: Value::Text("b".into()), + }); + let inserted = tracker.apply(TrackerEdit::Insert { + values: vec![Value::Null, Value::Text("c".into())], + }); + + assert_eq!(edited.rows, vec![row(1)]); + assert_eq!(edited.pending_count, 1); + assert_eq!(inserted.pending_count, 2); + assert!(matches!( + inserted.visual.as_slice(), + [VisualChange::InsertDraft { values, .. }] if values[1] == Value::Text("c".into()) + )); + assert_eq!(tracker.pending_count(), 2); + } + + #[test] + fn clones_share_one_tracker() { + let tracker = TabTrackerHandle::default(); + let grid_copy = tracker.clone(); + + let _ = tracker.apply(TrackerEdit::Delete { + row_key: row(4), + original_values: vec![Value::Int(4)], + }); + + assert_eq!(grid_copy.row_state(&row(4)), RowState::PendingDelete); + } + + #[test] + fn a_read_inside_a_delta_loop_finds_no_held_borrow() { + let tracker = TabTrackerHandle::default(); + let delta = tracker.apply(TrackerEdit::Cell { + row_key: row(1), + col: 0, + original: Value::Int(1), + value: Value::Int(2), + }); + + for key in &delta.rows { + let _ = tracker.apply(TrackerEdit::Delete { + row_key: key.clone(), + original_values: vec![Value::Int(1)], + }); + assert_eq!(tracker.cell_overlay(key, 0, &Value::Int(1)).value, Value::Int(2)); + } + } +} diff --git a/linux/crates/app/src/services/tracker_delta.rs b/linux/crates/app/src/services/tracker_delta.rs new file mode 100644 index 0000000000..0e13a043ad --- /dev/null +++ b/linux/crates/app/src/services/tracker_delta.rs @@ -0,0 +1,22 @@ +use tablepro_core::Value; + +use super::change_tracker::RowKey; + +/// What a tracker change means for the grid, handed back once the +/// tracker is no longer borrowed so the caller can touch GTK freely. +#[derive(Debug, Clone, Default, PartialEq)] +pub struct TrackerDelta { + /// Rows whose pending state changed and need a rebind. + pub rows: Vec, + /// Draft rows the list store has to gain, lose or update, since a + /// draft's cells live on its row object. + pub visual: Vec, + pub pending_count: usize, +} + +#[derive(Debug, Clone, PartialEq)] +pub enum VisualChange { + SetDraftCell { draft_id: u64, col: usize, value: Value }, + RemoveDraft { draft_id: u64 }, + InsertDraft { draft_id: u64, values: Vec }, +} diff --git a/linux/crates/app/src/services/tracker_edit.rs b/linux/crates/app/src/services/tracker_edit.rs new file mode 100644 index 0000000000..d8eca5ddd1 --- /dev/null +++ b/linux/crates/app/src/services/tracker_edit.rs @@ -0,0 +1,29 @@ +use tablepro_core::Value; + +use super::change_tracker::RowKey; + +/// One change a browse tab records against its pending edits. +#[derive(Debug, Clone, PartialEq)] +pub enum TrackerEdit { + Cell { + row_key: RowKey, + col: usize, + original: Value, + value: Value, + }, + DraftCell { + draft_id: u64, + col: usize, + value: Value, + }, + Insert { + values: Vec, + }, + Delete { + row_key: RowKey, + original_values: Vec, + }, + DiscardDraft { + draft_id: u64, + }, +} diff --git a/linux/crates/app/src/ui/app/browse.rs b/linux/crates/app/src/ui/app/browse.rs index 3b6e5deee7..0ba5bf1c1e 100644 --- a/linux/crates/app/src/ui/app/browse.rs +++ b/linux/crates/app/src/ui/app/browse.rs @@ -200,7 +200,10 @@ impl App { let Some(id) = self.selected_browse_tab_id() else { return; }; - let dirty = crate::services::change_tracker::with_tab_ref(id, |tr| tr.has_pending()).unwrap_or(false); + let dirty = self + .change_trackers + .get(id) + .is_some_and(|tracker| tracker.pending_count() > 0); if !dirty { self.dispatch_to_tab(id, BrowseTabInput::Refresh); return; @@ -223,11 +226,10 @@ impl App { let workspace_tabs = self.workspace_tabs.clone(); dialog.connect_response(None, move |dlg: &adw::AlertDialog, response: &str| { dlg.close(); - if response == "discard" { - crate::services::change_tracker::with_tab(id, |t| t.clear()); - if let Some(controller) = workspace_tabs.borrow().get(&id).and_then(|t| t.browse_controller()) { - let _ = controller.sender().send(BrowseTabInput::Refresh); - } + if response == "discard" + && let Some(controller) = workspace_tabs.borrow().get(&id).and_then(|t| t.browse_controller()) + { + let _ = controller.sender().send(BrowseTabInput::DiscardAndRefresh); } }); dialog.present(Some(&self.window)); diff --git a/linux/crates/app/src/ui/app/connect_flow.rs b/linux/crates/app/src/ui/app/connect_flow.rs new file mode 100644 index 0000000000..6701894e9d --- /dev/null +++ b/linux/crates/app/src/ui/app/connect_flow.rs @@ -0,0 +1,411 @@ +//! The work behind a connect the window starts, run off the GTK thread. +//! +//! Every step stops at the attempt's cancel: a cancelled connect asks +//! for nothing more and adds no entry or secret. + +use tablepro_core::DriverRegistry; +use tablepro_core::credentials::{CredentialInteraction, SecretError, SecretKind, SecretVault}; +use tablepro_session::connect::ConnectionWrites; +use tablepro_session::{ + ConnectError, ConnectRequest, Connector, CredentialSource, PendingSave, PendingSession, PersistReport, +}; +use tablepro_storage::{CredentialDemand, KeyringCredentials, SavedConnection, StorageError}; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +/// Why a submitted connection did not open. +#[derive(Debug)] +pub(super) enum SubmitError { + Connect(ConnectError), + Save(StorageError), +} + +/// Connect to what the dialog was filled in with, then save it. The +/// entry is only written once the connect works, so a typo never lands +/// in the connection list. +pub(super) async fn connect_then_save( + connector: &dyn Connector, + request: ConnectRequest, + interaction: CredentialInteraction, + cancel: CancellationToken, + secrets: &dyn SecretVault, + writes: &dyn ConnectionWrites, + save: PendingSave, +) -> Result<(PendingSession, PersistReport), SubmitError> { + let pending = connector + .establish(request, interaction, cancel.clone()) + .await + .map_err(SubmitError::Connect)?; + let pending = unless_cancelled(pending, &cancel).await.map_err(SubmitError::Connect)?; + let is_new = save.is_new; + let report = match tablepro_session::persist_pending_save(secrets, writes, save).await { + Ok(report) => report, + Err(error) => { + let report = pending.discard().await; + tracing::debug!(?report, "closed a connection whose entry could not be saved"); + return Err(SubmitError::Save(error)); + } + }; + // A keyring unlock prompt can hold the save up long enough for the + // user to cancel. An overwritten entry has no earlier version left + // to restore, so only a new one is taken back. + if is_new && cancel.is_cancelled() { + unsave(secrets, writes, report.saved.id).await; + } + unless_cancelled(pending, &cancel) + .await + .map(|pending| (pending, report)) + .map_err(SubmitError::Connect) +} + +async fn unsave(secrets: &dyn SecretVault, writes: &dyn ConnectionWrites, id: Uuid) { + // A secret with no entry is orphaned in the keyring, while an + // entry with no secrets only asks for them again. + if let Err(error) = secrets.delete_connection(id).await { + tracing::warn!(%error, "kept a cancelled connect's entry, whose secrets could not be deleted"); + return; + } + if let Err(error) = writes.remove(id).await { + tracing::warn!(%error, "a cancelled connect's entry is still saved"); + } +} + +/// A secret typed into a prompt is kept only once the login it was +/// typed for has worked. +pub(super) async fn connect_saved( + saved: &SavedConnection, + drivers: &DriverRegistry, + credentials: &KeyringCredentials, + connector: &dyn Connector, + interaction: CredentialInteraction, + cancel: CancellationToken, +) -> Result<(PendingSession, Vec<(SecretKind, SecretError)>), ConnectError> { + // Dropping the resolve on a cancel drops its prompt too, which is + // what takes a password dialog still on screen down with it. + let resolving = tablepro_session::resolve_saved( + saved, + drivers, + credentials, + interaction.clone(), + CredentialDemand::default(), + ); + let refreshed = cancel + .run_until_cancelled(resolving) + .await + .ok_or(ConnectError::Cancelled)??; + let pending = connector + .establish(refreshed.request, interaction, cancel.clone()) + .await?; + let pending = unless_cancelled(pending, &cancel).await?; + let refused = credentials.remember(refreshed.remember).await; + Ok((pending, refused)) +} + +/// A driver blocked in a C call can still connect after the user +/// cancelled. That connection is closed here rather than saved or kept. +async fn unless_cancelled(pending: PendingSession, cancel: &CancellationToken) -> Result { + if !cancel.is_cancelled() { + return Ok(pending); + } + let report = pending.discard().await; + tracing::debug!( + ?report, + "closed a connection that arrived after its connect was cancelled" + ); + Err(ConnectError::Cancelled) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::time::Duration; + + use futures::future::BoxFuture; + use secrecy::SecretString; + use tablepro_core::credentials::{CredentialPrompter, SecretSave}; + use tablepro_core::{AuthMode, TlsConfig}; + use tablepro_session::connect::StoreConnectionWrites; + use tablepro_session::fakes::{FakeConnector, FakeEstablish}; + use tablepro_storage::{ConnectionStore, StoragePaths}; + use tablepro_test_support::{ConnectionScript, FakeDriver, FakeSecretVault}; + + use super::*; + use crate::ui::credential_prompt::GtkCredentialPrompter; + + const LATE: Duration = Duration::from_secs(5); + + fn connection(id: Uuid) -> SavedConnection { + SavedConnection { + id, + name: "Prod".to_owned(), + driver_id: "fake".to_owned(), + host: "db.example.com".to_owned(), + port: 5432, + database: "shop".to_owned(), + username: "app".to_owned(), + tls: TlsConfig::verify_full(), + read_only: false, + auth_mode: AuthMode::Password, + ssh: None, + last_opened_at: None, + color: None, + group: None, + } + } + + fn save(id: Uuid) -> PendingSave { + PendingSave { + connection: connection(id), + secrets: vec![SecretSave { + kind: SecretKind::DatabasePassword, + secret: SecretString::from("s3cret"), + label: "Prod".to_owned(), + }], + forget: Vec::new(), + is_new: true, + } + } + + fn late_connector() -> FakeConnector { + FakeConnector::new([FakeEstablish::Late { + after: LATE, + script: ConnectionScript::default(), + }]) + } + + struct CancelledWhileSaving { + writes: StoreConnectionWrites, + cancel: CancellationToken, + } + + impl ConnectionWrites for CancelledWhileSaving { + fn upsert(&self, connection: SavedConnection) -> BoxFuture<'_, Result<(), StorageError>> { + self.cancel.cancel(); + self.writes.upsert(connection) + } + + fn remove(&self, id: Uuid) -> BoxFuture<'_, Result<(), StorageError>> { + self.writes.remove(id) + } + } + + fn drivers() -> DriverRegistry { + let mut drivers = DriverRegistry::new(); + drivers.register(Arc::new(FakeDriver::connecting( + &FakeDriver::NETWORK_CAPABILITIES, + ConnectionScript::default(), + ))); + drivers + } + + #[tokio::test(start_paused = true)] + async fn a_connect_that_lands_after_its_cancel_saves_nothing() { + let root = tempfile::tempdir().expect("a temporary directory"); + let store = ConnectionStore::new(&StoragePaths::under(root.path(), "tablepro-test", "app.tablepro.Test")); + let writes = StoreConnectionWrites::new(store.clone(), crate::test_support::paused_tasks()); + let vault = FakeSecretVault::new(); + let connector = late_connector(); + let cancel = CancellationToken::new(); + let request = FakeConnector::request().expect("a request"); + + let connecting = connect_then_save( + &connector, + request, + CredentialInteraction::Unattended, + cancel.clone(), + &vault, + &writes, + save(Uuid::new_v4()), + ); + cancel.cancel(); + let outcome = connecting.await; + + assert!( + matches!(outcome, Err(SubmitError::Connect(ConnectError::Cancelled))), + "{outcome:?}" + ); + assert!(vault.is_empty(), "a cancelled connect stored its secrets"); + assert!( + store.load_blocking().expect("the list").is_empty(), + "a cancelled connect saved its entry" + ); + let built = connector.connections(); + assert_eq!(built.len(), 1); + assert!(built[0].is_closed(), "the late connection was left open"); + } + + #[tokio::test(start_paused = true)] + async fn a_connect_that_is_not_cancelled_saves_its_entry_and_secrets() { + let root = tempfile::tempdir().expect("a temporary directory"); + let store = ConnectionStore::new(&StoragePaths::under(root.path(), "tablepro-test", "app.tablepro.Test")); + let writes = StoreConnectionWrites::new(store.clone(), crate::test_support::paused_tasks()); + let vault = FakeSecretVault::new(); + let connector = late_connector(); + let id = Uuid::new_v4(); + + let (pending, report) = connect_then_save( + &connector, + FakeConnector::request().expect("a request"), + CredentialInteraction::Unattended, + CancellationToken::new(), + &vault, + &writes, + save(id), + ) + .await + .expect("a saved connection"); + + assert_eq!(report.saved.id, id); + assert!(!vault.is_empty()); + assert_eq!(store.load_blocking().expect("the list").len(), 1); + let _closed = pending.discard().await; + } + + #[tokio::test(start_paused = true)] + async fn a_new_entry_saved_as_its_connect_is_cancelled_is_taken_back() { + let root = tempfile::tempdir().expect("a temporary directory"); + let store = ConnectionStore::new(&StoragePaths::under(root.path(), "tablepro-test", "app.tablepro.Test")); + let cancel = CancellationToken::new(); + let writes = CancelledWhileSaving { + writes: StoreConnectionWrites::new(store.clone(), crate::test_support::paused_tasks()), + cancel: cancel.clone(), + }; + let vault = FakeSecretVault::new(); + let connector = FakeConnector::new([FakeEstablish::Connect(ConnectionScript::default())]); + + let outcome = connect_then_save( + &connector, + FakeConnector::request().expect("a request"), + CredentialInteraction::Unattended, + cancel, + &vault, + &writes, + save(Uuid::new_v4()), + ) + .await; + + assert!( + matches!(outcome, Err(SubmitError::Connect(ConnectError::Cancelled))), + "{outcome:?}" + ); + assert!(vault.is_empty(), "the cancelled connect's secrets stayed"); + assert!( + store.load_blocking().expect("the list").is_empty(), + "the cancelled connect's entry stayed" + ); + let built = connector.connections(); + assert_eq!(built.len(), 1); + assert!(built[0].is_closed(), "the connection was left open"); + } + + #[tokio::test(start_paused = true)] + async fn an_existing_entry_saved_as_its_connect_is_cancelled_keeps_the_edit() { + let root = tempfile::tempdir().expect("a temporary directory"); + let store = ConnectionStore::new(&StoragePaths::under(root.path(), "tablepro-test", "app.tablepro.Test")); + let id = Uuid::new_v4(); + store.upsert_blocking(connection(id)).expect("the saved entry"); + let cancel = CancellationToken::new(); + let writes = CancelledWhileSaving { + writes: StoreConnectionWrites::new(store.clone(), crate::test_support::paused_tasks()), + cancel: cancel.clone(), + }; + let vault = FakeSecretVault::new(); + let connector = FakeConnector::new([FakeEstablish::Connect(ConnectionScript::default())]); + let mut edit = save(id); + edit.connection.name = "Prod (edited)".to_owned(); + edit.is_new = false; + + let outcome = connect_then_save( + &connector, + FakeConnector::request().expect("a request"), + CredentialInteraction::Unattended, + cancel, + &vault, + &writes, + edit, + ) + .await; + + assert!( + matches!(outcome, Err(SubmitError::Connect(ConnectError::Cancelled))), + "{outcome:?}" + ); + let list = store.load_blocking().expect("the list"); + assert_eq!(list.len(), 1); + assert_eq!(list[0].name, "Prod (edited)"); + assert!(!vault.is_empty(), "the entry lost its secrets"); + assert!(connector.connections()[0].is_closed(), "the connection was left open"); + } + + #[tokio::test(start_paused = true)] + async fn cancelling_while_a_password_is_asked_for_ends_the_prompt() { + let saved = connection(Uuid::new_v4()); + let credentials = KeyringCredentials::for_saved(Arc::new(FakeSecretVault::new()), &saved); + let (prompter, mut queue) = GtkCredentialPrompter::new(); + let prompter: Arc = Arc::new(prompter); + let connector = FakeConnector::new([FakeEstablish::Connect(ConnectionScript::default())]); + let cancel = CancellationToken::new(); + let drivers = drivers(); + + let connecting = connect_saved( + &saved, + &drivers, + &credentials, + &connector, + CredentialInteraction::Attended(prompter), + cancel.clone(), + ); + let asking = async { + let shown = queue.next().await.expect("the password prompt"); + cancel.cancel(); + shown + }; + let (outcome, shown) = tokio::time::timeout(LATE, async { tokio::join!(connecting, asking) }) + .await + .expect("the cancel did not end the connect while its prompt was up"); + + assert!(matches!(outcome, Err(ConnectError::Cancelled)), "{outcome:?}"); + assert!(shown.reply.is_closed(), "the prompt is still waited on"); + assert!(connector.calls().is_empty(), "a cancelled attempt still connected"); + } + + #[tokio::test(start_paused = true)] + async fn a_saved_connect_that_lands_after_its_cancel_keeps_no_typed_secret() { + let saved = connection(Uuid::new_v4()); + let vault = Arc::new(FakeSecretVault::new()); + let credentials = KeyringCredentials::for_saved(vault.clone(), &saved); + let (prompter, mut queue) = GtkCredentialPrompter::new(); + let prompter: Arc = Arc::new(prompter); + let connector = late_connector(); + let cancel = CancellationToken::new(); + let drivers = drivers(); + + let connecting = connect_saved( + &saved, + &drivers, + &credentials, + &connector, + CredentialInteraction::Attended(prompter), + cancel.clone(), + ); + let answering = async { + let shown = queue.next().await.expect("the password prompt"); + shown + .reply + .send(tablepro_core::credentials::PromptReply::Submitted { + values: vec![SecretString::from("typed")], + remember: true, + }) + .expect("the connect is waiting"); + tokio::time::sleep(LATE / 2).await; + cancel.cancel(); + }; + let (outcome, ()) = tokio::join!(connecting, answering); + + assert!(matches!(outcome, Err(ConnectError::Cancelled)), "{outcome:?}"); + assert!(vault.is_empty(), "a cancelled connect remembered the typed password"); + let built = connector.connections(); + assert_eq!(built.len(), 1); + assert!(built[0].is_closed(), "the late connection was left open"); + } +} diff --git a/linux/crates/app/src/ui/app/connection.rs b/linux/crates/app/src/ui/app/connection.rs index d5eb2e0c55..080df39e5c 100644 --- a/linux/crates/app/src/ui/app/connection.rs +++ b/linux/crates/app/src/ui/app/connection.rs @@ -6,22 +6,23 @@ use relm4::{ComponentController, ComponentSender, adw, gtk}; use tablepro_core::meta::ObjectInfo; use tablepro_session::connect::AttemptGeneration; -use tablepro_session::runtime::WatchForwarder; +use tablepro_session::runtime::{ProcessHold, WatchForwarder}; use tablepro_session::{ - ClaimError, ConnectError, CredentialSource, MonitorSignal, PendingSession, Session, SessionClaim, SessionHealth, - SessionKey, + ClaimError, ConnectAttempts, ConnectError, CredentialSource, MonitorSignal, PendingSession, Session, SessionClaim, + SessionHealth, SessionKey, }; -use tablepro_storage::{CredentialDemand, SavedConnection}; +use tablepro_storage::SavedConnection; use uuid::Uuid; use tablepro_session::RequestTarget; use tablepro_session::connect::StoreConnectionWrites; -use tablepro_storage::StorageError; +use crate::services::active_session::ActiveSession; use crate::ui::connect_dialog::{ConnectDialog, ConnectDialogInit, ConnectPrefill, ConnectSubmission, FormField}; use crate::ui::error_text::TlsRemedy; use crate::ui::ssh_inputs::SshEnvironment; +use super::connect_flow::{self, SubmitError}; use super::teardown_guard::Teardown; use super::{App, AppMsg, qualified_label}; @@ -33,12 +34,6 @@ pub(crate) struct ConnectRetry { pub remedy: Option, } -/// Why a submitted connection did not open. -enum SubmitError { - Connect(ConnectError), - Save(StorageError), -} - /// The password the user typed, so editing after a failure does not /// make them type it again. It stays in memory. fn typed_password(target: &RequestTarget) -> Option { @@ -158,18 +153,16 @@ impl App { let secrets = self.context.secrets.clone(); let writes = StoreConnectionWrites::new(self.storage.connections().clone(), self.tasks.clone()); let connecting = async move { - let pending = connector - .establish(request, interaction, cancel) - .await - .map_err(SubmitError::Connect)?; - match tablepro_session::persist_pending_save(&*secrets, &writes, pending_save).await { - Ok(report) => Ok((pending, report)), - Err(error) => { - let report = pending.discard().await; - tracing::debug!(?report, "closed a connection whose entry could not be saved"); - Err(SubmitError::Save(error)) - } - } + connect_flow::connect_then_save( + &*connector, + request, + interaction, + cancel, + &*secrets, + &writes, + pending_save, + ) + .await }; let context = self.context.clone(); let settings = self.settings.clone(); @@ -225,12 +218,8 @@ impl App { }); } - /// The Connecting toast's Cancel. The attempt is superseded, not - /// only cancelled: a driver blocked in a socket call can still - /// connect afterwards, and that late session must not be adopted. pub(super) fn on_cancel_connect(&mut self) { - drop(self.connect_attempts.begin()); - self.connect_attempts.cancel(); + stop_connecting(&self.connect_attempts); self.dismiss_loading_page(); } @@ -437,17 +426,7 @@ impl App { // different connection would target a non-existent table. self.clear_closed_tabs_stack(); self.session_links = None; - if let Some(session) = self.context.active.take() { - let hold = self.context.hold.acquire(); - let closing = session.begin_shutdown(); - glib::spawn_future_local(async move { - match closing.await { - Ok(report) => tracing::debug!(?report, "the session closed"), - Err(failure) => tracing::warn!(%failure, "the session close task failed"), - } - drop(hold); - }); - } + close_session(&self.context.active, &*self.context.hold); self.reconnect_banner.set_revealed(false); self.schema_buffer.set_text(crate::ui::editor::SQL_KEYWORDS); self.current_driver_id = None; @@ -810,19 +789,7 @@ impl App { // Reading the keyring, asking for what it lacks and connecting // all happen off the GTK thread; only the outcome comes back. let connecting = async move { - let refreshed = tablepro_session::resolve_saved( - &saved, - &drivers, - &credentials, - interaction.clone(), - CredentialDemand::default(), - ) - .await?; - let pending = connector.establish(refreshed.request, interaction, cancel).await?; - // A secret typed into a prompt is kept only once the login - // it was typed for has worked. - let refused = credentials.remember(refreshed.remember).await; - Ok::<_, ConnectError>((pending, refused)) + connect_flow::connect_saved(&saved, &drivers, &credentials, &*connector, interaction, cancel).await }; crate::runtime::spawn_and_deliver(&self.tasks, connecting, move |outcome| match outcome { Ok(Ok((pending, refused))) => sender.input(AppMsg::SessionReady(Box::new(ReadySession { @@ -911,7 +878,7 @@ impl App { // title with "• " when any open document has unsaved changes, // so the dirty state is visible from the Activities overview / // Alt-Tab without needing the tab to be focused. - if crate::services::change_tracker::any_pending_globally() { + if !self.dirty_tabs().is_empty() { os_title = format!("• {os_title}"); } self.window.set_title(Some(&os_title)); @@ -932,9 +899,44 @@ impl App { } } -/// Performs the actual disk + keyring teardown for a saved connection. -/// Extracted from `on_delete_connection` so the confirm-yes branch and -/// the prefs-disabled branch share one implementation. +/// Supersede the connect in flight, not only cancel it: a driver +/// blocked in a socket call can still connect afterwards, and that late +/// session must not be adopted. +fn stop_connecting(attempts: &ConnectAttempts) { + drop(attempts.begin()); + attempts.cancel(); +} + +/// Shut the window's session down and close its connection under the +/// close deadline. The process is held until the close ends, so the +/// server sees the connection closed rather than cut off. +fn close_session(active: &ActiveSession, hold: &dyn ProcessHold) { + let Some(session) = active.take() else { + return; + }; + let hold = hold.acquire(); + let closing = session.begin_shutdown(); + glib::spawn_future_local(async move { + match closing.await { + Ok(report) => tracing::debug!(?report, "the session closed"), + Err(failure) => tracing::warn!(%failure, "the session close task failed"), + } + drop(hold); + }); +} + +/// What a closing window still owes its connection: no connect left to +/// land in it, and the session closed rather than dropped. +/// +/// The window's close request is where this has to start. relm4 calls +/// a detached root's `shutdown` only after the main loop has returned, +/// when no close can run any more and a hold no longer keeps the +/// process alive. +pub(super) fn close_window_session(active: &ActiveSession, hold: &dyn ProcessHold, attempts: &ConnectAttempts) { + stop_connecting(attempts); + close_session(active, hold); +} + /// Move every secret the original holds onto the copy's own id. /// /// Returns whether all of them made it: a keyring that is locked or @@ -1138,11 +1140,55 @@ fn failed_banner_text(cause: tablepro_session::FailureCause) -> String { #[cfg(test)] mod tests { + use std::time::Duration; + + use tablepro_core::LivenessPolicy; use tablepro_core::meta::ObjectKind; - use tablepro_session::FailureCause; + use tablepro_session::fakes::{FakeConnector, FakeCredentialSource}; + use tablepro_session::operation::DefaultDeadlines; + use tablepro_session::runtime::CountingHold; + use tablepro_session::{FailureCause, SessionDeps}; + use tablepro_test_support::ConnectionScript; use super::*; + #[gtk4::test] + fn a_closing_window_closes_its_session_and_stops_its_connect() { + let runtime = crate::test_support::test_runtime(); + let (pending, connection) = FakeConnector::pending_session( + FakeConnector::request().expect("a request"), + ConnectionScript::default(), + ); + let session = Session::start( + pending, + SessionDeps { + connector: Arc::new(FakeConnector::new([])), + credentials: Arc::new(FakeCredentialSource::new()), + deadlines: Arc::new(DefaultDeadlines::new(LivenessPolicy::DESKTOP)), + tasks: runtime.tasks(), + prompter: None, + }, + ); + let active = ActiveSession::new(); + active.replace(session.clone()); + let hold = CountingHold::new(); + let attempts = ConnectAttempts::new(); + let connecting = attempts.begin(); + + close_window_session(&active, &hold, &attempts); + + assert!(connecting.is_cancelled(), "the connect in flight kept running"); + assert!( + !attempts.accepts(connecting.generation()), + "a late connect would still be adopted" + ); + assert!(!active.is_connected()); + assert_eq!(hold.live(), 1, "nothing kept the process alive for the close"); + crate::test_support::wait_until(Duration::from_secs(5), || hold.live() == 0).expect("the close to end"); + assert!(session.is_shut_down()); + assert!(connection.is_closed(), "the connection was cut off rather than closed"); + } + fn object(name: &str, kind: ObjectKind) -> ObjectInfo { ObjectInfo { schema: Some("public".to_owned()), diff --git a/linux/crates/app/src/ui/app/mod.rs b/linux/crates/app/src/ui/app/mod.rs index 0221d66c43..5923ef044a 100644 --- a/linux/crates/app/src/ui/app/mod.rs +++ b/linux/crates/app/src/ui/app/mod.rs @@ -1,4 +1,5 @@ mod browse; +mod connect_flow; mod connection; mod import; mod ran_statements; @@ -29,7 +30,9 @@ use super::editor::{SqlEditor, build_schema_buffer}; use super::history_dialog::HistoryDialog; use super::sidebar_row::{SidebarRow, SidebarRowOutput}; use super::welcome_view::{WelcomeView, WelcomeViewInit, WelcomeViewOutput}; +use crate::services::change_tracker_registry::ChangeTrackerRegistry; use crate::services::session_context::SessionContext; +use crate::services::structure_tracker_registry::StructureTrackerRegistry; pub(crate) use connection::{ConnectRetry, ReadySession}; @@ -72,7 +75,8 @@ pub struct App { /// Ends the credential prompt loop with the window. prompt_presenter: glib::JoinHandle<()>, /// Which connect attempt is current, so a late result is discarded. - connect_attempts: tablepro_session::ConnectAttempts, + /// Shared with the close handler, which stops the one in flight. + connect_attempts: std::rc::Rc, settings: std::rc::Rc, storage: crate::storage::SharedStorage, tasks: tablepro_session::runtime::Tasks, @@ -119,6 +123,10 @@ pub struct App { /// HashMap for O(1) tab_id lookup; display order is read from /// `tab_view.pages()` since HashMap is unordered. workspace_tabs: std::rc::Rc>>, + /// This window's pending row edits, one tracker per Table tab. + change_trackers: std::rc::Rc, + /// This window's pending DDL, one tracker per Structure tab. + structure_trackers: std::rc::Rc, dialog: Option, schema_buffer: gtk::TextBuffer, history_dialog: Option>, @@ -634,7 +642,20 @@ impl StatusKind { } } +/// Tabs in either registry with unsaved changes. +fn dirty_tabs(changes: &ChangeTrackerRegistry, structures: &StructureTrackerRegistry) -> Vec { + let mut dirty = changes.dirty(); + dirty.extend(structures.dirty()); + dirty.sort_unstable(); + dirty.dedup(); + dirty +} + impl App { + pub(super) fn dirty_tabs(&self) -> Vec { + dirty_tabs(&self.change_trackers, &self.structure_trackers) + } + /// The active driver id, or "postgres" if no connection is active. /// /// Single fallback site (was duplicated at 7 call sites). The @@ -851,7 +872,14 @@ impl SimpleComponent for App { let close_window_after_save_handle = close_window_after_save_for_close.clone(); let in_flight_saves_handle = in_flight_saves.clone(); let in_flight_saves_for_close = in_flight_saves.clone(); + let change_trackers = std::rc::Rc::new(ChangeTrackerRegistry::default()); + let structure_trackers = std::rc::Rc::new(StructureTrackerRegistry::default()); + let change_trackers_for_close = change_trackers.clone(); + let structure_trackers_for_close = structure_trackers.clone(); let close_request_input_sender = sender.input_sender().clone(); + let connect_attempts = std::rc::Rc::new(tablepro_session::ConnectAttempts::new()); + let context_for_close = context.clone(); + let connect_attempts_for_close = connect_attempts.clone(); widgets.window.connect_close_request(move |w| { // If a Save is mid-flight (async transaction running), block // the close until it resolves. Without this, the completion @@ -887,8 +915,7 @@ impl SimpleComponent for App { // below) — skip the guard, save state, allow close. // Browse + Structure tabs share the dirty-state guard: // either source of pending changes triggers the dialog. - let has_pending = crate::services::change_tracker::any_pending_globally() - || crate::services::structure_tracker::any_pending_globally(); + let has_pending = !dirty_tabs(&change_trackers_for_close, &structure_trackers_for_close).is_empty(); if !force_close_for_close.get() && has_pending { // Plural-form heading matches the per-tab dialog's // tone — factual GNOME HIG language rather than the @@ -913,15 +940,23 @@ impl SimpleComponent for App { let close_after_save_for_resp = close_after_save_for_close.clone(); let close_window_after_save_for_resp = close_window_after_save_for_close.clone(); let input_sender_for_resp = close_request_input_sender.clone(); + let change_trackers_for_resp = change_trackers_for_close.clone(); + let structure_trackers_for_resp = structure_trackers_for_close.clone(); dialog.connect_response(None, move |dlg, response| { dlg.close(); match response { "discard" => { - for tab_id in crate::services::change_tracker::pending_tabs() { - crate::services::change_tracker::with_tab(tab_id, |t| t.clear()); + // The window closes next, so no grid is left + // to show the discarded rows. + for tab_id in change_trackers_for_resp.dirty() { + if let Some(tracker) = change_trackers_for_resp.get(tab_id) { + let _ = tracker.discard(); + } } - for tab_id in crate::services::structure_tracker::pending_tabs() { - crate::services::structure_tracker::with_tab(tab_id, |t| t.clear()); + for tab_id in structure_trackers_for_resp.dirty() { + if let Some(tracker) = structure_trackers_for_resp.get(tab_id) { + tracker.clear(); + } } force_close_for_resp.set(true); // Re-fire close_request — guard sees the flag, @@ -936,8 +971,8 @@ impl SimpleComponent for App { // the SaveCompletedForTab / StructureSaveCompleted // handlers in App::update close the window once // the set drains. Any SaveFailed aborts. - let browse_tabs: Vec = crate::services::change_tracker::pending_tabs(); - let structure_tabs: Vec = crate::services::structure_tracker::pending_tabs(); + let browse_tabs = change_trackers_for_resp.dirty(); + let structure_tabs = structure_trackers_for_resp.dirty(); // Counter increments — a Table tab listed in both // sets bumps to 2 so the window close waits for // BOTH the browse save and the structure save. @@ -965,6 +1000,11 @@ impl SimpleComponent for App { return glib::Propagation::Stop; } crate::ui::window_geometry::persist(w, &settings_for_close); + connection::close_window_session( + &context_for_close.active, + &*context_for_close.hold, + &connect_attempts_for_close, + ); // The last column drag or filter change may still be on its // way to disk. The hold keeps the process alive until it // lands, so quitting does not lose it. @@ -1281,7 +1321,7 @@ impl SimpleComponent for App { context, session_links: None, prompt_presenter, - connect_attempts: tablepro_session::ConnectAttempts::new(), + connect_attempts, settings: settings.clone(), storage: storage.clone(), tasks: tasks.clone(), @@ -1311,6 +1351,8 @@ impl SimpleComponent for App { workspace_tab_view: None, workspace_root_added: std::cell::Cell::new(false), workspace_tabs: std::rc::Rc::new(std::cell::RefCell::new(std::collections::HashMap::new())), + change_trackers, + structure_trackers, dialog: None, schema_buffer: build_schema_buffer(), history_dialog: None, @@ -1363,6 +1405,9 @@ impl SimpleComponent for App { self.history_gate.abort(); self.prompt_presenter.abort(); self.session_links = None; + // The close request already did this; a window destroyed + // without one still stops its session's work here. + connection::close_window_session(&self.context.active, &*self.context.hold, &self.connect_attempts); } fn update(&mut self, msg: Self::Input, sender: ComponentSender) { diff --git a/linux/crates/app/src/ui/app/row_ops.rs b/linux/crates/app/src/ui/app/row_ops.rs index 941171ac80..4b009346fa 100644 --- a/linux/crates/app/src/ui/app/row_ops.rs +++ b/linux/crates/app/src/ui/app/row_ops.rs @@ -186,7 +186,9 @@ mod tests { use tablepro_core::column::{ColumnDefault, ColumnKind, ColumnType, IntegerKind, IntegerWidth}; use super::*; - use crate::services::change_tracker::{RowKey, TabChangeTracker}; + use crate::services::change_tracker::RowKey; + use crate::services::tab_tracker_handle::TabTrackerHandle; + use crate::services::tracker_edit::TrackerEdit; fn column(name: &str, primary_key: bool) -> ColumnInfo { let mut column_type = ColumnType::untyped_text("integer"); @@ -209,9 +211,16 @@ mod tests { fn planned() -> (SavePlan, TrackedChanges, RowKey) { let columns = vec![column("id", true), column("n", false)]; let edited = RowKey::from_pk_values(&[Value::Int(3)]).unwrap(); - let mut tracker = TabChangeTracker::new(); - tracker.track_insert(vec![Value::Int(10), Value::Int(1)]); - tracker.track_cell_edit(edited.clone(), 1, Value::Int(1), Value::Int(2)); + let tracker = TabTrackerHandle::default(); + let _ = tracker.apply(TrackerEdit::Insert { + values: vec![Value::Int(10), Value::Int(1)], + }); + let _ = tracker.apply(TrackerEdit::Cell { + row_key: edited.clone(), + col: 1, + original: Value::Int(1), + value: Value::Int(2), + }); let changes = tracker.change_set(&columns).unwrap(); let dialect = drivers_sqlite::SqliteDialect::new(drivers_sqlite::SqliteVersion::new(3, 45, 0)); let plan = plan_save( diff --git a/linux/crates/app/src/ui/app/structure.rs b/linux/crates/app/src/ui/app/structure.rs index 753ee2954c..31e1efe83d 100644 --- a/linux/crates/app/src/ui/app/structure.rs +++ b/linux/crates/app/src/ui/app/structure.rs @@ -23,7 +23,6 @@ use tablepro_core::{ForeignKeyMode, WriteBatch, WriteKind, WriteReport}; use tablepro_session::runtime::TaskFailure; use tablepro_session::{OperationError, OperationKind, Session}; -use crate::services::structure_tracker; use crate::ui::app::{App, AppMsg, WorkspaceTab}; use crate::ui::error_text; use crate::ui::structure_tab::{StructureMode, StructureTabInput}; @@ -180,24 +179,20 @@ impl App { )); return; }; - let driver_id = self.driver_id().to_string(); - let result = structure_tracker::with_tab_ref(tab_id, |t| t.materialize(&driver_id, &*dialect)); - match result { - Some(Ok(statements)) if !statements.is_empty() => { + let Some(tracker) = self.structure_trackers.get(tab_id) else { + return; + }; + match materialize_ops(&tracker.ops(), self.driver_id(), &*dialect) { + Ok(statements) if !statements.is_empty() => { self.on_execute_structure_transaction(tab_id, statements, sender); } - Some(Ok(_)) => { - // Nothing to save — short-circuit so close-after-save - // can proceed. - sender.input(AppMsg::StructureSaveCompleted { - tab_id, - new_table_name: None, - }); - } - Some(Err(e)) => { - sender.input(AppMsg::StructureSaveFailed(tab_id, format!("{e}"))); - } - None => {} + // Nothing to save, so a close waiting on this save can go + // ahead. + Ok(_) => sender.input(AppMsg::StructureSaveCompleted { + tab_id, + new_table_name: None, + }), + Err(e) => sender.input(AppMsg::StructureSaveFailed(tab_id, format!("{e}"))), } } @@ -276,7 +271,11 @@ impl App { } } }; - let ops = structure_tracker::with_tab_ref(tab_id, |t| t.ops().to_vec()).unwrap_or_default(); + let ops = self + .structure_trackers + .get(tab_id) + .map(|tracker| tracker.ops()) + .unwrap_or_default(); let new_table_name = saved_table_name(mode, &ops); // The batch names the table as it is before the write: a new // table by the name it is being created under. @@ -604,7 +603,10 @@ impl App { // Combine with the data-side dirty state — either // mode contributing pending changes prefixes the tab // with the "•" GNOME convention. - let data = crate::services::change_tracker::with_tab_ref(s.id, |tr| tr.has_pending()).unwrap_or(false); + let data = self + .change_trackers + .get(s.id) + .is_some_and(|tracker| tracker.pending_count() > 0); (&s.page, s.schema.as_deref(), s.table.clone(), dirty || data) } _ => return, diff --git a/linux/crates/app/src/ui/app/teardown_guard.rs b/linux/crates/app/src/ui/app/teardown_guard.rs index 941cfb0fc9..a83f17b8c3 100644 --- a/linux/crates/app/src/ui/app/teardown_guard.rs +++ b/linux/crates/app/src/ui/app/teardown_guard.rs @@ -3,8 +3,6 @@ use std::cell::Cell; use relm4::adw::prelude::*; use relm4::{ComponentSender, adw}; -use crate::services::{change_tracker, structure_tracker}; - use super::{App, AppMsg, WorkspaceTab}; /// What closing the workspace would throw away. @@ -41,7 +39,7 @@ pub(super) enum Teardown { impl App { pub(super) fn workspace_loss(&self) -> WorkspaceLoss { - let unsaved_edits = change_tracker::any_pending_globally() || structure_tracker::any_pending_globally(); + let unsaved_edits = !self.dirty_tabs().is_empty(); let open_transactions = self .workspace_tabs .borrow() diff --git a/linux/crates/app/src/ui/app/workspace_tabs.rs b/linux/crates/app/src/ui/app/workspace_tabs.rs index a6ac0f80ab..1f08564629 100644 --- a/linux/crates/app/src/ui/app/workspace_tabs.rs +++ b/linux/crates/app/src/ui/app/workspace_tabs.rs @@ -340,7 +340,7 @@ impl App { let tab_id = Uuid::new_v4(); let driver_id = self.driver_id().to_string(); let init = crate::ui::structure_tab::StructureTabInit { - tab_id, + tracker: self.structure_trackers.register(tab_id), schema: schema.clone(), table: String::new(), mode: crate::ui::structure_tab::StructureMode::New, @@ -418,7 +418,7 @@ impl App { let read_only = self.read_only; let browse_init = BrowseTabInit { - tab_id, + tracker: self.change_trackers.register(tab_id), schema: schema.clone(), table: table.clone(), connection_id, @@ -500,7 +500,7 @@ impl App { let driver_id = self.driver_id().to_string(); let mode = crate::ui::structure_tab::StructureMode::Edit; let init = crate::ui::structure_tab::StructureTabInit { - tab_id, + tracker: self.structure_trackers.register(tab_id), schema: schema.clone(), table: table.clone(), mode, @@ -621,28 +621,7 @@ impl App { return; }; - // If this is a Browse tab with pending changeset, intercept the - // close with an AdwAlertDialog (Discard / Cancel). The user can - // also cancel close, save manually, then close — no Save-and- - // close branch keeps the async commit logic out of this path. - let pending = self - .workspace_tabs - .borrow() - .get(&id) - .and_then(|t| match t { - WorkspaceTab::Structure(s) => { - crate::services::structure_tracker::with_tab_ref(s.id, |tr| tr.has_pending()) - } - WorkspaceTab::Table(s) => { - let data_dirty = - crate::services::change_tracker::with_tab_ref(s.id, |tr| tr.has_pending()).unwrap_or(false); - let struct_dirty = - crate::services::structure_tracker::with_tab_ref(s.id, |tr| tr.has_pending()).unwrap_or(false); - Some(data_dirty || struct_dirty) - } - _ => None, - }) - .unwrap_or(false); + let pending = self.dirty_tabs().contains(&id); let open_transaction = matches!( self.workspace_tabs.borrow().get(&id), Some(WorkspaceTab::Editor(slot)) if slot.editor.has_open_transaction() @@ -692,18 +671,23 @@ impl App { // branches so a Table tab whose only dirt is on the // Structure side doesn't try to save through the Browse // path (and vice versa). - let data_dirty = crate::services::change_tracker::with_tab_ref(id, |tr| tr.has_pending()).unwrap_or(false); - let struct_dirty = - crate::services::structure_tracker::with_tab_ref(id, |tr| tr.has_pending()).unwrap_or(false); + let data_tracker = self + .change_trackers + .get(id) + .filter(|tracker| tracker.pending_count() > 0); + let structure_tracker = self.structure_trackers.get(id).filter(|tracker| tracker.has_pending()); + let (data_dirty, struct_dirty) = (data_tracker.is_some(), structure_tracker.is_some()); dialog.connect_response(None, move |dlg, response| { dlg.close(); match response { "discard" => { - if data_dirty { - crate::services::change_tracker::with_tab(id, |t| t.clear()); + // The tab closes next, so no grid is left to + // show the discarded rows. + if let Some(tracker) = &data_tracker { + let _ = tracker.discard(); } - if struct_dirty { - crate::services::structure_tracker::with_tab(id, |t| t.clear()); + if let Some(tracker) = &structure_tracker { + tracker.clear(); } sender_for_resp.input(AppMsg::WorkspaceTabClosed(id)); } @@ -810,18 +794,11 @@ impl App { .discard(&tablepro_storage::DraftScope::default(), slot.draft); } WorkspaceTab::Structure(slot) => { - crate::services::structure_tracker::close_tab(slot.id); + self.structure_trackers.unregister(slot.id); self.structure_saves_in_flight.borrow_mut().remove(&slot.id); } WorkspaceTab::Table(slot) => { - // A Table tab owns BOTH a Browse-side row tracker and a - // Structure-side DDL tracker against the same uuid; close - // both registries. Also drop the in-flight-save guard - // entry so a re-opened tab with the same uuid (or a - // dropped async future from `drop_on_shutdown`) doesn't - // leave the save path permanently locked. - crate::services::change_tracker::close_tab(slot.id); - crate::services::structure_tracker::close_tab(slot.id); + self.change_trackers.unregister(slot.id); self.structure_saves_in_flight.borrow_mut().remove(&slot.id); } } @@ -1177,9 +1154,7 @@ impl App { return; }; let (page, schema, table) = (&s.page, s.schema.as_deref(), s.table.as_str()); - let data = crate::services::change_tracker::with_tab_ref(s.id, |tr| tr.has_pending()).unwrap_or(false); - let structure = crate::services::structure_tracker::with_tab_ref(s.id, |tr| tr.has_pending()).unwrap_or(false); - let combined_dirty = data || structure || dirty; + let combined_dirty = dirty || self.dirty_tabs().contains(&s.id); let base = qualified_browse_tab_label(schemas_count, schema, table); let title = if combined_dirty { format!("• {base}") } else { base }; page.set_title(&title); @@ -1198,16 +1173,12 @@ impl App { // teardown drops state before the 500ms timer would fire. self.do_persist_workspace_state_now(); self.cancel_all_editor_runs(); - // Drop per-tab pending-change trackers — disconnecting wipes - // the connection and its row identities, so any pending edits - // would no longer be commitable. + // The pending edits name rows of the connection going away, so + // they cannot be saved after it. for tab in self.workspace_tabs.borrow().values() { match tab { - WorkspaceTab::Structure(slot) => crate::services::structure_tracker::close_tab(slot.id), - WorkspaceTab::Table(slot) => { - crate::services::change_tracker::close_tab(slot.id); - crate::services::structure_tracker::close_tab(slot.id); - } + WorkspaceTab::Structure(slot) => self.structure_trackers.unregister(slot.id), + WorkspaceTab::Table(slot) => self.change_trackers.unregister(slot.id), WorkspaceTab::Editor(_) => {} } } diff --git a/linux/crates/app/src/ui/browse_tab.rs b/linux/crates/app/src/ui/browse_tab.rs index 8efbc107ae..5508117a2d 100644 --- a/linux/crates/app/src/ui/browse_tab.rs +++ b/linux/crates/app/src/ui/browse_tab.rs @@ -10,6 +10,10 @@ use tablepro_core::edit::{CellInput, EditContext, EditParseError}; use tablepro_core::{ColumnInfo, QueryResult, Value}; use super::grid::{CellPreset, GridMsg, TabGridContext, build_column_view}; +use crate::services::change_tracker::{RowKey, StatementSource, TrackedChanges}; +use crate::services::tab_tracker_handle::TabTrackerHandle; +use crate::services::tracker_delta::{TrackerDelta, VisualChange}; +use crate::services::tracker_edit::TrackerEdit; const PAGE_SIZE_OPTIONS: &[u64] = &[100, 500, 1_000, 5_000, 10_000]; const DEFAULT_PAGE_SIZE: u64 = 1_000; @@ -22,7 +26,7 @@ const DEFAULT_PAGE_SIZE: u64 = 1_000; const BULK_DELETE_CONFIRM_THRESHOLD: usize = 10; pub struct BrowseTabInit { - pub tab_id: Uuid, + pub tracker: TabTrackerHandle, pub schema: Option, pub table: String, pub connection_id: Option, @@ -35,7 +39,7 @@ pub struct BrowseTabInit { } pub struct BrowseTab { - tab_id: Uuid, + tracker: TabTrackerHandle, schema: Option, table: String, connection_id: Option, @@ -86,7 +90,7 @@ pub struct BrowseTab { /// once the new page lands. Persisted rows match by PK; drafts /// match by draft_id (for the rare case where a draft is focused /// when a sort happens). Cleared by `restore_focused_row`. - pending_focus_restore: std::cell::RefCell>, + pending_focus_restore: std::cell::RefCell>, paginator_label: gtk::Label, /// Live count of selected rows. Hidden when 0 or 1 rows are /// selected; shows "{n} selected" once the user shift-clicks @@ -224,17 +228,9 @@ pub enum BrowseTabInput { row_position: u32, }, GridCopyToClipboard(String), - /// Ctrl+Z on this tab. Pops one entry off the change tracker's - /// undo stack AND mirrors the visual revert in the grid: - /// CellEdit → reset the RowObject's cell + items_changed; - /// Insert → remove the draft RowObject from the ListStore; - /// Delete → re-bind the row so the strikethrough overlay drops. - /// Without the mirror the chrome (counter, .tp-cell-modified - /// class) updated correctly while the cell text stayed at the - /// post-edit value. + /// Ctrl+Z on this tab. Undo, - /// Ctrl+Shift+Z. Symmetric to Undo: re-applies the popped op - /// and mirrors the visual change forward. + /// Ctrl+Shift+Z. Redo, /// User clicked Save: hand the tracker's changes to the App as one /// `BrowseTabOutput::ExecuteTransaction`, which saves them as one @@ -243,14 +239,14 @@ pub enum BrowseTabInput { /// User clicked Discard — clear all pending edits and refetch /// the page so the grid shows committed values again. DiscardAll, - /// Pending count changed (from tracker subscription) — refresh - /// the Save / Discard / counter visibility. - PendingCountChanged(usize), - /// Specific rows mutated in the tracker (cell edit, set NULL, - /// insert, delete, undo, redo). Triggers a targeted re-bind of - /// just those rows so pending-state CSS classes update without - /// re-binding the entire visible viewport. - ChangedRows(Vec), + /// Drop every pending edit, then reload the page and count. + DiscardAndRefresh, + /// Mark persisted rows for deletion and drop drafts, once a bulk + /// delete is confirmed. + MarkRowsDeleted { + rows: Vec<(RowKey, Vec)>, + drafts: Vec, + }, /// Save command resolved successfully — clear tracker, refetch. SaveCompleted, /// Save command failed — surface error to the user; keep the @@ -262,7 +258,7 @@ pub enum BrowseTabInput { /// and deletes) and scroll-and-select it. Best-effort: if the row /// isn't on the current page (paginated past it, sorted away), /// the alert dialog still tells the user which statement failed. - FlashErrorRow(crate::services::change_tracker::StatementSource), + FlashErrorRow(StatementSource), /// Ctrl+C with row(s) selected: serialize each selected row as /// tab-separated cells and push to the system clipboard. Falls /// through to GTK's default Ctrl+C if no rows are selected so @@ -322,7 +318,7 @@ pub enum BrowseTabOutput { ExecuteTransaction { table: tablepro_core::meta::TableRef, columns: Vec, - changes: crate::services::change_tracker::TrackedChanges, + changes: TrackedChanges, }, } @@ -351,7 +347,7 @@ impl BrowseTab { /// is looking at, pending edits included. fn grid_context(&self) -> TabGridContext { TabGridContext { - tab_id: Some(self.tab_id), + tracker: Some(self.tracker.clone()), column_widths: self .connection_id .map(|connection_id| crate::ui::grid::ColumnWidthBinding { @@ -699,6 +695,71 @@ impl BrowseTab { self.refresh_banner_visibility(); } + /// Apply what a tracker call changed: the draft rows the store gains + /// or loses, then a rebind of each touched row, then the footer. + fn apply_tracker_delta(&self, delta: TrackerDelta, sender: &ComponentSender) { + if let Some(store) = self.list_store() { + for change in &delta.visual { + apply_visual_change(&store, change); + } + } + for key in &delta.rows { + if let Some(position) = self.find_row_position_by_key(key) { + self.refresh_row_at(position); + } + } + self.sync_pending_chrome(delta.pending_count, sender); + } + + /// The footer and the tab's dirty flag, which App turns into the + /// title bullet. App hears only real transitions. + fn sync_pending_chrome(&self, count: usize, sender: &ComponentSender) { + self.refresh_pending_bar(count); + let dirty = count > 0; + if dirty != self.was_dirty.replace(dirty) { + let _ = sender.output(BrowseTabOutput::DirtyChanged(dirty)); + } + } + + fn apply_edit(&self, edit: TrackerEdit, sender: &ComponentSender) { + let delta = self.tracker.apply(edit); + self.apply_tracker_delta(delta, sender); + } + + /// A typed or preset value for one cell. A draft keeps the value on + /// its own row; a persisted row keeps it in the tracker. + fn edit_cell(&self, row_position: u32, col: usize, value: Value, sender: &ComponentSender) { + if let Some(draft_id) = self.row_object_at(row_position).and_then(|row| row.draft_id()) { + self.apply_edit(TrackerEdit::DraftCell { draft_id, col, value }, sender); + return; + } + let Some((row_key, row)) = self.row_key_at(row_position) else { + return; + }; + let Some(original) = row.get(col).cloned() else { + return; + }; + self.apply_edit( + TrackerEdit::Cell { + row_key, + col, + original, + value, + }, + sender, + ); + } + + fn reload(&self, sender: &ComponentSender) { + self.capture_focus_for_restore(); + self.show_loading_inner( + &crate::i18n::gettext("Loading…"), + &crate::i18n::gettext_f("Fetching rows from {table}", &[("table", &self.table_label())]), + ); + let _ = sender.output(BrowseTabOutput::FetchPage); + let _ = sender.output(BrowseTabOutput::FetchRowCount); + } + /// Reveal at most ONE banner at a time. Read-only takes priority /// over no-PK because read-only blocks every kind of edit. fn refresh_banner_visibility(&self) { @@ -775,8 +836,8 @@ impl BrowseTab { /// recomputing the row's PK key-tuple from cell values and /// comparing. Returns the position in the (filtered) selection /// model, or `None` if the row isn't on the current page. - fn find_row_position_by_key(&self, key: &crate::services::change_tracker::RowKey) -> Option { - use crate::services::change_tracker::{KeyValue, RowKey}; + fn find_row_position_by_key(&self, key: &RowKey) -> Option { + use crate::services::change_tracker::KeyValue; let selection = self.current_selection.as_ref()?; let model = selection.model()?; let pk_indices: Vec = self @@ -813,16 +874,10 @@ impl BrowseTab { None } - /// Locate the row that produced a failing statement and scroll-and- - /// select it, plus apply a one-shot red flash animation. The flash - /// state lives on `TabChangeTracker.error_row` so the grid bind - /// callback picks it up via the existing tracker query path; a - /// 1.8s timeout clears the state afterwards (matches the CSS - /// animation duration). Best-effort lookup: a row that's been - /// paginated past, sorted away, or filtered out won't be found, - /// and we silently fall through to just the alert dialog. - fn flash_error_row(&self, source: &crate::services::change_tracker::StatementSource) { - use crate::services::change_tracker::{RowKey, StatementSource}; + /// Scroll to the row behind a failed statement and flash it. A row + /// paged, sorted or filtered away is not found, and the alert alone + /// tells the user which statement failed. + fn flash_error_row(&self, source: &StatementSource) { let key = match source { StatementSource::Insert { draft_id } => RowKey::Draft(*draft_id), StatementSource::Update { row_key } | StatementSource::Delete { row_key } => row_key.clone(), @@ -841,31 +896,15 @@ impl BrowseTab { None, ); } - // Mark the row as the error row and trigger a re-bind so the - // bind callback applies tp-row-leftmost-error-flash. Schedule - // a timeout to clear the state once the animation has played. - // The generation counter protects against a second flash that - // starts inside the 1.8s window: the older timer's clear runs - // but no-ops because gen no longer matches. - let tab_id = self.tab_id; - let generation = - crate::services::change_tracker::with_tab(tab_id, |t| t.set_error_row(key.clone())).unwrap_or(0); + let generation = self.tracker.flash_error_row(key); self.refresh_row_at(position); + let tracker = self.tracker.clone(); let selection_for_clear = self.current_selection.clone(); + // Matches the length of the flash animation in the stylesheet. glib::timeout_add_local_once(std::time::Duration::from_millis(1800), move || { - let cleared = crate::services::change_tracker::with_tab(tab_id, |t| { - let was_match = t.is_error_row_gen(generation); - t.clear_error_row_if_gen(generation); - was_match - }) - .unwrap_or(false); - if !cleared { - // A newer flash superseded ours; don't disturb its bind. + if !tracker.clear_error_row_if(generation) { return; } - // Inline the underlying-store hop because `self` is gone - // from the closure scope (it's a 1.8s deferred timer); - // the chain walk is the same as `refresh_row_at`. if let Some(selection) = selection_for_clear && let Some(model) = selection.model() && let Some(row_obj) = model @@ -884,7 +923,6 @@ impl BrowseTab { /// (sort flip, save, F5 refresh) so we can re-anchor the user's /// view to the same row after the new page renders. fn capture_focus_for_restore(&self) { - use crate::services::change_tracker::RowKey; // If a previous capture is still pending (sort+save fire in // quick succession before the first reload's restore runs), keep // the original capture. The first user-visible focus should @@ -1153,16 +1191,11 @@ impl BrowseTab { let Some(store) = self.list_store() else { return; }; - let drafts = crate::services::change_tracker::with_tab_ref(self.tab_id, |t| { - t.drafts() - .iter() - .map(|d| (d.draft_id, d.values.clone())) - .collect::>() - }) - .unwrap_or_default(); - for (draft_id, values) in drafts { - let draft_row = super::row_object::RowObject::new_draft(draft_id, values); - store.insert(0, &draft_row); + for draft in self.tracker.drafts() { + store.insert( + 0, + &super::row_object::RowObject::new_draft(draft.draft_id, draft.values), + ); } } @@ -1188,7 +1221,7 @@ impl BrowseTab { /// Returns None for drafts (caller is expected to handle the /// draft path before reaching here), for tables with no PK, or /// when the position is out of range. - fn row_key_at(&self, row_position: u32) -> Option<(crate::services::change_tracker::RowKey, Vec)> { + fn row_key_at(&self, row_position: u32) -> Option<(RowKey, Vec)> { let row_obj = self.row_object_at(row_position)?; if row_obj.draft_id().is_some() { return None; @@ -1357,11 +1390,6 @@ impl SimpleComponent for BrowseTab { } fn init(init: Self::Init, root: Self::Root, sender: ComponentSender) -> ComponentParts { - // Open this tab's pending-changeset tracker. Closed in - // workspace_tabs::close_workspace_tab_by_id when the tab is - // removed. Idempotent — calling twice is a no-op. - crate::services::change_tracker::open_tab(init.tab_id); - // Restore the saved filter for this (connection, schema, // table) up front so both the model field and the inline // strip start with the same FilterSet. @@ -1717,7 +1745,7 @@ impl SimpleComponent for BrowseTab { })); let model = BrowseTab { - tab_id: init.tab_id, + tracker: init.tracker, schema: init.schema, table: init.table, connection_id: init.connection_id, @@ -1765,25 +1793,6 @@ impl SimpleComponent for BrowseTab { // the moment the tab opens. model.refresh_filter_chrome(); - // Subscribe to the tracker so we can refresh the pending UI - // any time the user adds / undoes / commits a change. The - // channel is leaked into the GTK main loop via spawn_future_local, - // matching how the per-tab GridMsg channel above is wired. - let (tracker_sender, tracker_receiver) = relm4::channel::(); - crate::services::change_tracker::with_tab(init.tab_id, |t| t.subscribe(tracker_sender)); - let input_for_tracker = sender.input_sender().clone(); - glib::spawn_future_local(tracker_receiver.forward(input_for_tracker, move |event| match event { - crate::services::change_tracker::TrackerEvent::PendingCountChanged(n) => { - BrowseTabInput::PendingCountChanged(n) - } - crate::services::change_tracker::TrackerEvent::Cleared => BrowseTabInput::PendingCountChanged(0), - // ChangedRows drives targeted items_changed so only the - // affected rows re-bind, not the whole visible viewport. - // PendingCountChanged is emitted alongside by the tracker - // (see emit_changed) so banner / dirty-flag updates still - // run for the same mutation. - crate::services::change_tracker::TrackerEvent::ChangedRows(keys) => BrowseTabInput::ChangedRows(keys), - })); // Trigger the initial fetches the moment the parent attaches us. // The parent's forward closure wraps these in AppMsg::… with tab_id. let _ = sender.output(BrowseTabOutput::FetchColumns); @@ -1875,14 +1884,11 @@ impl SimpleComponent for BrowseTab { self.show_error_inner(&message); self.inner_stack.set_visible_child_name("error"); } - BrowseTabInput::Refresh => { - self.capture_focus_for_restore(); - self.show_loading_inner( - &crate::i18n::gettext("Loading…"), - &crate::i18n::gettext_f("Fetching rows from {table}", &[("table", &self.table_label())]), - ); - let _ = sender.output(BrowseTabOutput::FetchPage); - let _ = sender.output(BrowseTabOutput::FetchRowCount); + BrowseTabInput::Refresh => self.reload(&sender), + BrowseTabInput::DiscardAndRefresh => { + let delta = self.tracker.discard(); + self.apply_tracker_delta(delta, &sender); + self.reload(&sender); } BrowseTabInput::ClearSelection => { let Some(sel) = self.current_selection.as_ref() else { @@ -2020,18 +2026,7 @@ impl SimpleComponent for BrowseTab { } }) .collect(); - let key_opt = - crate::services::change_tracker::with_tab(self.tab_id, |t| t.track_insert(values.clone())); - let Some(key) = key_opt else { - return; - }; - let crate::services::change_tracker::RowKey::Draft(draft_id) = key else { - return; - }; - if let Some(store) = self.list_store() { - let draft_row = super::row_object::RowObject::new_draft(draft_id, values); - store.insert(0, &draft_row); - } + self.apply_edit(TrackerEdit::Insert { values }, &sender); self.refresh_inner_stack_visibility(); // Scroll + focus deferred via FocusInsertedDraft — see // the matching note in the `InsertRow` arm above. Same @@ -2043,25 +2038,8 @@ impl SimpleComponent for BrowseTab { if self.current_columns.is_empty() { return; } - // Inline draft: track in the changeset (returns a - // RowKey::Draft(N) handle), then prepend a fresh - // RowObject tagged with the same draft id to the - // grid's ListStore. The new row appears at the top - // with a green tint and editable cells; user fills - // them inline and clicks Save to commit. - let default_values: Vec = self.current_columns.iter().map(|_| Value::Null).collect(); - let key_opt = - crate::services::change_tracker::with_tab(self.tab_id, |t| t.track_insert(default_values.clone())); - let Some(key) = key_opt else { - return; - }; - let crate::services::change_tracker::RowKey::Draft(draft_id) = key else { - return; - }; - if let Some(store) = self.list_store() { - let draft_row = super::row_object::RowObject::new_draft(draft_id, default_values); - store.insert(0, &draft_row); - } + let values: Vec = self.current_columns.iter().map(|_| Value::Null).collect(); + self.apply_edit(TrackerEdit::Insert { values }, &sender); // The empty-state status page hides the column view — // if we were sitting on it (zero persisted rows on the // first page) the draft we just appended would be @@ -2112,7 +2090,7 @@ impl SimpleComponent for BrowseTab { let Some(model) = selection.model() else { return; }; - let mut snapshot: Vec<(crate::services::change_tracker::RowKey, Vec)> = Vec::new(); + let mut snapshot: Vec<(RowKey, Vec)> = Vec::new(); let mut draft_ids: Vec = Vec::new(); let mut had_persisted_row = false; for pos in &positions { @@ -2143,26 +2121,9 @@ impl SimpleComponent for BrowseTab { return; } let count = snapshot.len() + draft_ids.len(); - let tab_id = self.tab_id; - let selection_for_commit = selection.clone(); - let commit_delete = move |snapshot: Vec<(crate::services::change_tracker::RowKey, Vec)>, - draft_ids: Vec| { - crate::services::change_tracker::with_tab(tab_id, |t| { - for (key, row) in snapshot { - t.track_delete(key, row); - } - for id in draft_ids { - t.discard_draft(id); - } - }); - // Clear the multi-row selection so the - // "{n} selected" badge disappears and the Delete - // button tooltip resets. Without this the bitset - // still contains the now-strikethrough rows - // (they remain in the model until Save). Spreadsheet - // convention: a bulk action ends the selection it - // operated on. - selection_for_commit.unselect_all(); + let sender_for_commit = sender.clone(); + let commit_delete = move |rows: Vec<(RowKey, Vec)>, drafts: Vec| { + sender_for_commit.input(BrowseTabInput::MarkRowsDeleted { rows, drafts }); }; if count >= BULK_DELETE_CONFIRM_THRESHOLD { // Dialog parent is any descendant of the toplevel @@ -2245,27 +2206,7 @@ impl SimpleComponent for BrowseTab { return; } }; - let row_obj = self.row_object_at(row_position); - if let Some(row_obj) = &row_obj - && let Some(draft_id) = row_obj.draft_id() - { - // Draft row — mutate the tracker's draft buffer - // directly. The RowObject's own cells are also - // updated so the grid's display reflects the - // pending value without waiting for re-fetch. - crate::services::change_tracker::with_tab(self.tab_id, |t| { - t.track_draft_cell_edit(draft_id, col_index, new.clone()); - }); - row_obj.set_cell(col_index, new); - return; - } - let Some((key, row)) = self.row_key_at(row_position) else { - return; - }; - let original = row[col_index].clone(); - crate::services::change_tracker::with_tab(self.tab_id, |t| { - t.track_cell_edit(key, col_index, original, new); - }); + self.edit_cell(row_position, col_index, new, &sender); } BrowseTabInput::GridSetCellValue { row_position, @@ -2279,22 +2220,7 @@ impl SimpleComponent for BrowseTab { return; } }; - if let Some(row_obj) = self.row_object_at(row_position) - && let Some(draft_id) = row_obj.draft_id() - { - crate::services::change_tracker::with_tab(self.tab_id, |t| { - t.track_draft_cell_edit(draft_id, col_index, value.clone()); - }); - row_obj.set_cell(col_index, value); - return; - } - let Some((key, row)) = self.row_key_at(row_position) else { - return; - }; - let original = row[col_index].clone(); - crate::services::change_tracker::with_tab(self.tab_id, |t| { - t.track_cell_edit(key, col_index, original, value); - }); + self.edit_cell(row_position, col_index, value, &sender); } BrowseTabInput::GridShowToast(message) => { let _ = sender.output(BrowseTabOutput::ShowToast(message)); @@ -2323,12 +2249,16 @@ impl SimpleComponent for BrowseTab { }); } BrowseTabInput::GridDeleteRowAt { row_position } => { - let Some((key, row)) = self.row_key_at(row_position) else { + let Some((row_key, original_values)) = self.row_key_at(row_position) else { return; }; - crate::services::change_tracker::with_tab(self.tab_id, |t| { - t.track_delete(key, row); - }); + self.apply_edit( + TrackerEdit::Delete { + row_key, + original_values, + }, + &sender, + ); } BrowseTabInput::GridCopyRowAsInsert { row_position } => { let _ = sender.output(BrowseTabOutput::CopyRowAsInsert { row_position }); @@ -2417,9 +2347,8 @@ impl SimpleComponent for BrowseTab { } BrowseTabInput::CommitSave => { let columns = self.current_columns.clone(); - let result = crate::services::change_tracker::with_tab_ref(self.tab_id, |t| t.change_set(&columns)); - match result { - Some(Ok(changes)) if !changes.changes.is_empty() => { + match self.tracker.change_set(&columns) { + Ok(changes) if !changes.changes.is_empty() => { // Disable both buttons for the duration of the // in-flight transaction. SaveCompleted / // SaveFailed re-enable them. This prevents a @@ -2438,55 +2367,44 @@ impl SimpleComponent for BrowseTab { changes, }); } - Some(Ok(_)) => { - // Nothing to save (tracker empty) — refresh bar. - self.refresh_pending_bar(0); - } - Some(Err(e)) => { + Ok(_) => self.refresh_pending_bar(0), + Err(e) => { let _ = sender.output(BrowseTabOutput::ShowSelectionAlert { title: crate::i18n::gettext("Cannot save"), body: format!("{e}"), }); } - None => {} } } BrowseTabInput::DiscardAll => { - crate::services::change_tracker::with_tab(self.tab_id, |t| t.clear()); + let delta = self.tracker.discard(); + self.apply_tracker_delta(delta, &sender); let _ = sender.output(BrowseTabOutput::FetchPage); } - BrowseTabInput::PendingCountChanged(n) => { - self.refresh_pending_bar(n); - // Tell the App so it can prefix the tab title with the - // GNOME-Text-Editor "•" dot for dirty buffers. Only on - // real transitions (empty ↔ non-empty) so a count - // change like 2 → 3 doesn't re-rewrite the tab title. - let dirty = n > 0; - if dirty != self.was_dirty.get() { - self.was_dirty.set(dirty); - let _ = sender.output(BrowseTabOutput::DirtyChanged(dirty)); + BrowseTabInput::MarkRowsDeleted { rows, drafts } => { + for (row_key, original_values) in rows { + self.apply_edit( + TrackerEdit::Delete { + row_key, + original_values, + }, + &sender, + ); } - // Note: row re-binds are driven by the parallel - // ChangedRows event, not from here. PendingCountChanged - // fires on every tracker mutation so re-binding the - // viewport here would be wasteful — most edits affect - // exactly one row and ChangedRows hits only that row. - } - BrowseTabInput::ChangedRows(keys) => { - // Walk the model once per key. For typical interactive - // edits (one cell at a time) this is O(n) per keystroke - // where n = visible row count — bounded and cheap. - // Bulk operations (Discard) emit one ChangedRows per - // op via undo unwind, again bounded. - for key in &keys { - if let Some(pos) = self.find_row_position_by_key(key) { - self.refresh_row_at(pos); - } + for draft_id in drafts { + self.apply_edit(TrackerEdit::DiscardDraft { draft_id }, &sender); + } + // A bulk action ends the selection it acted on. + if let Some(selection) = self.current_selection.as_ref() { + selection.unselect_all(); } } BrowseTabInput::SaveCompleted => { - crate::services::change_tracker::with_tab(self.tab_id, |t| t.clear()); - self.refresh_pending_bar(0); + // The reload replaces every row; dropping the saved + // drafts from the store first would blank them until it + // lands. + let delta = self.tracker.discard(); + self.sync_pending_chrome(delta.pending_count, &sender); self.save_button.set_sensitive(true); self.discard_button.set_sensitive(true); let _ = sender.output(BrowseTabOutput::FetchPage); @@ -2507,102 +2425,44 @@ impl SimpleComponent for BrowseTab { self.focus_inserted_draft(); } BrowseTabInput::Undo => { - use crate::services::change_tracker::UndoOp; - let op = match crate::services::change_tracker::with_tab(self.tab_id, |t| t.undo()) { - Some(Some(op)) => op, - _ => return, - }; - match op { - UndoOp::CellEdit { - row_key, - col, - prev_value, - .. - } => { - // Drafts hold the post-edit value in - // RowObject.cells (mirrored at edit time); - // reverting the visible value requires - // mutating the cell back. Persisted rows - // never mutate RowObject — the tracker is - // the single source of truth and - // connect_bind queries - // `current_cell_value` to overlay the - // pending edit. set_cell on a persisted - // row is a harmless no-op (the cell - // already holds the original). - if let Some(pos) = self.find_row_position_by_key(&row_key) - && let Some(row_obj) = self.row_object_at(pos) - { - row_obj.set_cell(col, prev_value); - } - } - UndoOp::Insert { draft_id, .. } => { - // The draft RowObject is still in the - // ListStore (prepended by InsertRow). Walk - // the store, find the row whose draft_id - // matches, remove it. The subsequent - // ChangedRows event for `Draft(id)` is a - // no-op once the row is gone. - if let Some(store) = self.list_store() { - let n = store.n_items(); - for i in 0..n { - if let Some(obj) = store.item(i) - && let Ok(row) = obj.downcast::() - && row.draft_id() == Some(draft_id) - { - store.remove(i); - break; - } - } - } - } - UndoOp::Delete { row_key, .. } => { - // No RowObject mutation needed — the row was - // never visually removed; it stayed in the - // ListStore with a strikethrough overlay - // applied at bind time. The tracker's - // emit_changed → items_changed re-bind drops - // the strikethrough automatically because - // `row_state` returns Clean once the deletes - // entry is gone. - let _ = row_key; - } + if let Some(delta) = self.tracker.undo() { + self.apply_tracker_delta(delta, &sender); } } BrowseTabInput::Redo => { - use crate::services::change_tracker::UndoOp; - let op = match crate::services::change_tracker::with_tab(self.tab_id, |t| t.redo()) { - Some(Some(op)) => op, - _ => return, - }; - match op { - UndoOp::CellEdit { - row_key, - col, - new_value, - .. - } => { - if let Some(pos) = self.find_row_position_by_key(&row_key) - && let Some(row_obj) = self.row_object_at(pos) - { - row_obj.set_cell(col, new_value); - } - } - UndoOp::Insert { draft_id, values } => { - // Re-add the draft RowObject. Match the - // original insert path: prepend at position 0. - if let Some(store) = self.list_store() { - let draft_row = super::row_object::RowObject::new_draft(draft_id, values); - store.insert(0, &draft_row); - } - } - UndoOp::Delete { .. } => {} + if let Some(delta) = self.tracker.redo() { + self.apply_tracker_delta(delta, &sender); } } } } } +fn apply_visual_change(store: >k::gio::ListStore, change: &VisualChange) { + use super::row_object::RowObject; + let draft_at = |draft_id: u64| { + (0..store.n_items()).find_map(|position| { + let row = store.item(position).and_downcast::()?; + (row.draft_id() == Some(draft_id)).then_some((position, row)) + }) + }; + match change { + VisualChange::InsertDraft { draft_id, values } => { + store.insert(0, &RowObject::new_draft(*draft_id, values.clone())); + } + VisualChange::RemoveDraft { draft_id } => { + if let Some((position, _)) = draft_at(*draft_id) { + store.remove(position); + } + } + VisualChange::SetDraftCell { draft_id, col, value } => { + if let Some((_, row)) = draft_at(*draft_id) { + row.set_cell(*col, value.clone()); + } + } + } +} + fn clear_box(b: >k::Box) { while let Some(child) = b.first_child() { b.remove(&child); @@ -2633,15 +2493,12 @@ fn normalize_single_line_input(text: &str) -> String { /// / RowObject. Callers feed it a clone of the row's cells (already /// pulled from the model in selection-model space, see the /// row-position-vs-result-rows note in `DeleteSelectedRow`). -fn build_persisted_row_key( - cells: Vec, - pk_indices: &[usize], -) -> Option<(crate::services::change_tracker::RowKey, Vec)> { +fn build_persisted_row_key(cells: Vec, pk_indices: &[usize]) -> Option<(RowKey, Vec)> { let pk_values: Vec = pk_indices .iter() .map(|&i| cells.get(i).cloned()) .collect::>()?; - let key = crate::services::change_tracker::RowKey::from_pk_values(&pk_values)?; + let key = RowKey::from_pk_values(&pk_values)?; Some((key, cells)) } @@ -2735,8 +2592,12 @@ fn result_columns(columns: &[tablepro_core::ColumnInfo]) -> std::sync::Arc<[tabl #[cfg(test)] mod tests { - use super::{build_persisted_row_key, format_thousands, parse_input_for_column}; + use gtk4::prelude::*; + + use super::{apply_visual_change, build_persisted_row_key, format_thousands, parse_input_for_column}; use crate::services::change_tracker::RowKey; + use crate::services::tracker_delta::VisualChange; + use crate::ui::row_object::RowObject; use tablepro_core::edit::EditContext; use tablepro_core::{ColumnInfo, Value}; @@ -3011,4 +2872,36 @@ mod tests { let (key, _) = build_persisted_row_key(cells, &[0]).expect("Null PK is valid"); assert!(matches!(key, RowKey::Persisted(_))); } + + fn row_at(store: >k4::gio::ListStore, position: u32) -> RowObject { + store.item(position).and_downcast::().expect("a row") + } + + #[test] + fn visual_changes_add_update_and_remove_draft_rows() { + let store = gtk4::gio::ListStore::new::(); + store.append(&RowObject::new(vec![Value::Int(1)])); + + apply_visual_change( + &store, + &VisualChange::InsertDraft { + draft_id: 4, + values: vec![Value::Null], + }, + ); + apply_visual_change( + &store, + &VisualChange::SetDraftCell { + draft_id: 4, + col: 0, + value: Value::Int(9), + }, + ); + assert_eq!(row_at(&store, 0).draft_id(), Some(4)); + assert_eq!(row_at(&store, 0).cell_value(0), Value::Int(9)); + + apply_visual_change(&store, &VisualChange::RemoveDraft { draft_id: 4 }); + assert_eq!(store.n_items(), 1); + assert_eq!(row_at(&store, 0).draft_id(), None); + } } diff --git a/linux/crates/app/src/ui/credential_prompt/presenter.rs b/linux/crates/app/src/ui/credential_prompt/presenter.rs index 8b7b99e739..3cd0545d29 100644 --- a/linux/crates/app/src/ui/credential_prompt/presenter.rs +++ b/linux/crates/app/src/ui/credential_prompt/presenter.rs @@ -1,7 +1,9 @@ use std::cell::RefCell; +use std::pin::pin; use std::rc::Rc; use adw::prelude::*; +use futures::future::{Either, select}; use libadwaita as adw; use tablepro_core::credentials::PromptReply; use tokio::sync::oneshot; @@ -24,26 +26,43 @@ impl CredentialPromptPresenter { let parent = parent.upcast_ref::().downgrade(); glib::spawn_future_local(async move { while let Some(pending) = queue.next().await { - let PendingPrompt { prompt, reply } = pending; + let PendingPrompt { prompt, mut reply } = pending; if reply.is_closed() { continue; } let answer = match parent.upgrade() { Some(parent) => { let form = PromptForm::build(&prompt); - let response = ask(&form.dialog, &parent).await; + let Some(response) = response_while_wanted(&form.dialog, &parent, &mut reply).await else { + // The connect that asked was cancelled. Its + // dialog goes with it, or the next prompt + // would wait behind one nobody can answer. + form.abandon(); + continue; + }; form.reply(&response) } None => PromptReply::Cancelled, }; - // The connect may have been cancelled while the dialog - // was up; nobody is waiting then, which is fine. + // The asker can still go between the response and here; + // nobody is waiting then, which is fine. let _ = reply.send(answer); } }) } } +async fn response_while_wanted( + dialog: &adw::AlertDialog, + parent: >k4::Widget, + reply: &mut oneshot::Sender, +) -> Option { + match select(pin!(ask(dialog, parent)), pin!(reply.closed())).await { + Either::Left((response, _)) => Some(response), + Either::Right(((), _)) => None, + } +} + /// Present the dialog and wait for the response it closes with. /// /// The closed signal answers too, as a cancel: a dialog torn down with @@ -137,6 +156,41 @@ mod tests { crate::test_support::drain_main_context(); } + #[gtk4::test] + fn a_prompt_whose_connect_was_cancelled_closes_and_lets_the_next_one_show() { + adw::init().expect("libadwaita initialises under the test backend"); + let window = adw::Window::new(); + window.present(); + let (prompter, queue) = GtkCredentialPrompter::new(); + let presenter = CredentialPromptPresenter::start(queue, &window); + let cancelled = prompter.prompt(prompt("cancelled")); + crate::test_support::wait_until(Duration::from_secs(5), || open_alerts(&window).len() == 1) + .expect("the first prompt"); + + drop(cancelled); + + crate::test_support::wait_until(Duration::from_secs(5), || open_alerts(&window).is_empty()) + .expect("the dialog of a cancelled connect stayed on screen"); + let answered = std::rc::Rc::new(std::cell::Cell::new(false)); + let asking = prompter.prompt(prompt("next")); + let sink = answered.clone(); + glib::spawn_future_local(async move { + asking.await; + sink.set(true); + }); + crate::test_support::wait_until(Duration::from_secs(5), || { + open_alerts(&window).iter().any(|alert| alert.body().contains("next")) + }) + .expect("the next prompt behind the cancelled one"); + for alert in open_alerts(&window) { + alert.close(); + } + crate::test_support::wait_until(Duration::from_secs(5), || answered.get()).expect("the next prompt answered"); + presenter.abort(); + window.close(); + crate::test_support::drain_main_context(); + } + #[gtk4::test] fn a_prompt_whose_dialog_is_forced_shut_is_cancelled() { adw::init().expect("libadwaita initialises under the test backend"); diff --git a/linux/crates/app/src/ui/credential_prompt/prompt_form.rs b/linux/crates/app/src/ui/credential_prompt/prompt_form.rs index 4d160acf32..5c9e59039d 100644 --- a/linux/crates/app/src/ui/credential_prompt/prompt_form.rs +++ b/linux/crates/app/src/ui/credential_prompt/prompt_form.rs @@ -89,6 +89,13 @@ impl PromptForm { Self { dialog, rows, remember } } + pub fn abandon(&self) { + for row in &self.rows { + drop(row.take()); + } + self.dialog.force_close(); + } + /// The reply for the response the dialog closed with. Anything but /// the accept button, including Escape, is a cancel. pub fn reply(&self, response: &str) -> PromptReply { diff --git a/linux/crates/app/src/ui/editor/format_plan.rs b/linux/crates/app/src/ui/editor/format_plan.rs index 5e0d4445bc..d0e99e1e2f 100644 --- a/linux/crates/app/src/ui/editor/format_plan.rs +++ b/linux/crates/app/src/ui/editor/format_plan.rs @@ -9,10 +9,11 @@ use super::significant_tokens::significant_tokens; /// sqlformat has its own lexer, and it does not know every dialect the /// app connects to: it reads `//` as an operator, ends a nested block /// comment at the first `*/`, treats a backslash as an escape inside -/// quotes, and takes `$$` for a keyword. Reformatting through it can -/// therefore change what a statement means. Each statement is checked -/// against its own tokens afterwards, and one that came back different -/// is written out as the user typed it. +/// quotes, takes `$$` for a keyword, and puts a space inside `+=`, +/// `b'01'` and `GROUP_CONCAT(`. Reformatting through it can therefore +/// change what a statement means. Each statement is checked against +/// its own tokens afterwards, and one that came back different is +/// written out as the user typed it. pub fn format_script(text: &str, grammar: SqlGrammar, settings: LexicalSettings) -> String { let plan = ScriptPlan::build(text, grammar, settings); let mut out = String::with_capacity(text.len()); @@ -161,6 +162,55 @@ mod tests { assert!(out.contains("DELIMITER ;"), "the delimiter reset was lost: {out}"); } + #[test] + fn statements_sqlformat_would_respace_into_something_else_kept_verbatim() { + let cases = [ + // `+=` would become `+ =`. + ("SET @x += 1", SqlGrammar::MsSql), + // A bit literal and a charset introducer would become a + // column with a string alias. + ("select b'01', _utf8mb4'x' from t", SqlGrammar::MySql), + // `@@ version` is not a system variable. + ("select @@version", SqlGrammar::MySql), + // `U & '…'` is a bitwise and, not a Unicode escape string. + (r"select U&'d\0061t'", SqlGrammar::PostgreSql), + // A string continued on the next line is one string; on + // one line the two are a syntax error. + ("select 'a'\n'b'", SqlGrammar::PostgreSql), + // A query parameter's name, colon and type are one unit. + ("SELECT {id:UInt32}", SqlGrammar::ClickHouse), + ]; + + for (source, grammar) in cases { + assert_eq!(format(source, grammar), source, "{grammar:?}"); + } + } + + #[test] + fn a_mysql_call_keeps_its_name_against_its_paren() { + // sqlformat writes `GROUP_CONCAT (a)`, which MySQL reads as a + // stored function of that name. + let out = format("SELECT GROUP_CONCAT(a) FROM t", SqlGrammar::MySql); + + assert!( + out.contains("GROUP_CONCAT(a)"), + "the call was split from its paren: {out}" + ); + } + + #[test] + fn mysql_statements_with_untouched_calls_are_still_formatted() { + for source in [ + "select count(*), max(a) from t", + "insert into t (a, b) values (1, 2)", + "select ifnull(a, 0) from t where a in(1, 2)", + ] { + let out = format(source, SqlGrammar::MySql); + + assert!(out.contains('\n'), "the statement was not reflowed: {out}"); + } + } + #[test] fn an_empty_script_stays_empty() { assert_eq!(format("", SqlGrammar::PostgreSql), ""); diff --git a/linux/crates/app/src/ui/editor/mod.rs b/linux/crates/app/src/ui/editor/mod.rs index a4a3fd9dec..92020be2a4 100644 --- a/linux/crates/app/src/ui/editor/mod.rs +++ b/linux/crates/app/src/ui/editor/mod.rs @@ -15,6 +15,7 @@ mod run_scope; mod script_runner; mod sidebar_policy; mod significant_tokens; +mod sql_file; mod statement_cursor; #[cfg(test)] mod tests; @@ -186,18 +187,22 @@ impl SqlEditor { /// Run the statement the cursor is in, selecting it so the user sees /// which of the buffer's queries ran. pub fn run_at_cursor(&self) { + if self.is_running() { + return; + } let buffer = self.buffer(); let text = buffer_text(&buffer); let plan = statement_cursor::plan_for(&text, self.grammar()); let cursor = usize::try_from(buffer.iter_at_mark(&buffer.get_insert()).offset()).unwrap_or_default(); - let Some((scoped, range)) = run_scope::at_cursor(&text, &plan, cursor) else { + let Some((start, end)) = statement_cursor::statement_bounds_at(&buffer, &plan, &text, cursor) else { self.say(&crate::i18n::gettext("No statement at the cursor.")); return; }; - let start = buffer.iter_at_offset(i32::try_from(range.start).unwrap_or(i32::MAX)); - let end = buffer.iter_at_offset(i32::try_from(range.end).unwrap_or(i32::MAX)); buffer.select_range(&start, &end); - self.execute(scoped); + self.execute(ScopedText { + text: buffer.text(&start, &end, false).to_string(), + base_char: usize::try_from(start.offset()).unwrap_or_default(), + }); } /// Reformat the buffer per statement, as one undo step, keeping the @@ -223,7 +228,7 @@ impl SqlEditor { /// Read a dropped or chosen file and put it in the buffer: the whole /// buffer when it is empty, at the cursor otherwise. pub async fn insert_file(&self, file: gio::File, replace: bool) { - let text = match load_text(&file).await { + let text = match sql_file::load_text(&file).await { Ok(text) => text, Err(message) => { self.emit_by_name::<()>("toast", &[&message]); @@ -407,25 +412,26 @@ impl SqlEditor { self.show_new_entries(before); } - /// Add the log's lines from `from` on to the results list. fn show_new_entries(&self, from: usize) { let imp = self.imp(); - let Some(grid_sender) = imp.grid_sender.get() else { - return; - }; let log = imp.log.borrow(); let Some(log) = log.as_ref() else { return; }; - for entry in log.entries().get(from..).unwrap_or_default() { - let weak = self.downgrade(); - let widget = result_widgets::entry_widget(entry, grid_sender, move |position| { - if let Some(editor) = weak.upgrade() { - editor.go_to_char(position); - } - }); - imp.results.append(entry, &widget); - } + imp.results.append(log.entries().get(from..).unwrap_or_default()); + } + + fn result_page(&self, index: u32) -> Option { + let imp = self.imp(); + let grid_sender = imp.grid_sender.get()?; + let log = imp.log.borrow(); + let entry = log.as_ref()?.entries().get(usize::try_from(index).ok()?)?; + let weak = self.downgrade(); + Some(result_widgets::entry_widget(entry, grid_sender, move |position| { + if let Some(editor) = weak.upgrade() { + editor.go_to_char(position); + } + })) } fn finish_run(&self, query: String, metadata: &tablepro_session::ConnectionMetadata, started_at: SystemTime) { @@ -575,8 +581,12 @@ impl SqlEditor { imp.menu_button.set_icon_name(crate::ui::icons::VIEW_MORE); } - /// Context-menu actions from a result grid. + /// Result pages, and the context-menu actions from their grids. fn connect_grid(&self) { + let weak = self.downgrade(); + self.imp() + .results + .set_page_builder(move |index| weak.upgrade().and_then(|editor| editor.result_page(index))); let (sender, receiver) = relm4::channel::(); let _ = self.imp().grid_sender.set(sender); let weak = self.downgrade(); @@ -742,27 +752,6 @@ fn buffer_text(buffer: &impl IsA) -> String { buffer.text(&start, &end, false).to_string() } -/// The most a file may be before the editor refuses it. -/// -/// A dump can be gigabytes, and a text buffer holding one stops being -/// an editor. Refusing is better than a window that will not paint. -const MAX_SQL_FILE_BYTES: i64 = 8 * 1024 * 1024; - -/// Read a file as text off the main thread, or say why it cannot be. -async fn load_text(file: &gio::File) -> Result { - let info = file - .query_info_future("standard::size", gio::FileQueryInfoFlags::NONE, glib::Priority::DEFAULT) - .await; - if info.is_ok_and(|info| info.size() > MAX_SQL_FILE_BYTES) { - return Err(crate::i18n::gettext("That file is too large to open in the editor.")); - } - let (bytes, _) = file.load_contents_future().await.map_err(|error| { - tracing::warn!(%error, "could not read a file for the editor"); - crate::i18n::gettext("That file could not be read.") - })?; - String::from_utf8(bytes.to_vec()).map_err(|_| crate::i18n::gettext("That file is not UTF-8 text.")) -} - /// Toggle SQL line-comment (`-- `) for the lines in the buffer's /// current selection (or the cursor's line when nothing is selected). /// If every non-blank line in the range is already commented, strip diff --git a/linux/crates/app/src/ui/editor/results_pane.rs b/linux/crates/app/src/ui/editor/results_pane.rs index 2f37b7601e..82cae94399 100644 --- a/linux/crates/app/src/ui/editor/results_pane.rs +++ b/linux/crates/app/src/ui/editor/results_pane.rs @@ -7,8 +7,15 @@ use super::result_item::ResultItem; use super::run_log::{EntryKind, RunLogEntry}; use super::sidebar_policy::{self, SidebarState}; +/// How many results keep their page once shown, so going back to one +/// keeps its scroll position without every line holding a page. +const PAGES_KEPT: usize = 8; + +type PageBuilder = Box Option>; + mod imp { - use std::cell::Cell; + use std::cell::{Cell, RefCell}; + use std::collections::VecDeque; use libadwaita::subclass::prelude::*; @@ -34,6 +41,9 @@ mod imp { pub items: std::cell::OnceCell, pub selection: std::cell::OnceCell, pub sidebar: Cell, + pub page_builder: RefCell>, + /// The lines whose page is built, most recently shown first. + pub built: RefCell>, } #[glib::object_subclass] @@ -118,16 +128,10 @@ impl ResultsPane { if let Some(items) = imp.items.get() { items.remove_all(); } - let pages: Vec = imp - .stack - .pages() - .iter::() - .filter_map(Result::ok) - .filter(|page| page.name().is_none_or(|name| name != "empty")) - .map(|page| page.child()) - .collect(); - for child in pages { - imp.stack.remove(&child); + for index in imp.built.take() { + if let Some(page) = imp.stack.child_by_name(&page_name(index)) { + imp.stack.remove(&page); + } } imp.stack.set_visible_child_name("empty"); imp.sidebar.set(SidebarState::default()); @@ -136,6 +140,12 @@ impl ResultsPane { imp.transaction_label.set_visible(false); } + /// A page is built when its line is first shown, so a script with + /// thousands of statements builds one, not thousands. + pub fn set_page_builder(&self, builder: impl Fn(u32) -> Option + 'static) { + self.imp().page_builder.replace(Some(Box::new(builder))); + } + pub fn set_running(&self, running: bool) { self.imp().spinner.set_visible(running); } @@ -150,16 +160,24 @@ impl ResultsPane { imp.transaction_label.set_visible(label.is_some()); } - /// Add a line and the widget it shows. - pub fn append(&self, entry: &RunLogEntry, widget: >k4::Widget) { + pub fn append(&self, entries: &[RunLogEntry]) { let imp = self.imp(); let Some(items) = imp.items.get() else { return; }; - let index = items.n_items(); - let (title, icon) = entry_title(index as usize, entry); - items.append(&ResultItem::new(&title, &entry.preview, icon)); - imp.stack.add_named(widget, Some(&page_name(index))); + if entries.is_empty() { + return; + } + let first = items.n_items(); + let added: Vec = entries + .iter() + .enumerate() + .map(|(offset, entry)| { + let (title, icon) = entry_title(first as usize + offset, entry); + ResultItem::new(&title, &entry.preview, icon) + }) + .collect(); + items.splice(first, 0, &added); let previous = SidebarState { show_sidebar: imp.split_view.shows_sidebar(), @@ -169,7 +187,7 @@ impl ResultsPane { imp.sidebar.set(state); imp.sidebar_toggle.set_visible(state.toggle_visible); imp.split_view.set_show_sidebar(state.show_sidebar); - if index == 0 { + if first == 0 { self.select(0); } } @@ -196,7 +214,7 @@ impl ResultsPane { fn show_page(&self, index: u32) { let imp = self.imp(); - if index == gtk4::INVALID_LIST_POSITION { + if index == gtk4::INVALID_LIST_POSITION || !self.keep_page(index) { return; } imp.stack.set_visible_child_name(&page_name(index)); @@ -205,6 +223,35 @@ impl ResultsPane { imp.split_view.set_show_sidebar(false); } } + + fn keep_page(&self, index: u32) -> bool { + let imp = self.imp(); + { + let mut built = imp.built.borrow_mut(); + if let Some(position) = built.iter().position(|kept| *kept == index) { + built.remove(position); + built.push_front(index); + return true; + } + } + let page = imp.page_builder.borrow().as_ref().and_then(|build| build(index)); + let Some(page) = page else { + return false; + }; + imp.stack.add_named(&page, Some(&page_name(index))); + let evicted: Vec = { + let mut built = imp.built.borrow_mut(); + built.push_front(index); + let keep = built.len().min(PAGES_KEPT); + built.drain(keep..).collect() + }; + for oldest in evicted { + if let Some(page) = imp.stack.child_by_name(&page_name(oldest)) { + imp.stack.remove(&page); + } + } + true + } } fn page_name(index: u32) -> String { @@ -304,7 +351,14 @@ mod tests { fn pane() -> ResultsPane { crate::register_test_resources(); - ResultsPane::default() + let pane = ResultsPane::default(); + pane.set_page_builder(|index| Some(gtk4::Label::new(Some(&index.to_string())).upcast())); + pane + } + + fn built_pages(pane: &ResultsPane) -> u32 { + // The stack always holds the empty state as well. + pane.imp().stack.pages().n_items() - 1 } fn entry(kind: EntryKind) -> RunLogEntry { @@ -319,16 +373,12 @@ mod tests { entry(EntryKind::Command { rows_affected: Some(1) }) } - fn page() -> gtk4::Widget { - gtk4::Label::new(Some("page")).upcast() - } - #[gtk4::test] fn twenty_results_use_list_not_switcher() { let pane = pane(); for _ in 0..20 { - pane.append(&command(), &page()); + pane.append(&[command()]); } let imp = pane.imp(); @@ -342,7 +392,7 @@ mod tests { fn a_single_result_has_no_list() { let pane = pane(); - pane.append(&command(), &page()); + pane.append(&[command()]); assert!(!pane.imp().sidebar_toggle.is_visible()); assert!(!pane.imp().split_view.shows_sidebar()); @@ -351,15 +401,14 @@ mod tests { #[gtk4::test] fn first_failure_is_selected() { let pane = pane(); - pane.append(&command(), &page()); - pane.append( - &entry(EntryKind::Failed { + pane.append(&[command()]); + pane.append(&[ + entry(EntryKind::Failed { message: "boom".to_owned(), error_char: None, }), - &page(), - ); - pane.append(&entry(EntryKind::NotRun), &page()); + entry(EntryKind::NotRun), + ]); pane.select(1); @@ -373,7 +422,7 @@ mod tests { let imp = pane.imp(); imp.split_view.set_collapsed(true); for _ in 0..3 { - pane.append(&command(), &page()); + pane.append(&[command()]); } imp.split_view.set_show_sidebar(true); @@ -391,15 +440,16 @@ mod tests { #[gtk4::test] fn clearing_forgets_the_last_run() { let pane = pane(); - pane.append(&command(), &page()); - pane.append(&command(), &page()); + pane.append(&[command(), command()]); + pane.select(1); pane.clear(); assert_eq!(pane.n_entries(), 0); assert!(!pane.imp().sidebar_toggle.is_visible()); assert_eq!(pane.imp().stack.visible_child_name().as_deref(), Some("empty")); - pane.append(&command(), &page()); + assert_eq!(built_pages(&pane), 0, "a page outlived its run"); + pane.append(&[command()]); assert_eq!(pane.imp().stack.visible_child_name().as_deref(), Some("entry-0")); } @@ -412,4 +462,47 @@ mod tests { gtk4::AccessibleProperty::Label )); } + + #[gtk4::test] + fn a_long_list_builds_only_the_page_it_shows() { + let pane = pane(); + let not_run = entry(EntryKind::NotRun); + let entries = vec![not_run; 100_000]; + + pane.append(&entries); + + assert_eq!(pane.n_entries(), 100_000); + assert_eq!(built_pages(&pane), 1); + assert_eq!(pane.imp().stack.visible_child_name().as_deref(), Some("entry-0")); + } + + #[gtk4::test] + fn paging_through_the_list_keeps_a_bounded_number_of_pages() { + let pane = pane(); + pane.append(&vec![command(); 50]); + + for index in 0..50 { + pane.select(index); + } + + assert_eq!(built_pages(&pane), PAGES_KEPT as u32); + assert_eq!(pane.imp().stack.visible_child_name().as_deref(), Some("entry-49")); + assert!( + pane.imp().stack.child_by_name("entry-0").is_none(), + "the oldest page was kept" + ); + } + + #[gtk4::test] + fn going_back_to_a_recent_result_keeps_its_page() { + let pane = pane(); + pane.append(&[command(), command()]); + pane.select(0); + let first = pane.imp().stack.visible_child().expect("the first page"); + + pane.select(1); + pane.select(0); + + assert_eq!(pane.imp().stack.visible_child().as_ref(), Some(&first)); + } } diff --git a/linux/crates/app/src/ui/editor/run_scope.rs b/linux/crates/app/src/ui/editor/run_scope.rs index 479022c345..658a00bb63 100644 --- a/linux/crates/app/src/ui/editor/run_scope.rs +++ b/linux/crates/app/src/ui/editor/run_scope.rs @@ -1,6 +1,6 @@ use std::ops::Range; -use tablepro_core::sql_syntax::script::{ScriptPlan, text_offsets}; +use tablepro_core::sql_syntax::script::text_offsets; /// The SQL one run sends, and where it starts in the buffer. #[derive(Debug, Clone, PartialEq, Eq)] @@ -27,26 +27,6 @@ pub fn for_run(buffer_text: &str, selection: Option>) -> Option Option<(ScopedText, Range)> { - let byte = text_offsets::char_to_byte(buffer_text, cursor_char); - let statement = plan.statement_at(byte)?; - let start = text_offsets::byte_to_char(buffer_text, statement.range.start); - let end = text_offsets::byte_to_char(buffer_text, statement.range.end); - let text = statement.text(buffer_text); - if text.trim().is_empty() { - return None; - } - Some(( - ScopedText { - text: text.to_owned(), - base_char: start, - }, - start..end, - )) -} - fn char_slice(text: &str, range: Range) -> &str { let start = text_offsets::char_to_byte(text, range.start); let end = text_offsets::char_to_byte(text, range.end); @@ -55,9 +35,6 @@ fn char_slice(text: &str, range: Range) -> &str { #[cfg(test)] mod tests { - use tablepro_core::sql_syntax::SqlGrammar; - use tablepro_core::sql_syntax::script::LexicalSettings; - use super::*; #[test] @@ -83,23 +60,4 @@ mod tests { assert_eq!(selected.text, "SELECT 2"); } - - #[test] - fn run_scope_statement_at_cursor_uses_plan() { - let buffer = "CREATE FUNCTION f() RETURNS int AS $$ SELECT 1; $$ LANGUAGE sql;\nSELECT 2;"; - let plan = ScriptPlan::build( - buffer, - SqlGrammar::PostgreSql, - LexicalSettings::default_for(SqlGrammar::PostgreSql), - ); - - let (inside_body, range) = at_cursor(buffer, &plan, 40).expect("a statement"); - assert!(inside_body.text.starts_with("CREATE FUNCTION"), "{}", inside_body.text); - assert!(inside_body.text.ends_with("LANGUAGE sql"), "{}", inside_body.text); - assert_eq!(range.start, 0); - - let (second, _) = at_cursor(buffer, &plan, buffer.chars().count()).expect("the last statement"); - assert_eq!(second.text, "SELECT 2"); - assert_eq!(second.base_char, buffer.find("SELECT 2").unwrap_or_default()); - } } diff --git a/linux/crates/app/src/ui/editor/significant_tokens.rs b/linux/crates/app/src/ui/editor/significant_tokens.rs index fdb929501b..6d4586bfd0 100644 --- a/linux/crates/app/src/ui/editor/significant_tokens.rs +++ b/linux/crates/app/src/ui/editor/significant_tokens.rs @@ -1,46 +1,139 @@ use tablepro_core::sql_syntax::SqlGrammar; use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan, ScriptTokenKind}; -/// What a statement says, with the spacing taken out. +/// What a statement says, with the spacing taken out where spacing +/// cannot change it. /// /// Two texts with the same list are the same statement written /// differently, which is the only thing a formatter is allowed to do. -pub fn significant_tokens( - text: &str, - grammar: SqlGrammar, - settings: LexicalSettings, -) -> Vec<(ScriptTokenKind, String)> { +pub fn significant_tokens(text: &str, grammar: SqlGrammar, settings: LexicalSettings) -> Vec { let plan = ScriptPlan::build(text, grammar, settings); - plan.tokens(0..text.len()) - .filter(|token| token.kind != ScriptTokenKind::Whitespace) - .map(|token| { - let slice = text.get(token.range.clone()).unwrap_or_default(); - let body = match token.kind { - // A formatter may move the spaces at the end of a - // comment without changing what the comment says. - ScriptTokenKind::LineComment | ScriptTokenKind::BlockComment => slice.trim_end(), - _ => slice, - }; - (token.kind, body.to_owned()) - }) - .collect() + let mut tokens: Vec = Vec::new(); + let mut spacing = Spacing::default(); + for token in plan.tokens(0..text.len()) { + let slice = text.get(token.range.clone()).unwrap_or_default(); + if token.kind == ScriptTokenKind::Whitespace { + spacing.spaced = true; + spacing.line_break |= slice.contains('\n'); + continue; + } + let body = match token.kind { + // A formatter may move the spaces at the end of a + // comment without changing what the comment says. + ScriptTokenKind::LineComment | ScriptTokenKind::BlockComment => slice.trim_end(), + _ => slice, + }; + let gap = tokens.last().map_or(Gap::Free, |previous| { + gap_between(grammar, previous, token.kind, body, spacing) + }); + tokens.push(SignificantToken { + kind: token.kind, + text: body.to_owned(), + gap, + }); + spacing = Spacing::default(); + } + tokens +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SignificantToken { + kind: ScriptTokenKind, + text: String, + gap: Gap, +} + +/// The space between two tokens, kept only where an engine reads the +/// two differently with and without it. +/// +/// The script lexer reads every operator character on its own, where +/// an engine reads `+=` or `->>` as one operator; and it reads `b'01'` +/// as a word and a string, where MySQL reads one bit literal. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Gap { + Free, + Touching, + Spaced, + /// Two string literals with a line break between are one string in + /// PostgreSQL; on one line they are a syntax error. + LineBreak, +} + +/// Words sqlformat spaces from their paren that MySQL 8.4 and MariaDB +/// 11.4 were checked to read the same either way. +const MYSQL_SPACED_CALLS: [&str; 7] = ["EXISTS", "IF", "IFNULL", "IN", "REPLACE", "USING", "VALUES"]; + +#[derive(Debug, Clone, Copy, Default)] +struct Spacing { + spaced: bool, + line_break: bool, +} + +fn gap_between( + grammar: SqlGrammar, + previous: &SignificantToken, + kind: ScriptTokenKind, + text: &str, + spacing: Spacing, +) -> Gap { + let strings = previous.kind == ScriptTokenKind::StringLiteral && kind == ScriptTokenKind::StringLiteral; + if strings && spacing.line_break { + return Gap::LineBreak; + } + let operator = previous.kind == ScriptTokenKind::Punct + && kind == ScriptTokenKind::Punct + && previous.text != "," + && text != ","; + // Without IGNORE_SPACE, MySQL reads `count(` as the built-in and + // `count (` as a stored function named count. + let call = grammar == SqlGrammar::MySql + && previous.kind == ScriptTokenKind::Word + && kind == ScriptTokenKind::OpenParen + && !MYSQL_SPACED_CALLS + .iter() + .any(|word| word.eq_ignore_ascii_case(&previous.text)); + let free = spacing_is_free(previous.kind, &previous.text) || spacing_is_free(kind, text); + if !strings && !operator && !call && free { + return Gap::Free; + } + if spacing.spaced { Gap::Spaced } else { Gap::Touching } +} + +/// Whether spacing next to this token changes nothing an engine reads, +/// outside the pairs `gap_between` checks first: a bracket, a +/// separator, a comment, or an operator character that is never part +/// of a name, a literal or a parameter. +fn spacing_is_free(kind: ScriptTokenKind, text: &str) -> bool { + match kind { + ScriptTokenKind::OpenParen + | ScriptTokenKind::CloseParen + | ScriptTokenKind::Semicolon + | ScriptTokenKind::LineComment + | ScriptTokenKind::BlockComment => true, + ScriptTokenKind::Punct => matches!( + text, + "," | "." | "=" | "<" | ">" | "+" | "-" | "*" | "/" | "%" | "^" | "|" | "!" | "~" | "[" | "]" + ), + _ => false, + } } #[cfg(test)] mod tests { use super::*; - fn tokens(text: &str) -> Vec<(ScriptTokenKind, String)> { - significant_tokens( - text, - SqlGrammar::PostgreSql, - LexicalSettings::default_for(SqlGrammar::PostgreSql), - ) + fn tokens_for(text: &str, grammar: SqlGrammar) -> Vec { + significant_tokens(text, grammar, LexicalSettings::default_for(grammar)) + } + + fn tokens(text: &str) -> Vec { + tokens_for(text, SqlGrammar::PostgreSql) } #[test] fn spacing_alone_does_not_change_a_statement() { assert_eq!(tokens("select a ,b from t"), tokens("select a, b\nfrom t")); + assert_eq!(tokens("select a=-1,(1,-1)"), tokens("select a =- 1, (1, -1)")); } #[test] @@ -53,4 +146,49 @@ mod tests { assert_eq!(tokens("select 1 -- note "), tokens("select 1 -- note")); assert_ne!(tokens("select 1 -- note"), tokens("select 1 -- other")); } + + #[test] + fn an_operator_split_by_a_space_changes_the_statement() { + assert_ne!( + tokens_for("SET @x += 1", SqlGrammar::MsSql), + tokens_for("SET @x + = 1", SqlGrammar::MsSql) + ); + assert_ne!(tokens("select a->>'k'"), tokens("select a- >>'k'")); + assert_eq!(tokens("select a->>'k'"), tokens("select a ->> 'k'")); + } + + #[test] + fn a_prefix_split_from_its_string_changes_the_statement() { + assert_ne!( + tokens_for("select b'01'", SqlGrammar::MySql), + tokens_for("select b '01'", SqlGrammar::MySql) + ); + assert_ne!( + tokens_for("select @@version", SqlGrammar::MySql), + tokens_for("select @@ version", SqlGrammar::MySql) + ); + } + + #[test] + fn a_space_before_a_mysql_call_changes_the_statement() { + assert_ne!( + tokens_for("select count(a) from t", SqlGrammar::MySql), + tokens_for("select count (a) from t", SqlGrammar::MySql) + ); + assert_eq!( + tokens_for("select count(a) from t", SqlGrammar::MySql), + tokens_for("select\n count(a)\nfrom t", SqlGrammar::MySql) + ); + assert_eq!(tokens("select count(a) from t"), tokens("select count (a) from t")); + assert_eq!( + tokens_for("select ifnull(a, 0) from t where a in(1)", SqlGrammar::MySql), + tokens_for("select ifnull (a, 0) from t where a in (1)", SqlGrammar::MySql) + ); + } + + #[test] + fn strings_joined_onto_one_line_change_the_statement() { + assert_ne!(tokens("select 'a'\n'b'"), tokens("select 'a' 'b'")); + assert_eq!(tokens("select 'a'\n'b'"), tokens("select\n 'a'\n 'b'")); + } } diff --git a/linux/crates/app/src/ui/editor/sql_file.rs b/linux/crates/app/src/ui/editor/sql_file.rs new file mode 100644 index 0000000000..d2ca5d9aed --- /dev/null +++ b/linux/crates/app/src/ui/editor/sql_file.rs @@ -0,0 +1,169 @@ +use gtk4::glib; +use gtk4::prelude::*; + +/// The most a file may be before the editor refuses it. +/// +/// A dump can be gigabytes, and a text buffer holding one stops being +/// an editor. Refusing is better than a window that will not paint. +const MAX_SQL_FILE_BYTES: usize = 8 * 1024 * 1024; + +const READ_CHUNK_BYTES: usize = 256 * 1024; + +#[derive(Debug)] +enum ReadError { + TooLarge, + Io(glib::Error), +} + +/// Read a file as text off the main thread, or say why it cannot be. +pub async fn load_text(file: &gio::File) -> Result { + let info = file + .query_info_future( + "standard::type,standard::size", + gio::FileQueryInfoFlags::NONE, + glib::Priority::DEFAULT, + ) + .await; + if let Ok(info) = info { + // Opening a pipe waits for a writer that may never come, and a + // device has no end to read to. + if info.has_attribute(gio::FILE_ATTRIBUTE_STANDARD_TYPE) && info.file_type() == gio::FileType::Special { + return Err(crate::i18n::gettext("Only regular files can be opened in the editor.")); + } + if info.has_attribute(gio::FILE_ATTRIBUTE_STANDARD_SIZE) + && u64::try_from(info.size()).is_ok_and(|size| size > MAX_SQL_FILE_BYTES as u64) + { + return Err(too_large()); + } + } + let stream = file.read_future(glib::Priority::DEFAULT).await.map_err(unreadable)?; + let read = read_capped(stream.upcast_ref(), MAX_SQL_FILE_BYTES).await; + if let Err(error) = stream.close_future(glib::Priority::DEFAULT).await { + tracing::debug!(%error, "closing a file the editor read failed"); + } + let bytes = match read { + Ok(bytes) => bytes, + Err(ReadError::TooLarge) => return Err(too_large()), + Err(ReadError::Io(error)) => return Err(unreadable(error)), + }; + String::from_utf8(bytes).map_err(|_| crate::i18n::gettext("That file is not UTF-8 text.")) +} + +/// Read to the end of the stream, giving up once it passes `limit`. +/// +/// The size a file reports is not enough: a remote file may report +/// none, and a file can grow between being measured and being read. +async fn read_capped(stream: &gio::InputStream, limit: usize) -> Result, ReadError> { + let mut contents = Vec::new(); + loop { + let chunk = stream + .read_bytes_future(READ_CHUNK_BYTES, glib::Priority::DEFAULT) + .await + .map_err(ReadError::Io)?; + if chunk.is_empty() { + return Ok(contents); + } + if contents.len() + chunk.len() > limit { + return Err(ReadError::TooLarge); + } + contents.extend_from_slice(&chunk); + } +} + +fn too_large() -> String { + crate::i18n::gettext("That file is too large to open in the editor.") +} + +fn unreadable(error: glib::Error) -> String { + tracing::warn!(%error, "could not read a file for the editor"); + crate::i18n::gettext("That file could not be read.") +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use super::*; + use crate::test_support::wait_until; + + const WAIT: Duration = Duration::from_secs(10); + + fn load(file: gio::File) -> Result { + let outcome = std::rc::Rc::new(std::cell::RefCell::new(None)); + let slot = outcome.clone(); + glib::spawn_future_local(async move { + let loaded = load_text(&file).await; + slot.replace(Some(loaded)); + }); + wait_until(WAIT, || outcome.borrow().is_some()).expect("the load to end"); + outcome.take().expect("an outcome") + } + + fn capped(bytes: Vec, limit: usize) -> Result, ReadError> { + let stream = gio::MemoryInputStream::from_bytes(&glib::Bytes::from_owned(bytes)); + let outcome = std::rc::Rc::new(std::cell::RefCell::new(None)); + let slot = outcome.clone(); + glib::spawn_future_local(async move { + let read = read_capped(stream.upcast_ref(), limit).await; + slot.replace(Some(read)); + }); + wait_until(WAIT, || outcome.borrow().is_some()).expect("the read to end"); + outcome.take().expect("an outcome") + } + + #[gtk4::test] + fn a_stream_past_the_limit_is_refused_without_a_reported_size() { + let read = capped(vec![b'x'; READ_CHUNK_BYTES * 3], READ_CHUNK_BYTES * 2); + + assert!(matches!(read, Err(ReadError::TooLarge)), "{read:?}"); + } + + #[gtk4::test] + fn a_stream_at_the_limit_is_read_whole() { + let read = capped(vec![b'x'; READ_CHUNK_BYTES * 2], READ_CHUNK_BYTES * 2).expect("the contents"); + + assert_eq!(read.len(), READ_CHUNK_BYTES * 2); + } + + #[gtk4::test] + fn a_device_is_refused_before_it_is_read() { + let refused = load(gio::File::for_path("/dev/zero")).expect_err("a refusal"); + + assert!(refused.contains("regular files"), "{refused}"); + } + + #[gtk4::test] + fn a_pipe_is_refused_without_waiting_for_a_writer() { + let root = tempfile::tempdir().expect("tempdir"); + let path = root.path().join("script.sql"); + let made = std::process::Command::new("mkfifo") + .arg(&path) + .status() + .expect("mkfifo"); + assert!(made.success()); + + let refused = load(gio::File::for_path(&path)).expect_err("a refusal"); + + assert!(refused.contains("regular files"), "{refused}"); + } + + #[gtk4::test] + fn a_file_over_the_limit_is_refused() { + let root = tempfile::tempdir().expect("tempdir"); + let path = root.path().join("dump.sql"); + std::fs::write(&path, vec![b' '; MAX_SQL_FILE_BYTES + 1]).expect("write"); + + let refused = load(gio::File::for_path(&path)).expect_err("a refusal"); + + assert!(refused.contains("too large"), "{refused}"); + } + + #[gtk4::test] + fn a_small_file_is_read_as_text() { + let root = tempfile::tempdir().expect("tempdir"); + let path = root.path().join("query.sql"); + std::fs::write(&path, "SELECT 'tên';").expect("write"); + + assert_eq!(load(gio::File::for_path(&path)).as_deref(), Ok("SELECT 'tên';")); + } +} diff --git a/linux/crates/app/src/ui/editor/statement_cursor.rs b/linux/crates/app/src/ui/editor/statement_cursor.rs index 4c634575ad..7d97bd7255 100644 --- a/linux/crates/app/src/ui/editor/statement_cursor.rs +++ b/linux/crates/app/src/ui/editor/statement_cursor.rs @@ -1,5 +1,6 @@ +use gtk4::prelude::*; use tablepro_core::sql_syntax::SqlGrammar; -use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan}; +use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan, text_offsets}; /// The script the buffer holds, read with the engine's own rules: a /// dollar-quoted body, a `GO` batch or a procedure between `DELIMITER` @@ -8,46 +9,117 @@ pub fn plan_for(text: &str, grammar: SqlGrammar) -> ScriptPlan { ScriptPlan::build(text, grammar, LexicalSettings::default_for(grammar)) } +/// Where the statement under the cursor starts and ends in the buffer. +/// +/// The buffer counts characters and the plan counts bytes, so every +/// mapping between the two goes through here: a cursor in a Vietnamese +/// identifier lands mid-character otherwise. +pub fn statement_bounds_at( + buffer: >k4::TextBuffer, + plan: &ScriptPlan, + text: &str, + cursor_char: usize, +) -> Option<(gtk4::TextIter, gtk4::TextIter)> { + let byte = text_offsets::char_to_byte(text, cursor_char); + let statement = plan.statement_at(byte)?; + Some(( + iter_for_byte(buffer, text, statement.range.start), + iter_for_byte(buffer, text, statement.range.end), + )) +} + +/// The buffer position of a byte offset into `text`, which must be the +/// buffer's text. Past the end is the end. +pub fn iter_for_byte(buffer: >k4::TextBuffer, text: &str, byte: usize) -> gtk4::TextIter { + let chars = i32::try_from(text_offsets::byte_to_char(text, byte)).unwrap_or(i32::MAX); + buffer.iter_at_offset(chars) +} + #[cfg(test)] mod tests { - use super::super::run_scope::at_cursor; use super::*; - fn statement_at(text: &str, cursor_char: usize) -> Option { + fn buffer(text: &str) -> gtk4::TextBuffer { + let buffer = gtk4::TextBuffer::new(None); + buffer.set_text(text); + buffer + } + + fn bounds_at(text: &str, cursor_char: usize) -> Option<(i32, i32, String)> { + let buffer = buffer(text); let plan = plan_for(text, SqlGrammar::PostgreSql); - at_cursor(text, &plan, cursor_char).map(|(scoped, _)| scoped.text) + let (start, end) = statement_bounds_at(&buffer, &plan, text, cursor_char)?; + Some(( + start.offset(), + end.offset(), + buffer.text(&start, &end, false).to_string(), + )) } - #[test] + #[gtk4::test] + fn statement_bounds_multibyte_text() { + let text = "SELECT 'tên'; SELECT \"bảng\" FROM t"; + + let (start, end, statement) = bounds_at(text, 30).expect("a statement"); + + assert_eq!(statement, "SELECT \"bảng\" FROM t"); + assert_eq!((start, end), (14, 34), "the bounds were not mapped to characters"); + } + + #[gtk4::test] + fn a_cursor_in_a_multibyte_identifier_stays_in_its_statement() { + // Character 9 is inside the quoted identifier, which is three + // bytes further along than it is characters. + let (_, _, statement) = bounds_at("SELECT \"tên\" FROM t; SELECT 2", 9).expect("a statement"); + + assert_eq!(statement, "SELECT \"tên\" FROM t"); + } + + #[gtk4::test] fn the_cursor_picks_the_statement_it_sits_in() { - let sql = "SELECT 1; SELECT 2"; + let text = "SELECT 1; SELECT 2"; - assert_eq!(statement_at(sql, 4).as_deref(), Some("SELECT 1")); - assert_eq!(statement_at(sql, 17).as_deref(), Some("SELECT 2")); + assert_eq!(bounds_at(text, 4).map(|(_, _, sql)| sql).as_deref(), Some("SELECT 1")); + assert_eq!(bounds_at(text, 17).map(|(_, _, sql)| sql).as_deref(), Some("SELECT 2")); } - #[test] + #[gtk4::test] fn a_cursor_past_the_end_picks_the_last_statement() { - assert_eq!(statement_at("SELECT 1; SELECT 2", 9999).as_deref(), Some("SELECT 2")); + let statement = bounds_at("SELECT 1; SELECT 2", 9999).map(|(_, _, sql)| sql); + + assert_eq!(statement.as_deref(), Some("SELECT 2")); } - #[test] + #[gtk4::test] fn a_semicolon_inside_a_string_is_not_a_boundary() { - let sql = "INSERT INTO t VALUES ('a;b'); SELECT 2"; + let (_, _, statement) = bounds_at("INSERT INTO t VALUES ('a;b'); SELECT 2", 24).expect("a statement"); + + assert_eq!(statement, "INSERT INTO t VALUES ('a;b')"); + } - let statement = statement_at(sql, 24).expect("a statement"); + #[gtk4::test] + fn a_dollar_quoted_body_is_one_statement() { + let text = "CREATE FUNCTION f() RETURNS int AS $$ SELECT 1; $$ LANGUAGE sql;\nSELECT 2;"; - assert!(statement.starts_with("INSERT INTO t VALUES"), "{statement}"); - assert!(statement.contains("'a;b'"), "{statement}"); + let (start, _, statement) = bounds_at(text, 40).expect("a statement"); + + assert_eq!(start, 0); + assert!(statement.ends_with("LANGUAGE sql"), "{statement}"); } - #[test] - fn a_cursor_in_a_multibyte_identifier_stays_in_its_statement() { - let sql = "SELECT \"tên\" FROM t; SELECT 2"; - // Character 9 is inside the quoted identifier, which is three - // bytes further along than it is characters. - let statement = statement_at(sql, 9).expect("a statement"); + #[gtk4::test] + fn an_empty_buffer_has_no_statement() { + assert_eq!(bounds_at("", 0), None); + assert_eq!(bounds_at(" \n -- only a comment\n", 3), None); + } + + #[gtk4::test] + fn iter_for_byte_at_end() { + let text = "SELECT 'tên'"; + let buffer = buffer(text); - assert!(statement.starts_with("SELECT \"tên\""), "{statement}"); + assert!(iter_for_byte(&buffer, text, text.len()).is_end()); + assert!(iter_for_byte(&buffer, text, text.len() + 10).is_end()); + assert_eq!(iter_for_byte(&buffer, text, text.len() - 1).offset(), 11); } } diff --git a/linux/crates/app/src/ui/editor/tests.rs b/linux/crates/app/src/ui/editor/tests.rs index 155f5c6985..8db240ce2f 100644 --- a/linux/crates/app/src/ui/editor/tests.rs +++ b/linux/crates/app/src/ui/editor/tests.rs @@ -95,6 +95,43 @@ fn selected_text_is_the_only_sql_sent() { assert_eq!(sent(&harness.fake), ["SELECT 2"]); } +#[gtk4::test] +fn run_at_cursor_selects_executed_statement() { + let harness = editor_with(answering(), "SELECT 'tên';\nSELECT 2 FROM t;\nSELECT 3;"); + let buffer = harness.editor.buffer(); + buffer.place_cursor(&buffer.iter_at_offset(20)); + + harness.editor.run_at_cursor(); + assert!(harness.editor.is_running(), "the run did not start"); + wait_until(WAIT, || !harness.editor.is_running()).expect("the run to end"); + + assert_eq!(sent(&harness.fake), ["SELECT 2 FROM t"]); + let (start, end) = buffer.selection_bounds().expect("the statement is selected"); + assert_eq!((start.offset(), end.offset()), (14, 29)); +} + +#[gtk4::test] +fn run_at_cursor_during_a_run_leaves_the_buffer_alone() { + let mut runs = Replies::default(); + runs.always(FakeRun::until_stopped()); + let harness = editor_with(runs, "SELECT pg_sleep(60);\nSELECT 2;"); + let buffer = harness.editor.buffer(); + harness.editor.run(); + wait_until(WAIT, || !harness.fake.texts().is_empty()).expect("the first statement to start"); + let sent_before = sent(&harness.fake); + buffer.place_cursor(&buffer.iter_at_offset(23)); + + harness.editor.run_at_cursor(); + + assert!( + buffer.selection_bounds().is_none(), + "a statement that did not run was selected" + ); + assert_eq!(sent(&harness.fake), sent_before); + harness.editor.stop(); + wait_until(WAIT, || !harness.editor.is_running()).expect("the run to end"); +} + #[gtk4::test] fn a_script_fills_the_results_list_one_line_per_statement() { let mut runs = Replies::default(); @@ -118,6 +155,30 @@ fn a_script_fills_the_results_list_one_line_per_statement() { assert!(harness.editor.imp().run_button.is_sensitive()); } +#[gtk4::test] +fn a_long_script_that_fails_early_builds_one_result_page() { + let mut runs = Replies::default(); + runs.when_sql_contains( + "boom", + FakeRun::failed(DriverError::server("relation \"boom\" does not exist")), + ); + runs.always(FakeRun::completed(Vec::new())); + let script: Vec = std::iter::once("SELECT * FROM boom;".to_owned()) + .chain((1..2_000).map(|n| format!("SELECT {n};"))) + .collect(); + let harness = editor_with(runs, &script.join("\n")); + + run_and_wait(&harness); + + let results = &harness.editor.imp().results; + assert_eq!(results.n_entries(), 2_000); + assert_eq!( + results.imp().stack.pages().n_items(), + 2, + "pages were built for lines nobody opened" + ); +} + #[gtk4::test] fn an_open_transaction_marks_the_tab() { let mut runs = Replies::default(); diff --git a/linux/crates/app/src/ui/grid.rs b/linux/crates/app/src/ui/grid.rs index 75366366d8..b06d4b68f7 100644 --- a/linux/crates/app/src/ui/grid.rs +++ b/linux/crates/app/src/ui/grid.rs @@ -12,6 +12,9 @@ use tablepro_core::column::ResultColumn; use tablepro_core::{ColumnInfo, QueryResult, Value}; use super::row_object::RowObject; +use crate::services::cell_overlay::CellOverlay; +use crate::services::change_tracker::{CellState, RowKey, RowState}; +use crate::services::tab_tracker_handle::TabTrackerHandle; #[derive(Debug)] pub enum GridMsg { @@ -71,13 +74,11 @@ pub enum CellPreset { Null, } -/// Per-tab context plumbed into the grid factory so cell bind-time -/// callbacks can query the change tracker for pending-state CSS -/// classes. `tab_id == None` means the grid is read-only / not -/// associated with a tracked Browse tab (e.g., editor results). +/// What a browse grid draws pending edits from. `tracker` is `None` for +/// a grid nothing tracks, such as editor results. #[derive(Debug, Clone, Default)] pub struct TabGridContext { - pub tab_id: Option, + pub tracker: Option, pub pk_col_indices: Vec, /// Where a dragged column edge is remembered. `None` for a grid /// with nothing to remember it against, such as editor results. @@ -119,31 +120,18 @@ impl ColumnWidthBinding { } impl TabGridContext { - /// The tracker key for a persisted row. `None` when the grid isn't - /// backed by a tracked tab, when the row is a draft (its own cells - /// are the tracker's mirror and already authoritative), or when - /// the row carries no usable primary key. - fn tracked_key(&self, row: &RowObject) -> Option<(uuid::Uuid, crate::services::change_tracker::RowKey)> { - let tab_id = self.tab_id?; - if row.draft_id().is_some() { - return None; - } - let pk_values: Vec = self.pk_col_indices.iter().map(|&i| row.cell_value(i)).collect(); - let key = crate::services::change_tracker::RowKey::from_pk_values(&pk_values)?; - Some((tab_id, key)) - } - - /// What the grid is showing for one cell: the tracker's pending - /// edit when there is one, else the row's stored value. - fn effective_cell(&self, row: &RowObject, idx: usize) -> Value { - let raw = row.cell_value(idx); - let Some((tab_id, key)) = self.tracked_key(row) else { - return raw; + /// The tracker and the row's key in it. `None` when nothing tracks + /// the grid or a persisted row has no usable primary key. + fn tracked_key(&self, row: &RowObject) -> Option<(&TabTrackerHandle, RowKey)> { + let tracker = self.tracker.as_ref()?; + let key = match row.draft_id() { + Some(draft_id) => RowKey::Draft(draft_id), + None => { + let pk_values: Vec = self.pk_col_indices.iter().map(|&i| row.cell_value(i)).collect(); + RowKey::from_pk_values(&pk_values)? + } }; - match crate::services::change_tracker::with_tab_ref(tab_id, |t| t.current_cell_value(&key, idx, &raw).clone()) { - Some(value) => value, - None => raw, - } + Some((tracker, key)) } /// The whole row as the grid is showing it. Every copy and export @@ -151,19 +139,41 @@ impl TabGridContext { /// a persisted row holds the untouched values the fetch returned. pub(super) fn effective_cells(&self, row: &RowObject) -> Vec { let raw = row.cells_clone(); - let Some((tab_id, key)) = self.tracked_key(row) else { + let Some((tracker, key)) = self.tracked_key(row) else { return raw; }; - match crate::services::change_tracker::with_tab_ref(tab_id, |t| { - raw.iter() - .enumerate() - .map(|(i, v)| t.current_cell_value(&key, i, v).clone()) - .collect::>() - }) { - Some(cells) => cells, - None => raw, - } + raw.iter() + .enumerate() + .map(|(col, value)| tracker.cell_overlay(&key, col, value).value) + .collect() + } + + /// The value a bind draws and the pending-state classes it carries. + fn bind_state(&self, row: &RowObject, col: usize) -> (Value, Vec<&'static str>) { + let raw = row.cell_value(col); + let Some((tracker, key)) = self.tracked_key(row) else { + return (raw, Vec::new()); + }; + let overlay = tracker.cell_overlay(&key, col, &raw); + let classes = pending_classes(tracker.row_state(&key), &overlay, col); + (overlay.value, classes) + } +} + +/// Pending state shows as the row's tint alone: insert green, modified +/// orange, delete red with a strikethrough. +fn pending_classes(row_state: RowState, overlay: &CellOverlay, col: usize) -> Vec<&'static str> { + let mut classes = Vec::new(); + match (row_state, overlay.state) { + (RowState::PendingDelete, _) => classes.push("tp-row-pending-delete"), + (RowState::InsertDraft, _) => classes.push("tp-row-pending-insert"), + (_, CellState::Modified) => classes.push("tp-cell-modified"), + _ => {} + } + if col == 0 && overlay.error_flash { + classes.push("tp-row-leftmost-error-flash"); } + classes } pub fn build_column_view( @@ -377,20 +387,7 @@ fn build_column( let Some(row) = item.item().and_downcast::() else { return; }; - // For persisted rows, RowObject.cells holds the immutable - // original DB values — the tracker is the single source of - // truth for pending edits. Override with the tracker's - // pending value when one exists so the cell visibly - // reflects what the user typed (with `tp-cell-modified` - // applied below for the orange tint). Without this, edits - // appear to "disappear" the moment the cell rebinds, which - // is exactly the symptom the user hit on undo. - // - // Drafts skip this branch — their cells live inside the - // tracker's `inserts.values` AND mirror onto RowObject - // (set_cell at edit time), so `row.cell_value(idx)` is - // already authoritative for them. - let value = tab_ctx_for_bind.effective_cell(&row, idx); + let (value, pending_classes) = tab_ctx_for_bind.bind_state(&row, idx); let is_null = matches!(value, Value::Null); // Editable cells render NULL as the italic sentinel — // distinguishes a true NULL from an empty string visually. @@ -411,55 +408,6 @@ fn build_column( value_to_display_text(&value) }; - // Query the per-tab change tracker for pending state on this - // (row, col). Apply tp-cell-modified / tp-row-pending-delete / - // tp-row-pending-insert CSS classes accordingly. Done at bind - // time so scroll-recycled widgets always reflect current - // tracker state without needing per-cell signal subscriptions. - // - // Draft rows are detected via RowObject's draft_id field - // (set when the row was created via the inline-Insert flow). - // Persisted rows are keyed by PK column values via - // RowKey::from_pk_values. - // - // Pending state is communicated through the row's BACKGROUND - // TINT alone (insert = green, modified = orange via - // `.tp-cell-modified`, delete = red+strikethrough). No - // leftmost-cell ribbon — that custom GNOME-Builder-style - // gutter felt foreign next to the AdwListView idiom, and - // the tint+strikethrough already make the row state legible - // at a glance. - let pending_classes: Vec<&'static str> = if tab_ctx_for_bind.tab_id.is_none() { - Vec::new() - } else if row.draft_id().is_some() { - vec!["tp-row-pending-insert"] - } else { - match tab_ctx_for_bind.tracked_key(&row) { - None => Vec::new(), - Some((tab_id, key)) => crate::services::change_tracker::with_tab_ref(tab_id, |t| { - let mut v: Vec<&'static str> = Vec::new(); - let row_state = t.row_state(&key); - let cell_state = t.cell_state(&key, idx); - use crate::services::change_tracker::{CellState, RowState}; - match (row_state, cell_state) { - (RowState::PendingDelete, _) => v.push("tp-row-pending-delete"), - (RowState::InsertDraft, _) => v.push("tp-row-pending-insert"), - (_, CellState::Modified) => v.push("tp-cell-modified"), - _ => {} - } - // Error-flash overlay on the leftmost cell — a - // transient background-only animation pulls the - // eye to the row that failed to commit. No - // accompanying gutter ribbon (see comment above). - if idx == 0 && t.is_error_row(&key) { - v.push("tp-row-leftmost-error-flash"); - } - v - }) - .unwrap_or_default(), - } - }; - let is_pending_delete = pending_classes.contains(&"tp-row-pending-delete"); let Some(child) = item.child() else { return }; if let Ok(label) = child.clone().downcast::() { @@ -2210,6 +2158,100 @@ mod tests { } } + fn cell_texts(root: &impl IsA) -> Vec { + crate::test_support::descendants(root) + .into_iter() + .filter_map(|widget| widget.downcast::().ok()) + .map(|cell| cell.text().to_string()) + .collect() + } + + #[gtk4::test] + fn edit_during_rebind_shows_pending_value() { + use crate::services::tracker_edit::TrackerEdit; + + let mut id = col("integer", true); + id.name = "id".into(); + let mut name = col("text", false); + name.name = "name".into(); + let columns = vec![id, name]; + let result = QueryResult::new( + columns.iter().map(ColumnInfo::result_column).collect::>(), + vec![vec![Value::Int(1), Value::Text("old".into())]], + ); + let tracker = TabTrackerHandle::default(); + let (sender, _receiver) = relm4::channel::(); + let context = TabGridContext { + tracker: Some(tracker.clone()), + pk_col_indices: vec![0], + column_widths: None, + }; + let (view, selection) = build_column_view(&result, &columns, sender, true, None, None, context); + let window = gtk::Window::new(); + window.set_child(Some(&view)); + window.present(); + crate::test_support::wait_until(std::time::Duration::from_secs(5), || { + cell_texts(&view).iter().any(|text| text == "old") + }) + .expect("the fetched value is drawn"); + + let delta = tracker.apply(TrackerEdit::Cell { + row_key: RowKey::from_pk_values(&[Value::Int(1)]).expect("a key"), + col: 1, + original: Value::Text("old".into()), + value: Value::Text("new".into()), + }); + // A fresh row object, because the list item manager skips the + // rebind for an items-changed that keeps the same item. + let store = selection + .model() + .and_downcast::() + .expect("a list store"); + let row = store.item(0).and_downcast::().expect("a row"); + store.splice(0, 1, &[RowObject::new(row.cells_clone())]); + + crate::test_support::wait_until(std::time::Duration::from_secs(5), || { + cell_texts(&view).iter().any(|text| text == "new") + }) + .expect("the rebind draws the pending value"); + assert_eq!(delta.pending_count, 1); + window.close(); + crate::test_support::drain_main_context(); + } + + #[test] + fn a_tracked_draft_carries_its_insert_tint_and_error_flash() { + let tracker = TabTrackerHandle::default(); + let delta = tracker.apply(crate::services::tracker_edit::TrackerEdit::Insert { + values: vec![Value::Null], + }); + let Some(crate::services::tracker_delta::VisualChange::InsertDraft { draft_id, values }) = + delta.visual.into_iter().next() + else { + panic!("an insert adds a draft row"); + }; + let _ = tracker.flash_error_row(RowKey::Draft(draft_id)); + let context = TabGridContext { + tracker: Some(tracker), + pk_col_indices: vec![0], + column_widths: None, + }; + + let (_, classes) = context.bind_state(&RowObject::new_draft(draft_id, values), 0); + + assert_eq!(classes, vec!["tp-row-pending-insert", "tp-row-leftmost-error-flash"]); + } + + #[test] + fn an_untracked_grid_draws_the_fetched_value_plainly() { + let row = RowObject::new(vec![Value::Int(1)]); + + let (value, classes) = TabGridContext::default().bind_state(&row, 0); + + assert_eq!(value, Value::Int(1)); + assert!(classes.is_empty()); + } + #[test] fn editable_for_normal_column() { assert!(is_cell_editable(&col("text", false))); diff --git a/linux/crates/app/src/ui/structure_tab/mod.rs b/linux/crates/app/src/ui/structure_tab/mod.rs index ddb3d8e549..1a6b3daeae 100644 --- a/linux/crates/app/src/ui/structure_tab/mod.rs +++ b/linux/crates/app/src/ui/structure_tab/mod.rs @@ -42,9 +42,8 @@ use tablepro_core::ddl::{ColumnDefinition, TableDefinition}; use tablepro_core::dialect::SqlDialect; use tablepro_core::sql_ddl::{BuildDdlError, DraftColumn}; use tablepro_core::{ForeignKeyInfo, IndexInfo}; -use uuid::Uuid; -use crate::services::structure_tracker; +use crate::services::structure_tracker_handle::StructureTrackerHandle; use crate::ui::structure_tab_dialogs::{present_fk_dialog, present_index_dialog}; use tablepro_core::sql_ddl::{StructureOp, diff_to_ops, materialize_ops}; @@ -66,7 +65,7 @@ pub enum StructureMode { #[derive(Debug)] pub struct StructureTabInit { - pub tab_id: Uuid, + pub tracker: StructureTrackerHandle, pub schema: Option, pub table: String, pub mode: StructureMode, @@ -82,7 +81,7 @@ pub struct StructureTabInit { } pub struct StructureTab { - tab_id: Uuid, + tracker: StructureTrackerHandle, schema: Option, table_name: Rc>, mode: Rc>, @@ -278,9 +277,7 @@ impl StructureTab { let count = ops.len(); self.refresh_buttons(count); self.regenerate_sql_preview_from(&ops); - - let ops_for_cache = ops.clone(); - structure_tracker::with_tab(self.tab_id, |t| t.set_ops(ops_for_cache)); + self.tracker.set_ops(ops); let dirty = count > 0; let mut last = self.last_dirty.borrow_mut(); @@ -520,8 +517,6 @@ impl SimpleComponent for StructureTab { } fn init(init: Self::Init, root: Self::Root, sender: ComponentSender) -> ComponentParts { - structure_tracker::open_tab(init.tab_id); - let view_stack = adw::ViewStack::new(); let columns_box = gtk::Box::builder() @@ -814,7 +809,7 @@ impl SimpleComponent for StructureTab { } let model = StructureTab { - tab_id: init.tab_id, + tracker: init.tracker, schema: init.schema, original_table_name: Rc::new(RefCell::new(init.table.clone())), table_name: Rc::new(RefCell::new(init.table)), @@ -1103,7 +1098,7 @@ impl SimpleComponent for StructureTab { // a New→Edit promotion doesn't leave a stale CREATE // TABLE statement visible in the preview pane until // StructureLoaded arrives. - structure_tracker::with_tab(self.tab_id, |t| t.clear()); + self.tracker.clear(); self.refresh_buttons(0); self.regenerate_sql_preview_from(&[]); let mut last = self.last_dirty.borrow_mut(); diff --git a/linux/crates/core/src/commit.rs b/linux/crates/core/src/commit.rs index 14b17c6a68..b3b09c3dd8 100644 --- a/linux/crates/core/src/commit.rs +++ b/linux/crates/core/src/commit.rs @@ -1,3 +1,5 @@ +use crate::meta::TableRef; + /// What the app needs to ask the server whether a commit landed. /// /// A connection lost between COMMIT and its acknowledgement leaves the @@ -12,9 +14,13 @@ pub enum CommitToken { PostgresXid { xid: u64, legacy_txid: bool }, /// ClickHouse has no transactions, so the token is the query ids /// that were sent, and how many of them are known to have finished. + /// `table` is the table they write, qualified: a mutation is looked + /// up by table, and the log of an ALTER that failed while waiting on + /// its mutation names none. ClickHouseQuery { query_ids: Vec, first_unconfirmed: usize, + table: TableRef, }, } diff --git a/linux/crates/core/src/dialect/test_dialect.rs b/linux/crates/core/src/dialect/test_dialect.rs index a754cfb272..054b93f6fd 100644 --- a/linux/crates/core/src/dialect/test_dialect.rs +++ b/linux/crates/core/src/dialect/test_dialect.rs @@ -22,6 +22,7 @@ pub struct TestDialect { row_ids: bool, default_row: bool, count_all: &'static str, + grammar: SqlGrammar, binds: Mutex>, ddl: Option, } @@ -72,6 +73,7 @@ impl Default for TestDialect { row_ids: true, default_row: true, count_all: "COUNT(*)", + grammar: SqlGrammar::PostgreSql, binds: Mutex::new(Vec::new()), ddl: Some(TestDdl::default()), } @@ -114,6 +116,10 @@ impl TestDialect { Self { count_all, ..self } } + pub fn parsing_as(self, grammar: SqlGrammar) -> Self { + Self { grammar, ..self } + } + pub fn targets(&self) -> Vec { self.recorded().into_iter().map(|bind| bind.target).collect() } @@ -149,7 +155,7 @@ impl SqlDialect for TestDialect { } fn grammar(&self) -> SqlGrammar { - SqlGrammar::PostgreSql + self.grammar } fn quote_identifier(&self, name: &str) -> String { @@ -257,12 +263,12 @@ impl SqlDialect for TestDialect { } fn parse_type(&self, text: &str) -> Result { - let parsed = parse_type(SqlGrammar::PostgreSql, text, crate::sql_syntax::MAX_TYPE_LEN)?; + let parsed = parse_type(self.grammar, text, crate::sql_syntax::MAX_TYPE_LEN)?; Ok(ColumnType::untyped_text(parsed.text)) } fn parse_expression(&self, text: &str) -> Result { - let parsed = parse_expression(SqlGrammar::PostgreSql, text, crate::sql_syntax::MAX_EXPRESSION_LEN)?; + let parsed = parse_expression(self.grammar, text, crate::sql_syntax::MAX_EXPRESSION_LEN)?; Ok(SqlExpression::from_catalog_text(parsed.text)) } diff --git a/linux/crates/core/src/export/csv.rs b/linux/crates/core/src/export/csv.rs index 213c725181..44e72c4ea8 100644 --- a/linux/crates/core/src/export/csv.rs +++ b/linux/crates/core/src/export/csv.rs @@ -36,9 +36,10 @@ pub fn csv_writer_builder(opts: &CsvOptions) -> csv::WriterBuilder { /// Only free text can smuggle a formula into a spreadsheet. A typed /// number, date or UUID, and text that is itself a plain number, is data /// the spreadsheet already reads correctly, and prefixing it would turn -/// `-5` into the string `'-5`. +/// `-5` into the string `'-5`. A type the driver could not place is the +/// server's own text, which is as free as any. pub fn neutralise_formula(value: &Value, text: String, is_header: bool) -> String { - if is_header || matches!(value, Value::Text(_) | Value::Json(_)) { + if is_header || matches!(value, Value::Text(_) | Value::Json(_) | Value::Other(_)) { neutralise_text(text) } else { text @@ -395,6 +396,37 @@ mod tests { } } + #[test] + fn csv_formula_rule_unchanged_after_split() { + let columns = cols(&["a"]); + let other = |text: &str| { + Value::Other(Box::new(crate::value::OtherValue { + type_name: "citext".to_owned(), + text: text.to_owned(), + })) + }; + let neutralised = [ + Value::Text("=cmd".into()), + Value::Json(crate::value::JsonText::parse("\t{}".to_owned()).expect("valid json")), + other("-2+cmd"), + ]; + for value in neutralised { + let out = render_csv(&columns, &[vec![value.clone()]], &CsvOptions::default()).unwrap(); + assert!(out.contains("\"'"), "{value:?} reached the sheet as a formula: {out}"); + } + let kept = [ + (Value::Int(-5), "-5"), + (Value::Float64(-1.5), "-1.5"), + (Value::Decimal("-12.30".parse().expect("a decimal")), "-12.30"), + (Value::Text("-12".into()), "-12"), + (other("+1.5e3"), "+1.5e3"), + ]; + for (value, text) in kept { + let out = render_csv(&columns, &[vec![value]], &CsvOptions::default()).unwrap(); + assert_eq!(out, format!("\"a\"\n\"{text}\"\n"), "{text} is data, not a formula"); + } + } + #[test] fn csv_neutralises_formula_header() { let columns = cols(&["=HYPERLINK(\"x\")"]); diff --git a/linux/crates/core/src/export/in_clause.rs b/linux/crates/core/src/export/in_clause.rs index ac5b1bd3a3..15ddaf3ac2 100644 --- a/linux/crates/core/src/export/in_clause.rs +++ b/linux/crates/core/src/export/in_clause.rs @@ -10,12 +10,11 @@ fn in_clause_literal(v: &Value) -> Option { // the caller rather than written. Value::Null | Value::Bytes(_) => None, Value::Bool(b) => Some(if *b { "TRUE".to_string() } else { "FALSE".to_string() }), - Value::Int(_) - | Value::UInt(_) - | Value::WideInt(_) - | Value::Float32(_) - | Value::Float64(_) - | Value::Decimal(_) => value_to_text(v), + // The widened double, not the real's own digits: an engine + // compares a `real` with a numeric literal as a double, and + // `0.1` as a double is not the `real` 0.1. + Value::Float32(real) => value_to_text(&Value::Float64(f64::from(*real))), + Value::Int(_) | Value::UInt(_) | Value::WideInt(_) | Value::Float64(_) | Value::Decimal(_) => value_to_text(v), other => value_to_text(other).map(|s| format!("'{}'", s.replace('\'', "''"))), } } @@ -62,6 +61,15 @@ mod tests { assert_eq!(out.skipped, 1); } + #[test] + fn in_clause_writes_a_real_as_the_double_it_compares_as() { + let rows = vec![vec![Value::Float32(0.1)]]; + + let out = render_in_clause(&rows, 0); + + assert_eq!(out.sql, format!("({})", f64::from(0.1_f32))); + } + #[test] fn in_clause_is_empty_rather_than_invalid_when_all_skipped() { let rows = vec![vec![Value::Null], vec![Value::Bytes(vec![1, 2])]]; diff --git a/linux/crates/core/src/export/json.rs b/linux/crates/core/src/export/json.rs index faed9fffe7..81585d0424 100644 --- a/linux/crates/core/src/export/json.rs +++ b/linux/crates/core/src/export/json.rs @@ -1,12 +1,12 @@ use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer}; use serde_json::value::RawValue; -use crate::column::{ColumnType, ResultColumn}; +use crate::column::{ColumnKind, ResultColumn}; use crate::value::Value; use super::column_type; use super::json_field_names::json_field_names; -use super::value_text::{value_text, value_to_text}; +use super::value_text::{element_kind, kind_text}; /// One row as a JSON object, keyed by the names the columns resolve to. struct JsonRow<'a> { @@ -24,7 +24,7 @@ impl Serialize for JsonRow<'_> { name, &JsonCell { value, - column: column_type(self.columns, index), + kind: &column_type(self.columns, index).kind, }, )?, // A row shorter than its header: the cell is missing, @@ -43,7 +43,7 @@ impl Serialize for JsonRow<'_> { /// value on the way through one. struct JsonCell<'a> { value: &'a Value, - column: &'a ColumnType, + kind: &'a ColumnKind, } impl Serialize for JsonCell<'_> { @@ -54,10 +54,12 @@ impl Serialize for JsonCell<'_> { Value::Int(number) => serializer.serialize_i64(*number), Value::UInt(number) => serializer.serialize_u64(*number), Value::WideInt(number) => number_token(serializer, &number.to_string()), - Value::Float32(number) => finite_number(serializer, f64::from(*number)), - Value::Float64(number) => finite_number(serializer, *number), + // Each width at its own shortest digits: a `real` widened + // to a double prints `0.10000000149011612` for 0.1. + Value::Float32(number) if number.is_finite() => serializer.serialize_f32(*number), + Value::Float64(number) if number.is_finite() => serializer.serialize_f64(*number), Value::Decimal(decimal) if decimal.is_finite() => { - number_token(serializer, &value_text(self.value, self.column).unwrap_or_default()) + number_token(serializer, &kind_text(self.value, self.kind).unwrap_or_default()) } // A JSON column already holds a document, so it goes in as // one rather than as a string of one. @@ -66,20 +68,21 @@ impl Serialize for JsonCell<'_> { Err(_) => serializer.serialize_str(json.as_str()), }, Value::Array(values) => { + let element = element_kind(self.kind); let mut seq = serializer.serialize_seq(Some(values.len()))?; for value in values { - seq.serialize_element(&JsonCell { - value, - column: self.column, - })?; + seq.serialize_element(&JsonCell { value, kind: element })?; } seq.end() } Value::Timestamp(_) | Value::TimestampTz(_) => { - let text = value_text(self.value, self.column).unwrap_or_default(); + let text = kind_text(self.value, self.kind).unwrap_or_default(); serializer.serialize_str(&text.replacen(' ', "T", 1)) } - other => match value_text(other, self.column).or_else(|| value_to_text(other)) { + // JSON has no NaN or Infinity, so a non-finite number goes + // out as its name rather than as a null that reads like a + // missing value. + other => match kind_text(other, self.kind) { Some(text) => serializer.serialize_str(&text), None => serializer.serialize_none(), }, @@ -98,22 +101,6 @@ fn number_token(serializer: S, digits: &str) -> Result(serializer: S, number: f64) -> Result { - if number.is_finite() { - return serializer.serialize_f64(number); - } - let text = if number.is_nan() { - "NaN" - } else if number.is_sign_negative() { - "-Infinity" - } else { - "Infinity" - }; - serializer.serialize_str(text) -} - /// One row as a document a person reads, which is the only place a /// single row is rendered on its own. pub fn row_to_json(columns: &[ResultColumn], row: &[Value]) -> String { @@ -169,7 +156,7 @@ mod tests { } #[test] - fn a_number_wider_than_a_double_keeps_its_digits() { + fn json_emits_raw_number_tokens_for_u64_max_and_decimal_40_20() { let columns = vec![ typed( "bigint unsigned", @@ -224,25 +211,73 @@ mod tests { } #[test] - fn a_json_column_embeds_its_document_rather_than_a_string_of_one() { + fn json_embeds_json_column_verbatim() { + let document = r#"{"b": 1, "a": [1, 2]}"#; let columns = vec![typed("payload", "jsonb", ColumnKind::Json { binary: true })]; let row = vec![Value::Json( - crate::value::JsonText::parse(r#"{"a": [1, 2]}"#.to_owned()).expect("valid json"), + crate::value::JsonText::parse(document.to_owned()).expect("valid json"), )]; + let text = row_to_json(&columns, &row); + + assert!( + text.contains(&format!("\"payload\": {document}")), + "the document was rewritten or quoted: {text}" + ); + } + + #[test] + fn json_non_finite_is_string() { + let columns = cols(&["double", "real", "nan", "negative"]); + let row = vec![ + Value::Float64(f64::INFINITY), + Value::Float32(f32::NAN), + Value::Decimal("NaN".parse().expect("a decimal")), + Value::Decimal("-Infinity".parse().expect("a decimal")), + ]; + let json = parsed(&columns, &row); - assert_eq!(json["payload"], serde_json::json!({"a": [1, 2]})); + assert_eq!(json["double"], serde_json::json!("Infinity")); + assert_eq!(json["real"], serde_json::json!("NaN")); + assert_eq!(json["nan"], serde_json::json!("NaN")); + assert_eq!(json["negative"], serde_json::json!("-Infinity")); + } + + #[test] + fn a_real_goes_out_at_its_own_digits() { + let columns = cols(&["r"]); + let row = vec![Value::Float32(0.1)]; + + let text = row_to_json(&columns, &row); + + assert!( + text.contains("\"r\": 0.1\n"), + "the real was widened to a double: {text}" + ); } #[test] - fn json_non_finite_float_is_string() { - let columns = cols(&["v"]); - let rows = vec![vec![Value::Float64(f64::INFINITY)]]; + fn array_items_go_out_at_their_element_scale() { + let columns = vec![typed( + "amounts", + "numeric(5,2)[]", + ColumnKind::Array { + element: Box::new(ColumnKind::Decimal { + precision: Some(5), + scale: Some(2), + }), + }, + )]; + let row = vec![Value::Array(vec![ + Value::Decimal("1.5".parse().expect("a decimal")), + Value::Null, + ])]; - let out = render_json(&columns, &rows); + let text = row_to_json(&columns, &row); - assert!(out.contains("\"Infinity\""), "{out}"); + assert!(text.contains("1.50,"), "the item lost its column's scale: {text}"); + assert_eq!(parsed(&columns, &row)["amounts"], serde_json::json!([1.5, null])); } #[test] diff --git a/linux/crates/core/src/export/markdown.rs b/linux/crates/core/src/export/markdown.rs index 73ce932309..b97c7f0cf1 100644 --- a/linux/crates/core/src/export/markdown.rs +++ b/linux/crates/core/src/export/markdown.rs @@ -5,7 +5,12 @@ use super::column_type; use super::value_text::value_text; fn markdown_cell(value: &Value, column: &ColumnType) -> String { - let text = value_text(value, column).unwrap_or_else(|| "NULL".to_string()); + markdown_text(&value_text(value, column).unwrap_or_else(|| "NULL".to_string())) +} + +/// A pipe or a line break, in a cell or a column name alike, would end +/// the cell or the row early. +fn markdown_text(text: &str) -> String { text.replace('|', "\\|") .replace("\r\n", "
") .replace(['\r', '\n'], "
") @@ -13,7 +18,7 @@ fn markdown_cell(value: &Value, column: &ColumnType) -> String { pub fn render_markdown(columns: &[ResultColumn], rows: &[Vec]) -> String { let mut lines: Vec = Vec::new(); - let header: Vec<&str> = columns.iter().map(|c| c.name.as_str()).collect(); + let header: Vec = columns.iter().map(|c| markdown_text(&c.name)).collect(); lines.push(format!("| {} |", header.join(" | "))); let separator: Vec<&str> = columns.iter().map(|_| "---").collect(); lines.push(format!("| {} |", separator.join(" | "))); @@ -42,4 +47,14 @@ mod tests { assert_eq!(out, "| a |\n| --- |\n| has\\|pipe
and newline |"); } + + #[test] + fn markdown_escapes_a_pipe_in_a_column_name() { + let columns = cols(&["a|b", "c"]); + let rows = vec![vec![Value::Int(1), Value::Int(2)]]; + + let out = render_markdown(&columns, &rows); + + assert_eq!(out, "| a\\|b | c |\n| --- | --- |\n| 1 | 2 |"); + } } diff --git a/linux/crates/core/src/export/value_text.rs b/linux/crates/core/src/export/value_text.rs index 32b0478e23..d9684088e1 100644 --- a/linux/crates/core/src/export/value_text.rs +++ b/linux/crates/core/src/export/value_text.rs @@ -1,6 +1,6 @@ use chrono::Timelike; -use crate::column::ColumnType; +use crate::column::{ColumnKind, ColumnType}; use crate::value::{SqlDecimal, Value}; /// Full text of a value for export, at the precision the column @@ -12,18 +12,22 @@ use crate::value::{SqlDecimal, Value}; /// `numeric(38,10)` holding `1`. Export writes what the column says, /// because that is what the server would print. pub fn value_text(value: &Value, column: &ColumnType) -> Option { + kind_text(value, &column.kind) +} + +pub(super) fn kind_text(value: &Value, kind: &ColumnKind) -> Option { match value { - Value::Time(time) => Some(time.format(second_digits(column))), + Value::Time(time) => Some(time.format(second_digits(kind))), Value::TimeTz(time) => Some(format!( "{}{}", - time.time.format(second_digits(column)), + time.time.format(second_digits(kind)), offset_text(time.offset) )), Value::Timestamp(stamp) => temporal_text(stamp, |stamp| { format!( "{}{}", stamp.format("%Y-%m-%d %H:%M:%S"), - fraction_text(stamp.nanosecond(), column.kind.fractional_digits()) + fraction_text(stamp.nanosecond(), kind.fractional_digits()) ) }), Value::TimestampTz(stamp) => temporal_text(stamp, |stamp| { @@ -31,18 +35,29 @@ pub fn value_text(value: &Value, column: &ColumnType) -> Option { format!( "{}{}{}", stamp.format("%Y-%m-%d %H:%M:%S"), - fraction_text(stamp.nanosecond(), column.kind.fractional_digits()), + fraction_text(stamp.nanosecond(), kind.fractional_digits()), stamp.format("%:z") ) }), - Value::Decimal(decimal) => Some(decimal_text(decimal, column.kind.decimal_scale())), + Value::Decimal(decimal) => Some(decimal_text(decimal, kind.decimal_scale())), + Value::Array(values) => Some(array_text(values, Some(element_kind(kind)))), other => value_to_text(other), } } +/// The kind each item of an array holds. A PostgreSQL `int4[][]` is +/// declared once for every dimension, so a kind that is not itself an +/// array passes down unchanged. +pub(super) fn element_kind(kind: &ColumnKind) -> &ColumnKind { + match kind { + ColumnKind::Array { element } => element, + other => other, + } +} + /// The digits a time type declares, in the width `SqlTime` formats at. -fn second_digits(column: &ColumnType) -> Option { - column.kind.fractional_digits().map(|digits| digits.min(9)) +fn second_digits(kind: &ColumnKind) -> Option { + kind.fractional_digits().map(|digits| digits.min(9)) } /// The fractional second, at the digits the column declares. @@ -99,15 +114,13 @@ pub fn value_to_text(v: &Value) -> Option { Value::Int(i) => Some(i.to_string()), Value::UInt(i) => Some(i.to_string()), Value::WideInt(i) => Some(i.to_string()), - // Shortest round-trip form, so the text parses back to the - // same bits. The non-finite ones have no numeric spelling. - Value::Float32(f) => Some(float_text( - f64::from(*f), - f.is_nan(), - f.is_infinite(), - f.is_sign_negative(), - )), - Value::Float64(f) => Some(float_text(*f, f.is_nan(), f.is_infinite(), f.is_sign_negative())), + // Shortest round-trip form at the value's own width: an f32 + // widened first prints `0.10000000149011612` for a `real` + // holding 0.1, digits the column never held. + Value::Float32(f) if f.is_finite() => Some(f.to_string()), + Value::Float64(f) if f.is_finite() => Some(f.to_string()), + Value::Float32(f) => Some(non_finite_text(f.is_nan(), f.is_sign_negative())), + Value::Float64(f) => Some(non_finite_text(f.is_nan(), f.is_sign_negative())), // Full scale: trimming a trailing zero loses the precision // the column declared. Value::Decimal(d) => Some(d.to_string()), @@ -127,7 +140,7 @@ pub fn value_to_text(v: &Value) -> Option { }), Value::Interval(i) => Some(i.to_string()), Value::Bits(b) => Some(b.to_string()), - Value::Array(values) => Some(array_text(values)), + Value::Array(values) => Some(array_text(values, None)), Value::Other(other) => Some(other.text.clone()), // The driver could not read it, so there is no text to write. // The caller reports the cell rather than inventing one. @@ -135,14 +148,13 @@ pub fn value_to_text(v: &Value) -> Option { } } -fn float_text(value: f64, is_nan: bool, is_infinite: bool, is_negative: bool) -> String { - if is_nan { - return "NaN".to_owned(); - } - if is_infinite { - return if is_negative { "-Infinity" } else { "Infinity" }.to_owned(); +fn non_finite_text(is_nan: bool, is_negative: bool) -> String { + match (is_nan, is_negative) { + (true, _) => "NaN", + (false, true) => "-Infinity", + (false, false) => "Infinity", } - value.to_string() + .to_owned() } /// An infinite date or timestamp has a keyword rather than a number. @@ -162,13 +174,18 @@ fn offset_text(offset: chrono::FixedOffset) -> String { } /// An array as JSON array text, which is the one spelling every engine -/// reads back. -fn array_text(values: &[Value]) -> String { +/// reads back. Each item is written at the precision its element kind +/// declares, when there is one, and an inner dimension stays an array +/// rather than becoming the string of one. +fn array_text(values: &[Value], element: Option<&ColumnKind>) -> String { let parts: Vec = values .iter() - .map(|value| match value_to_text(value) { - Some(text) => serde_json::Value::String(text).to_string(), - None => "null".to_owned(), + .map(|value| match value { + Value::Array(inner) => array_text(inner, element.map(element_kind)), + other => match element.map_or_else(|| value_to_text(other), |kind| kind_text(other, kind)) { + Some(text) => serde_json::Value::String(text).to_string(), + None => "null".to_owned(), + }, }) .collect(); format!("[{}]", parts.join(",")) @@ -314,7 +331,7 @@ mod tests { } #[test] - fn a_timestamp_keeps_the_microseconds_its_column_declares() { + fn value_text_timestamp_keeps_declared_microseconds() { let column = typed("timestamp(6)", stamp(Some(6), TemporalZone::Naive)); let stamp = Value::Timestamp(Temporal::Finite( NaiveDate::from_ymd_opt(2024, 6, 15) @@ -330,7 +347,7 @@ mod tests { } #[test] - fn a_time_takes_its_precision_from_its_column() { + fn value_text_time_precision_from_kind() { let value = Value::Time(SqlTime::new(false, 12, 0, 0, 123_456_000).expect("a time")); assert_eq!( @@ -349,7 +366,7 @@ mod tests { } #[test] - fn a_timestamp_with_a_zone_keeps_its_offset() { + fn value_text_timestamptz_keeps_offset() { let column = typed("timestamptz(3)", stamp(Some(3), TemporalZone::Offset)); let stamp = chrono::DateTime::parse_from_rfc3339("2024-06-15T12:00:00.5+05:30").expect("a timestamp"); let value = Value::TimestampTz(Temporal::Finite(crate::value::OffsetTimestamp::from_datetime(stamp))); @@ -361,7 +378,7 @@ mod tests { } #[test] - fn a_decimal_fills_out_the_scale_its_column_declares() { + fn value_text_numeric_38_10_full_scale() { let column = typed("numeric(38,10)", decimal(38, 10)); assert_eq!( @@ -385,4 +402,67 @@ mod tests { "the server's own digits were rounded away" ); } + + #[test] + fn a_real_prints_its_own_shortest_digits() { + for real in [0.1_f32, 1.1, -3.3, 16_777_216.0] { + let text = value_to_text(&Value::Float32(real)).expect("a real has text"); + + assert_eq!(text, real.to_string(), "{real} was widened before printing"); + assert_eq!(text.parse::().ok(), Some(real), "{text} does not read back"); + } + assert_eq!( + value_to_text(&Value::Float32(f32::NEG_INFINITY)).as_deref(), + Some("-Infinity") + ); + } + + #[test] + fn array_items_take_the_precision_of_their_element() { + let column = typed( + "timestamp(6)[]", + ColumnKind::Array { + element: Box::new(stamp(Some(6), TemporalZone::Naive)), + }, + ); + let value = Value::Array(vec![ + Value::Timestamp(Temporal::Finite( + NaiveDate::from_ymd_opt(2024, 6, 15) + .expect("a date") + .and_hms_nano_opt(12, 0, 0, 100_000_000) + .expect("a time"), + )), + Value::Null, + ]); + + assert_eq!( + value_text(&value, &column).as_deref(), + Some(r#"["2024-06-15 12:00:00.100000",null]"#) + ); + } + + #[test] + fn an_inner_dimension_stays_an_array() { + let column = typed( + "numeric(5,2)[]", + ColumnKind::Array { + element: Box::new(decimal(5, 2)), + }, + ); + let value = Value::Array(vec![ + Value::Array(vec![Value::Decimal("1".parse().expect("a decimal"))]), + Value::Array(vec![Value::Decimal("2.5".parse().expect("a decimal"))]), + ]); + + assert_eq!( + value_text(&value, &column).as_deref(), + Some(r#"[["1.00"],["2.50"]]"#), + "a row of a two-dimensional array was written as a string" + ); + assert_eq!( + value_to_text(&value).as_deref(), + Some(r#"[["1"],["2.5"]]"#), + "without a column, a row of the array was written as a string" + ); + } } diff --git a/linux/crates/core/src/export/xlsx.rs b/linux/crates/core/src/export/xlsx.rs index cb102b5233..8dac182854 100644 --- a/linux/crates/core/src/export/xlsx.rs +++ b/linux/crates/core/src/export/xlsx.rs @@ -72,10 +72,6 @@ fn write_cell( // read the two differently, and NULL is the absent one. Value::Null => Ok(()), Value::Bool(flag) => sheet.write_boolean(row, column, *flag).map(drop), - Value::Int(number) => sheet.write_number(row, column, *number as f64).map(drop), - Value::UInt(number) => sheet.write_number(row, column, *number as f64).map(drop), - Value::Float32(number) => sheet.write_number(row, column, f64::from(*number)).map(drop), - Value::Float64(number) => sheet.write_number(row, column, *number).map(drop), Value::Date(Temporal::Finite(day)) => sheet.write_datetime_with_format(row, column, day, date).map(drop), Value::Timestamp(Temporal::Finite(at)) => { sheet.write_datetime_with_format(row, column, at, timestamp).map(drop) @@ -85,20 +81,30 @@ fn write_cell( Value::TimestampTz(Temporal::Finite(at)) => sheet .write_datetime_with_format(row, column, at.utc(), timestamp) .map(drop), - // A decimal is written as a number only when Excel can hold it: - // a sheet stores every number as an f64, so one that does not - // round-trip would be silently changed. The rest keep their - // digits as text, which is what the user asked the database - // for. - Value::Decimal(_) | Value::WideInt(_) => match exact_number(columns, column, value) { + _ => match sheet_number(columns, column, value) { Some(number) => sheet.write_number(row, column, number).map(drop), None => write_text(sheet, row, column, columns, value), }, - _ => write_text(sheet, row, column, columns, value), } .map_err(xlsx_error) } +/// The number a cell goes in as, or `None` for one that goes in as +/// text. +/// +/// An integer or decimal is a number only when Excel can hold it: a +/// sheet stores every number as an f64, so a `bigint` past 2^53 or a +/// long decimal would be silently changed. The rest keep their digits +/// as text, which is what the user asked the database for. +fn sheet_number(columns: &[ResultColumn], column: u16, value: &Value) -> Option { + match value { + Value::Float32(number) => Some(real_number(*number)), + Value::Float64(number) => Some(*number), + Value::Int(_) | Value::UInt(_) | Value::Decimal(_) | Value::WideInt(_) => exact_number(columns, column, value), + _ => None, + } +} + /// Everything else goes in as the same text the other exports write, /// so a cell reads the same whichever format the user picked. fn write_text( @@ -114,19 +120,33 @@ fn write_text( } } -/// The value as an f64, when that f64 prints back as the same digits. +/// The value as an f64, when that f64 prints back as the same number. /// `None` when it does not, which is the case this exists to catch. fn exact_number(columns: &[ResultColumn], column: u16, value: &Value) -> Option { let text = cell_text(columns, column as usize, value)?; let parsed: f64 = text.parse().ok()?; // `{}` on an f64 prints the shortest form that round-trips, so // comparing against it is comparing the value, not the spelling. - match format!("{parsed}") == text { + // The zeros a `numeric(10,2)` pads `1.50` with are spelling. + match without_scale_padding(&format!("{parsed}")) == without_scale_padding(&text) { true => Some(parsed), false => None, } } +fn without_scale_padding(text: &str) -> &str { + match text.contains('.') { + true => text.trim_end_matches('0').trim_end_matches('.'), + false => text, + } +} + +/// The double a `real`'s own shortest digits name. Widening its bits +/// instead hands the sheet `0.100000001490116` for a `real` holding 0.1. +fn real_number(number: f32) -> f64 { + number.to_string().parse().unwrap_or(f64::from(number)) +} + /// A sheet name Excel accepts: at most 31 characters, none of /// `[]:*?/\`, not empty, and not wrapped in apostrophes. fn sheet_title(name: &str) -> String { @@ -152,7 +172,8 @@ fn xlsx_error(error: rust_xlsxwriter::XlsxError) -> EncodeError { #[cfg(test)] mod tests { use super::*; - use crate::export::test_columns::{col, cols}; + use crate::column::ColumnKind; + use crate::export::test_columns::{col, cols, typed}; fn workbook_bytes(columns: &[ResultColumn], rows: &[Vec]) -> Vec { render_xlsx(columns, rows, "Results").expect("render") @@ -209,6 +230,46 @@ mod tests { assert_eq!(exact_number(&columns, 0, &value), None); } + #[test] + fn a_decimal_padded_to_its_scale_is_still_a_number() { + let columns = vec![typed( + "amount", + "numeric(10,2)", + ColumnKind::Decimal { + precision: Some(10), + scale: Some(2), + }, + )]; + + for (digits, number) in [("1.5", 1.5), ("2", 2.0), ("-0.10", -0.1), ("100", 100.0)] { + let value = Value::Decimal(digits.parse().expect("decimal")); + assert_eq!(sheet_number(&columns, 0, &value), Some(number), "{digits}"); + } + } + + #[test] + fn an_integer_past_an_f64_stays_text() { + let columns = cols(&["id"]); + + assert_eq!(sheet_number(&columns, 0, &Value::Int(42)), Some(42.0)); + assert_eq!( + sheet_number(&columns, 0, &Value::Int(-(1 << 53))), + Some(-9_007_199_254_740_992.0) + ); + assert_eq!(sheet_number(&columns, 0, &Value::Int((1 << 53) + 1)), None); + assert_eq!(sheet_number(&columns, 0, &Value::UInt(u64::MAX)), None); + } + + #[test] + fn a_real_is_written_as_the_number_it_prints_as() { + let columns = cols(&["r"]); + + assert_eq!(sheet_number(&columns, 0, &Value::Float32(0.1)), Some(0.1)); + assert_eq!(sheet_number(&columns, 0, &Value::Float32(-3.3)), Some(-3.3)); + assert!(sheet_number(&columns, 0, &Value::Float32(f32::NAN)).is_some_and(f64::is_nan)); + assert_eq!(sheet_number(&columns, 0, &Value::Text("1".to_owned())), None); + } + #[test] fn a_result_wider_than_a_sheet_is_refused() { let columns: Vec = (0..MAX_COLUMNS + 1).map(|index| col(&index.to_string())).collect(); diff --git a/linux/crates/core/src/filter/validate_raw_predicate.rs b/linux/crates/core/src/filter/validate_raw_predicate.rs index 2cb8db3945..9b929698d0 100644 --- a/linux/crates/core/src/filter/validate_raw_predicate.rs +++ b/linux/crates/core/src/filter/validate_raw_predicate.rs @@ -17,6 +17,7 @@ pub fn validate_raw_predicate(dialect: &dyn SqlDialect, text: &str) -> Result<() #[cfg(test)] mod tests { use crate::dialect::TestDialect; + use crate::sql_syntax::SqlGrammar; use super::*; @@ -47,6 +48,16 @@ mod tests { assert_eq!(validate("a = 1 --"), Err(ExpressionSyntaxError::Comment)); } + #[test] + fn a_mysql_hash_comment_is_refused() { + let mysql = TestDialect::exact().parsing_as(SqlGrammar::MySql); + + assert_eq!( + validate_raw_predicate(&mysql, "a = 1 # x"), + Err(ExpressionSyntaxError::Comment) + ); + } + #[test] fn a_plain_expression_passes() { assert_eq!(validate("a OR b"), Ok(())); diff --git a/linux/crates/core/src/sql_syntax/parse_expression.rs b/linux/crates/core/src/sql_syntax/parse_expression.rs index e5eb9a9ca3..d5e292c5d3 100644 --- a/linux/crates/core/src/sql_syntax/parse_expression.rs +++ b/linux/crates/core/src/sql_syntax/parse_expression.rs @@ -5,7 +5,7 @@ use sqlparser::tokenizer::Token; use super::SqlGrammar; use super::errors::ExpressionSyntaxError; use super::literal_token::LiteralToken; -use super::parse_type::{RECURSION_LIMIT, reject_comment_tokens}; +use super::parse_type::{RECURSION_LIMIT, contains_comment}; use crate::value::SqlDecimal; /// An expression the engine's own parser accepted, plus the literal it @@ -33,7 +33,7 @@ pub fn parse_expression( if text.chars().count() > max_len { return Err(ExpressionSyntaxError::TooLong { max: max_len }); } - if reject_comment_tokens(grammar, text).is_err() && (text.contains("--") || text.contains("/*")) { + if contains_comment(grammar, text) { return Err(ExpressionSyntaxError::Comment); } @@ -152,4 +152,14 @@ mod tests { Err(ExpressionSyntaxError::TooLong { max: 2 }) ); } + + #[test] + fn a_hash_is_a_comment_only_where_the_engine_reads_one() { + assert_eq!( + parse_expression(SqlGrammar::MySql, "a = 1 # x", MAX_EXPRESSION_LEN), + Err(ExpressionSyntaxError::Comment) + ); + assert!(parse_expression(SqlGrammar::MySql, "a = '#'", MAX_EXPRESSION_LEN).is_ok()); + assert!(parse("a # b").is_ok(), "PostgreSQL reads # as XOR"); + } } diff --git a/linux/crates/core/src/sql_syntax/parse_type.rs b/linux/crates/core/src/sql_syntax/parse_type.rs index c55e9ed8c0..b08c2d4728 100644 --- a/linux/crates/core/src/sql_syntax/parse_type.rs +++ b/linux/crates/core/src/sql_syntax/parse_type.rs @@ -49,7 +49,9 @@ pub fn parse_type(grammar: SqlGrammar, text: &str, max_len: usize) -> Result max_len { return Err(TypeSyntaxError::TooLong { max: max_len }); } - reject_comments(grammar, text)?; + if contains_comment(grammar, text) { + return Err(TypeSyntaxError::Comment); + } let mut parser = Parser::new(grammar.parser_dialect()) .with_recursion_limit(RECURSION_LIMIT) @@ -71,28 +73,18 @@ pub fn parse_type(grammar: SqlGrammar, text: &str, max_len: usize) -> Result Result<(), ()> { - let tokens = Tokenizer::new(grammar.parser_dialect(), text) - .tokenize_with_location() - .map_err(|_| ())?; - let has_comment = tokens.iter().any(|token| { - matches!( - token.token, - Token::Whitespace(Whitespace::SingleLineComment { .. } | Whitespace::MultiLineComment(_)) - ) - }); - if has_comment { Err(()) } else { Ok(()) } -} - -fn reject_comments(grammar: SqlGrammar, text: &str) -> Result<(), TypeSyntaxError> { - match reject_comment_tokens(grammar, text) { - Ok(()) => Ok(()), - // A tokenizer failure is reported by the parser below with a - // better message, so only the comment case stops here. - Err(()) if text.contains("--") || text.contains("/*") => Err(TypeSyntaxError::Comment), - Err(()) => Ok(()), +/// A comment, MySQL's `#` included, swallows every clause after it. +/// Text the tokenizer refuses is left to the parser's message unless it +/// opens a comment, such as an unterminated `/*`. +pub(crate) fn contains_comment(grammar: SqlGrammar, text: &str) -> bool { + match Tokenizer::new(grammar.parser_dialect(), text).tokenize_with_location() { + Ok(tokens) => tokens.iter().any(|token| { + matches!( + token.token, + Token::Whitespace(Whitespace::SingleLineComment { .. } | Whitespace::MultiLineComment(_)) + ) + }), + Err(_) => text.contains("--") || text.contains("/*"), } } @@ -241,6 +233,11 @@ mod tests { } } + #[test] + fn parse_type_rejects_a_mysql_hash_comment() { + assert_eq!(parse(SqlGrammar::MySql, "int # x"), Err(TypeSyntaxError::Comment)); + } + #[test] fn parse_rejects_empty_and_over_length() { assert_eq!(parse(SqlGrammar::PostgreSql, " "), Err(TypeSyntaxError::Empty)); diff --git a/linux/crates/drivers/clickhouse/src/call_client.rs b/linux/crates/drivers/clickhouse/src/call_client.rs index 328d102dfe..61d92cc536 100644 --- a/linux/crates/drivers/clickhouse/src/call_client.rs +++ b/linux/crates/drivers/clickhouse/src/call_client.rs @@ -3,6 +3,7 @@ use std::time::Duration; use clickhouse::query::Query; use tablepro_core::statement::Statement; +use tablepro_core::{DriverError, StopRequest}; use tokio::time::Instant; use uuid::Uuid; @@ -11,37 +12,46 @@ use crate::param_value::bind_params; /// The client one call sends its queries through, each under a query id /// of its own that the call's stop can name. +/// +/// A KILL stops only the query running when it arrives, so a call of +/// several queries would go on to the next one after a stop. Once the +/// call is stopped, no further query is handed out. pub(crate) struct CallClient { client: clickhouse::Client, stop: Arc, + request: StopRequest, } impl CallClient { - pub(crate) fn new(client: clickhouse::Client, stop: Arc) -> Self { - Self { client, stop } + pub(crate) fn new(client: clickhouse::Client, stop: Arc, request: StopRequest) -> Self { + Self { client, stop, request } } /// SQL sent as written: a `?` in it is the user's, not a marker. - pub(crate) fn raw(&self, sql: &str) -> Query { + pub(crate) fn raw(&self, sql: &str) -> Result { self.tracked(self.client.query_raw(sql)) } /// A generated statement, with its values bound as the server-side /// parameters its `{pN:Type}` markers name. - pub(crate) fn statement(&self, statement: &Statement) -> Query { - bind_params(self.raw(statement.sql()), statement.params()) + pub(crate) fn statement(&self, statement: &Statement) -> Result { + Ok(bind_params(self.raw(statement.sql())?, statement.params())) } /// SQL whose `?` markers the client fills, for identifiers bound /// through `clickhouse::sql::Identifier`. - pub(crate) fn template(&self, sql: &str) -> Query { + pub(crate) fn template(&self, sql: &str) -> Result { self.tracked(self.client.query(sql)) } - fn tracked(&self, query: Query) -> Query { + fn tracked(&self, query: Query) -> Result { + if self.request.is_requested() { + self.stop.withhold(); + return Err(self.stop.interrupted(&self.request)); + } let id = Uuid::new_v4(); self.stop.track(id); - query.with_setting("query_id", id.to_string()) + Ok(query.with_setting("query_id", id.to_string())) } } @@ -79,8 +89,46 @@ pub(crate) fn whole_seconds(bound: Option) -> u64 { #[cfg(test)] mod tests { + use tablepro_core::StopReason; + use super::*; + fn call(request: &StopRequest) -> (CallClient, Arc) { + let stop = Arc::new(ClickHouseStop::new(clickhouse::Client::default())); + let calls = CallClient::new(clickhouse::Client::default(), stop.clone(), request.clone()); + (calls, stop) + } + + #[test] + fn a_query_goes_out_until_the_call_is_stopped() { + let request = StopRequest::new(); + let (calls, stop) = call(&request); + + assert!(calls.raw("SELECT 1").is_ok()); + assert!(!stop.withheld()); + request.request(StopReason::Cancelled); + + assert_eq!(calls.raw("SELECT 2").err(), Some(DriverError::Cancelled)); + assert!(calls.template("SHOW CREATE TABLE ?.?").is_err()); + assert!(stop.withheld()); + } + + #[test] + fn a_withheld_query_reads_as_what_stopped_the_call() { + let request = StopRequest::new(); + let (calls, _stop) = call(&request); + + request.request(StopReason::DeadlineElapsed); + + assert_eq!( + calls.raw("SELECT 1").err(), + Some(DriverError::Timeout { + phase: tablepro_core::TimeoutPhase::Statement, + server_cancelled: false, + }) + ); + } + #[test] fn a_bound_rounds_up_to_whole_seconds_and_never_reaches_zero() { assert_eq!(whole_seconds(None), 0); diff --git a/linux/crates/drivers/clickhouse/src/cancel.rs b/linux/crates/drivers/clickhouse/src/cancel.rs index 40a790b3da..df41287ccc 100644 --- a/linux/crates/drivers/clickhouse/src/cancel.rs +++ b/linux/crates/drivers/clickhouse/src/cancel.rs @@ -1,3 +1,4 @@ +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Mutex, MutexGuard, PoisonError}; use async_trait::async_trait; @@ -30,6 +31,7 @@ pub(crate) struct ClickHouseStop { control: clickhouse::Client, running: Mutex>, progress: Mutex, + withheld: AtomicBool, } impl ClickHouseStop { @@ -38,6 +40,7 @@ impl ClickHouseStop { control, running: Mutex::new(None), progress: Mutex::new(StopProgress::NotSent), + withheld: AtomicBool::new(false), } } @@ -56,6 +59,17 @@ impl ClickHouseStop { *lock(&self.progress) } + /// The call ended at a stop without sending its next query. + pub(crate) fn withhold(&self) { + self.withheld.store(true, Ordering::SeqCst); + } + + /// Whether the call ended by not sending a query. That was the + /// driver's own doing, so it says nothing about the route. + pub(crate) fn withheld(&self) -> bool { + self.withheld.load(Ordering::SeqCst) + } + /// What a call stopped before it reached the server reads as. pub(crate) fn interrupted(&self, request: &StopRequest) -> DriverError { match request.reason() { diff --git a/linux/crates/drivers/clickhouse/src/catalog/mod.rs b/linux/crates/drivers/clickhouse/src/catalog/mod.rs index 52369579fb..8368a02e73 100644 --- a/linux/crates/drivers/clickhouse/src/catalog/mod.rs +++ b/linux/crates/drivers/clickhouse/src/catalog/mod.rs @@ -43,14 +43,14 @@ pub(crate) async fn column_rows( calls: &CallClient, table: &CatalogTable, ) -> Result, DriverError> { - let result = read_table(table.bind(calls.raw(COLUMN_SELECT)), None).await?; + let result = read_table(table.bind(calls.raw(COLUMN_SELECT)?), None).await?; result.rows.iter().map(|row| ClickHouseColumnRow::read(row)).collect() } pub(crate) async fn table_keys(calls: &CallClient, table: &CatalogTable) -> Result { let sql = "SELECT sorting_key, primary_key, total_rows FROM system.tables \ WHERE database = {db:String} AND name = {tbl:String}"; - let result = read_table(table.bind(calls.raw(sql)), None).await?; + let result = read_table(table.bind(calls.raw(sql)?), None).await?; let Some(row) = result.rows.first() else { return Ok(TableKeys::default()); }; @@ -80,7 +80,7 @@ pub(crate) async fn fetch_indexes(calls: &CallClient, table: &CatalogTable) -> R } let sql = "SELECT name, expr FROM system.data_skipping_indices \ WHERE database = {db:String} AND table = {tbl:String} ORDER BY name"; - let skipping = read_table(table.bind(calls.raw(sql)), None).await?; + let skipping = read_table(table.bind(calls.raw(sql)?), None).await?; for row in &skipping.rows { let row = CatalogRow(row); indexes.push(IndexInfo { diff --git a/linux/crates/drivers/clickhouse/src/catalog/table_definition.rs b/linux/crates/drivers/clickhouse/src/catalog/table_definition.rs index cd17209b6a..aca01745e9 100644 --- a/linux/crates/drivers/clickhouse/src/catalog/table_definition.rs +++ b/linux/crates/drivers/clickhouse/src/catalog/table_definition.rs @@ -50,7 +50,7 @@ pub(crate) async fn fetch_table_definition( /// identifier by the client rather than quoted here. async fn create_statement(calls: &CallClient, catalog: &CatalogTable) -> Result, DriverError> { let query = calls - .template("SHOW CREATE TABLE ?.?") + .template("SHOW CREATE TABLE ?.?")? .bind(Identifier(&catalog.database)) .bind(Identifier(&catalog.name)); let result = read_table(query, None).await?; diff --git a/linux/crates/drivers/clickhouse/src/client_factory.rs b/linux/crates/drivers/clickhouse/src/client_factory.rs index 3d8af4f0f2..275615edd1 100644 --- a/linux/crates/drivers/clickhouse/src/client_factory.rs +++ b/linux/crates/drivers/clickhouse/src/client_factory.rs @@ -23,10 +23,14 @@ const POOL_IDLE_TIMEOUT: Duration = Duration::from_secs(2); const CERTIFICATE_AUTH_HEADER: &str = "X-ClickHouse-SSL-Certificate-Auth"; /// The settings every request carries. -const COMMON_SETTINGS: [(&str, &str); 6] = [ +const COMMON_SETTINGS: [(&str, &str); 7] = [ // A Decimal is written at its column's scale, so a value read back // keeps the scale it was stored with and compares equal to it. ("output_format_decimal_trailing_zeros", "1"), + // A DateTime is read as `YYYY-MM-DD hh:mm:ss` in its own time zone. + // A profile or session that set `iso` would send it in UTC instead, + // and a save keyed on it would miss its row. + ("date_time_output_format", "simple"), // An error after the first rows is written as plain exception text, // which the clickhouse crate detects, rather than as one more row in // the output format, which would read as data. diff --git a/linux/crates/drivers/clickhouse/src/commit_resolution.rs b/linux/crates/drivers/clickhouse/src/commit_resolution.rs index 58237fcb85..18f31c4c9c 100644 --- a/linux/crates/drivers/clickhouse/src/commit_resolution.rs +++ b/linux/crates/drivers/clickhouse/src/commit_resolution.rs @@ -5,10 +5,12 @@ //! waits the query log has no entry for it at all, and a wait that fails //! is logged as ExceptionBeforeStart with no table named. The statement //! is therefore looked for in `system.processes` when the log is silent, -//! and its mutation in `system.mutations` by creation time when its ALTER -//! failed while the mutation may live on. `system.mutations` has no +//! and its mutation in `system.mutations` by table and creation time when +//! its ALTER failed while the mutation may live on. The table comes from +//! the write itself when the log names none. `system.mutations` has no //! column linking a mutation to its ALTER, so any doubt is Unresolvable. +use tablepro_core::meta::TableRef; use tablepro_core::{CommitResolution, CommitToken, DriverError, Value}; use uuid::Uuid; @@ -105,6 +107,19 @@ fn mutated_table(entry: &LogEntry) -> Option { }) } +/// The table to look for the mutation in: the log's where it names one, +/// otherwise the one the write was for. Searching every table instead +/// would take another session's mutation in the same seconds for this +/// one. +pub(crate) fn mutation_table(logged: Option, written: &TableRef) -> Option { + logged.or_else(|| { + Some(MutatedTable { + database: written.schema.clone().filter(|database| !database.is_empty())?, + table: written.name.clone(), + }) + }) +} + /// One mutation in the window is the ALTER's; none means someone ran /// KILL MUTATION, which removes the row but keeps the parts it already /// changed, and more than one cannot be told apart. @@ -132,6 +147,7 @@ pub(crate) async fn resolve(calls: &CallClient, token: &CommitToken) -> Result Result {} Err(error) if server_code(&error) == Some(code::ACCESS_DENIED) => {} Err(error) => return Err(error), @@ -154,13 +170,16 @@ pub(crate) async fn resolve(calls: &CallClient, token: &CommitToken) -> Result resolution, LogVerdict::NotEnded => still_running(calls, *query_id).await?, - LogVerdict::Mutation { table, started, ended } => mutation(calls, table.as_ref(), &started, &ended).await?, + LogVerdict::Mutation { table, started, ended } => match mutation_table(table, written) { + Some(table) => mutation(calls, &table, &started, &ended).await?, + None => CommitResolution::Unresolvable, + }, }; Ok(compose(*first_unconfirmed, query_ids.len(), statement)) } async fn log_entries(calls: &CallClient, query_id: Uuid) -> Result, DriverError> { - let query = calls.raw(LOG_SELECT).param("id", query_id.to_string()); + let query = calls.raw(LOG_SELECT)?.param("id", query_id.to_string()); let result = read_table(query, None).await?; result .rows @@ -184,7 +203,7 @@ async fn log_entries(calls: &CallClient, query_id: Uuid) -> Result /// lists it, and lost otherwise. async fn still_running(calls: &CallClient, query_id: Uuid) -> Result { let query = calls - .raw("SELECT count() FROM system.processes WHERE query_id = {id:String}") + .raw("SELECT count() FROM system.processes WHERE query_id = {id:String}")? .param("id", query_id.to_string()); let result = read_table(query, None).await?; let running = result @@ -202,21 +221,19 @@ async fn still_running(calls: &CallClient, query_id: Uuid) -> Result, + table: &MutatedTable, started: &str, ended: &str, ) -> Result { - let window = "create_time >= {started:DateTime} AND create_time <= {ended:DateTime}"; - let query = match table { - Some(table) => calls - .raw(&format!( - "SELECT is_done FROM system.mutations WHERE database = {{db:String}} AND table = {{tbl:String}} AND {window}" - )) - .param("db", table.database.as_str()) - .param("tbl", table.table.as_str()), - None => calls.raw(&format!("SELECT is_done FROM system.mutations WHERE {window}")), - }; - let query = query.param("started", started).param("ended", ended); + let query = calls + .raw( + "SELECT is_done FROM system.mutations WHERE database = {db:String} AND table = {tbl:String} \ + AND create_time >= {started:DateTime} AND create_time <= {ended:DateTime}", + )? + .param("db", table.database.as_str()) + .param("tbl", table.table.as_str()) + .param("started", started) + .param("ended", ended); let result = read_table(query, None).await?; let done = result .rows @@ -335,6 +352,27 @@ mod tests { } } + #[test] + fn a_mutation_is_looked_for_in_the_logged_table_then_the_written_one() { + let written = TableRef::new(Some("tablepro".to_owned()), "orders"); + let orders = Some(MutatedTable { + database: "tablepro".to_owned(), + table: "orders".to_owned(), + }); + + assert_eq!(mutation_table(events_table(), &written), events_table()); + assert_eq!( + mutation_table(None, &written), + orders, + "an ALTER whose wait failed names no table" + ); + assert_eq!( + mutation_table(None, &TableRef::unqualified("orders")), + None, + "a table without its database could be any database's" + ); + } + #[test] fn a_statement_answer_composes_over_the_batch() { use CommitResolution::{Applied, NotApplied, PartiallyApplied, StillRunning, Unresolvable}; diff --git a/linux/crates/drivers/clickhouse/src/connection.rs b/linux/crates/drivers/clickhouse/src/connection.rs index 1a906004c1..103bacf63a 100644 --- a/linux/crates/drivers/clickhouse/src/connection.rs +++ b/linux/crates/drivers/clickhouse/src/connection.rs @@ -119,15 +119,15 @@ impl ClickHouseConnection { Client::Reads => reading(self.endpoint.workspace(), read_bound(options.deadline(), bound)), Client::Control => self.endpoint.control().clone(), }; - let calls = CallClient::new(base, stop.clone()); + let work_stop = stop.clone(); let outcome = self .calls .run_read( options, HttpCall::new(), stop.clone(), - move |call, _request| async move { - let result = work(calls).await; + move |call, request| async move { + let result = work(CallClient::new(base, work_stop, request)).await; (call, result) }, ) @@ -160,10 +160,9 @@ impl ClickHouseConnection { if !suspects_link(&error) { return Err(error); } - let lost = match stop.progress() { - StopProgress::InFlight | StopProgress::Unreachable => true, - StopProgress::Delivered => false, - StopProgress::NotSent | StopProgress::NothingToStop => { + let lost = match known_lost(stop.progress(), stop.withheld()) { + Some(lost) => lost, + None => { let checked = self.endpoint.check(self.liveness().cancel_grace).await; checked.is_err_and(|failure| means_link_lost(&failure)) } @@ -193,6 +192,21 @@ fn suspects_link(error: &DriverError) -> bool { } } +/// What the stop already says about the route, or `None` when only a +/// request of its own can tell. +/// +/// A call that waits on its KILL and ends with it still out means the +/// route stopped answering. A call that ended by holding back its next +/// query waited on nothing, so a KILL still out says nothing yet. +fn known_lost(progress: StopProgress, withheld: bool) -> Option { + match progress { + StopProgress::Unreachable => Some(true), + StopProgress::InFlight if !withheld => Some(true), + StopProgress::Delivered => Some(false), + StopProgress::InFlight | StopProgress::NotSent | StopProgress::NothingToStop => None, + } +} + fn means_link_lost(error: &DriverError) -> bool { matches!( error.category(), @@ -324,7 +338,7 @@ impl Connection for ClickHouseConnection { ) -> Result { let statement = statement.clone(); self.workspace_read(options, move |calls| async move { - let table = read_table(calls.statement(&statement), Some(limit.get())).await?; + let table = read_table(calls.statement(&statement)?, Some(limit.get())).await?; Ok(QueryResult::new(table.columns, table.rows).truncated(table.truncated)) }) .await @@ -333,7 +347,7 @@ impl Connection for ClickHouseConnection { async fn count_rows(&self, statement: &Statement, options: CallOptions) -> Result { let statement = statement.clone(); self.workspace_read(options, move |calls| async move { - let table = read_table(calls.statement(&statement), Some(1)).await?; + let table = read_table(calls.statement(&statement)?, Some(1)).await?; let value = table .rows .first() @@ -367,10 +381,12 @@ impl Connection for ClickHouseConnection { let stop = Arc::new(ClickHouseStop::new(self.endpoint.control().clone())); let ledger = Arc::new(WriteLedger::new(WriteAtomicity::StatementByStatement)); let probe_bound = read_bound(options.deadline(), Some(self.settings.statement_timeout)); + let written = self.catalog_table(&batch.table); let plan = WritePlan { batch: batch.clone(), + table: TableRef::new(Some(written.database), written.name), ledger: ledger.clone(), - probes: CallClient::new(reading(self.endpoint.workspace(), probe_bound), stop.clone()), + probes: reading(self.endpoint.workspace(), probe_bound), writes: self .endpoint .workspace() @@ -422,3 +438,20 @@ impl Connection for ClickHouseConnection { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_kill_still_out_after_a_held_back_query_is_not_a_lost_route() { + assert_eq!(known_lost(StopProgress::InFlight, true), None); + assert_eq!(known_lost(StopProgress::Unreachable, true), Some(true)); + assert_eq!(known_lost(StopProgress::Delivered, true), Some(false)); + assert_eq!(known_lost(StopProgress::InFlight, false), Some(true)); + assert_eq!(known_lost(StopProgress::Unreachable, false), Some(true)); + assert_eq!(known_lost(StopProgress::Delivered, false), Some(false)); + assert_eq!(known_lost(StopProgress::NotSent, false), None); + assert_eq!(known_lost(StopProgress::NothingToStop, false), None); + } +} diff --git a/linux/crates/drivers/clickhouse/src/editor_run.rs b/linux/crates/drivers/clickhouse/src/editor_run.rs index e16223361c..983d41c95c 100644 --- a/linux/crates/drivers/clickhouse/src/editor_run.rs +++ b/linux/crates/drivers/clickhouse/src/editor_run.rs @@ -12,6 +12,7 @@ use uuid::Uuid; use crate::call_client::whole_seconds; use crate::cancel::{acknowledges_stop, kill_query}; +use crate::errors::{code, server_code}; use crate::result_reader::{ReadEvent, ResultReader}; /// Rows per `ResultEvent::Rows`. The sink is awaited per batch, so a slow @@ -90,8 +91,7 @@ impl EditorRun { emitter.limit_reached().await; // The rest of the result is not wanted, so the read // stops on the server before its body is dropped. - let killing = kill_query(&self.control, query_id); - let _ = tokio::time::timeout(self.grace, killing).await; + kill_within(&self.control, query_id, self.grace).await; break stop.fired.map_or(RunEnd::Completed, |_| stop.end(None)); } } @@ -105,10 +105,25 @@ impl EditorRun { during: LossPhase::Statement, }); } + // The server let the session expire, and with it what the + // tab built up there; carrying on in a new one would hide + // that. + Err(error) if server_code(&error) == Some(code::SESSION_NOT_FOUND) => { + return Err(DriverError::ConnectionLost { + during: LossPhase::Idle, + }); + } Err(DriverError::Timeout { phase: TimeoutPhase::Statement, server_cancelled, }) => break RunEnd::TimedOut { server_cancelled }, + Err(error) if unreadable(&error) => { + // The server is still sending what the driver could not + // read, so the query is stopped there before its body + // is dropped. + kill_within(&self.control, query_id, self.grace).await; + break RunEnd::Failed(error); + } Err(error) => break RunEnd::Failed(error), } if emitter.consumer_gone && stop.fired.is_none() { @@ -137,6 +152,22 @@ fn lost(error: &DriverError) -> bool { ) } +/// A body the driver gave up reading, while the server may still be +/// producing it. +fn unreadable(error: &DriverError) -> bool { + matches!(error, DriverError::Protocol(_) | DriverError::Decode { .. }) +} + +/// A KILL that gives up after `grace`: on a route that stopped +/// answering, it would otherwise wait, and hold its connection, for good. +async fn kill_within(control: &clickhouse::Client, query_id: Uuid, grace: Duration) { + match tokio::time::timeout(grace, kill_query(control, query_id)).await { + Ok(Ok(())) => {} + Ok(Err(error)) => tracing::debug!(%error, "the KILL QUERY for an editor run failed"), + Err(_) => tracing::debug!("the KILL QUERY for an editor run got no answer"), + } +} + /// What stops a run, and when the stop was sent. struct Stop { cancel: CancellationToken, @@ -179,11 +210,8 @@ impl Stop { self.fired = Some(interrupt); self.give_up = Some(Instant::now() + self.grace); let control = control.clone(); - tokio::spawn(async move { - if let Err(error) = kill_query(&control, query_id).await { - tracing::debug!(%error, "the KILL QUERY for an editor run failed"); - } - }); + let grace = self.grace; + tokio::spawn(async move { kill_within(&control, query_id, grace).await }); } async fn gave_up(&self) { @@ -307,6 +335,42 @@ mod tests { use super::*; + #[tokio::test] + async fn a_kill_the_server_never_answers_gives_up_at_the_grace() { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("a listener"); + let address = listener.local_addr().expect("an address"); + let silent = tokio::spawn(async move { + let mut held = Vec::new(); + while let Ok((socket, _)) = listener.accept().await { + held.push(socket); + } + }); + let control = clickhouse::Client::default().with_url(format!("http://{address}")); + + let killing = kill_within(&control, Uuid::new_v4(), Duration::from_millis(200)); + + assert!( + tokio::time::timeout(Duration::from_secs(10), killing).await.is_ok(), + "the KILL waited on a server that never answered" + ); + silent.abort(); + } + + #[test] + fn only_a_body_the_driver_could_not_read_is_killed_after_it_fails() { + let refused = DriverError::reported(ServerDiagnostics::new( + Some(ServerCode::ClickHouse { code: 62 }), + "Code: 62. DB::Exception: Syntax error", + )); + + assert!(unreadable(&DriverError::Protocol("a row ran past 256 MiB".to_owned()))); + assert!(unreadable(&DriverError::Decode { + column: "c".to_owned(), + detail: "not a number".to_owned(), + })); + assert!(!unreadable(&refused)); + } + fn stopped(interrupt: Interrupt) -> Stop { let mut stop = Stop::new( CancellationToken::new(), diff --git a/linux/crates/drivers/clickhouse/src/editor_session.rs b/linux/crates/drivers/clickhouse/src/editor_session.rs index b48b090482..33a78ab98d 100644 --- a/linux/crates/drivers/clickhouse/src/editor_session.rs +++ b/linux/crates/drivers/clickhouse/src/editor_session.rs @@ -55,7 +55,9 @@ impl ClickHouseEditorSession { } Ok(Arc::new(Self { id: EditorSessionId::new(), - session, + // Without the check, a run on a session the server let expire + // quietly starts a new, empty one under the same id. + session: session.with_setting("session_check", "1"), control: endpoint.control().clone(), statement_timeout: options.statement_timeout, grace: liveness.cancel_grace, @@ -128,3 +130,74 @@ impl EditorSession for ClickHouseEditorSession { self.status.lose(); } } + +#[cfg(test)] +mod tests { + use async_trait::async_trait; + use secrecy::SecretString; + use tablepro_core::{ + ConnectEnvironment, Credentials, NetworkEndpoint, NetworkTarget, ResultEvent, RunEnd, SinkFlow, TlsConfig, + }; + use tablepro_net::DirectTcpTransport; + use tablepro_test_fixtures::ClickHouseFixture; + use tablepro_test_support::conformance::test_liveness; + + use super::*; + + struct Discard; + + #[async_trait] + impl ResultSink for Discard { + async fn accept(&mut self, _event: ResultEvent) -> SinkFlow { + SinkFlow::Continue + } + } + + async fn run(session: &ClickHouseEditorSession, sql: &str) -> Result { + session + .run( + UserSql::new(sql), + RunOptions::new(CancellationToken::new()), + Box::new(Discard), + ) + .await + } + + #[tokio::test] + #[ignore = "requires docker"] + async fn a_session_the_server_let_expire_is_lost_not_replaced() { + let fixture = ClickHouseFixture::start().await.expect("a ClickHouse container"); + let liveness = test_liveness(); + let address = NetworkEndpoint::new(fixture.host(), fixture.http_port()).expect("an endpoint"); + let credentials = fixture.password_credentials(); + let target = NetworkTarget { + transport: Arc::new(DirectTcpTransport::new(address, &liveness)), + credentials: Credentials::Password { + username: credentials.username.clone(), + password: SecretString::from(credentials.password.clone()), + }, + database: Some("tablepro".to_owned()), + tls: TlsConfig::disabled(), + }; + let endpoint = ClickHouseEndpoint::new(&target, liveness, &ConnectEnvironment::new("test"), false) + .expect("an endpoint") + .with_session_timeout(Duration::from_secs(1)); + let session = + ClickHouseEditorSession::open(&endpoint, EditorSessionOptions::default(), CancellationToken::new()) + .await + .expect("an editor session"); + + let created = run(&session, "CREATE TEMPORARY TABLE scratch (x UInt8)").await; + assert_eq!(created.map(|summary| summary.end), Ok(RunEnd::Completed)); + tokio::time::sleep(Duration::from_secs(4)).await; + let outcome = run(&session, "SELECT count() FROM scratch").await; + + assert_eq!( + outcome.map(|summary| summary.end), + Err(DriverError::ConnectionLost { + during: LossPhase::Idle + }) + ); + assert_eq!(session.state(), EditorSessionState::Lost); + } +} diff --git a/linux/crates/drivers/clickhouse/src/endpoint.rs b/linux/crates/drivers/clickhouse/src/endpoint.rs index 9c697df431..6d75c008fa 100644 --- a/linux/crates/drivers/clickhouse/src/endpoint.rs +++ b/linux/crates/drivers/clickhouse/src/endpoint.rs @@ -11,7 +11,7 @@ use crate::errors::{ErrorContext, map_error_in}; use crate::result_reader::drain; /// How long the server keeps an editor session's state between runs. -const SESSION_TIMEOUT_SECONDS: &str = "3600"; +const SESSION_TIMEOUT: Duration = Duration::from_secs(3600); /// Everything needed to reach one server: the workspace's client, the /// sessionless control client that KILLs and resolves, and the @@ -22,6 +22,7 @@ pub(crate) struct ClickHouseEndpoint { transport: Arc, liveness: LivenessPolicy, tls_requested: bool, + session_timeout: Duration, } impl ClickHouseEndpoint { @@ -44,9 +45,16 @@ impl ClickHouseEndpoint { transport: target.transport.clone(), liveness, tls_requested: target.tls.mode != TlsMode::Disable, + session_timeout: SESSION_TIMEOUT, }) } + #[cfg(test)] + pub(crate) fn with_session_timeout(mut self, timeout: Duration) -> Self { + self.session_timeout = timeout; + self + } + /// The client of the workspace's own requests, read-only when the /// connection is. pub(crate) fn workspace(&self) -> &clickhouse::Client { @@ -63,13 +71,14 @@ impl ClickHouseEndpoint { &self.liveness } - /// The client of one editor session: the server keeps a session's - /// settings and temporary tables under its id. + /// The client that creates one editor session: the server keeps a + /// session's settings and temporary tables under its id, for as long + /// as the request that created it asked. pub(crate) fn session(&self, id: Uuid) -> clickhouse::Client { self.workspace .clone() .with_setting("session_id", id.to_string()) - .with_setting("session_timeout", SESSION_TIMEOUT_SECONDS) + .with_setting("session_timeout", self.session_timeout.as_secs().max(1).to_string()) } /// What the server says it is, and the database unqualified names diff --git a/linux/crates/drivers/clickhouse/src/errors.rs b/linux/crates/drivers/clickhouse/src/errors.rs index ca06befa74..e17dc431b6 100644 --- a/linux/crates/drivers/clickhouse/src/errors.rs +++ b/linux/crates/drivers/clickhouse/src/errors.rs @@ -19,6 +19,7 @@ pub(crate) mod code { pub(crate) const SOCKET_TIMEOUT: i32 = 209; pub(crate) const NETWORK_ERROR: i32 = 210; pub(crate) const UNFINISHED: i32 = 341; + pub(crate) const SESSION_NOT_FOUND: i32 = 372; pub(crate) const QUERY_IS_PROHIBITED: i32 = 392; pub(crate) const QUERY_WAS_CANCELLED: i32 = 394; pub(crate) const ACCESS_DENIED: i32 = 497; diff --git a/linux/crates/drivers/clickhouse/src/line_reader.rs b/linux/crates/drivers/clickhouse/src/line_reader.rs index d027d1467d..b09e6f8402 100644 --- a/linux/crates/drivers/clickhouse/src/line_reader.rs +++ b/linux/crates/drivers/clickhouse/src/line_reader.rs @@ -1,45 +1,118 @@ use clickhouse::query::BytesCursor; use tablepro_core::DriverError; -use tokio::io::AsyncBufReadExt; +use tokio::io::{AsyncBufRead, AsyncBufReadExt, AsyncReadExt}; use crate::errors::map_io_error; +/// What a response body yields next. +#[derive(Debug, PartialEq, Eq)] +pub(crate) enum Line { + /// One line, without its terminator. + Complete(Vec), + /// The line ran past the limit without ending. + TooLong, + End, +} + /// A response body read one `\n`-terminated line at a time, so a caller /// can stop at a row limit without reading the rest of the result. -pub(crate) struct LineReader { - cursor: BytesCursor, +pub(crate) struct LineReader { + cursor: R, line: Vec, } -impl LineReader { - pub(crate) fn new(cursor: BytesCursor) -> Self { +impl LineReader { + pub(crate) fn new(cursor: R) -> Self { Self { cursor, line: Vec::new(), } } - /// The next line, without its terminator, or `None` at the end. + /// The next line, which may hold at most `limit` bytes. + /// + /// A body in a format with no line breaks, such as RowBinary, is one + /// endless line, so without the limit it would be buffered until the + /// process ran out of memory. At most one byte past the limit is ever + /// held. /// /// Safe to use in `select!`: `read_until` appends to the buffer, so a /// read cut short keeps what it had and the next call carries on. - pub(crate) async fn next_line(&mut self) -> Result>, DriverError> { - self.cursor + pub(crate) async fn next_line(&mut self, limit: usize) -> Result { + let room = limit.saturating_add(1).saturating_sub(self.line.len()); + (&mut self.cursor) + .take(u64::try_from(room).unwrap_or(u64::MAX)) .read_until(b'\n', &mut self.line) .await .map_err(map_io_error)?; - if self.line.is_empty() { - return Ok(None); - } - let mut line = std::mem::take(&mut self.line); - if line.last() == Some(&b'\n') { + if self.line.last() == Some(&b'\n') { + let mut line = std::mem::take(&mut self.line); line.pop(); + return Ok(Line::Complete(line)); + } + if self.line.len() > limit { + return Ok(Line::TooLong); } - Ok(Some(line)) + if self.line.is_empty() { + return Ok(Line::End); + } + Ok(Line::Complete(std::mem::take(&mut self.line))) } +} +impl LineReader { /// The `X-ClickHouse-Summary` the server sent with the headers. pub(crate) fn written_rows(&self) -> Option { self.cursor.summary().and_then(clickhouse::QuerySummary::written_rows) } } + +#[cfg(test)] +mod tests { + use super::*; + + async fn lines(body: &[u8], limit: usize) -> Vec { + let mut reader = LineReader::new(body); + let mut lines = Vec::new(); + loop { + let line = reader.next_line(limit).await.expect("a line"); + let last = matches!(line, Line::End | Line::TooLong); + lines.push(line); + if last { + return lines; + } + } + } + + #[tokio::test] + async fn lines_split_on_newlines_and_the_last_needs_none() { + assert_eq!( + lines(b"[1]\n[22]\n[3]", 4).await, + vec![ + Line::Complete(b"[1]".to_vec()), + Line::Complete(b"[22]".to_vec()), + Line::Complete(b"[3]".to_vec()), + Line::End, + ] + ); + assert_eq!(lines(b"", 4).await, vec![Line::End]); + } + + #[tokio::test] + async fn a_line_past_the_limit_is_refused_whether_or_not_it_ends() { + assert_eq!( + lines(b"[1]\n12345\n", 4).await, + vec![Line::Complete(b"[1]".to_vec()), Line::TooLong] + ); + assert_eq!(lines(b"12345", 4).await, vec![Line::TooLong]); + } + + #[tokio::test] + async fn an_endless_line_buffers_no_more_than_one_byte_past_the_limit() { + let endless = tokio::io::BufReader::new(tokio::io::repeat(b'x')); + let mut reader = LineReader::new(endless); + + assert_eq!(reader.next_line(1024).await.expect("a line"), Line::TooLong); + assert_eq!(reader.line.len(), 1025); + } +} diff --git a/linux/crates/drivers/clickhouse/src/metadata.rs b/linux/crates/drivers/clickhouse/src/metadata.rs index 229e175236..7a1afdd965 100644 --- a/linux/crates/drivers/clickhouse/src/metadata.rs +++ b/linux/crates/drivers/clickhouse/src/metadata.rs @@ -10,7 +10,7 @@ const SYSTEM_DATABASES: [&str; 3] = ["system", "INFORMATION_SCHEMA", "informatio /// Every database, with the session's own as the default. pub(crate) async fn list_schemas(calls: &CallClient) -> Result, DriverError> { let sql = "SELECT name, name = currentDatabase() FROM system.databases ORDER BY name"; - let result = read_table(calls.raw(sql), None).await?; + let result = read_table(calls.raw(sql)?, None).await?; result .rows .iter() @@ -38,15 +38,15 @@ pub(crate) async fn list_objects( let base = "SELECT database, name, engine FROM system.tables WHERE is_temporary = 0"; let query = match scope { ObjectScope::DefaultSchema => calls - .raw(&format!("{base} AND database = {{db:String}} ORDER BY name")) + .raw(&format!("{base} AND database = {{db:String}} ORDER BY name"))? .param("db", default_database), ObjectScope::Schema(database) => calls - .raw(&format!("{base} AND database = {{db:String}} ORDER BY name")) + .raw(&format!("{base} AND database = {{db:String}} ORDER BY name"))? .param("db", database.as_str()), ObjectScope::AllSchemas => calls .raw(&format!( "{base} AND database NOT IN {{system:Array(String)}} ORDER BY database, name" - )) + ))? .param("system", SYSTEM_DATABASES), }; let mut objects = Vec::new(); @@ -60,7 +60,7 @@ pub(crate) async fn list_objects( } if !matches!(scope, ObjectScope::Schema(_)) { let sql = "SELECT name FROM system.functions WHERE origin = 'SQLUserDefined' ORDER BY name"; - for row in &read_table(calls.raw(sql), None).await?.rows { + for row in &read_table(calls.raw(sql)?, None).await?.rows { objects.push(ObjectInfo { schema: None, name: CatalogRow(row).text(0)?, diff --git a/linux/crates/drivers/clickhouse/src/result_reader.rs b/linux/crates/drivers/clickhouse/src/result_reader.rs index bcc03a232b..49d9566995 100644 --- a/linux/crates/drivers/clickhouse/src/result_reader.rs +++ b/linux/crates/drivers/clickhouse/src/result_reader.rs @@ -3,11 +3,12 @@ use std::sync::Arc; use clickhouse::query::{BytesCursor, Query}; use tablepro_core::column::ResultColumn; use tablepro_core::{DriverError, Value}; +use tokio::io::AsyncBufRead; use crate::cell_seed::CellPlan; use crate::clickhouse_type::ClickHouseType; use crate::errors::map_error; -use crate::line_reader::LineReader; +use crate::line_reader::{Line, LineReader}; use crate::row_seed::decode_row; /// Rows one JSON array per line, after a line of column names and a line @@ -15,6 +16,24 @@ use crate::row_seed::decode_row; /// types come with the result, so each cell is read by its own type. pub(crate) const RESULT_FORMAT: &str = "JSONCompactEachRowWithNamesAndTypes"; +/// How long a line of the body may grow before the read gives up. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct LineLimits { + /// A header line holds the column names or types, which no real + /// result makes this long. A longer one is a body in another format, + /// which a FORMAT clause in user SQL asks for. + pub(crate) header: usize, + /// One row, which the read has to hold whole to decode. + pub(crate) row: usize, +} + +impl LineLimits { + pub(crate) const DEFAULT: Self = Self { + header: 16 << 20, + row: 256 << 20, + }; +} + /// What a response body yields next. #[derive(Debug)] pub(crate) enum ReadEvent { @@ -25,46 +44,68 @@ pub(crate) enum ReadEvent { /// A response body read as events. The state lives here rather than in /// the future reading it, so it may be raced against a stop. -pub(crate) struct ResultReader { - lines: LineReader, +pub(crate) struct ResultReader { + lines: LineReader, + limits: LineLimits, names: Option>, plans: Option>, } impl ResultReader { - pub(crate) fn new(cursor: BytesCursor) -> Self { - Self { - lines: LineReader::new(cursor), - names: None, - plans: None, - } - } - /// Send the query and read its body as events. pub(crate) fn open(query: Query) -> Result { query .fetch_bytes(RESULT_FORMAT) - .map(Self::new) + .map(|cursor| Self::new(cursor, LineLimits::DEFAULT)) .map_err(|error| map_error(&error)) } + pub(crate) fn written_rows(&self) -> Option { + self.lines.written_rows() + } +} + +impl ResultReader { + pub(crate) fn new(cursor: R, limits: LineLimits) -> Self { + Self { + lines: LineReader::new(cursor), + limits, + names: None, + plans: None, + } + } + /// Whether the body began with a result set. A statement without one, /// such as an INSERT or a CREATE, sends an empty body. pub(crate) fn has_result_set(&self) -> bool { self.plans.is_some() } - pub(crate) fn written_rows(&self) -> Option { - self.lines.written_rows() - } - pub(crate) async fn next_event(&mut self) -> Result { loop { - let Some(line) = self.lines.next_line().await? else { - if self.names.is_some() && self.plans.is_none() { - return Err(DriverError::Protocol("the result ended inside its header".to_owned())); + let limit = match self.plans { + Some(_) => self.limits.row, + None => self.limits.header, + }; + let line = match self.lines.next_line(limit).await? { + Line::Complete(line) => line, + Line::TooLong if self.plans.is_some() => { + return Err(DriverError::Protocol(format!( + "a row ran past {} MiB, the most the driver reads into memory", + self.limits.row >> 20 + ))); + } + Line::TooLong => { + return Err(DriverError::Protocol(format!( + "the result did not come back as {RESULT_FORMAT}; a FORMAT clause in the query changes the format" + ))); + } + Line::End => { + if self.names.is_some() && self.plans.is_none() { + return Err(DriverError::Protocol("the result ended inside its header".to_owned())); + } + return Ok(ReadEvent::End); } - return Ok(ReadEvent::End); }; if let Some(plans) = &self.plans { return decode_row(&line, plans).map(ReadEvent::Row); @@ -205,3 +246,57 @@ impl CatalogRow<'_> { fn unexpected(index: usize, value: Option<&Value>) -> DriverError { DriverError::Protocol(format!("a catalogue column {index} held {value:?}")) } + +#[cfg(test)] +mod tests { + use super::*; + + const SMALL: LineLimits = LineLimits { header: 64, row: 16 }; + + fn reader(body: &[u8]) -> ResultReader<&[u8]> { + ResultReader::new(body, SMALL) + } + + #[tokio::test] + async fn a_body_reads_as_columns_rows_and_an_end() { + let mut reader = reader(b"[\"n\"]\n[\"UInt8\"]\n[1]\n[2]\n"); + + let ReadEvent::Columns(columns) = reader.next_event().await.expect("the header") else { + panic!("the header was not read first"); + }; + assert_eq!(columns.len(), 1); + for expected in [1, 2] { + let ReadEvent::Row(row) = reader.next_event().await.expect("a row") else { + panic!("row {expected} was not read"); + }; + assert_eq!(row, vec![Value::Int(expected)]); + } + assert!(matches!(reader.next_event().await, Ok(ReadEvent::End))); + } + + #[tokio::test] + async fn a_body_in_another_format_is_refused_at_the_header_limit() { + let row_binary: Vec = b"\x01x".repeat(64); + + let refused = reader(&row_binary).next_event().await; + + let Err(DriverError::Protocol(message)) = refused else { + panic!("a RowBinary body gave {refused:?}"); + }; + assert!(message.contains("FORMAT"), "{message}"); + } + + #[tokio::test] + async fn a_row_past_the_row_limit_is_refused_after_the_rows_before_it() { + let body = format!("[\"s\"]\n[\"String\"]\n[\"a\"]\n[\"{}\"]\n", "x".repeat(16)); + let mut reader = reader(body.as_bytes()); + + assert!(matches!(reader.next_event().await, Ok(ReadEvent::Columns(_)))); + assert!(matches!(reader.next_event().await, Ok(ReadEvent::Row(_)))); + let refused = reader.next_event().await; + assert!( + matches!(&refused, Err(DriverError::Protocol(message)) if message.contains("MiB")), + "{refused:?}" + ); + } +} diff --git a/linux/crates/drivers/clickhouse/src/stream_read.rs b/linux/crates/drivers/clickhouse/src/stream_read.rs index 16916100fa..8382de6497 100644 --- a/linux/crates/drivers/clickhouse/src/stream_read.rs +++ b/linux/crates/drivers/clickhouse/src/stream_read.rs @@ -14,8 +14,8 @@ pub(crate) async fn stream( sink: &mut dyn ResultSink, ) -> Result { let query = match read { - ReadStatement::Generated(statement) => calls.statement(statement), - ReadStatement::User(sql) => calls.raw(sql.as_str()), + ReadStatement::Generated(statement) => calls.statement(statement)?, + ReadStatement::User(sql) => calls.raw(sql.as_str())?, }; let mut reader = ResultReader::open(query)?; let mut emitter = Emitter { diff --git a/linux/crates/drivers/clickhouse/src/write_batch.rs b/linux/crates/drivers/clickhouse/src/write_batch.rs index c8b33cf2b2..d71d349869 100644 --- a/linux/crates/drivers/clickhouse/src/write_batch.rs +++ b/linux/crates/drivers/clickhouse/src/write_batch.rs @@ -1,6 +1,7 @@ use std::sync::Arc; use clickhouse::error::Error as ClickHouseError; +use tablepro_core::meta::TableRef; use tablepro_core::statement::{RowGuard, Statement}; use tablepro_core::{ CommitToken, DriverError, ReadOnlyRefusal, SessionFate, StopRequest, WriteBatch, WriteLedger, WriteReport, @@ -35,9 +36,11 @@ pub(crate) fn precheck(batch: &WriteBatch, read_only: bool) -> Result<(), Driver /// Everything one write needs once its task runs. pub(crate) struct WritePlan { pub(crate) batch: WriteBatch, + /// The batch's table, qualified, for the commit token. + pub(crate) table: TableRef, pub(crate) ledger: Arc, /// Reads that may be stopped, for the probes. - pub(crate) probes: CallClient, + pub(crate) probes: clickhouse::Client, /// Waits for each statement to finish, and never bounds it: a /// mutation the server gives up waiting on keeps running anyway. pub(crate) writes: clickhouse::Client, @@ -54,9 +57,10 @@ pub(crate) struct WritePlan { pub(crate) async fn perform( plan: &WritePlan, request: &StopRequest, - stop: &ClickHouseStop, + stop: &Arc, ) -> Result { let ledger = &plan.ledger; + let probes = CallClient::new(plan.probes.clone(), stop.clone(), request.clone()); for (index, step) in plan.batch.steps.iter().enumerate() { let Some((probe, expected)) = step.guard.probe() else { continue; @@ -64,7 +68,7 @@ pub(crate) async fn perform( if request.is_requested() { return Err(ledger.interrupted(stop.interrupted(request), SessionFate::RolledBack)); } - match count(&plan.probes, probe).await { + match count(&probes, probe).await { Ok(actual) if actual == expected => {} Ok(actual) => return Err(ledger.guard_failed(index, expected, actual, SessionFate::RolledBack)), Err(error) if request.is_requested() && acknowledges_stop(&error) => { @@ -83,6 +87,7 @@ pub(crate) async fn perform( ledger.attach_token(CommitToken::ClickHouseQuery { query_ids: query_ids.clone(), first_unconfirmed: index, + table: plan.table.clone(), }); ledger.statement_sending(index); match execute(&plan.writes, &step.statement, *query_id).await { @@ -131,7 +136,7 @@ async fn execute( } async fn count(calls: &CallClient, probe: &Statement) -> Result { - let result = read_table(calls.statement(probe), None).await?; + let result = read_table(calls.statement(probe)?, None).await?; let value = result .rows .first() diff --git a/linux/crates/drivers/clickhouse/tests/integration/sessions.rs b/linux/crates/drivers/clickhouse/tests/integration/sessions.rs index 44d5711c46..5f4c5fbee0 100644 --- a/linux/crates/drivers/clickhouse/tests/integration/sessions.rs +++ b/linux/crates/drivers/clickhouse/tests/integration/sessions.rs @@ -3,10 +3,45 @@ use std::time::Duration; -use tablepro_core::{ResultEvent, RowLimit, RunEnd, RunOptions, Truncation, Value}; +use async_trait::async_trait; +use tablepro_core::value::Temporal; +use tablepro_core::{ + Connection, DriverError, ReadStatement, ResultEvent, ResultSink, RowLimit, RunEnd, RunOptions, SinkFlow, + Truncation, UserSql, Value, +}; +use tokio::time::Instant; use tokio_util::sync::CancellationToken; -use crate::support::{connect, count, flush_logs, run, run_ok, run_with, scalar, session, start, text, unique}; +use crate::support::{ + CALL, connect, count, flush_logs, run, run_ok, run_with, scalar, session, start, text, unique, within, +}; + +/// A result that holds no line break, so one read as lines is one line +/// that never ends. +fn row_binary(marker: &str) -> String { + format!("SELECT '{marker}' FROM numbers(10000000000) FORMAT RowBinary") +} + +async fn wait_until_gone(connection: &dyn Connection, marker: &str) { + let running = format!( + "SELECT count() FROM system.processes WHERE query LIKE '%{marker}%' \ + AND query NOT LIKE '%system.processes%'" + ); + let give_up = Instant::now() + Duration::from_secs(10); + while count(connection, &running).await > 0 { + assert!(Instant::now() < give_up, "the statement is still running"); + tokio::time::sleep(Duration::from_millis(100)).await; + } +} + +struct Discard; + +#[async_trait] +impl ResultSink for Discard { + async fn accept(&mut self, _event: ResultEvent) -> SinkFlow { + SinkFlow::Continue + } +} #[tokio::test] #[ignore = "requires docker"] @@ -155,3 +190,71 @@ async fn a_question_mark_in_editor_sql_is_data() { editor.close().await; connection.close().await.expect("a close"); } + +#[tokio::test] +#[ignore = "requires docker"] +async fn a_format_clause_fails_the_run_instead_of_filling_memory() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + let editor = session(connection.as_ref()).await; + let marker = unique("format"); + + let (summary, _) = tokio::time::timeout(Duration::from_secs(60), run(editor.as_ref(), &row_binary(&marker))) + .await + .expect("the run ended"); + + assert!( + matches!(&summary.end, RunEnd::Failed(DriverError::Protocol(message)) if message.contains("FORMAT")), + "{:?}", + summary.end + ); + wait_until_gone(connection.as_ref(), &marker).await; + run_ok(editor.as_ref(), "SELECT 1").await; + editor.close().await; + connection.close().await.expect("a close"); +} + +#[tokio::test] +#[ignore = "requires docker"] +async fn a_format_clause_fails_an_export_instead_of_filling_memory() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + let marker = unique("export"); + let read = ReadStatement::User(UserSql::new(row_binary(&marker))); + + let outcome = tokio::time::timeout( + Duration::from_secs(60), + connection.stream_read(&read, Box::new(Discard), within(CALL)), + ) + .await + .expect("the export ended"); + + assert!( + matches!(&outcome, Err(DriverError::Protocol(message)) if message.contains("FORMAT")), + "{outcome:?}" + ); + wait_until_gone(connection.as_ref(), &marker).await; + connection.close().await.expect("a close"); +} + +#[tokio::test] +#[ignore = "requires docker"] +async fn a_session_set_to_iso_still_reads_timestamps_in_their_own_zone() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + let editor = session(connection.as_ref()).await; + + run_ok(editor.as_ref(), "SET date_time_output_format = 'iso'").await; + let rows = run_ok( + editor.as_ref(), + "SELECT toDateTime('2024-06-15 08:30:00', 'Asia/Tokyo')", + ) + .await; + + let stamp = chrono::NaiveDate::from_ymd_opt(2024, 6, 15) + .and_then(|date| date.and_hms_opt(8, 30, 0)) + .expect("a timestamp"); + assert_eq!(rows.rows(), vec![vec![Value::Timestamp(Temporal::Finite(stamp))]]); + editor.close().await; + connection.close().await.expect("a close"); +} diff --git a/linux/crates/drivers/clickhouse/tests/integration/support.rs b/linux/crates/drivers/clickhouse/tests/integration/support.rs index f07d604865..ac33d64396 100644 --- a/linux/crates/drivers/clickhouse/tests/integration/support.rs +++ b/linux/crates/drivers/clickhouse/tests/integration/support.rs @@ -145,6 +145,13 @@ pub fn batch(statements: &[&str]) -> WriteBatch { ) } +/// Row changes to the table `name` in the test database. +pub fn batch_on(name: &str, statements: &[&str]) -> WriteBatch { + let mut batch = batch(statements); + batch.table = table(name); + batch +} + /// The save the grid would plan for `changes` on `table`. pub async fn planned(connection: &dyn Connection, table: &TableRef, changes: &ChangeSet) -> WriteBatch { let columns = connection diff --git a/linux/crates/drivers/clickhouse/tests/integration/writes.rs b/linux/crates/drivers/clickhouse/tests/integration/writes.rs index 97162172cd..f09c159a24 100644 --- a/linux/crates/drivers/clickhouse/tests/integration/writes.rs +++ b/linux/crates/drivers/clickhouse/tests/integration/writes.rs @@ -11,7 +11,7 @@ use tablepro_core::{ use tokio::time::Instant; use crate::support::{ - CALL, DATABASE, admin, apply, batch, connect, count, flush_logs, planned, quoted, run, scalar, session, setup, + CALL, DATABASE, admin, apply, batch_on, connect, count, flush_logs, planned, quoted, run, scalar, session, setup, start, table, text, unique, within, }; @@ -197,10 +197,14 @@ async fn write_statements_never_carry_max_execution_time() { async fn alter_update_deadline_commit_outcome_unknown_resolves_still_running_then_applied() { let fixture = start().await; let connection = connect(&fixture, false).await; - let qualified = slow_table(connection.as_ref(), &unique("deadline")).await; + let name = unique("deadline"); + let qualified = slow_table(connection.as_ref(), &name).await; let outcome = connection - .write_batch(&batch(&[&slow_update(&qualified)]), within(Duration::from_secs(1))) + .write_batch( + &batch_on(&name, &[&slow_update(&qualified)]), + within(Duration::from_secs(1)), + ) .await; let Err(DriverError::CommitOutcomeUnknown { token: Some(token), .. }) = outcome else { @@ -233,8 +237,8 @@ async fn alter_update_killed_by_other_session_is_unknown_not_cancelled() { let writing = { let connection = connection.clone(); - let update = slow_update(&qualified); - tokio::spawn(async move { connection.write_batch(&batch(&[&update]), within(CALL)).await }) + let update = batch_on(&name, &[&slow_update(&qualified)]); + tokio::spawn(async move { connection.write_batch(&update, within(CALL)).await }) }; let monitor = session(connection.as_ref()).await; let find = format!( @@ -288,12 +292,15 @@ async fn failing_mutation_unfinished_is_unknown_and_still_running() { ) .await; - let outcome = apply( - connection.as_ref(), - WriteKind::RowChanges, - &[&format!("ALTER TABLE {qualified} UPDATE n = toUInt8(s) WHERE 1")], - ) - .await; + let outcome = connection + .write_batch( + &batch_on( + &name, + &[&format!("ALTER TABLE {qualified} UPDATE n = toUInt8(s) WHERE 1")], + ), + within(CALL), + ) + .await; let Err(DriverError::CommitOutcomeUnknown { token: Some(token), .. }) = outcome else { panic!("a failing mutation gave {outcome:?}"); @@ -310,6 +317,77 @@ async fn failing_mutation_unfinished_is_unknown_and_still_running() { connection.close().await.expect("a close"); } +/// The ALTER's mutation never runs and is killed, while another table's +/// mutation lands inside the ALTER's window. Its log names no table, so +/// only the table the write was for can keep the other one out. +#[tokio::test] +#[ignore = "requires docker"] +async fn killed_mutation_unresolvable_despite_another_tables_mutation_in_its_window() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + let held = unique("held"); + let other = unique("other"); + let (held_table, other_table) = (quoted(&held), quoted(&other)); + setup( + connection.as_ref(), + &[ + &format!("CREATE TABLE {held_table} (id Int32, n Int32) ENGINE = MergeTree ORDER BY id"), + &format!("INSERT INTO {held_table} VALUES (1, 0)"), + &format!("CREATE TABLE {other_table} (id Int32, n Int32) ENGINE = MergeTree ORDER BY id"), + &format!("INSERT INTO {other_table} VALUES (1, 0)"), + ], + ) + .await; + admin(connection.as_ref(), &format!("SYSTEM STOP MERGES {held_table}")).await; + + let writing = { + let connection = connection.clone(); + let update = batch_on(&held, &[&format!("ALTER TABLE {held_table} UPDATE n = 1 WHERE 1")]); + tokio::spawn(async move { connection.write_batch(&update, within(CALL)).await }) + }; + let waiting = format!( + "SELECT count() FROM system.processes WHERE query LIKE 'ALTER TABLE%{held}%' \ + AND query NOT LIKE '%system.processes%'" + ); + let give_up = Instant::now() + Duration::from_secs(10); + while count(connection.as_ref(), &waiting).await == 0 { + assert!( + Instant::now() < give_up, + "the ALTER never showed up in system.processes" + ); + tokio::time::sleep(Duration::from_millis(100)).await; + } + admin( + connection.as_ref(), + &format!("ALTER TABLE {other_table} UPDATE n = 1 WHERE 1"), + ) + .await; + admin( + connection.as_ref(), + &format!("KILL MUTATION WHERE database = '{DATABASE}' AND table = '{held}' SYNC"), + ) + .await; + + let outcome = writing.await.expect("the write task"); + let Err(DriverError::CommitOutcomeUnknown { token: Some(token), .. }) = outcome else { + panic!("a mutation killed under its ALTER gave {outcome:?}"); + }; + assert_eq!( + count( + connection.as_ref(), + &format!("SELECT count() FROM system.mutations WHERE table = '{other}' AND is_done") + ) + .await, + 1, + "the other table's mutation did not land" + ); + assert_eq!( + resolution(connection.as_ref(), &token).await, + CommitResolution::Unresolvable + ); + connection.close().await.expect("a close"); +} + #[tokio::test] #[ignore = "requires docker"] async fn resolve_finished_insert_applied() { @@ -334,6 +412,7 @@ async fn resolve_finished_insert_applied() { let token = CommitToken::ClickHouseQuery { query_ids: vec![query_id], first_unconfirmed: 0, + table: table(&name), }; assert_eq!(resolution(connection.as_ref(), &token).await, CommitResolution::Applied); @@ -348,6 +427,7 @@ async fn resolve_random_id_unresolvable() { let token = CommitToken::ClickHouseQuery { query_ids: vec![uuid::Uuid::new_v4()], first_unconfirmed: 0, + table: table("t"), }; assert_eq!( @@ -390,6 +470,7 @@ async fn resolve_syntax_error_insert_not_applied() { let token = CommitToken::ClickHouseQuery { query_ids: vec![query_id], first_unconfirmed: 0, + table: table(&name), }; assert_eq!( diff --git a/linux/crates/drivers/mssql/src/catalog/mod.rs b/linux/crates/drivers/mssql/src/catalog/mod.rs index 4647009080..ac5f77f3b4 100644 --- a/linux/crates/drivers/mssql/src/catalog/mod.rs +++ b/linux/crates/drivers/mssql/src/catalog/mod.rs @@ -108,8 +108,8 @@ pub(crate) fn result_column(column: &Column) -> ResultColumn { TdsType::Int4 | TdsType::Intn => integer(IntegerWidth::W32, true), TdsType::Int8 => integer(IntegerWidth::W64, true), TdsType::Float4 => ColumnKind::Float(FloatWidth::F32), - // tiberius reads MONEY as a float, so here it is one. - TdsType::Float8 | TdsType::Floatn | TdsType::Money | TdsType::Money4 => ColumnKind::Float(FloatWidth::F64), + TdsType::Float8 | TdsType::Floatn => ColumnKind::Float(FloatWidth::F64), + TdsType::Money | TdsType::Money4 => ColumnKind::Money { scale: Some(4) }, TdsType::Decimaln | TdsType::Numericn => ColumnKind::Decimal { precision: None, scale: None, @@ -143,14 +143,10 @@ pub(crate) fn result_column(column: &Column) -> ResultColumn { TdsType::SSVariant => ColumnKind::Variant, TdsType::Null => ColumnKind::Other, }; - let name = match tds { - TdsType::Money | TdsType::Money4 => format!("{} (approximate)", type_label(tds)), - other => type_label(other).to_owned(), - }; ResultColumn::new( column.name(), ColumnType { - name, + name: type_label(tds).to_owned(), kind, catalog: None, dynamic_storage: false, @@ -255,11 +251,11 @@ mod tests { } #[test] - fn editor_money_is_labelled_approximate() { + fn editor_money_is_money_of_scale_4() { let money = result_column(&Column::new("price".to_owned(), TdsType::Money)); - assert_eq!(money.column_type.name, "money (approximate)"); - assert_eq!(money.column_type.kind, ColumnKind::Float(FloatWidth::F64)); + assert_eq!(money.column_type.name, "money"); + assert_eq!(money.column_type.kind, ColumnKind::Money { scale: Some(4) }); assert_eq!( result_column(&Column::new("v".to_owned(), TdsType::SSVariant)) .column_type diff --git a/linux/crates/drivers/mssql/src/catalog/mssql_column_row.rs b/linux/crates/drivers/mssql/src/catalog/mssql_column_row.rs index 2db4d683e1..2f5814a79c 100644 --- a/linux/crates/drivers/mssql/src/catalog/mssql_column_row.rs +++ b/linux/crates/drivers/mssql/src/catalog/mssql_column_row.rs @@ -9,7 +9,8 @@ use super::sql_server_type::{CatalogFacts, SqlServerType}; /// An alias type is classified by the system type it is stored as, /// which `TYPE_NAME(system_type_id)` names; a CLR type keeps its own /// name. A collation is reported only where it differs from the -/// database's, which is the default a column takes. +/// database's, which is the default a column takes, and the last column +/// says whether its code page differs too. pub(crate) const COLUMNS_SQL: &str = "SELECT \ c.name, \ t.name, \ @@ -34,7 +35,10 @@ pub(crate) const COLUMNS_SQL: &str = "SELECT \ THEN c.collation_name END, \ CAST(c.is_sparse AS int), \ cc.definition, \ - CAST(ISNULL(cc.is_persisted, 0) AS int) \ + CAST(ISNULL(cc.is_persisted, 0) AS int), \ + CAST(CASE WHEN COLLATIONPROPERTY(c.collation_name, 'CodePage') \ + <> COLLATIONPROPERTY(CAST(DATABASEPROPERTYEX(DB_NAME(), 'Collation') AS nvarchar(128)), 'CodePage') \ + THEN 1 ELSE 0 END AS int) \ FROM sys.columns c \ JOIN sys.types t ON t.user_type_id = c.user_type_id \ LEFT JOIN sys.default_constraints dc ON dc.object_id = c.default_object_id \ @@ -85,6 +89,7 @@ impl MssqlColumnRow { max_length: cells.int(7).unwrap_or(0), precision: cells.int(8).unwrap_or(0), scale: cells.int(9).unwrap_or(0), + foreign_code_page: cells.flag(21), }); Ok(Self { name: cells.required_text(0)?, @@ -136,6 +141,7 @@ mod tests { Value::Int(0), Value::Null, Value::Int(0), + Value::Int(0), ]; let row = MssqlColumnRow::from_values(&values).unwrap(); diff --git a/linux/crates/drivers/mssql/src/catalog/sql_server_type.rs b/linux/crates/drivers/mssql/src/catalog/sql_server_type.rs index 3571b5e87b..726f613639 100644 --- a/linux/crates/drivers/mssql/src/catalog/sql_server_type.rs +++ b/linux/crates/drivers/mssql/src/catalog/sql_server_type.rs @@ -26,6 +26,9 @@ pub(crate) struct SqlServerType { /// the server, or an alias typed by name. It is read as the server's /// text. opaque: bool, + /// A character type whose collation has another code page than the + /// database's. + foreign_code_page: bool, } /// What the app makes of a type. @@ -100,6 +103,7 @@ impl SqlServerType { scale: u32::try_from(facts.scale).ok(), alias, opaque: facts.user_clr, + foreign_code_page: facts.foreign_code_page, } } @@ -131,6 +135,7 @@ impl SqlServerType { scale: None, alias: Some(text.to_owned()), opaque: true, + foreign_code_page: false, }; } let arguments: Vec<&str> = arguments @@ -155,14 +160,20 @@ impl SqlServerType { scale, alias: None, opaque: false, + foreign_code_page: false, } } /// The type as DDL spells it, modifiers included. pub(crate) fn declaration(&self) -> String { - if let Some(alias) = &self.alias { - return alias.clone(); + match &self.alias { + Some(alias) => alias.clone(), + None => self.system_declaration(), } + } + + /// The system type values are stored as, modifiers included. + fn system_declaration(&self) -> String { let base = &self.base; match base.as_str() { "char" | "varchar" | "nchar" | "nvarchar" | "binary" | "varbinary" => match self.length { @@ -189,15 +200,21 @@ impl SqlServerType { /// The type a bound text parameter is cast to. A length would /// truncate silently, so text casts to its max form, while a decimal /// keeps its digits because a bare `decimal` means `decimal(18,0)`. + /// An alias casts to the system type under it, which the server + /// converts to the alias on the way in. A varchar in another code + /// page than the database's stays nvarchar: a cast would go through + /// the database's code page and could turn one character into + /// another. pub(crate) fn cast_target(&self) -> String { - if self.alias.is_some() { + if self.opaque { return self.declaration(); } match self.base.as_str() { + "char" | "varchar" | "text" if self.foreign_code_page => "nvarchar(max)".to_owned(), "char" | "varchar" | "text" => "varchar(max)".to_owned(), "nchar" | "nvarchar" | "ntext" | "sysname" => "nvarchar(max)".to_owned(), "binary" | "varbinary" | "image" => "varbinary(max)".to_owned(), - _ => self.declaration(), + _ => self.system_declaration(), } } @@ -333,6 +350,7 @@ pub(crate) struct CatalogFacts<'a> { pub(crate) max_length: i64, pub(crate) precision: i64, pub(crate) scale: i64, + pub(crate) foreign_code_page: bool, } fn bracket(name: &str) -> String { @@ -343,6 +361,21 @@ fn bracket(name: &str) -> String { mod tests { use super::*; + fn facts(base: &str, max_length: i64) -> CatalogFacts<'_> { + CatalogFacts { + type_name: base, + base_type: base, + type_schema: Some("sys"), + user_defined: false, + alias: false, + user_clr: false, + max_length, + precision: 0, + scale: 0, + foreign_code_page: false, + } + } + fn system(base: &str, max_length: i64, precision: i64, scale: i64) -> SqlServerType { SqlServerType::from_catalog(CatalogFacts { type_name: base, @@ -354,6 +387,7 @@ mod tests { max_length, precision, scale, + foreign_code_page: false, }) } @@ -397,6 +431,7 @@ mod tests { max_length: 40, precision: 0, scale: 0, + foreign_code_page: false, }); assert_eq!(phone.declaration(), "[dbo].[Phone]"); assert_eq!( @@ -415,6 +450,7 @@ mod tests { max_length: 256, precision: 0, scale: 0, + foreign_code_page: false, }); assert_eq!(sysname.declaration(), "sysname"); assert_eq!( @@ -437,6 +473,7 @@ mod tests { max_length: 64, precision: 0, scale: 0, + foreign_code_page: false, }); assert_eq!(clr.declaration(), "[geo].[Point]"); @@ -490,4 +527,31 @@ mod tests { assert_eq!(system("decimal", 9, 10, 2).cast_target(), "decimal(10,2)"); assert_eq!(system("hierarchyid", 892, 0, 0).cast_target(), "hierarchyid"); } + + #[test] + fn a_varchar_in_another_code_page_and_an_alias_cast_to_what_holds_them() { + let greek = SqlServerType::from_catalog(CatalogFacts { + foreign_code_page: true, + ..facts("varchar", 10) + }); + let code = SqlServerType::from_catalog(CatalogFacts { + type_name: "Code", + type_schema: Some("dbo"), + user_defined: true, + alias: true, + ..facts("varchar", 10) + }); + let stamp = SqlServerType::from_catalog(CatalogFacts { + type_name: "Stamp", + type_schema: Some("dbo"), + user_defined: true, + alias: true, + ..facts("datetime", 8) + }); + + assert_eq!(greek.cast_target(), "nvarchar(max)"); + assert_eq!(code.declaration(), "[dbo].[Code]"); + assert_eq!(code.cast_target(), "varchar(max)"); + assert_eq!(stamp.cast_target(), "datetime"); + } } diff --git a/linux/crates/drivers/mssql/src/decode.rs b/linux/crates/drivers/mssql/src/decode.rs index 18457b8c2d..2056e4bcd5 100644 --- a/linux/crates/drivers/mssql/src/decode.rs +++ b/linux/crates/drivers/mssql/src/decode.rs @@ -1,22 +1,30 @@ use chrono::{DateTime, Days, FixedOffset, NaiveDate, NaiveDateTime, NaiveTime}; use tablepro_core::value::{OffsetTimestamp, SqlDecimal, SqlTime, Temporal, UndecodableReason, UndecodedValue, Value}; use tiberius::time::{Date, DateTime as TdsDateTime, DateTime2, DateTimeOffset, SmallDateTime, Time}; -use tiberius::{ColumnData, Row}; +use tiberius::{ColumnData, ColumnType as TdsType, Row}; use crate::catalog::type_label; const NANOS_PER_SECOND: u64 = 1_000_000_000; +/// MONEY and SMALLMONEY carry ten-thousandths on the wire. +const MONEY_SCALE: u32 = 4; +const MONEY_UNITS_PER_ONE: f64 = 10_000.0; + +/// 2^53: past it, neighbouring integers share a float. +const EXACT_FLOAT_INTEGERS: f64 = 9_007_199_254_740_992.0; + /// Every cell of a row, in column order. pub(crate) fn decode_row(row: &Row) -> Vec { row.cells() - .map(|(column, data)| decode_cell(data, type_label(column.column_type()))) + .map(|(column, data)| decode_cell(data, column.column_type())) .collect() } /// One cell, exactly. A NULL is the only thing that reads as `Null`; a /// value the driver cannot represent says so instead. -pub(crate) fn decode_cell(data: &ColumnData<'static>, type_name: &str) -> Value { +pub(crate) fn decode_cell(data: &ColumnData<'static>, wire_type: TdsType) -> Value { + let type_name = type_label(wire_type); let decoded = match data { ColumnData::Bit(value) => value.map(Value::Bool), ColumnData::U8(value) => value.map(|integer| Value::Int(i64::from(integer))), @@ -24,9 +32,10 @@ pub(crate) fn decode_cell(data: &ColumnData<'static>, type_name: &str) -> Value ColumnData::I32(value) => value.map(|integer| Value::Int(i64::from(integer))), ColumnData::I64(value) => value.map(Value::Int), ColumnData::F32(value) => value.map(Value::Float32), - // MONEY arrives here too: tiberius reads it as a float. Browse - // reads money columns through a decimal cast instead. - ColumnData::F64(value) => value.map(Value::Float64), + ColumnData::F64(value) => value.map(|real| match wire_type { + TdsType::Money | TdsType::Money4 => money(real, type_name), + _ => Value::Float64(real), + }), ColumnData::String(value) => value.as_ref().map(|text| Value::Text(text.to_string())), ColumnData::Guid(value) => value.map(Value::Uuid), // CLR types such as geography and hierarchyid arrive as their @@ -79,6 +88,33 @@ pub(crate) fn decode_cell(data: &ColumnData<'static>, type_name: &str) -> Value decoded.unwrap_or(Value::Null) } +/// tiberius reads MONEY into a float, dividing the scaled integer on the +/// wire by 10^4. Where only one integer divides to that float, it is the +/// value exactly; past about 5.5 * 10^11 two can, and the cell says so +/// rather than pick one. +fn money(real: f64, type_name: &str) -> Value { + match money_units(real) { + Some(units) => Value::Decimal(SqlDecimal::from_scaled_i128(i128::from(units), MONEY_SCALE)), + None => Value::Undecodable(Box::new(UndecodedValue { + type_name: type_name.to_owned(), + reason: UndecodableReason::OutOfRange, + })), + } +} + +/// The scaled integer that tiberius turned into `real`, when no other +/// one turns into the same float. +fn money_units(real: f64) -> Option { + let estimate = (real * MONEY_UNITS_PER_ONE).round(); + if !estimate.is_finite() || estimate.abs() >= EXACT_FLOAT_INTEGERS { + return None; + } + let estimate = estimate as i64; + let reads_as = |units: i64| units as f64 / MONEY_UNITS_PER_ONE == real; + let units = (estimate - 2..=estimate + 2).find(|units| reads_as(*units))?; + (!reads_as(units - 1) && !reads_as(units + 1)).then_some(units) +} + fn temporal(value: Option, type_name: &str) -> Value { value.unwrap_or_else(|| { Value::Undecodable(Box::new(UndecodedValue { @@ -154,7 +190,16 @@ mod tests { use super::*; fn decode(data: ColumnData<'static>) -> Value { - decode_cell(&data, "test") + decode_cell(&data, TdsType::Null) + } + + /// MONEY as tiberius hands it over: the wire's integer over 10^4. + fn money_as_read(units: i64) -> Value { + decode_cell(&ColumnData::F64(Some(units as f64 / 10_000.0)), TdsType::Money) + } + + fn exact_money(units: i64) -> Value { + Value::Decimal(SqlDecimal::from_scaled_i128(i128::from(units), 4)) } /// 2024-06-15 counted in days from 0001-01-01. @@ -215,6 +260,33 @@ mod tests { assert_eq!(min, Value::Decimal("-922337203685477.5808".parse().unwrap())); } + #[test] + fn money_through_the_editor_is_exact_below_2_pow_39() { + // 549755813887.9999, the last value before floats grow coarser + // than a ten-thousandth. + let widest = 5_497_558_138_879_999; + for units in [0, 1, -1, 12_345, 2_147_483_647, -2_147_483_648, widest, -widest] { + assert_eq!(money_as_read(units), exact_money(units), "{units}"); + } + assert_eq!(decode_cell(&ColumnData::F64(None), TdsType::Money4), Value::Null); + assert_eq!(decode(ColumnData::F64(Some(0.1))), Value::Float64(0.1)); + } + + #[test] + fn money_too_wide_for_a_float_is_never_a_wrong_decimal() { + let near_the_ends = (i64::MAX - 1_000..=i64::MAX).chain(i64::MIN..=i64::MIN + 1_000); + let ambiguous = 8_000_000_000_000_000..8_000_000_000_001_000; + for units in near_the_ends.chain(ambiguous) { + let decoded = money_as_read(units); + assert!( + decoded == exact_money(units) + || matches!(&decoded, Value::Undecodable(undecoded) if undecoded.reason == UndecodableReason::OutOfRange), + "{units}: {decoded:?}" + ); + } + assert!(matches!(money_as_read(i64::MAX), Value::Undecodable(_))); + } + #[test] fn datetimeoffset_plus_0530_keeps_its_offset() { let utc_time = Time::new((8 * 3600 + 15 * 60) * 10_000_000 + 1_234_567, 7); diff --git a/linux/crates/drivers/mssql/src/dialect.rs b/linux/crates/drivers/mssql/src/dialect.rs index f5310946f3..3a6d729cdc 100644 --- a/linux/crates/drivers/mssql/src/dialect.rs +++ b/linux/crates/drivers/mssql/src/dialect.rs @@ -75,6 +75,9 @@ impl SqlDialect for MssqlDialect { Some(legacy) => format!("CAST({marker} AS {legacy})"), None => marker, }, + (ParamType::Text | ParamType::FixedText, BindTarget::Column(column)) if non_unicode(column) => { + format!("CAST({marker} AS {NON_UNICODE_TEXT})") + } _ => marker, }; Ok(Placeholder::bound(sql, BoundParam::new(value, param_type)?)) @@ -287,6 +290,24 @@ fn legacy_datetime(column: &ColumnInfo) -> Option<&'static str> { .find(|legacy| declared.eq_ignore_ascii_case(legacy)) } +/// What `SqlServerType::cast_target` gives a char or varchar column whose +/// code page is the database's. +const NON_UNICODE_TEXT: &str = "varchar(max)"; + +/// tiberius sends text as nvarchar, which outranks varchar, so +/// `[code] = @P1` converts the column. Under a SQL_* collation that +/// cannot seek, and the save probe's UPDLOCK, HOLDLOCK then scans and +/// locks the whole table until commit. Cast to varchar, the parameter +/// meets the column as its index has it. +fn non_unicode(column: &ColumnInfo) -> bool { + matches!(column.column_type.kind, ColumnKind::Text { .. }) + && column + .column_type + .catalog + .as_ref() + .is_some_and(|catalog| catalog.cast_target.as_sql().eq_ignore_ascii_case(NON_UNICODE_TEXT)) +} + /// The parameter type a column takes. fn param_type_for(column: &ColumnInfo) -> Result { let column_type = &column.column_type; @@ -384,6 +405,31 @@ mod tests { ); } + #[test] + fn text_for_a_varchar_column_is_cast_so_the_column_can_seek() { + let text = || Value::Text("k1".to_owned()); + let mut foreign = column("varchar(10)"); + if let Some(catalog) = foreign.column_type.catalog.as_mut() { + catalog.cast_target = SqlTypeExpr::from_catalog_text("nvarchar(max)"); + } + + assert_eq!(bound("varchar(10)", text()).sql, "CAST(@P1 AS varchar(max))"); + assert_eq!(bound("char(4)", text()).sql, "CAST(@P1 AS varchar(max))"); + assert_eq!(bound("varchar(10)", Value::Null).sql, "CAST(@P1 AS varchar(max))"); + assert_eq!(bound("nvarchar(10)", text()).sql, "@P1"); + assert_eq!( + dialect() + .placeholder(0, text(), BindTarget::Column(&foreign)) + .unwrap() + .sql, + "@P1" + ); + assert_eq!( + dialect().placeholder(0, text(), BindTarget::Pattern).unwrap().sql, + "@P1" + ); + } + #[test] fn markers_count_from_the_params_already_bound() { let placeholder = dialect() diff --git a/linux/crates/drivers/mssql/src/in_band_request.rs b/linux/crates/drivers/mssql/src/in_band_request.rs index 204988c715..63d5ea2186 100644 --- a/linux/crates/drivers/mssql/src/in_band_request.rs +++ b/linux/crates/drivers/mssql/src/in_band_request.rs @@ -184,7 +184,10 @@ impl Watch { } /// A request still waiting has consumed nothing if nothing arrived, - /// and may be inside a token if anything did. + /// and may be inside a token if anything did. Nothing new since the + /// last row proves less: the server fills each packet whatever token + /// that splits, and tiberius takes a token in pieces as packets + /// come, so the row after it may be half read already. fn halt_while_waiting(&self) -> Halt { match self.tally.bytes() == self.sent_at.load(Ordering::SeqCst) { true => Halt::Boundary, @@ -309,8 +312,10 @@ mod tests { use std::time::Duration; use tablepro_core::StopReason; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; use super::*; + use crate::transport_io::TransportIo; fn watch(tally: &ReadTally) -> Watch { Watch::new( @@ -362,6 +367,28 @@ mod tests { assert!(matches!(outcome, Err(Interrupt::Stopped(Halt::Boundary)))); } + /// The bytes stand for the first half of a row, split across two + /// packets, which the decoder has already taken. + #[tokio::test(start_paused = true)] + async fn a_request_that_went_quiet_after_any_answer_may_be_mid_token() { + let (client, mut server) = tokio::io::duplex(64); + let tally = ReadTally::new(); + let mut io = TransportIo::new(Box::new(client), tally.clone()); + let watch = watch(&tally); + let request = watch.request().clone(); + + let outcome = watch + .start(async move { + server.write_all(b"half a row").await.unwrap(); + io.read_exact(&mut [0u8; 10]).await.unwrap(); + request.request(StopReason::Cancelled); + std::future::pending::>().await + }) + .await; + + assert!(matches!(outcome, Err(Interrupt::Stopped(Halt::MidToken)))); + } + #[tokio::test] async fn a_request_that_ends_while_stopping_finished_on_its_own() { let watch = watch(&ReadTally::new()); diff --git a/linux/crates/drivers/mssql/tests/integration/decode.rs b/linux/crates/drivers/mssql/tests/integration/decode.rs index 18bad3d3c9..28753336be 100644 --- a/linux/crates/drivers/mssql/tests/integration/decode.rs +++ b/linux/crates/drivers/mssql/tests/integration/decode.rs @@ -5,7 +5,7 @@ use chrono::DateTime; use tablepro_core::value::{OffsetTimestamp, SqlDecimal, Temporal}; use tablepro_core::{Connection, Value}; -use crate::support::{browse, columns, connect, dbo, insert_one, rows, scalar, setup, start, write}; +use crate::support::{browse, columns, connect, dbo, insert_one, rows, run_ok, scalar, session, setup, start, write}; fn decimal(text: &str) -> Value { Value::Decimal(text.parse::().expect("a decimal")) @@ -101,6 +101,29 @@ async fn money_max_min_exact() { .await; } +/// The editor and plain queries read MONEY without the decimal cast +/// browse uses, through the float tiberius makes of it. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn money_in_the_editor_and_queries_reads_exactly() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + let session = session(connection.as_ref()).await; + let sql = "SELECT CAST(123456789.1234 AS money), CAST(-214748.3648 AS smallmoney), \ + CAST(549755813887.9999 AS money), CAST(NULL AS money), CAST(922337203685477.5807 AS money)"; + + let edited = run_ok(session.as_ref(), sql).await.rows().remove(0); + let queried = rows(connection.as_ref(), sql).await.remove(0); + + for row in [edited, queried] { + assert_eq!(row[0], decimal("123456789.1234")); + assert_eq!(row[1], decimal("-214748.3648")); + assert_eq!(row[2], decimal("549755813887.9999")); + assert_eq!(row[3], Value::Null); + assert!(matches!(&row[4], Value::Undecodable(_)), "{:?}", row[4]); + } +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[ignore = "requires docker"] async fn smallmoney() { diff --git a/linux/crates/drivers/mssql/tests/integration/write_batch.rs b/linux/crates/drivers/mssql/tests/integration/write_batch.rs index 63fc3f1955..ff481dd2d4 100644 --- a/linux/crates/drivers/mssql/tests/integration/write_batch.rs +++ b/linux/crates/drivers/mssql/tests/integration/write_batch.rs @@ -10,9 +10,9 @@ use tablepro_core::{CallOptions, DriverError, ErrorCategory, Value, WriteBatch}; use tokio::time::Instant; use tokio_util::sync::CancellationToken; -use crate::support::{CALL, batch, browse, columns, connect, dbo, scalar, setup, start, within}; +use crate::support::{CALL, batch, browse, columns, connect, dbo, run_ok, scalar, session, setup, start, within}; -async fn update_label(connection: &dyn tablepro_core::Connection, table: &TableRef, id: i64) -> WriteBatch { +async fn update_label(connection: &dyn tablepro_core::Connection, table: &TableRef, key: Value) -> WriteBatch { let columns = columns(connection, table).await; let identity = connection.row_identity(table, within(CALL)).await.expect("an identity"); let label = columns @@ -21,7 +21,7 @@ async fn update_label(connection: &dyn tablepro_core::Connection, table: &TableR .expect("a label column"); let changes = ChangeSet { updates: vec![RowUpdate { - key: vec![Value::Int(id)], + key: vec![key], assignments: vec![(label, Value::Text("changed".to_owned()))], }], ..ChangeSet::default() @@ -111,6 +111,42 @@ async fn datetime_key_with_a_third_of_a_millisecond_saves() { ); } +/// Under a SQL_* collation an nvarchar parameter cannot seek a varchar +/// key, so the probe's UPDLOCK, HOLDLOCK would scan into a row another +/// transaction holds and wait for it. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn varchar_key_save_locks_only_its_own_row() { + let fixture = start().await; + let connection = connect(&fixture, false).await; + setup( + connection.as_ref(), + &[ + "CREATE TABLE dbo.codes (code varchar(10) COLLATE SQL_Latin1_General_CP1_CI_AS PRIMARY KEY, \ + label nvarchar(50) NOT NULL)", + "INSERT INTO dbo.codes VALUES ('k1', N'one'), ('k2', N'two'), ('k3', N'three'), ('k4', N'four'), \ + ('k5', N'five')", + ], + ) + .await; + let holder = session(connection.as_ref()).await; + run_ok( + holder.as_ref(), + "BEGIN TRANSACTION; UPDATE dbo.codes SET label = N'held' WHERE code = 'k5'", + ) + .await; + let save = update_label(connection.as_ref(), &dbo("codes"), Value::Text("k1".to_owned())).await; + + let saved = connection.write_batch(&save, within(Duration::from_secs(5))).await; + run_ok(holder.as_ref(), "ROLLBACK").await; + + assert!(saved.is_ok(), "{saved:?}"); + assert_eq!( + scalar(connection.as_ref(), "SELECT label FROM dbo.codes WHERE code = 'k1'").await, + Value::Text("changed".to_owned()) + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[ignore = "requires docker"] async fn after_update_trigger_saves_without_false_conflict() { @@ -129,7 +165,7 @@ async fn after_update_trigger_saves_without_false_conflict() { .await; let table = dbo("items"); - let save = update_label(connection.as_ref(), &table, 1).await; + let save = update_label(connection.as_ref(), &table, Value::Int(1)).await; connection.write_batch(&save, within(CALL)).await.expect("the save"); assert_eq!( @@ -156,7 +192,7 @@ async fn probe_mismatch_rolls_back() { ) .await; let table = dbo("items"); - let mut save = update_label(connection.as_ref(), &table, 999).await; + let mut save = update_label(connection.as_ref(), &table, Value::Int(999)).await; let first = crate::support::statement("INSERT INTO dbo.items VALUES (2, N'new')"); save.steps .insert(0, tablepro_core::statement::GuardedStatement::unguarded(first)); diff --git a/linux/crates/drivers/mysql/src/editor_run.rs b/linux/crates/drivers/mysql/src/editor_run.rs index 61e0dc82c0..dab336105b 100644 --- a/linux/crates/drivers/mysql/src/editor_run.rs +++ b/linux/crates/drivers/mysql/src/editor_run.rs @@ -15,6 +15,7 @@ use tablepro_core::column::ResultColumn; use tablepro_core::sql_syntax::SqlGrammar; use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan, ScriptTokenKind}; use tablepro_core::{DriverError, ResultEvent, ResultSink, RowLimit, SinkFlow, Truncation, Value}; +use tokio_util::sync::CancellationToken; use crate::catalog::result_column; use crate::decode::decode_row; @@ -47,12 +48,17 @@ pub(crate) fn batches(sql: &str) -> Vec<&str> { plan.statements().iter().map(|statement| statement.text(sql)).collect() } +/// Run the script. When the sink stops reading, `consumer_gone` is +/// cancelled so the caller can stop the server. The batch already sent +/// is still read to its end, unshown, since only that frees the +/// connection, and no later batch is sent. pub(crate) async fn run_script( connection: &mut MySqlConnection, flavour: MySqlFlavour, sql: &str, limit: RowLimit, sink: &mut dyn ResultSink, + consumer_gone: &CancellationToken, ) -> Result { let mut emitter = Emitter { sink, @@ -62,6 +68,9 @@ pub(crate) async fn run_script( gone: false, }; for text in batches(sql) { + if emitter.gone { + break; + } let results = connection.fetch_many(sqlx::raw_sql(AssertSqlSafe(text.to_owned()))); let mut results = std::pin::pin!(results); while let Some(item) = results.try_next().await.map_err(|error| map_error(&error, flavour))? { @@ -70,11 +79,13 @@ pub(crate) async fn run_script( Either::Left(done) => emitter.done(done.rows_affected()).await, } if emitter.gone { - return Ok(RunOutcome { consumer_gone: true }); + consumer_gone.cancel(); } } } - Ok(RunOutcome::default()) + Ok(RunOutcome { + consumer_gone: emitter.gone, + }) } struct OpenSet { @@ -93,6 +104,9 @@ struct Emitter<'a> { impl Emitter<'_> { async fn row(&mut self, row: &MySqlRow) { + if self.gone { + return; + } if self.open.is_none() { let columns: Arc<[ResultColumn]> = row .columns() @@ -125,6 +139,9 @@ impl Emitter<'_> { } async fn done(&mut self, rows_affected: u64) { + if self.gone { + return; + } let Some(set) = self.open.take() else { self.send(ResultEvent::CommandCompleted { rows_affected: Some(rows_affected), diff --git a/linux/crates/drivers/mysql/src/editor_session.rs b/linux/crates/drivers/mysql/src/editor_session.rs index 4b9568f816..5a0150a5e0 100644 --- a/linux/crates/drivers/mysql/src/editor_session.rs +++ b/linux/crates/drivers/mysql/src/editor_session.rs @@ -82,9 +82,10 @@ impl MysqlEditorSession { })) } - /// Run the script, stopping it on a cancel, at the deadline or when - /// the session closes. A stop is KILL QUERY on a connection of the - /// session's own; the statement then has the cancel grace to end. + /// Run the script, stopping it on a cancel, at the deadline, when + /// the session closes or when the sink stops reading. A stop is KILL + /// QUERY on a connection of the session's own; the statement then + /// has the cancel grace to end. async fn supervise( &self, held: &mut Held, @@ -97,26 +98,32 @@ impl MysqlEditorSession { return finished(Err(error)).map(|end| (end, true)); } let deadline = options.deadline.map(|limit| Instant::now() + limit); - let running = run_script(&mut held.connection, self.flavour, sql.as_str(), limit, sink); + let consumer_gone = CancellationToken::new(); + let running = run_script( + &mut held.connection, + self.flavour, + sql.as_str(), + limit, + sink, + &consumer_gone, + ); let mut running = std::pin::pin!(running); let stop = tokio::select! { outcome = &mut running => return finished(outcome).map(|end| (end, true)), () = options.cancel.cancelled() => Stop::Cancelled, () = self.closing.cancelled() => Stop::Cancelled, + () = consumer_gone.cancelled() => Stop::Cancelled, () = elapsed(deadline) => Stop::DeadlineElapsed, }; let until = Instant::now() + self.liveness.cancel_grace; - let killing = self.control.kill_query(self.connection_id); - let mut killing = std::pin::pin!(killing); - let mut delivered = None; - let settled = loop { - tokio::select! { - biased; - sent = &mut killing, if delivered.is_none() => delivered = Some(sent.is_ok()), - outcome = &mut running => break Some(outcome), - () = tokio::time::sleep_until(until) => break None, - } - }; + // Nothing more is read until the KILL has landed. A batch the + // server is still sending then waits on the socket, so it cannot + // move on to its next statement before the KILL stops it. + let delivered = tokio::time::timeout_at(until, self.control.kill_query(self.connection_id)) + .await + .ok() + .map(|sent| sent.is_ok()); + let settled = tokio::time::timeout_at(until, &mut running).await.ok(); let server_cancelled = delivered == Some(true); let end = match stop { Stop::Cancelled => RunEnd::Cancelled, diff --git a/linux/crates/drivers/mysql/src/errors.rs b/linux/crates/drivers/mysql/src/errors.rs index 1f60d2226d..e922034a93 100644 --- a/linux/crates/drivers/mysql/src/errors.rs +++ b/linux/crates/drivers/mysql/src/errors.rs @@ -23,6 +23,9 @@ pub(crate) const READ_ONLY_TRANSACTION: u16 = 1792; /// ER_WARNING_NOT_COMPLETE_ROLLBACK: a table that cannot roll back was /// written inside the transaction. pub(crate) const INCOMPLETE_ROLLBACK: u16 = 1196; +/// ER_UNSUPPORTED_PS: the server parsed the statement and cannot run it +/// as a prepared statement. +pub(crate) const UNSUPPORTED_PS: u16 = 1295; /// MySQL's ER_QUERY_TIMEOUT for `max_execution_time`. pub(crate) const MYSQL_QUERY_TIMEOUT: u16 = 3024; /// MariaDB's ER_STATEMENT_TIMEOUT for `max_statement_time`. diff --git a/linux/crates/drivers/mysql/src/physical.rs b/linux/crates/drivers/mysql/src/physical.rs index ae20b92460..4558e1a208 100644 --- a/linux/crates/drivers/mysql/src/physical.rs +++ b/linux/crates/drivers/mysql/src/physical.rs @@ -48,6 +48,7 @@ pub(crate) struct MysqlEndpoint { transport: Arc, options: MySqlConnectOptions, tls_server_name: String, + verifies_name: bool, liveness: LivenessPolicy, } @@ -102,6 +103,7 @@ impl MysqlEndpoint { transport: target.transport.clone(), options, tls_server_name: target.tls_server_name().to_owned(), + verifies_name: target.tls.mode == TlsMode::VerifyFull, liveness, }) } @@ -117,15 +119,26 @@ impl MysqlEndpoint { /// The options for one login over the transport's route. Over a /// socket the `host` is not dialled, so it carries the name TLS /// verifies: the service's own, never the forward's. - fn options_for(&self, kind: SessionKind) -> MySqlConnectOptions { + /// + /// Over TCP, sqlx verifies the certificate against the host it dials + /// and takes no other name, so a route whose name differs is refused + /// rather than checked against the wrong one. + fn options_for(&self, kind: SessionKind) -> Result { let options = match self.transport.route() { - TransportRoute::Tcp(endpoint) => self.options.clone().host(endpoint.host()).port(endpoint.port()), + TransportRoute::Tcp(endpoint) => { + if self.verifies_name && endpoint.host() != self.tls_server_name { + return Err(DriverError::Config(ConfigError::ServerNameOverrideUnsupported { + route: self.transport.kind().class(), + })); + } + self.options.clone().host(endpoint.host()).port(endpoint.port()) + } TransportRoute::UnixSocket(path) => self.options.clone().socket(path).host(&self.tls_server_name), }; - match kind { + Ok(match kind { SessionKind::Workspace => options.timezone(Some(WORKSPACE_TIME_ZONE.to_owned())), SessionKind::Editor | SessionKind::Control => options, - } + }) } /// Log in and set the session up, within the connect and login @@ -152,7 +165,7 @@ impl MysqlEndpoint { } async fn open(&self, kind: SessionKind, limits: &SessionLimits) -> Result { - let options = self.options_for(kind); + let options = self.options_for(kind)?; let mut connection = match options.connect().await { Ok(connection) => connection, Err(error) => return Err(self.explain(map_connect_error(&error, self.service())).await), @@ -253,3 +266,141 @@ async fn identify(connection: &mut MySqlConnection, endpoint: &NetworkEndpoint) .map_err(|_| DriverError::Protocol(format!("CONNECTION_ID() returned {id:?}")))?; Ok((connection_id, text(1)?)) } + +#[cfg(test)] +mod tests { + use async_trait::async_trait; + use secrecy::SecretString; + use tablepro_core::{TlsConfig, TransportClass, TransportError, TransportKind, TransportStream}; + + use super::*; + + /// A route whose dial target is not the service, as a forward that + /// listens on a local port would be. + #[derive(Debug)] + struct Route { + service: NetworkEndpoint, + kind: TransportKind, + route: TransportRoute, + } + + #[async_trait] + impl Transport for Route { + fn service_endpoint(&self) -> &NetworkEndpoint { + &self.service + } + + fn kind(&self) -> TransportKind { + self.kind.clone() + } + + fn route(&self) -> TransportRoute { + self.route.clone() + } + + async fn open(&self) -> Result, TransportError> { + Err(TransportError::Refused { + endpoint: self.service.clone(), + }) + } + + async fn reroute(&self, _endpoint: NetworkEndpoint) -> Result, TransportError> { + Err(TransportError::Refused { + endpoint: self.service.clone(), + }) + } + + async fn explain_closed_stream(&self) -> Option { + None + } + } + + fn endpoint(host: &str) -> NetworkEndpoint { + NetworkEndpoint::new(host, 3306).unwrap() + } + + fn mysql_endpoint(route: Route, server_name: Option<&str>) -> MysqlEndpoint { + let target = NetworkTarget { + transport: Arc::new(route), + credentials: Credentials::Password { + username: "app".to_owned(), + password: SecretString::from("secret"), + }, + database: Some("app".to_owned()), + tls: TlsConfig { + server_name: server_name.map(str::to_owned), + ..TlsConfig::verify_full() + }, + }; + MysqlEndpoint::new(&target, LivenessPolicy::DESKTOP).unwrap() + } + + #[test] + fn a_direct_route_verifies_the_host_it_dials() { + let direct = mysql_endpoint( + Route { + service: endpoint("db.prod"), + kind: TransportKind::DirectTcp, + route: TransportRoute::Tcp(endpoint("db.prod")), + }, + None, + ); + + let options = direct.options_for(SessionKind::Editor).unwrap(); + + assert_eq!(options.get_host(), "db.prod"); + assert_eq!(options.get_socket(), None); + } + + /// sqlx would check the certificate against 127.0.0.1, not the name + /// the user gave, so the login is refused before it is attempted. + #[test] + fn a_tcp_route_to_another_host_than_the_name_is_refused() { + let forwarded = |server_name| { + mysql_endpoint( + Route { + service: endpoint("db.prod"), + kind: TransportKind::SshForward { + destination: "bastion".to_owned(), + }, + route: TransportRoute::Tcp(endpoint("127.0.0.1")), + }, + server_name, + ) + }; + + for server_name in [None, Some("db.internal")] { + assert!( + matches!( + forwarded(server_name).options_for(SessionKind::Workspace), + Err(DriverError::Config(ConfigError::ServerNameOverrideUnsupported { + route: TransportClass::SshForward + })) + ), + "{server_name:?}" + ); + } + } + + #[test] + fn a_socket_route_carries_the_name_tls_verifies() { + let socket = mysql_endpoint( + Route { + service: endpoint("db.prod"), + kind: TransportKind::SshForward { + destination: "bastion".to_owned(), + }, + route: TransportRoute::UnixSocket("/run/user/1000/forward.sock".into()), + }, + Some("db.internal"), + ); + + let options = socket.options_for(SessionKind::Control).unwrap(); + + assert_eq!(options.get_host(), "db.internal"); + assert_eq!( + options.get_socket().map(|path| path.as_path()), + Some(std::path::Path::new("/run/user/1000/forward.sock")) + ); + } +} diff --git a/linux/crates/drivers/mysql/src/write_batch.rs b/linux/crates/drivers/mysql/src/write_batch.rs index ff7cbc4e48..1a249cb9fd 100644 --- a/linux/crates/drivers/mysql/src/write_batch.rs +++ b/linux/crates/drivers/mysql/src/write_batch.rs @@ -17,7 +17,7 @@ use tablepro_core::{ use crate::cancel::{MySqlStop, acknowledges_stop, interrupted}; use crate::decode::decode_row; -use crate::errors::{INCOMPLETE_ROLLBACK, map_error, reply_lost}; +use crate::errors::{INCOMPLETE_ROLLBACK, UNSUPPORTED_PS, map_error, reply_lost, server_number}; use crate::flavour::MySqlFlavour; use crate::mysql_call::Tainter; use crate::mysql_timestamp_mode::MySqlTimestampMode; @@ -261,21 +261,22 @@ async fn schema_changes(context: &mut WriteContext<'_>, plan: &WritePlan) -> Res outcome } -/// One step. A statement with parameters is prepared; one without goes -/// over the text protocol, which takes statements MySQL cannot prepare, -/// such as CREATE TRIGGER. +/// One step, prepared even without parameters: a prepare takes exactly +/// one statement, so a stray `;` in generated SQL fails to parse instead +/// of running a second one. Only a statement the server parsed whole +/// and cannot prepare, such as CREATE TRIGGER on MySQL, goes over the +/// text protocol, which runs every statement it is given. async fn execute(context: &mut WriteContext<'_>, statement: &Statement) -> Result { - let done = match statement.params().is_empty() { - true => { - context - .connection - .execute(sqlx::raw_sql(AssertSqlSafe(statement.sql().to_owned()))) - .await - } - false => prepared(statement)?.execute(&mut *context.connection).await, + let outcome = prepared(statement)?.execute(&mut *context.connection).await; + let done = match outcome.map_err(|error| map_error(&error, context.flavour)) { + Err(error) if statement.params().is_empty() && server_number(&error) == Some(UNSUPPORTED_PS) => context + .connection + .execute(sqlx::raw_sql(AssertSqlSafe(statement.sql().to_owned()))) + .await + .map_err(|error| map_error(&error, context.flavour))?, + other => other?, }; - done.map(|result| result.rows_affected()) - .map_err(|error| map_error(&error, context.flavour)) + Ok(done.rows_affected()) } async fn run(context: &mut WriteContext<'_>, sql: &str) -> Result<(), DriverError> { diff --git a/linux/crates/drivers/mysql/tests/integration/editor_session.rs b/linux/crates/drivers/mysql/tests/integration/editor_session.rs index cb3652e421..ecb850891d 100644 --- a/linux/crates/drivers/mysql/tests/integration/editor_session.rs +++ b/linux/crates/drivers/mysql/tests/integration/editor_session.rs @@ -1,10 +1,25 @@ //! Where an editor session stands with a transaction, asked of the -//! server: MariaDB's @@in_transaction, MySQL's performance_schema. +//! server: MariaDB's @@in_transaction, MySQL's performance_schema. And +//! what a run does when its reader goes away. -use tablepro_core::{CallOptions, EditorSession, TransactionState}; +use tablepro_core::{ + CallOptions, EditorSession, EditorSessionState, RowLimit, RunEnd, RunOptions, TransactionState, UserSql, +}; use tablepro_test_fixtures::MysqlFlavour; +use tokio_util::sync::CancellationToken; -use crate::support::{CALL, connect, connect_root, run_ok, session, start, within}; +use crate::support::{ + CALL, EventSink, Events, FLAVOURS, connect, connect_root, count, run_ok, session, setup, start, within, +}; + +/// A hundred thousand rows of a kilobyte each: far more than the socket +/// buffers hold, so the server is still sending the SELECT when the +/// reader leaves. The DELETE behind it must never run. +const WIDE_THEN_DELETE: &str = "WITH d AS (SELECT 0 AS n UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 \ + UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 \ + UNION ALL SELECT 9) \ + SELECT REPEAT('x', 1000) AS filler FROM d a, d b, d c, d e, d f; \ + DELETE FROM t"; async fn state_after(session: &dyn EditorSession, sql: &str) -> TransactionState { let (summary, _) = crate::support::run(session, sql).await; @@ -73,3 +88,43 @@ async fn mysql_user_without_ps_select_unknown() { editor.close().await; connection.close().await.expect("a close"); } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn consumer_gone_kills_the_rest_of_the_batch() { + for flavour in FLAVOURS { + let fixture = start(flavour).await; + let connection = connect(&fixture, false).await; + setup( + connection.as_ref(), + &["CREATE TABLE t (id int PRIMARY KEY)", "INSERT INTO t VALUES (1)"], + ) + .await; + let editor = session(connection.as_ref()).await; + let events = Events::default(); + let options = + RunOptions::new(CancellationToken::new()).with_row_limit(RowLimit::new(1_000_000).expect("a limit")); + + let summary = editor + .run( + UserSql::new(WIDE_THEN_DELETE), + options, + Box::new(EventSink::new(&events).gone_after(1)), + ) + .await + .expect("the run"); + + assert_eq!(summary.end, RunEnd::Cancelled, "{flavour:?}"); + assert_eq!(editor.state(), EditorSessionState::Ready, "{flavour:?}"); + // The next run reads whatever the batch left on the wire first, + // so by now a DELETE that was still coming has run. + run_ok(editor.as_ref(), "SELECT 1").await; + assert_eq!( + count(connection.as_ref(), "SELECT COUNT(*) FROM t").await, + 1, + "{flavour:?}" + ); + editor.close().await; + connection.close().await.expect("a close"); + } +} diff --git a/linux/crates/drivers/mysql/tests/integration/main.rs b/linux/crates/drivers/mysql/tests/integration/main.rs index 35ba8817c5..3db3ab525d 100644 --- a/linux/crates/drivers/mysql/tests/integration/main.rs +++ b/linux/crates/drivers/mysql/tests/integration/main.rs @@ -15,3 +15,4 @@ mod non_transactional; mod row_limit; mod support; mod tls; +mod write_batch; diff --git a/linux/crates/drivers/mysql/tests/integration/support.rs b/linux/crates/drivers/mysql/tests/integration/support.rs index edf1d3333d..83a11ad6f8 100644 --- a/linux/crates/drivers/mysql/tests/integration/support.rs +++ b/linux/crates/drivers/mysql/tests/integration/support.rs @@ -258,19 +258,38 @@ impl Events { pub struct EventSink { events: Events, + gone_after_rows: Option, + seen_rows: usize, } impl EventSink { pub fn new(events: &Events) -> Self { - Self { events: events.clone() } + Self { + events: events.clone(), + gone_after_rows: None, + seen_rows: 0, + } + } + + /// Stop reading once this many rows have arrived, as a closed tab + /// does. + pub fn gone_after(mut self, rows: usize) -> Self { + self.gone_after_rows = Some(rows); + self } } #[async_trait] impl ResultSink for EventSink { async fn accept(&mut self, event: ResultEvent) -> SinkFlow { + if let ResultEvent::Rows(rows) = &event { + self.seen_rows += rows.len(); + } self.events.0.lock().unwrap_or_else(PoisonError::into_inner).push(event); - SinkFlow::Continue + match self.gone_after_rows { + Some(limit) if self.seen_rows >= limit => SinkFlow::ConsumerGone, + _ => SinkFlow::Continue, + } } } diff --git a/linux/crates/drivers/mysql/tests/integration/write_batch.rs b/linux/crates/drivers/mysql/tests/integration/write_batch.rs new file mode 100644 index 0000000000..ffac594d96 --- /dev/null +++ b/linux/crates/drivers/mysql/tests/integration/write_batch.rs @@ -0,0 +1,49 @@ +//! A save sends each statement on its own, so a stray `;` in one can +//! never run a second statement behind it. + +use tablepro_core::WriteKind; + +use crate::support::{FLAVOURS, apply, connect, count, setup, start}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn a_statement_after_a_semicolon_never_runs() { + for flavour in FLAVOURS { + let fixture = start(flavour).await; + let connection = connect(&fixture, false).await; + setup( + connection.as_ref(), + &["CREATE TABLE t (id int PRIMARY KEY)", "INSERT INTO t VALUES (1)"], + ) + .await; + + let smuggled = [ + (WriteKind::RowChanges, "INSERT INTO t VALUES (2); DELETE FROM t"), + (WriteKind::SchemaChanges, "CREATE TABLE u (id int); DROP TABLE t"), + // MySQL cannot prepare CREATE TRIGGER, but this text is two + // statements, so it never reaches the text protocol. + ( + WriteKind::SchemaChanges, + "CREATE TRIGGER t_same BEFORE INSERT ON t FOR EACH ROW SET NEW.id = NEW.id; DROP TABLE t", + ), + ]; + for (kind, sql) in smuggled { + let outcome = apply(connection.as_ref(), kind, &[sql]).await; + assert!(outcome.is_err(), "{flavour:?} `{sql}`: {outcome:?}"); + } + apply( + connection.as_ref(), + WriteKind::SchemaChanges, + &["CREATE TRIGGER t_same BEFORE INSERT ON t FOR EACH ROW SET NEW.id = NEW.id"], + ) + .await + .expect("one trigger, sent as one statement"); + + assert_eq!( + count(connection.as_ref(), "SELECT COUNT(*) FROM t").await, + 1, + "{flavour:?}" + ); + connection.close().await.expect("a close"); + } +} diff --git a/linux/crates/drivers/postgres/src/cancel.rs b/linux/crates/drivers/postgres/src/cancel.rs index bfc16afca5..df2d70ffc7 100644 --- a/linux/crates/drivers/postgres/src/cancel.rs +++ b/linux/crates/drivers/postgres/src/cancel.rs @@ -6,7 +6,7 @@ use tablepro_core::{ }; use tokio_postgres::CancelToken; -use crate::errors::sqlstate; +use crate::errors::{QUERY_CANCELED, sqlstate}; use crate::physical::PgEndpoint; /// How far a call's cancel request got. @@ -98,7 +98,7 @@ impl EngineStop for PgStop { /// Whether the server answered a statement with 57014, which is how it /// reports a cancel request or a statement timeout taking effect. pub(crate) fn acknowledges_stop(error: &DriverError) -> bool { - sqlstate(error) == Some("57014") + sqlstate(error) == Some(QUERY_CANCELED) } fn lock(mutex: &Mutex) -> MutexGuard<'_, T> { diff --git a/linux/crates/drivers/postgres/src/connection.rs b/linux/crates/drivers/postgres/src/connection.rs index 08e002dfc8..8f1486f338 100644 --- a/linux/crates/drivers/postgres/src/connection.rs +++ b/linux/crates/drivers/postgres/src/connection.rs @@ -29,7 +29,7 @@ use crate::physical::PgEndpoint; use crate::pool::{probe_timeouts, workspace_timeouts}; use crate::standalone::Standalone; use crate::stream_read::stream; -use crate::workspace_call::{Access, ReadOp, count_op, flight, preamble, query_bound_op, read_op, statement_bound}; +use crate::workspace_call::{Access, Preamble, ReadOp, count_op, flight, query_bound_op, read_op, statement_bound}; use crate::write_batch::{WritePlan, perform, precheck}; /// A workspace's link to one PostgreSQL server: a pool of four @@ -114,8 +114,8 @@ impl PgConnection { self.settle(outcome, &stop).await } - fn read_preamble(&self, options: &CallOptions) -> String { - preamble( + fn read_preamble(&self, options: &CallOptions) -> Preamble { + Preamble::new( Access::ReadOnly, Some(statement_bound(&self.settings, options.deadline())), self.settings.lock_timeout, @@ -362,7 +362,7 @@ impl Connection for PgConnection { return (link, Err(error)); } let result = match link.client() { - Ok(client) => stream(client, &read, sink.as_mut()).await, + Ok(client) => stream(client, link.standard_conforming_strings(), &read, sink.as_mut()).await, Err(error) => Err(error), }; (link, result) diff --git a/linux/crates/drivers/postgres/src/editor/run.rs b/linux/crates/drivers/postgres/src/editor/run.rs index 5a7c71a95e..86c1bd2050 100644 --- a/linux/crates/drivers/postgres/src/editor/run.rs +++ b/linux/crates/drivers/postgres/src/editor/run.rs @@ -5,8 +5,7 @@ use futures_util::StreamExt; use tablepro_core::sql_syntax::SqlGrammar; use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan, ScriptToken}; use tablepro_core::{ - DriverError, ReadOnlyRefusal, ResultEvent, ResultSink, RowLimit, RunEnd, ServerCode, ServerNotice, SinkFlow, - Truncation, Value, + DriverError, ResultEvent, ResultSink, RowLimit, RunEnd, ServerCode, ServerNotice, SinkFlow, Truncation, Value, }; use tokio_postgres::SimpleQueryMessage; @@ -20,7 +19,7 @@ use super::stop_watch::{Interrupt, StopWatch}; use super::transaction_tracker::{KnownState, Next, PROBE_RELEASE, PROBE_SAVEPOINT, probe_answer}; use crate::cancel::acknowledges_stop; use crate::catalog::text_column; -use crate::errors::{map_error, sqlstate}; +use crate::errors::{map_error, read_only_refusal, sqlstate}; /// Rows per `ResultEvent::Rows`. The sink is awaited per batch, so a slow /// grid slows the read instead of growing a queue. @@ -403,18 +402,6 @@ fn end_after_stop(interrupt: Interrupt, error: &DriverError) -> RunEnd { } } -/// Inside the read-only wrapper, a statement that cannot run in a -/// transaction block at all (VACUUM, CREATE DATABASE) fails with 25001, -/// which here means the read-only rule refused it. -fn read_only_refusal(error: DriverError) -> DriverError { - match error { - DriverError::Server(diagnostics) if diagnostics.sqlstate() == Some("25001") => { - DriverError::ReadOnly(ReadOnlyRefusal::ServerRejected(diagnostics)) - } - other => other, - } -} - impl SessionCore { fn standard_conforming_strings(&self) -> bool { self.conforming.load(Ordering::SeqCst) @@ -423,7 +410,7 @@ impl SessionCore { #[cfg(test)] mod tests { - use tablepro_core::{ServerDiagnostics, TimeoutPhase}; + use tablepro_core::TimeoutPhase; use super::*; @@ -450,17 +437,4 @@ mod tests { } ); } - - #[test] - fn vacuum_in_the_read_only_wrapper_is_a_read_only_refusal() { - let vacuum = DriverError::reported(ServerDiagnostics::new( - Some(ServerCode::SqlState("25001".to_owned())), - "VACUUM cannot run inside a transaction block", - )); - - assert!(matches!( - read_only_refusal(vacuum), - DriverError::ReadOnly(ReadOnlyRefusal::ServerRejected(_)) - )); - } } diff --git a/linux/crates/drivers/postgres/src/errors.rs b/linux/crates/drivers/postgres/src/errors.rs index a1075eea13..d014a168a7 100644 --- a/linux/crates/drivers/postgres/src/errors.rs +++ b/linux/crates/drivers/postgres/src/errors.rs @@ -12,6 +12,9 @@ use tokio_postgres::error::{DbError, ErrorPosition}; const AUTHENTICATION_KIND: &str = "authentication error"; const TLS_KIND: &str = "error performing TLS handshake"; +/// query_canceled: a cancel request or a statement timeout took effect. +pub(crate) const QUERY_CANCELED: &str = "57014"; + /// What a tokio-postgres error means to the app. pub(crate) fn map_error(error: &tokio_postgres::Error) -> DriverError { if let Some(carried) = carried(error) { @@ -71,12 +74,6 @@ pub(crate) fn map_diagnostics(diagnostics: ServerDiagnostics) -> DriverError { match diagnostics.sqlstate().unwrap_or_default() { "28P01" | "28000" => DriverError::auth(Some(diagnostics)), "25006" => DriverError::ReadOnly(ReadOnlyRefusal::server(diagnostics)), - // The SQLSTATE of a statement_timeout is the one a cancel request - // raises too, and only the message tells them apart. - "57014" if diagnostics.message.contains("statement timeout") => DriverError::Timeout { - phase: TimeoutPhase::Statement, - server_cancelled: true, - }, "55P03" => DriverError::Timeout { phase: TimeoutPhase::LockWait, server_cancelled: true, @@ -119,7 +116,7 @@ pub(crate) fn sqlstate(error: &DriverError) -> Option<&str> { DriverError::Timeout { phase: TimeoutPhase::Statement, server_cancelled: true, - } => Some("57014"), + } => Some(QUERY_CANCELED), DriverError::Timeout { phase: TimeoutPhase::LockWait, .. @@ -128,6 +125,19 @@ pub(crate) fn sqlstate(error: &DriverError) -> Option<&str> { } } +/// Inside a read-only transaction the driver opened, a statement that +/// cannot run in a transaction block at all (VACUUM, CREATE DATABASE), +/// or one that tries to make it writable, fails with 25001, which there +/// means the read-only rule refused it. +pub(crate) fn read_only_refusal(error: DriverError) -> DriverError { + match error { + DriverError::Server(diagnostics) if diagnostics.sqlstate() == Some("25001") => { + DriverError::ReadOnly(ReadOnlyRefusal::ServerRejected(diagnostics)) + } + other => other, + } +} + /// What a pool refusal means. A wait that runs out means every slot is /// busy, which is the server having no room rather than a dead link. pub(crate) fn map_pool_error(error: &PoolError) -> DriverError { @@ -194,15 +204,13 @@ mod tests { ("25006", "cannot execute INSERT in a read-only transaction", |e| { matches!(e, DriverError::ReadOnly(ReadOnlyRefusal::ServerRejected(_))) }), - ("57014", "canceling statement due to statement timeout", |e| { - matches!( - e, - DriverError::Timeout { - phase: TimeoutPhase::Statement, - server_cancelled: true - } - ) - }), + // Only the call that set the timeout can tell it from a + // cancel, so here both stay the server's own answer. + ( + "57014", + "canceling statement due to statement timeout", + |e| matches!(e, DriverError::Server(d) if d.sqlstate() == Some("57014")), + ), ( "57014", "canceling statement due to user request", @@ -243,6 +251,25 @@ mod tests { assert_eq!(sqlstate(&DriverError::Busy), None); } + #[test] + fn a_25001_in_a_read_only_transaction_is_a_read_only_refusal() { + let refused = [ + "VACUUM cannot run inside a transaction block", + "transaction read-write mode must be set before any query", + ]; + for message in refused { + assert!( + matches!( + read_only_refusal(map_diagnostics(diagnostics("25001", message))), + DriverError::ReadOnly(ReadOnlyRefusal::ServerRejected(_)) + ), + "{message}" + ); + } + let other = map_diagnostics(diagnostics("42P01", "relation does not exist")); + assert_eq!(read_only_refusal(other.clone()), other); + } + #[test] fn a_pool_wait_that_runs_out_is_busy() { assert_eq!( diff --git a/linux/crates/drivers/postgres/src/metadata/mod.rs b/linux/crates/drivers/postgres/src/metadata/mod.rs index 07b845a6d9..e8386e44ce 100644 --- a/linux/crates/drivers/postgres/src/metadata/mod.rs +++ b/linux/crates/drivers/postgres/src/metadata/mod.rs @@ -69,9 +69,17 @@ pub(crate) async fn fetch_table_definition( table: &TableRef, relation: &str, ) -> Result { - let columns = column_rows(client, relation).await?; - let primary_key = text_rows(client, PRIMARY_KEY_NAME_SQL, &[relation]) - .await? + // Every query is sent before the first answer is awaited, so all of + // them run inside the flight's transaction, ahead of its COMMIT. + let params = [relation]; + let (columns, primary_key, indexes, foreign_keys, inbound_foreign_keys) = futures_util::try_join!( + column_rows(client, relation), + text_rows(client, PRIMARY_KEY_NAME_SQL, ¶ms), + fetch_indexes(client, relation), + fetch_foreign_keys(client, relation), + text_rows(client, INBOUND_FOREIGN_KEYS_SQL, ¶ms), + )?; + let primary_key = primary_key .first() .map(|row| -> Result { let field = decode_error("pg_constraint"); @@ -81,10 +89,7 @@ pub(crate) async fn fetch_table_definition( }) }) .transpose()?; - let indexes = fetch_indexes(client, relation).await?; - let foreign_keys = fetch_foreign_keys(client, relation).await?; - let inbound_foreign_keys = text_rows(client, INBOUND_FOREIGN_KEYS_SQL, &[relation]) - .await? + let inbound_foreign_keys = inbound_foreign_keys .iter() .map(inbound_foreign_key) .collect::>()?; diff --git a/linux/crates/drivers/postgres/src/standalone.rs b/linux/crates/drivers/postgres/src/standalone.rs index 01efef7935..5bf59de050 100644 --- a/linux/crates/drivers/postgres/src/standalone.rs +++ b/linux/crates/drivers/postgres/src/standalone.rs @@ -17,6 +17,7 @@ pub(crate) struct Standalone { client: Option, driving: Option>, abandon: Arc, + conforming: bool, } impl Standalone { @@ -25,6 +26,7 @@ impl Standalone { client: None, driving: None, abandon: Arc::new(AtomicBool::new(false)), + conforming: true, } } @@ -44,6 +46,7 @@ impl Standalone { opened = endpoint.open(&never) => opened?, }; let (client, connection) = opened; + self.conforming = connection.parameter("standard_conforming_strings") != Some("off"); stop.arm(client.cancel_token()); self.driving = Some(tracker.spawn(async move { if let Err(error) = connection.await { @@ -57,6 +60,12 @@ impl Standalone { Ok(()) } + /// The session's `standard_conforming_strings` as the login reported + /// it. + pub(crate) fn standard_conforming_strings(&self) -> bool { + self.conforming + } + pub(crate) fn client(&self) -> Result<&Client, DriverError> { self.client .as_ref() diff --git a/linux/crates/drivers/postgres/src/stream_read.rs b/linux/crates/drivers/postgres/src/stream_read.rs index db4d2749f9..714d882ba8 100644 --- a/linux/crates/drivers/postgres/src/stream_read.rs +++ b/linux/crates/drivers/postgres/src/stream_read.rs @@ -13,26 +13,31 @@ use tokio_postgres::{Client, SimpleQueryMessage}; use crate::catalog::{result_column, text_column}; use crate::codec::typed; use crate::editor::statement_class::{StatementClass, classify}; -use crate::errors::map_error; +use crate::errors::{map_error, read_only_refusal}; use crate::workspace_call::{decode_row, describe}; const BATCH_ROWS: usize = 256; +/// The read's transaction. Its SELECT takes the snapshot, after which +/// the server refuses SET TRANSACTION READ WRITE and any other way of +/// making the transaction writable, so user SQL cannot write in it. +const BEGIN: &str = "START TRANSACTION READ ONLY; SET LOCAL statement_timeout = 0; SELECT 1"; + /// Every row of a read, into the sink, inside a read-only transaction /// with no server timeout: an export can take minutes, and the caller's -/// deadline and cancel still stop it. +/// deadline and cancel still stop it. `conforming` is the session's +/// `standard_conforming_strings`, which decides how the server splits +/// the user's SQL. pub(crate) async fn stream( client: &Client, + conforming: bool, read: &ReadStatement, sink: &mut dyn ResultSink, ) -> Result { if let ReadStatement::User(sql) = read { - refuse_transaction_control(sql.as_str())?; + refuse_transaction_control(sql.as_str(), conforming)?; } - client - .batch_execute("START TRANSACTION READ ONLY; SET LOCAL statement_timeout = 0") - .await - .map_err(|error| map_error(&error))?; + client.batch_execute(BEGIN).await.map_err(|error| map_error(&error))?; let mut emitter = Emitter { sink, batch: Vec::new(), @@ -56,20 +61,21 @@ pub(crate) async fn stream( } }; let committed = client.batch_execute("COMMIT").await.map_err(|error| map_error(&error)); - streamed?; + streamed.map_err(read_only_refusal)?; committed?; Ok(StreamSummary::new(emitter.rows)) } /// The read-only transaction is the export's whole guarantee, so user /// SQL that would end it, or a routine that may, is refused before -/// anything is sent. -fn refuse_transaction_control(sql: &str) -> Result<(), DriverError> { +/// anything is sent. The server reads the whole text before it runs any +/// of it, so the split is the one the session's setting gives. +fn refuse_transaction_control(sql: &str, conforming: bool) -> Result<(), DriverError> { let plan = ScriptPlan::build( sql, SqlGrammar::PostgreSql, LexicalSettings { - backslash_escapes: false, + backslash_escapes: !conforming, }, ); for statement in plan.statements() { @@ -185,17 +191,28 @@ impl Emitter<'_> { mod tests { use super::*; + fn guarded(sql: &str, conforming: bool) -> bool { + matches!( + refuse_transaction_control(sql, conforming), + Err(DriverError::ReadOnly(ReadOnlyRefusal::ClientGuard)) + ) + } + #[test] fn user_sql_that_would_end_the_read_only_transaction_is_refused() { - assert!(refuse_transaction_control("SELECT 1; SELECT 2").is_ok()); + assert!(refuse_transaction_control("SELECT 1; SELECT 2", true).is_ok()); for sql in ["COMMIT; DELETE FROM t", "SELECT 1; ROLLBACK", "CALL p()", "BEGIN"] { - assert!( - matches!( - refuse_transaction_control(sql), - Err(DriverError::ReadOnly(ReadOnlyRefusal::ClientGuard)) - ), - "{sql}" - ); + assert!(guarded(sql, true), "{sql}"); } } + + /// With `standard_conforming_strings` off the server reads `\'` as an + /// escaped quote, so the COMMIT here is a statement of its own. + #[test] + fn the_guard_splits_the_way_a_non_conforming_server_does() { + let sql = "SELECT 'a\\', '; COMMIT; DELETE FROM t; SELECT '1'"; + + assert!(guarded(sql, false), "{sql}"); + assert!(!guarded(sql, true), "{sql}"); + } } diff --git a/linux/crates/drivers/postgres/src/workspace_call.rs b/linux/crates/drivers/postgres/src/workspace_call.rs index e7d179d681..274266f90f 100644 --- a/linux/crates/drivers/postgres/src/workspace_call.rs +++ b/linux/crates/drivers/postgres/src/workspace_call.rs @@ -5,13 +5,13 @@ use std::time::Duration; use futures_util::{StreamExt, TryStreamExt}; use tablepro_core::column::ResultColumn; use tablepro_core::statement::Statement; -use tablepro_core::{DriverError, QueryResult, RowLimit, Value, WorkspaceSettings}; +use tablepro_core::{DriverError, QueryResult, RowLimit, TimeoutPhase, Value, WorkspaceSettings}; use tokio::time::Instant; use tokio_postgres::Client; use crate::catalog::result_column; use crate::codec::{RawCell, typed}; -use crate::errors::map_error; +use crate::errors::{QUERY_CANCELED, map_error}; use crate::pg_call::PgCall; pub(crate) type BoxFuture<'a, T> = Pin + Send + 'a>>; @@ -40,16 +40,52 @@ pub(crate) enum Access { /// /// `statement_timeout` of `None` sends 0, the server's "no limit". A /// bound is never rendered as 0 otherwise, since that would lift it. -pub(crate) fn preamble(access: Access, statement_timeout: Option, lock_timeout: Duration) -> String { - let mode = match access { - Access::ReadOnly => "READ ONLY", - Access::ReadWrite => "READ WRITE", - }; - let statement = statement_timeout.map_or(0, millis); - format!( - "START TRANSACTION {mode}; SET LOCAL statement_timeout = {statement}; SET LOCAL lock_timeout = {}", - millis(lock_timeout) - ) +#[derive(Debug, Clone)] +pub(crate) struct Preamble { + sql: String, + statement_timeout_ms: Option, +} + +impl Preamble { + pub(crate) fn new(access: Access, statement_timeout: Option, lock_timeout: Duration) -> Self { + let mode = match access { + Access::ReadOnly => "READ ONLY", + Access::ReadWrite => "READ WRITE", + }; + let statement_timeout_ms = statement_timeout.map(millis); + let sql = format!( + "START TRANSACTION {mode}; SET LOCAL statement_timeout = {}; SET LOCAL lock_timeout = {}", + statement_timeout_ms.unwrap_or(0), + millis(lock_timeout) + ); + Self { + sql, + statement_timeout_ms, + } + } + + pub(crate) fn sql(&self) -> &str { + &self.sql + } + + /// A 57014 no stop asked for is this transaction's statement timeout + /// or someone else's cancel, and only the message, which + /// `lc_messages` translates, tells them apart. The server never fires + /// the timeout early, so one that came after the bound is the timeout. + pub(crate) fn explain(&self, error: DriverError, started: Instant) -> DriverError { + let ran_out = self + .statement_timeout_ms + .is_some_and(|bound| started.elapsed().as_millis() >= bound); + match error { + DriverError::Server(diagnostics) if ran_out && diagnostics.sqlstate() == Some(QUERY_CANCELED) => { + DriverError::Timeout { + phase: TimeoutPhase::Statement, + server_cancelled: true, + } + } + other => other, + } + } } /// Whole milliseconds, at least one: a sub-millisecond remainder still @@ -72,10 +108,15 @@ pub(crate) fn statement_bound(settings: &WorkspaceSettings, deadline: Option(call: &PgCall, preamble: &str, op: ReadOp) -> Result { +/// +/// The COMMIT is queued as soon as the work first waits for an answer, +/// so the work must send every query before that: one sent later lands +/// after the COMMIT, outside the transaction and its bounds. +pub(crate) async fn flight(call: &PgCall, preamble: &Preamble, op: ReadOp) -> Result { let client = call.client()?; + let started = Instant::now(); let (begun, result, committed) = futures_util::join!( - client.batch_execute(preamble), + client.batch_execute(preamble.sql()), op(client), client.batch_execute("COMMIT") ); @@ -84,7 +125,7 @@ pub(crate) async fn flight(call: &PgCall, preamble: &str, op: ReadOp) -> R } match (begun, result, committed) { (Err(error), _, _) => Err(map_error(&error)), - (Ok(()), Err(error), _) => Err(error), + (Ok(()), Err(error), _) => Err(preamble.explain(error, started)), (Ok(()), Ok(_), Err(error)) => Err(map_error(&error)), (Ok(()), Ok(value), Ok(())) => Ok(value), } @@ -92,7 +133,7 @@ pub(crate) async fn flight(call: &PgCall, preamble: &str, op: ReadOp) -> R /// Read at most `limit` rows of a generated statement, and one more to /// learn whether the result was cut short. -pub(crate) fn query_bound_op(statement: Statement, limit: RowLimit, begin: String) -> ReadOp { +pub(crate) fn query_bound_op(statement: Statement, limit: RowLimit, begin: Preamble) -> ReadOp { read_op(move |client| { Box::pin(async move { let stream = client @@ -132,9 +173,9 @@ pub(crate) fn query_bound_op(statement: Statement, limit: RowLimit, begin: Strin /// row of NULLs, typed as its columns, and the planner skips the /// subquery itself. The outer flight's COMMIT has already gone out, so /// this one is its own read-only transaction. -async fn describe_in_own_flight(client: &Client, statement: &Statement, begin: &str) -> Vec { +async fn describe_in_own_flight(client: &Client, statement: &Statement, begin: &Preamble) -> Vec { let (_, described, _) = futures_util::join!( - client.batch_execute(begin), + client.batch_execute(begin.sql()), describe(client, statement), client.batch_execute("COMMIT") ); @@ -192,11 +233,31 @@ pub(crate) fn decode_row(row: &tokio_postgres::Row) -> Result, Driver #[cfg(test)] mod tests { + use tablepro_core::{ServerCode, ServerDiagnostics}; + use super::*; + fn sql(access: Access, statement_timeout: Option, lock_timeout: Duration) -> String { + Preamble::new(access, statement_timeout, lock_timeout).sql().to_owned() + } + + fn cancelled(message: &str) -> DriverError { + DriverError::reported(ServerDiagnostics::new( + Some(ServerCode::SqlState(QUERY_CANCELED.to_owned())), + message, + )) + } + + fn timed_out() -> DriverError { + DriverError::Timeout { + phase: TimeoutPhase::Statement, + server_cancelled: true, + } + } + #[test] fn preamble_uses_integer_milliseconds() { - let preamble = preamble( + let preamble = sql( Access::ReadOnly, Some(Duration::from_millis(1_500)), Duration::from_secs(10), @@ -210,7 +271,7 @@ mod tests { #[test] fn a_sub_millisecond_bound_never_renders_as_zero() { - let preamble = preamble(Access::ReadWrite, Some(Duration::from_micros(300)), Duration::ZERO); + let preamble = sql(Access::ReadWrite, Some(Duration::from_micros(300)), Duration::ZERO); assert!(preamble.contains("statement_timeout = 1;"), "{preamble}"); assert!(preamble.ends_with("lock_timeout = 1"), "{preamble}"); @@ -219,7 +280,7 @@ mod tests { #[test] fn an_unbounded_schema_change_sends_the_servers_no_limit() { - assert!(preamble(Access::ReadWrite, None, Duration::from_secs(10)).contains("statement_timeout = 0;")); + assert!(sql(Access::ReadWrite, None, Duration::from_secs(10)).contains("statement_timeout = 0;")); } #[tokio::test(start_paused = true)] @@ -230,4 +291,51 @@ mod tests { let near = Instant::now() + Duration::from_secs(2); assert_eq!(statement_bound(&settings, Some(near)), Duration::from_secs(2)); } + + /// The message is the server's, in whatever language `lc_messages` + /// picked, so only the time tells the timeout apart. + #[tokio::test(start_paused = true)] + async fn a_57014_once_the_bound_ran_out_is_the_timeout_in_any_language() { + let preamble = Preamble::new( + Access::ReadOnly, + Some(Duration::from_millis(1_500)), + Duration::from_secs(10), + ); + let started = Instant::now(); + tokio::time::advance(Duration::from_millis(1_500)).await; + + for message in [ + "canceling statement due to statement timeout", + "Anweisung wird abgebrochen wegen Zeitüberschreitung", + "annulation de la requête à cause du délai écoulé pour l'exécution de l'instruction", + ] { + assert_eq!(preamble.explain(cancelled(message), started), timed_out(), "{message}"); + } + } + + #[tokio::test(start_paused = true)] + async fn a_57014_before_the_bound_or_without_one_is_the_servers_answer() { + let bounded = Preamble::new( + Access::ReadOnly, + Some(Duration::from_millis(1_500)), + Duration::from_secs(10), + ); + let unbounded = Preamble::new(Access::ReadWrite, None, Duration::from_secs(10)); + let started = Instant::now(); + tokio::time::advance(Duration::from_millis(1_499)).await; + + assert_eq!( + bounded.explain(cancelled("canceling statement due to statement timeout"), started), + cancelled("canceling statement due to statement timeout") + ); + tokio::time::advance(Duration::from_secs(60)).await; + assert_eq!( + unbounded.explain(cancelled("canceling statement due to user request"), started), + cancelled("canceling statement due to user request") + ); + assert_eq!( + bounded.explain(DriverError::server("relation does not exist"), started), + DriverError::server("relation does not exist") + ); + } } diff --git a/linux/crates/drivers/postgres/src/write_batch.rs b/linux/crates/drivers/postgres/src/write_batch.rs index dab114b87d..591c7f07d8 100644 --- a/linux/crates/drivers/postgres/src/write_batch.rs +++ b/linux/crates/drivers/postgres/src/write_batch.rs @@ -7,13 +7,14 @@ use tablepro_core::{ CommitToken, DriverError, ReadOnlyRefusal, SessionFate, StopRequest, WorkspaceSettings, WriteBatch, WriteKind, WriteLedger, WriteReport, }; +use tokio::time::Instant; use tokio_postgres::Client; use crate::cancel::{PgStop, acknowledges_stop}; use crate::codec::{RawCell, typed}; use crate::errors::map_error; use crate::pg_call::PgCall; -use crate::workspace_call::{Access, preamble}; +use crate::workspace_call::{Access, Preamble}; /// Refuse a batch before anything is sent: a read-only connection never /// writes, and PostgreSQL has no way to suspend foreign keys. @@ -49,8 +50,9 @@ pub(crate) async fn perform( WriteKind::RowChanges => Some(plan.settings.statement_timeout), WriteKind::SchemaChanges => None, }; - let begin = preamble(Access::ReadWrite, statement_timeout, plan.settings.lock_timeout); - if let Err(error) = client.batch_execute(&begin).await { + let begin = Preamble::new(Access::ReadWrite, statement_timeout, plan.settings.lock_timeout); + let started = Instant::now(); + if let Err(error) = client.batch_execute(begin.sql()).await { let error = map_error(&error); let fate = rollback(call, client).await; return Err(ledger.failed(0, error, fate)); @@ -68,7 +70,10 @@ pub(crate) async fn perform( let fate = rollback(call, client).await; return Err(ledger.guard_failed(index, expected, actual, fate)); } - Err(error) => return Err(step_failed(call, client, plan, index, error, request, stop).await), + Err(error) => { + let error = begin.explain(error, started); + return Err(step_failed(call, client, plan, index, error, request, stop).await); + } } } ledger.statement_sending(index); @@ -80,7 +85,7 @@ pub(crate) async fn perform( let affected = match client.execute_typed(step.statement.sql(), &borrowed).await { Ok(affected) => affected, Err(error) => { - let error = map_error(&error); + let error = begin.explain(map_error(&error), started); return Err(step_failed(call, client, plan, index, error, request, stop).await); } }; @@ -107,7 +112,7 @@ pub(crate) async fn perform( // constraint or a serialization failure rolled it all back. Err(error) if error.as_db_error().is_some() => Err(DriverError::RolledBack { statement_index: plan.batch.steps.len(), - source: Box::new(map_error(&error)), + source: Box::new(begin.explain(map_error(&error), started)), }), Err(error) => { call.taint(); diff --git a/linux/crates/drivers/postgres/tests/integration/flight.rs b/linux/crates/drivers/postgres/tests/integration/flight.rs new file mode 100644 index 0000000000..b0b3f06e33 --- /dev/null +++ b/linux/crates/drivers/postgres/tests/integration/flight.rs @@ -0,0 +1,192 @@ +//! A workspace read is one transaction on the wire: its preamble, every +//! query it makes, then its COMMIT. + +use std::io; +use std::pin::Pin; +use std::sync::{Arc, Mutex, PoisonError}; +use std::task::{Context, Poll}; + +use async_trait::async_trait; +use tablepro_core::meta::TableRef; +use tablepro_core::{ + NetworkEndpoint, TlsConfig, Transport, TransportError, TransportKind, TransportRoute, TransportStream, + WorkspaceSettings, +}; +use tablepro_test_support::conformance::test_liveness; +use tokio::io::{AsyncRead, AsyncWrite, ReadBuf}; + +use crate::support::{CALL, connect_with, direct, options_over, password, setup, start, within}; + +type Log = Arc>>; + +/// A direct route that keeps what the client writes on each stream. +#[derive(Debug)] +struct RecordingTransport { + inner: Arc, + streams: Mutex>, +} + +impl RecordingTransport { + fn written(&self) -> Vec> { + self.streams + .lock() + .unwrap_or_else(PoisonError::into_inner) + .iter() + .map(|log| log.lock().unwrap_or_else(PoisonError::into_inner).clone()) + .collect() + } +} + +#[async_trait] +impl Transport for RecordingTransport { + fn service_endpoint(&self) -> &NetworkEndpoint { + self.inner.service_endpoint() + } + + fn kind(&self) -> TransportKind { + self.inner.kind() + } + + fn route(&self) -> TransportRoute { + self.inner.route() + } + + async fn open(&self) -> Result, TransportError> { + let inner = self.inner.open().await?; + let log = Log::default(); + self.streams + .lock() + .unwrap_or_else(PoisonError::into_inner) + .push(log.clone()); + Ok(Box::new(Recording { inner, log })) + } + + async fn reroute(&self, endpoint: NetworkEndpoint) -> Result, TransportError> { + self.inner.reroute(endpoint).await + } + + async fn explain_closed_stream(&self) -> Option { + None + } +} + +struct Recording { + inner: Box, + log: Log, +} + +impl AsyncRead for Recording { + fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { + Pin::new(&mut self.inner).poll_read(cx, buf) + } +} + +impl AsyncWrite for Recording { + fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { + let written = Pin::new(&mut self.inner).poll_write(cx, buf); + if let Poll::Ready(Ok(count)) = &written + && let Some(bytes) = buf.get(..*count) + { + self.log + .lock() + .unwrap_or_else(PoisonError::into_inner) + .extend_from_slice(bytes); + } + written + } + + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.inner).poll_flush(cx) + } + + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.inner).poll_shutdown(cx) + } +} + +fn length(bytes: &[u8]) -> Option { + let prefix: [u8; 4] = bytes.get(..4)?.try_into().ok()?; + usize::try_from(u32::from_be_bytes(prefix)).ok() +} + +/// The frontend messages a client wrote, as type and body. The startup +/// packet has no type byte; every message after it has one. +fn messages(written: &[u8]) -> Vec<(u8, Vec)> { + let mut rest = length(written) + .and_then(|startup| written.get(startup..)) + .unwrap_or_default(); + let mut parsed = Vec::new(); + while let Some((&kind, tail)) = rest.split_first() { + let Some(len) = length(tail) else { break }; + let (Some(body), Some(next)) = (tail.get(4..len), tail.get(len..)) else { + break; + }; + parsed.push((kind, body.to_vec())); + rest = next; + } + parsed +} + +fn query_starting(message: &(u8, Vec), text: &str) -> bool { + message.0 == b'Q' && message.1.starts_with(text.as_bytes()) +} + +/// The Parse messages of the last read-only flight on a stream: those +/// sent before its COMMIT, and those sent after it. +fn last_read_flight(messages: &[(u8, Vec)]) -> Option<(usize, usize)> { + let start = messages + .iter() + .rposition(|message| query_starting(message, "START TRANSACTION READ ONLY"))?; + let flight = messages.get(start..)?; + let commit = flight.iter().position(|message| query_starting(message, "COMMIT"))?; + let parses = |part: &[(u8, Vec)]| part.iter().filter(|(kind, _)| *kind == b'P').count(); + Some(( + parses(flight.get(..commit).unwrap_or_default()), + parses(flight.get(commit..).unwrap_or_default()), + )) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn table_definition_queries_all_run_before_the_commit() { + let fixture = start().await; + let liveness = test_liveness(); + let transport = Arc::new(RecordingTransport { + inner: direct(fixture.host(), fixture.port(), &liveness), + streams: Mutex::new(Vec::new()), + }); + let connection = connect_with(options_over( + transport.clone(), + password(&fixture.password_credentials()), + fixture.database(), + TlsConfig::disabled(), + WorkspaceSettings::workspace(false), + liveness, + )) + .await; + setup( + connection.as_ref(), + &[ + "CREATE TABLE parent (id int PRIMARY KEY)", + "CREATE TABLE child (id int PRIMARY KEY, parent_id int REFERENCES parent (id))", + "CREATE INDEX child_parent ON child (parent_id)", + ], + ) + .await; + + connection + .fetch_table_definition(&TableRef::new(Some("public".to_owned()), "child"), within(CALL)) + .await + .expect("the definition"); + + let flights: Vec<(usize, usize)> = transport + .written() + .iter() + .filter_map(|written| last_read_flight(&messages(written))) + .collect(); + assert_eq!(flights.len(), 1, "{flights:?}"); + let (inside, after) = flights.first().copied().unwrap_or_default(); + assert!(inside > 1, "the definition read sent {inside} queries in its flight"); + assert_eq!(after, 0, "{after} queries went out after the COMMIT"); + connection.close().await.expect("a close"); +} diff --git a/linux/crates/drivers/postgres/tests/integration/main.rs b/linux/crates/drivers/postgres/tests/integration/main.rs index e4e3f54539..18c41628ce 100644 --- a/linux/crates/drivers/postgres/tests/integration/main.rs +++ b/linux/crates/drivers/postgres/tests/integration/main.rs @@ -11,6 +11,7 @@ mod conformance_engine; mod decode; mod editor_session; mod filter; +mod flight; mod liveness; mod metadata; mod pgbouncer; diff --git a/linux/crates/drivers/postgres/tests/integration/read_only.rs b/linux/crates/drivers/postgres/tests/integration/read_only.rs index 070a62a3b7..6fccc5e06c 100644 --- a/linux/crates/drivers/postgres/tests/integration/read_only.rs +++ b/linux/crates/drivers/postgres/tests/integration/read_only.rs @@ -1,9 +1,14 @@ //! Read-only is held per transaction, so no statement escapes it and no //! setting outlives it. -use tablepro_core::{DriverError, ReadOnlyRefusal, RowLimit, RunEnd, TransactionState, Value, WriteKind}; +use tablepro_core::{ + Connection, DriverError, ReadOnlyRefusal, ReadStatement, RowLimit, RunEnd, StreamSummary, TransactionState, + UserSql, Value, WriteKind, +}; -use crate::support::{CALL, apply, connect, run, run_ok, scalar, session, setup, start, statement, within}; +use crate::support::{ + CALL, EventSink, Events, apply, connect, run, run_ok, scalar, session, setup, start, statement, within, +}; fn refused_by_server(end: &RunEnd) -> bool { matches!( @@ -127,3 +132,66 @@ async fn query_bound_nextval_is_25006() { outcome.map(|result| result.rows) ); } + +async fn stream_user_sql(connection: &dyn Connection, sql: &str) -> Result { + connection + .stream_read( + &ReadStatement::User(UserSql::new(sql)), + Box::new(EventSink::new(&Events::default())), + within(CALL), + ) + .await +} + +/// The read's own transaction has taken its snapshot before user SQL +/// runs, so none of these can make it writable. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn stream_read_user_sql_cannot_turn_the_transaction_writable() { + let fixture = start().await; + let owner = connect(&fixture, false).await; + setup(owner.as_ref(), &["CREATE TABLE t (id int)", "INSERT INTO t VALUES (1)"]).await; + + for sql in [ + "SET TRANSACTION READ WRITE; DELETE FROM t", + "SET transaction_read_only = off; DELETE FROM t", + "SET LOCAL transaction_read_only TO false; DELETE FROM t", + "SELECT set_config('transaction_read_only', 'off', true); DELETE FROM t", + "SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE; DELETE FROM t", + ] { + let outcome = stream_user_sql(owner.as_ref(), sql).await; + + assert!( + matches!(outcome, Err(DriverError::ReadOnly(ReadOnlyRefusal::ServerRejected(_)))), + "{sql}: {outcome:?}" + ); + } + assert_eq!(scalar(owner.as_ref(), "SELECT count(*) FROM t").await, Value::Int(1)); +} + +/// With `standard_conforming_strings` off the server reads `\'` as an +/// escaped quote, so the COMMIT is a statement of its own and the guard +/// has to see it that way too. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "requires docker"] +async fn stream_read_guard_splits_as_a_non_conforming_session_does() { + let fixture = start().await; + let owner = connect(&fixture, false).await; + setup( + owner.as_ref(), + &[ + "CREATE TABLE t (id int)", + "INSERT INTO t VALUES (1)", + "ALTER ROLE CURRENT_USER SET standard_conforming_strings = off", + ], + ) + .await; + + let outcome = stream_user_sql(owner.as_ref(), "SELECT 'a\\', '; COMMIT; DELETE FROM t; SELECT '1'").await; + + assert!( + matches!(outcome, Err(DriverError::ReadOnly(ReadOnlyRefusal::ClientGuard))), + "{outcome:?}" + ); + assert_eq!(scalar(owner.as_ref(), "SELECT count(*) FROM t").await, Value::Int(1)); +} diff --git a/linux/crates/drivers/sqlite/Cargo.toml b/linux/crates/drivers/sqlite/Cargo.toml index 4d2647c6c6..3bba6c801c 100644 --- a/linux/crates/drivers/sqlite/Cargo.toml +++ b/linux/crates/drivers/sqlite/Cargo.toml @@ -13,9 +13,10 @@ path = "src/lib.rs" tablepro-core = { path = "../../core" } async-trait.workspace = true futures.workspace = true -# One direct call, sqlite3_get_autocommit, until sqlx exposes the -# transaction state of a connection. The range follows sqlx-sqlite's own, -# because only one libsqlite3-sys can be linked. +# Two direct calls sqlx does not wrap: sqlite3_get_autocommit, until sqlx +# exposes the transaction state of a connection, and sqlite3_busy_handler. +# The range follows sqlx-sqlite's own, because only one libsqlite3-sys can +# be linked. libsqlite3-sys.workspace = true sqlparser.workspace = true sqlx = { workspace = true, features = ["sqlite-unbundled"] } diff --git a/linux/crates/drivers/sqlite/src/autocommit.rs b/linux/crates/drivers/sqlite/src/autocommit.rs index 61d3c74e05..beb762b5cd 100644 --- a/linux/crates/drivers/sqlite/src/autocommit.rs +++ b/linux/crates/drivers/sqlite/src/autocommit.rs @@ -2,9 +2,8 @@ //! //! sqlx tracks only the transactions it began itself and keeps its own //! flag crate-private, so a BEGIN the user typed in the editor is -//! invisible to it. SQLite's own answer is `sqlite3_get_autocommit`. -//! This is the crate's one unsafe call, an interim until sqlx exposes -//! the state on `LockedSqliteHandle`. +//! invisible to it. SQLite's own answer is `sqlite3_get_autocommit`, +//! called directly until sqlx exposes the state on `LockedSqliteHandle`. use sqlx::sqlite::LockedSqliteHandle; diff --git a/linux/crates/drivers/sqlite/src/connection.rs b/linux/crates/drivers/sqlite/src/connection.rs index 547981d4de..9a3e25e0a1 100644 --- a/linux/crates/drivers/sqlite/src/connection.rs +++ b/linux/crates/drivers/sqlite/src/connection.rs @@ -159,9 +159,10 @@ impl SqliteConnection { { let flag = InterruptFlag::new(); let stop = Arc::new(SqliteStop::new(flag.clone())); + let lock_timeout = self.settings.lock_timeout; self.calls .run_read(options, pooled, stop, move |mut pooled, _request| async move { - let result = match flag.install(pooled.connection()).await { + let result = match flag.install(pooled.connection(), lock_timeout).await { Ok(()) => work(pooled.connection()).await, Err(error) => Err(error), }; @@ -368,6 +369,7 @@ impl Connection for SqliteConnection { let connection = self.open_standalone(&options).await?; let flag = InterruptFlag::new(); let stop = Arc::new(SqliteStop::new(flag.clone())); + let lock_timeout = self.settings.lock_timeout; let read = read.clone(); let streamed = self .calls @@ -377,7 +379,7 @@ impl Connection for SqliteConnection { stop, move |mut held, _request| async move { let mut sink = sink; - let result = match flag.install(held.connection()).await { + let result = match flag.install(held.connection(), lock_timeout).await { Ok(()) => match &read { ReadStatement::Generated(statement) => { run_bound(held.connection(), statement, sink.as_mut()).await @@ -416,11 +418,12 @@ impl Connection for SqliteConnection { let ledger = Arc::new(WriteLedger::new(WriteAtomicity::Transactional)); let flag = InterruptFlag::new(); let stop = Arc::new(SqliteStop::new(flag.clone())); + let lock_timeout = self.settings.lock_timeout; let batch = batch.clone(); let recorded = ledger.clone(); self.calls .run_write(options, pooled, stop, ledger, move |mut pooled, request| async move { - let result = write_batch::run(&mut pooled, &batch, &recorded, &flag, &request).await; + let result = write_batch::run(&mut pooled, &batch, &recorded, &flag, &request, lock_timeout).await; (pooled, result) }) .await @@ -447,6 +450,7 @@ impl Connection for SqliteConnection { &self.path, options, &self.liveness, + self.settings.lock_timeout, cancel, ) .await?; diff --git a/linux/crates/drivers/sqlite/src/dialect.rs b/linux/crates/drivers/sqlite/src/dialect.rs index 3d55d4e896..dfaf3d3005 100644 --- a/linux/crates/drivers/sqlite/src/dialect.rs +++ b/linux/crates/drivers/sqlite/src/dialect.rs @@ -119,7 +119,7 @@ impl SqlDialect for SqliteDialect { // Text and the temporal values, stored as the ISO text // SQLite's date functions read. other => match value_to_text(other) { - Some(text) => Ok(format!("'{}'", text.replace('\'', "''"))), + Some(text) => Ok(text_literal(&text)), None => Ok("NULL".to_owned()), }, } @@ -202,6 +202,20 @@ impl SqlDialect for SqliteDialect { } } +/// SQLite ends SQL text at the first NUL, wherever it stands, so a NUL +/// inside quotes would cut the literal short. Each one is joined in +/// with `char(0)` instead. +fn text_literal(text: &str) -> String { + let quoted = |piece: &str| format!("'{}'", piece.replace('\'', "''")); + match text.contains('\0') { + false => quoted(text), + true => { + let pieces: Vec = text.split('\0').map(quoted).collect(); + format!("({})", pieces.join(" || char(0) || ")) + } + } +} + /// The parameter a value takes for a column. /// /// A column of an ordinary table stores whatever class it is given, so @@ -391,6 +405,20 @@ mod tests { assert!(dialect().literal(&Value::Float64(f64::NAN), &text).is_err()); } + #[test] + fn a_nul_in_text_is_joined_in_rather_than_quoted() { + let text = ColumnType::untyped_text("TEXT"); + + assert_eq!( + dialect().literal(&Value::Text("a\0b'c".to_owned()), &text).as_deref(), + Ok("('a' || char(0) || 'b''c')") + ); + assert_eq!( + dialect().literal(&Value::Text("\0".to_owned()), &text).as_deref(), + Ok("('' || char(0) || '')") + ); + } + #[test] fn literal_round_trips_through_the_sqlite_parser() { use sqlparser::dialect::SQLiteDialect; diff --git a/linux/crates/drivers/sqlite/src/editor_session.rs b/linux/crates/drivers/sqlite/src/editor_session.rs index f913d6fe40..f68b9423f3 100644 --- a/linux/crates/drivers/sqlite/src/editor_session.rs +++ b/linux/crates/drivers/sqlite/src/editor_session.rs @@ -15,7 +15,7 @@ use tokio_util::sync::CancellationToken; use crate::autocommit::is_autocommit; use crate::deadline; -use crate::errors::{is_interrupt, map_error, map_open_error}; +use crate::errors::{map_error, map_open_error}; use crate::interrupt::InterruptFlag; use crate::result_stream::{StreamOutcome, run_script}; @@ -33,6 +33,7 @@ pub(crate) struct SqliteEditorSession { /// The running statement's stop, so closing the tab stops it. current: Mutex>, statement_timeout: Option, + lock_timeout: Duration, cancel_grace: Duration, close_timeout: Duration, } @@ -50,6 +51,7 @@ impl SqliteEditorSession { path: &Path, session: EditorSessionOptions, liveness: &LivenessPolicy, + lock_timeout: Duration, cancel: CancellationToken, ) -> Result { let connecting = SqlxConnection::connect_with(options); @@ -73,6 +75,7 @@ impl SqliteEditorSession { lost: AtomicBool::new(false), current: Mutex::new(None), statement_timeout: session.statement_timeout, + lock_timeout, cancel_grace: liveness.cancel_grace, close_timeout: liveness.close_timeout, }) @@ -108,8 +111,8 @@ impl SqliteEditorSession { let settled = tokio::time::timeout(self.cancel_grace, &mut running).await; Ok(match (stop, settled) { (_, Ok(Ok(outcome))) => completed(outcome), - (Stop::Cancelled, Ok(Err(error))) if is_interrupt(&error) => RunEnd::Cancelled, - (Stop::DeadlineElapsed, Ok(Err(error))) if is_interrupt(&error) => { + (Stop::Cancelled, Ok(Err(error))) if flag.stopped(&error) => RunEnd::Cancelled, + (Stop::DeadlineElapsed, Ok(Err(error))) if flag.stopped(&error) => { RunEnd::TimedOut { server_cancelled: true } } (_, Ok(Err(error))) => return session_failure(error), @@ -194,7 +197,7 @@ impl EditorSession for SqliteEditorSession { let flag = InterruptFlag::new(); self.set_current(Some(flag.clone())); - let ended = match flag.install(connection).await { + let ended = match flag.install(connection, self.lock_timeout).await { Ok(()) => self.supervise(connection, &flag, &sql, &options, sink.as_mut()).await, Err(error) => session_failure(error), }; diff --git a/linux/crates/drivers/sqlite/src/interrupt.rs b/linux/crates/drivers/sqlite/src/interrupt.rs index 21724b8c22..6f654fdd5a 100644 --- a/linux/crates/drivers/sqlite/src/interrupt.rs +++ b/linux/crates/drivers/sqlite/src/interrupt.rs @@ -5,16 +5,20 @@ //! through it would arrive only once the statement it was meant to stop //! had ended. The handler runs on the worker thread itself, between //! virtual machine instructions, and stops the statement when the call -//! has raised its flag. +//! has raised its flag. A statement still waiting for another +//! connection's lock runs no instructions, so the lock wait checks the +//! same flag (`lock_wait`). use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; use async_trait::async_trait; use sqlx::sqlite::SqliteConnection as SqlxConnection; use tablepro_core::{DriverError, EngineStop, StopDelivery, StopScope}; use crate::errors::{is_interrupt, map_error}; +use crate::lock_wait::LockWait; /// How many virtual machine instructions run between two looks at the /// flag: often enough that a stop lands within milliseconds, rarely @@ -41,11 +45,31 @@ impl InterruptFlag { self.0.store(false, Ordering::SeqCst); } - /// Make this flag the one the connection's statements answer to. - pub(crate) async fn install(&self, connection: &mut SqlxConnection) -> Result<(), DriverError> { - let flag = self.0.clone(); + pub(crate) fn is_raised(&self) -> bool { + self.0.load(Ordering::SeqCst) + } + + /// Whether `error` is SQLite giving way to this flag: a running + /// statement is interrupted, and one waiting for a lock gives up + /// as busy. + pub(crate) fn stopped(&self, error: &DriverError) -> bool { + is_interrupt(error) || (self.is_raised() && *error == DriverError::Busy) + } + + /// Make this flag the one the connection's statements answer to, + /// with `lock_timeout` as the longest wait for another connection's + /// lock. + pub(crate) async fn install( + &self, + connection: &mut SqlxConnection, + lock_timeout: Duration, + ) -> Result<(), DriverError> { + let wait = Arc::new(LockWait::new(self.0.clone(), lock_timeout)); let mut handle = connection.lock_handle().await.map_err(map_error)?; - handle.set_progress_handler(INSTRUCTIONS_PER_CHECK, move || !flag.load(Ordering::Relaxed)); + wait.install(&mut handle)?; + // The progress handler owns the wait, so it lives exactly as long + // as this call's statements can ask for it. + handle.set_progress_handler(INSTRUCTIONS_PER_CHECK, move || !wait.stopped()); Ok(()) } } @@ -75,6 +99,6 @@ impl EngineStop for SqliteStop { } fn acknowledged(&self, error: &DriverError) -> bool { - is_interrupt(error) || matches!(error, DriverError::RolledBack { source, .. } if is_interrupt(source)) + self.flag.stopped(error) || matches!(error, DriverError::RolledBack { source, .. } if self.flag.stopped(source)) } } diff --git a/linux/crates/drivers/sqlite/src/lib.rs b/linux/crates/drivers/sqlite/src/lib.rs index f78ce9a67b..a4cba99071 100644 --- a/linux/crates/drivers/sqlite/src/lib.rs +++ b/linux/crates/drivers/sqlite/src/lib.rs @@ -16,6 +16,7 @@ mod errors; mod foreign_key_verification; mod index_sql; mod interrupt; +mod lock_wait; mod metadata; mod open; mod result_stream; diff --git a/linux/crates/drivers/sqlite/src/lock_wait.rs b/linux/crates/drivers/sqlite/src/lock_wait.rs new file mode 100644 index 0000000000..7941758f61 --- /dev/null +++ b/linux/crates/drivers/sqlite/src/lock_wait.rs @@ -0,0 +1,146 @@ +//! Waiting for another connection's lock in a way a stop can end. +//! +//! SQLite's own busy timeout sleeps inside the call that wants the +//! lock, where the progress handler never runs, so a stop could not +//! reach a waiting statement before the lock timeout had passed. This +//! busy handler sleeps the same steps and looks at the call's flag +//! between them. +//! +//! The handler is a C callback, and its argument is the connection's own +//! handle used only as a key. What it reads is found through a weak +//! reference, so it never follows a pointer the wait may have outlived. + +use std::ffi::{c_int, c_void}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex, PoisonError, Weak}; +use std::time::Duration; + +use sqlx::sqlite::LockedSqliteHandle; +use tablepro_core::DriverError; + +/// The sleeps between tries, in milliseconds, as SQLite's own handler +/// takes them; past the end, the last one repeats. +const DELAYS_MS: [u64; 12] = [1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50, 100]; + +/// Each connection's installed wait, by the address of its handle. +static WAITS: Mutex)>> = Mutex::new(Vec::new()); + +/// One call's lock wait: as long as the lock timeout, and over as soon +/// as the call's flag is raised. +#[derive(Debug)] +pub(crate) struct LockWait { + stop: Arc, + timeout: Duration, +} + +impl LockWait { + pub(crate) fn new(stop: Arc, timeout: Duration) -> Self { + Self { stop, timeout } + } + + pub(crate) fn stopped(&self) -> bool { + self.stop.load(Ordering::Relaxed) + } + + /// Make this the wait the connection's lock attempts answer to. The + /// caller keeps it alive for as long as it is installed; once it is + /// dropped, a lock attempt gives up at once. + pub(crate) fn install(self: &Arc, handle: &mut LockedSqliteHandle<'_>) -> Result<(), DriverError> { + let raw = handle.as_raw_handle(); + let key = raw.as_ptr().addr(); + { + let mut waits = WAITS.lock().unwrap_or_else(PoisonError::into_inner); + waits.retain(|(other, wait)| *other != key && wait.strong_count() > 0); + waits.push((key, Arc::downgrade(self))); + } + // SAFETY: `raw` comes from the locked handle, and the lock guard + // is held for the whole call, so the connection is open and no + // statement runs on it meanwhile. The argument handed to the + // callback is never dereferenced: `on_busy` uses it as a key. + let code = unsafe { libsqlite3_sys::sqlite3_busy_handler(raw.as_ptr(), Some(on_busy), raw.as_ptr().cast()) }; + match code { + libsqlite3_sys::SQLITE_OK => Ok(()), + code => Err(DriverError::Protocol(format!( + "SQLite refused the lock wait with code {code}" + ))), + } + } + + /// Sleep before the next try for the lock, or say to stop trying. + fn keep_waiting(&self, attempts: c_int) -> bool { + let Some(delay) = next_delay(attempts, self.timeout) else { + return false; + }; + if self.stopped() { + return false; + } + std::thread::sleep(delay); + !self.stopped() + } +} + +/// How long to sleep after `attempts` failed tries, or nothing once the +/// sleeps so far have used the timeout up. +fn next_delay(attempts: c_int, timeout: Duration) -> Option { + let attempts = usize::try_from(attempts).ok()?; + let ramp = DELAYS_MS.len().saturating_sub(1); + let last = DELAYS_MS.last().copied().unwrap_or(0); + let delay = DELAYS_MS.get(attempts).copied().unwrap_or(last); + let repeats = u64::try_from(attempts.saturating_sub(ramp)).unwrap_or(u64::MAX); + let prior = DELAYS_MS + .iter() + .take(attempts.min(ramp)) + .sum::() + .saturating_add(last.saturating_mul(repeats)); + let timeout = u64::try_from(timeout.as_millis()).unwrap_or(u64::MAX); + let delay = delay.min(timeout.saturating_sub(prior)); + (delay > 0).then(|| Duration::from_millis(delay)) +} + +/// SQLite's busy handler: nonzero to try for the lock again. +extern "C" fn on_busy(connection: *mut c_void, attempts: c_int) -> c_int { + let key = connection.addr(); + let wait = WAITS + .lock() + .unwrap_or_else(PoisonError::into_inner) + .iter() + .find(|(other, _)| *other == key) + .and_then(|(_, wait)| wait.upgrade()); + match wait { + Some(wait) => c_int::from(wait.keep_waiting(attempts)), + None => 0, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn millis(value: u64) -> Option { + Some(Duration::from_millis(value)) + } + + #[test] + fn the_sleeps_follow_sqlites_own_until_the_timeout() { + let timeout = Duration::from_millis(400); + + assert_eq!(next_delay(0, timeout), millis(1)); + assert_eq!(next_delay(5, timeout), millis(20)); + assert_eq!(next_delay(11, timeout), millis(100)); + // The first twelve sleeps took 328 ms, so 72 are left. + assert_eq!(next_delay(12, timeout), millis(72)); + assert_eq!(next_delay(13, timeout), None); + assert_eq!(next_delay(0, Duration::ZERO), None); + assert_eq!(next_delay(-1, timeout), None); + } + + #[test] + fn a_raised_flag_ends_the_wait() { + let stop = Arc::new(AtomicBool::new(false)); + let wait = LockWait::new(stop.clone(), Duration::from_secs(10)); + + assert!(wait.keep_waiting(0)); + stop.store(true, Ordering::SeqCst); + assert!(!wait.keep_waiting(1)); + } +} diff --git a/linux/crates/drivers/sqlite/src/result_stream.rs b/linux/crates/drivers/sqlite/src/result_stream.rs index 744519e667..68fedd0aa5 100644 --- a/linux/crates/drivers/sqlite/src/result_stream.rs +++ b/linux/crates/drivers/sqlite/src/result_stream.rs @@ -12,7 +12,9 @@ use sqlx::sqlite::{SqliteColumn, SqliteConnection as SqlxConnection}; use sqlx::{AssertSqlSafe, Column, Executor, SqlSafeStr, Statement as _, TypeInfo}; use tablepro_core::column::{ColumnType, ResultColumn}; use tablepro_core::sql_syntax::SqlGrammar; -use tablepro_core::sql_syntax::script::{LexicalSettings, ScriptPlan, ScriptTokenKind}; +use tablepro_core::sql_syntax::script::{ + LexicalSettings, OpenConstruct, ScriptDiagnostic, ScriptPlan, ScriptTokenKind, +}; use tablepro_core::statement::{BoundParam, Statement}; use tablepro_core::value::Value; use tablepro_core::{DriverError, QueryResult, ResultEvent, ResultSink, RowLimit, SinkFlow, Truncation}; @@ -111,19 +113,43 @@ pub(crate) async fn run_bound( } /// The statements of a script, split where SQLite itself would end one -/// (a trigger body's semicolons do not end it). Text the splitter -/// cannot finish, such as an unterminated string, goes to SQLite whole -/// so the error is SQLite's own. +/// (a trigger body's semicolons do not end it). +/// +/// An unterminated string, identifier or comment runs to the end of the +/// text. The statements before it still run one at a time, and the one +/// it opens goes to SQLite as it stands, so the error is SQLite's own. +/// Sent whole, the script would run as one statement under the first +/// statement's columns. SQLite reads an open comment as a comment, so +/// only the text before one is sent. pub(crate) fn split_statements(sql: &str) -> Vec<&str> { let plan = ScriptPlan::build( sql, SqlGrammar::Sqlite, LexicalSettings::default_for(SqlGrammar::Sqlite), ); - if !plan.diagnostics().is_empty() { - return vec![sql]; + let open = plan + .diagnostics() + .iter() + .map(|ScriptDiagnostic::Unterminated { construct, start }| (*start, *construct)) + .min_by_key(|(start, _)| *start); + let (open_at, tail_end) = match open { + Some((start, OpenConstruct::BlockComment)) => (start, start), + Some((start, _)) => (start, sql.len()), + None => (sql.len(), sql.len()), + }; + let mut statements = Vec::with_capacity(plan.statements().len()); + for statement in plan.statements() { + if statement.range.end <= open_at { + statements.push(statement.text(sql)); + continue; + } + let tail = sql.get(statement.range.start..tail_end).unwrap_or_default().trim_end(); + if !tail.is_empty() { + statements.push(tail); + } + break; } - plan.statements().iter().map(|statement| statement.text(sql)).collect() + statements } /// Whether SQLite's change count describes this statement. It counts @@ -300,6 +326,19 @@ mod tests { assert!(split_statements(" -- only a comment\n").is_empty()); } + #[test] + fn statements_before_an_unfinished_one_still_run_alone() { + assert_eq!( + split_statements("SELECT 1 AS a; SELECT 2 AS b, 3 AS c /* open"), + ["SELECT 1 AS a", "SELECT 2 AS b, 3 AS c"] + ); + assert_eq!( + split_statements("SELECT 1; SELECT 'open; still open"), + ["SELECT 1", "SELECT 'open; still open"] + ); + assert_eq!(split_statements("SELECT 1; /* open"), ["SELECT 1"]); + } + #[test] fn only_row_changes_report_a_change_count() { assert!(reports_changes(" INSERT INTO t VALUES (1)")); diff --git a/linux/crates/drivers/sqlite/src/write_batch.rs b/linux/crates/drivers/sqlite/src/write_batch.rs index e123083e91..db9ab73e03 100644 --- a/linux/crates/drivers/sqlite/src/write_batch.rs +++ b/linux/crates/drivers/sqlite/src/write_batch.rs @@ -7,6 +7,7 @@ //! as Busy after the lock timeout, before anything has run. use std::collections::HashMap; +use std::time::Duration; use sqlx::sqlite::SqliteConnection as SqlxConnection; use sqlx::{AssertSqlSafe, Connection as _, Row}; @@ -17,6 +18,7 @@ use tablepro_core::{ WriteBatch, WriteLedger, WriteReport, }; +use crate::autocommit::is_autocommit; use crate::catalog::{FOREIGN_KEY_CHECK_SQL, MAIN_SCHEMA}; use crate::encode::bind_all; use crate::errors::{SQLITE_IOERR, is_interrupt, map_error, primary_of}; @@ -32,8 +34,9 @@ pub(crate) async fn run( ledger: &WriteLedger, flag: &InterruptFlag, request: &StopRequest, + lock_timeout: Duration, ) -> Result { - flag.install(pooled.connection()).await?; + flag.install(pooled.connection(), lock_timeout).await?; let checks = match &batch.foreign_keys { ForeignKeyMode::Enforced => None, ForeignKeyMode::SuspendedThenVerified { checks } => Some(checks.as_slice()), @@ -219,8 +222,13 @@ impl Write<'_> { Ok(reports_changes(sql).then_some(affected)) } - async fn rollback(&mut self, transaction: sqlx::Transaction<'_, sqlx::Sqlite>) -> SessionFate { + async fn rollback(&mut self, mut transaction: sqlx::Transaction<'_, sqlx::Sqlite>) -> SessionFate { self.flag.lower(); + if let Err(error) = reopen_if_ended(&mut transaction).await { + tracing::warn!(%error, "the transaction state could not be read; closing the connection"); + self.discard = true; + return SessionFate::Discarded; + } match transaction.rollback().await { Ok(()) => SessionFate::RolledBack, Err(error) => { @@ -240,16 +248,37 @@ impl Write<'_> { } } - /// What the caller is told when `error` is the interrupt a stop - /// raised. + /// What the caller is told when `error` is how SQLite gave way to a + /// stop: an interrupt, or a lock wait given up as busy. The flag is + /// lowered for the rollback by then, so the request says whether a + /// stop came. fn stopped(&self, error: &DriverError) -> Option { - match is_interrupt(error) { + match is_interrupt(error) || *error == DriverError::Busy { true => self.request.reason().map(stop_cause), false => None, } } } +/// Some errors end the transaction inside SQLite itself: an interrupted +/// write, a full disk, a conflict resolved with ROLLBACK. sqlx still +/// counts it open, so the ROLLBACK it sends would be refused and leave +/// that count wrong for every later BEGIN. An empty transaction opened +/// in its place is what that ROLLBACK then ends. +async fn reopen_if_ended(connection: &mut SqlxConnection) -> Result<(), DriverError> { + let ended = { + let mut handle = connection.lock_handle().await.map_err(map_error)?; + is_autocommit(&mut handle) + }; + if ended { + sqlx::query("BEGIN") + .execute(&mut *connection) + .await + .map_err(map_error)?; + } + Ok(()) +} + enum StepFailure { Guard { expected: u64, actual: u64 }, Error(DriverError), diff --git a/linux/crates/drivers/sqlite/tests/integration/dialect.rs b/linux/crates/drivers/sqlite/tests/integration/dialect.rs index 03a3efed93..c55ab8a7f2 100644 --- a/linux/crates/drivers/sqlite/tests/integration/dialect.rs +++ b/linux/crates/drivers/sqlite/tests/integration/dialect.rs @@ -1,11 +1,12 @@ //! The dialect's filters against SQLite itself. -use tablepro_core::Connection; use tablepro_core::browse::build_count; +use tablepro_core::column::ColumnType; use tablepro_core::filter::{FilterOp, FilterRule, FilterSet, FilterValue}; use tablepro_core::meta::TableRef; +use tablepro_core::{Connection, Value}; -use crate::support::{TestFile, apply, call}; +use crate::support::{TestFile, apply, call, query}; async fn matching(connection: &dyn Connection, table: &TableRef, op: FilterOp, text: &str) -> u64 { let columns = connection.fetch_columns(table, call()).await.unwrap(); @@ -70,3 +71,19 @@ async fn insensitive_filter_matches_mixed_case() { assert_eq!(matching(connection.as_ref(), &table, FilterOp::Ilike, "%aLP%").await, 1); connection.close().await.unwrap(); } + +#[tokio::test] +async fn a_text_literal_with_a_nul_reads_back_whole() { + let file = TestFile::new().await; + let connection = file.open().await; + let text = "before\0after 'quoted'"; + let literal = connection + .dialect() + .literal(&Value::Text(text.to_owned()), &ColumnType::untyped_text("TEXT")) + .unwrap(); + + let read = query(connection.as_ref(), &format!("SELECT {literal}")).await; + + assert_eq!(read.rows[0][0], Value::Text(text.to_owned())); + connection.close().await.unwrap(); +} diff --git a/linux/crates/drivers/sqlite/tests/integration/lock_wait.rs b/linux/crates/drivers/sqlite/tests/integration/lock_wait.rs new file mode 100644 index 0000000000..caf2758980 --- /dev/null +++ b/linux/crates/drivers/sqlite/tests/integration/lock_wait.rs @@ -0,0 +1,104 @@ +//! A stop that lands while a statement waits for another connection's +//! lock, which is longer than the cancel grace. + +use std::time::Duration; + +use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection}; +use sqlx::{Connection as _, Executor as _}; +use tablepro_core::{ + CallOptions, DriverError, EditorSessionOptions, EditorSessionState, RunEnd, RunOptions, Value, WriteKind, +}; +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +use crate::support::{TestFile, apply, batch, query, run, run_with}; + +/// Well inside the ten-second lock timeout, and short of the second of +/// cancel grace a stop that never lands would use up. +const PROMPT: Duration = Duration::from_millis(900); + +async fn hold_write_lock(file: &TestFile) -> SqliteConnection { + let mut holder = SqliteConnection::connect_with(&SqliteConnectOptions::new().filename(&file.path)) + .await + .unwrap(); + holder.execute("BEGIN IMMEDIATE").await.unwrap(); + holder +} + +fn cancel_after(delay: Duration) -> CancellationToken { + let cancel = CancellationToken::new(); + let stopper = cancel.clone(); + tokio::spawn(async move { + tokio::time::sleep(delay).await; + stopper.cancel(); + }); + cancel +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_save_waiting_for_a_lock_stops_and_keeps_its_connection() { + let file = TestFile::new().await; + let connection = file.open().await; + apply( + connection.as_ref(), + &["CREATE TABLE t (a)", "CREATE TEMP TABLE kept (a)"], + ) + .await + .unwrap(); + let mut holder = hold_write_lock(&file).await; + + let started = Instant::now(); + let written = connection + .write_batch( + &batch(WriteKind::RowChanges, &["INSERT INTO t VALUES (1)"]), + CallOptions::new(None, cancel_after(Duration::from_millis(200))), + ) + .await; + let elapsed = started.elapsed(); + holder.execute("ROLLBACK").await.unwrap(); + holder.close().await.unwrap(); + + assert_eq!( + written, + Err(DriverError::RolledBack { + statement_index: 0, + source: Box::new(DriverError::Cancelled), + }) + ); + assert!(elapsed < PROMPT, "the stop took {elapsed:?}"); + let kept = query(connection.as_ref(), "SELECT count(*) FROM temp.kept").await; + assert_eq!(kept.rows[0][0], Value::Int(0), "the workspace connection was replaced"); + connection.close().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn an_editor_run_waiting_for_a_lock_stops_and_the_next_run_starts() { + let file = TestFile::new().await; + let connection = file.open().await; + apply(connection.as_ref(), &["CREATE TABLE t (a)"]).await.unwrap(); + let session = connection + .open_editor_session(EditorSessionOptions::default(), CancellationToken::new()) + .await + .unwrap(); + let mut holder = hold_write_lock(&file).await; + + let started = Instant::now(); + let (stopped, _) = run_with( + session.as_ref(), + "INSERT INTO t VALUES (1)", + RunOptions::new(cancel_after(Duration::from_millis(200))), + ) + .await; + let (next, collected) = run(session.as_ref(), "SELECT 1").await; + let elapsed = started.elapsed(); + holder.execute("ROLLBACK").await.unwrap(); + holder.close().await.unwrap(); + + assert_eq!(stopped.end, RunEnd::Cancelled); + assert_eq!(next.end, RunEnd::Completed); + assert_eq!(collected.rows(), vec![vec![Value::Int(1)]]); + assert!(elapsed < PROMPT, "the stop and the next run took {elapsed:?}"); + assert_eq!(session.state(), EditorSessionState::Ready); + session.close().await; + connection.close().await.unwrap(); +} diff --git a/linux/crates/drivers/sqlite/tests/integration/main.rs b/linux/crates/drivers/sqlite/tests/integration/main.rs index 3009721528..6cf5e195d0 100644 --- a/linux/crates/drivers/sqlite/tests/integration/main.rs +++ b/linux/crates/drivers/sqlite/tests/integration/main.rs @@ -13,6 +13,8 @@ mod decode; #[cfg(test)] mod dialect; #[cfg(test)] +mod lock_wait; +#[cfg(test)] mod metadata; #[cfg(test)] mod open_semantics; diff --git a/linux/crates/drivers/sqlite/tests/integration/sessions.rs b/linux/crates/drivers/sqlite/tests/integration/sessions.rs index e18a214ffb..41d5605ec9 100644 --- a/linux/crates/drivers/sqlite/tests/integration/sessions.rs +++ b/linux/crates/drivers/sqlite/tests/integration/sessions.rs @@ -57,6 +57,39 @@ async fn select_star_after_add_column_reads_the_new_column() { connection.close().await.unwrap(); } +/// A comment left open at the end makes the script unfinished, which +/// must not merge its statements into one run under the first header. +#[tokio::test] +async fn an_unterminated_comment_keeps_each_statement_its_own_columns() { + let file = TestFile::new().await; + let connection = file.open().await; + let session = connection + .open_editor_session(EditorSessionOptions::default(), CancellationToken::new()) + .await + .unwrap(); + + let (summary, collected) = run(session.as_ref(), "SELECT 1 AS a; SELECT 2 AS b, 3 AS c /* open").await; + + assert_eq!(summary.end, RunEnd::Completed); + let headers: Vec> = collected + .events() + .into_iter() + .filter_map(|event| match event { + ResultEvent::ResultSetStarted { columns } => { + Some(columns.iter().map(|column| column.name.clone()).collect()) + } + _ => None, + }) + .collect(); + assert_eq!(headers, [vec!["a".to_owned()], vec!["b".to_owned(), "c".to_owned()]]); + assert_eq!( + collected.rows(), + vec![vec![Value::Int(1)], vec![Value::Int(2), Value::Int(3)]] + ); + session.close().await; + connection.close().await.unwrap(); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn recursive_cte_cancelled() { let file = TestFile::new().await; @@ -243,6 +276,49 @@ async fn write_interrupt_before_commit_rolls_back() { connection.close().await.unwrap(); } +/// ON CONFLICT ROLLBACK ends the transaction inside SQLite before the +/// batch gets to roll back itself; the workspace connection, and the +/// temporary table on it, must survive that. +#[tokio::test] +async fn a_batch_sqlite_already_rolled_back_keeps_its_connection() { + let file = TestFile::new().await; + let connection = file.open().await; + apply( + connection.as_ref(), + &[ + "CREATE TABLE t (a INTEGER UNIQUE ON CONFLICT ROLLBACK)", + "INSERT INTO t VALUES (1)", + "CREATE TEMP TABLE kept (a)", + ], + ) + .await + .unwrap(); + + let written = connection + .write_batch( + &batch( + WriteKind::RowChanges, + &["INSERT INTO t VALUES (2)", "INSERT INTO t VALUES (1)"], + ), + call(), + ) + .await; + + assert!( + matches!(&written, Err(DriverError::RolledBack { statement_index: 1, .. })), + "{written:?}" + ); + let rows = query(connection.as_ref(), "SELECT count(*) FROM t").await; + assert_eq!(rows.rows[0][0], Value::Int(1)); + let kept = query(connection.as_ref(), "SELECT count(*) FROM temp.kept").await; + assert_eq!(kept.rows[0][0], Value::Int(0), "the workspace connection was replaced"); + let after = connection + .write_batch(&batch(WriteKind::RowChanges, &["INSERT INTO t VALUES (3)"]), call()) + .await; + assert!(after.is_ok(), "{after:?}"); + connection.close().await.unwrap(); +} + #[tokio::test] async fn a_second_run_while_one_is_going_is_refused() { let file = TestFile::new().await; diff --git a/linux/crates/session/src/connect/connection_writes.rs b/linux/crates/session/src/connect/connection_writes.rs index 1dc682c451..b5b52f76bb 100644 --- a/linux/crates/session/src/connect/connection_writes.rs +++ b/linux/crates/session/src/connect/connection_writes.rs @@ -1,11 +1,13 @@ use futures::future::BoxFuture; use tablepro_storage::{ConnectionStore, SavedConnection, StorageError}; +use uuid::Uuid; use crate::runtime::Tasks; /// Where a saved connection entry is written. pub trait ConnectionWrites: Send + Sync { fn upsert(&self, connection: SavedConnection) -> BoxFuture<'_, Result<(), StorageError>>; + fn remove(&self, id: Uuid) -> BoxFuture<'_, Result<(), StorageError>>; } /// The connection list on disk, written on the blocking pool because a @@ -34,4 +36,15 @@ impl ConnectionWrites for StoreConnectionWrites { .map_err(|failure| StorageError::Schema(format!("the connections write task failed: {failure}")))? }) } + + fn remove(&self, id: Uuid) -> BoxFuture<'_, Result<(), StorageError>> { + let store = self.store.clone(); + let removed = self.tasks.spawn_blocking_task(move || store.remove_blocking(id)); + Box::pin(async move { + removed + .await + .map_err(|failure| StorageError::Schema(format!("the connections write task failed: {failure}")))? + .map(|_| ()) + }) + } } diff --git a/linux/crates/session/src/connect/persist_pending_save.rs b/linux/crates/session/src/connect/persist_pending_save.rs index a703ec7ddd..690c74c771 100644 --- a/linux/crates/session/src/connect/persist_pending_save.rs +++ b/linux/crates/session/src/connect/persist_pending_save.rs @@ -122,6 +122,11 @@ mod tests { } }) } + + fn remove(&self, id: Uuid) -> BoxFuture<'_, Result<(), StorageError>> { + self.writes.lock().unwrap().retain(|(_, written)| written.id != id); + Box::pin(async { Ok(()) }) + } } fn connection(id: Uuid) -> SavedConnection { diff --git a/linux/crates/session/src/health/health_monitor.rs b/linux/crates/session/src/health/health_monitor.rs index 0f6222c3e7..d2373e6e34 100644 --- a/linux/crates/session/src/health/health_monitor.rs +++ b/linux/crates/session/src/health/health_monitor.rs @@ -160,10 +160,17 @@ async fn reconnect(ctx: &MonitorContext, user_asked: bool) -> Option>>, + } + + impl UnansweredPrompt { + fn still_asking(&self) -> bool { + self.questions + .lock() + .unwrap() + .iter() + .any(|question| !question.is_closed()) + } + } + + impl CredentialSource for UnansweredPrompt { + fn refresh( + &self, + request: ConnectRequest, + _interaction: tablepro_core::credentials::CredentialInteraction, + _demand: CredentialDemand, + ) -> futures::future::BoxFuture<'static, Result> { + let (question, answer) = tokio::sync::oneshot::channel(); + self.questions.lock().unwrap().push(question); + Box::pin(async move { + let _unanswered = answer.await; + Ok(RefreshedRequest { + request, + remember: Vec::new(), + }) + }) + } + } + fn refused() -> ConnectError { ConnectError::Driver(DriverError::Transport(TransportError::Refused { endpoint: NetworkEndpoint::new("db.example.com", 5432).unwrap(), @@ -489,6 +533,33 @@ mod tests { assert!(started.connection.is_closed(), "the replaced connection was left open"); } + #[tokio::test(start_paused = true)] + async fn shutting_down_ends_a_prompt_a_retry_raised() { + let prompt = Arc::new(UnansweredPrompt::default()); + let (pending, _connection) = FakeConnector::pending_session(FakeConnector::request().unwrap(), healthy()); + let connector = Arc::new(FakeConnector::new([])); + let session = Session::start( + pending, + SessionDeps { + connector: connector.clone(), + credentials: prompt.clone(), + deadlines: Arc::new(DefaultDeadlines::new(LivenessPolicy::DESKTOP)), + tasks: paused_tasks(), + prompter: Some(Arc::new(FakePrompter::new([]))), + }, + ); + session.signal(MonitorSignal::RetryNow); + settle().await; + assert!(prompt.still_asking(), "the retry never asked"); + + let report = session.begin_shutdown(); + tokio::time::sleep(Duration::from_secs(30)).await; + + assert!(report.await.is_ok()); + assert!(!prompt.still_asking(), "the prompt outlived the session that raised it"); + assert!(connector.calls().is_empty()); + } + #[tokio::test(start_paused = true)] async fn late_reconnect_after_cancel_is_discarded_and_closed() { let connector = FakeConnector::new([FakeEstablish::Late { diff --git a/linux/crates/session/tests/session_postgres.rs b/linux/crates/session/tests/session_postgres.rs index 8b956bbdc7..bf7adfb2c2 100644 --- a/linux/crates/session/tests/session_postgres.rs +++ b/linux/crates/session/tests/session_postgres.rs @@ -58,20 +58,31 @@ fn password(credentials: &FixtureCredentials) -> Credentials { } /// SSH services that run the system client with a private config: no -/// agent, a throwaway known_hosts, and new host keys accepted, so no -/// prompt is ever raised. +/// agent and a throwaway known_hosts. The app asks about every host key +/// it does not know, so a test trusts its server's key first. struct IsolatedSsh { - _dir: tempfile::TempDir, + dir: tempfile::TempDir, services: SshServices, } +impl IsolatedSsh { + async fn trust(&self, sshd: &OpenSshFixture) { + let key = sshd.host_public_key().await.expect("the server's host key"); + std::fs::write( + self.dir.path().join("known_hosts"), + format!("[{}]:{} {key}\n", sshd.host(), sshd.port()), + ) + .expect("the known_hosts file"); + } +} + fn isolated_ssh() -> IsolatedSsh { let dir = tempfile::tempdir().expect("a temporary directory"); let config = dir.path().join("ssh_config"); std::fs::write( &config, format!( - "Host *\n UserKnownHostsFile {}\n GlobalKnownHostsFile /dev/null\n StrictHostKeyChecking accept-new\n IdentityAgent none\n IdentitiesOnly yes\n", + "Host *\n UserKnownHostsFile {}\n GlobalKnownHostsFile /dev/null\n IdentityAgent none\n IdentitiesOnly yes\n", dir.path().join("known_hosts").display() ), ) @@ -91,7 +102,7 @@ fn isolated_ssh() -> IsolatedSsh { ssh_keygen_program: PathBuf::from("ssh-keygen"), askpass_program: PathBuf::from("/nonexistent/tablepro-askpass"), }; - IsolatedSsh { _dir: dir, services } + IsolatedSsh { dir, services } } fn connector(ssh: SshServices) -> Arc { @@ -305,6 +316,7 @@ async fn session_through_openssh_reconnects_after_the_master_dies() { .await .expect("an OpenSSH server"); let ssh = isolated_ssh(); + ssh.trust(&sshd).await; let connector = connector(ssh.services.clone()); let pending = establish(&connector, request(ssh_target(&sshd, &postgres, &private_key))).await; diff --git a/linux/crates/ssh/src/argv.rs b/linux/crates/ssh/src/argv.rs index 478301d4dd..441791cbfb 100644 --- a/linux/crates/ssh/src/argv.rs +++ b/linux/crates/ssh/src/argv.rs @@ -4,6 +4,7 @@ use std::path::Path; use tablepro_core::{LivenessPolicy, NetworkEndpoint}; +use crate::host_key_checking::HostKeyChecking; use crate::{SshAuth, SshConfig}; const FORCED_OPTIONS: [&str; 14] = [ @@ -37,12 +38,21 @@ pub(crate) enum ControlOp<'a> { Exit, } -pub(crate) fn master_args(config: &SshConfig, control: &Path, liveness: &LivenessPolicy) -> Vec { +pub(crate) fn master_args( + config: &SshConfig, + control: &Path, + liveness: &LivenessPolicy, + host_keys: Option, +) -> Vec { let mut args: Vec = ["-M", "-N", "-T"].into_iter().map(OsString::from).collect(); push_option(&mut args, control_path_option(control)); for option in &FORCED_OPTIONS[..FORCED_OPTIONS.len() - 1] { push_option(&mut args, OsString::from(option)); } + // The app's dialog is the trust decision, so accept-new and no are raised to ask; a config's yes stays a refusal. + if matches!(host_keys, Some(HostKeyChecking::AcceptNew | HostKeyChecking::No)) { + push_option(&mut args, OsString::from("StrictHostKeyChecking=ask")); + } push_option( &mut args, format!("ConnectTimeout={}", liveness.connect_timeout.as_secs()).into(), @@ -163,6 +173,7 @@ mod tests { &config(None, None, SshAuth::Agent), Path::new("/run/user/1000/app/ssh/0123456789abcdef/89abcdef/control"), &LivenessPolicy::DESKTOP, + None, ); assert_eq!( strings(&args), @@ -212,6 +223,43 @@ mod tests { ); } + #[test] + fn only_a_config_that_would_skip_the_host_key_dialog_is_overridden() { + let cases = [ + (None, None), + (Some(HostKeyChecking::Yes), None), + (Some(HostKeyChecking::Ask), None), + (Some(HostKeyChecking::AcceptNew), Some("StrictHostKeyChecking=ask")), + (Some(HostKeyChecking::No), Some("StrictHostKeyChecking=ask")), + ]; + for (host_keys, expected) in cases { + let auths = [ + SshAuth::Agent, + SshAuth::PrivateKey { + path: None, + passphrase: None, + }, + SshAuth::Password { + password: SecretString::from("secret"), + }, + SshAuth::KeyboardInteractive, + ]; + for auth in auths { + let args = master_args( + &config(None, None, auth), + Path::new("/tmp/control"), + &LivenessPolicy::DESKTOP, + host_keys, + ); + let strict: Vec<&str> = strings(&args) + .into_iter() + .filter(|arg| arg.starts_with("StrictHostKeyChecking=")) + .collect(); + assert_eq!(strict, expected.into_iter().collect::>(), "{host_keys:?}"); + } + } + } + #[test] fn port_and_user_only_when_set() { let control = Path::new("/tmp/control"); @@ -219,6 +267,7 @@ mod tests { &config(None, None, SshAuth::Agent), control, &LivenessPolicy::DESKTOP, + None, )) .join(" "); assert!(!without.contains(" -p ") && !without.contains(" -l ")); @@ -226,6 +275,7 @@ mod tests { &config(Some(2222), Some("deploy"), SshAuth::Agent), control, &LivenessPolicy::DESKTOP, + None, ); assert_eq!( strings(&with[with.len() - 6..]), @@ -237,7 +287,7 @@ mod tests { fn jump_hosts_render_one_minus_j() { let mut with_jumps = config(None, None, SshAuth::Agent); with_jumps.jump_hosts = SshDestination::parse_jump_list("ops@jump1:2200,[fd00::1]").unwrap(); - let args = master_args(&with_jumps, Path::new("/tmp/control"), &LivenessPolicy::DESKTOP); + let args = master_args(&with_jumps, Path::new("/tmp/control"), &LivenessPolicy::DESKTOP, None); let args = strings(&args); assert_eq!(args.iter().filter(|arg| **arg == "-J").count(), 1); let position = args.iter().position(|arg| *arg == "-J").unwrap(); @@ -254,6 +304,7 @@ mod tests { &config(None, None, auth), Path::new("/tmp/control"), &LivenessPolicy::DESKTOP, + None, ); let joined = strings(&args).join(" "); assert!( @@ -269,6 +320,7 @@ mod tests { &config(None, None, without_path), Path::new("/tmp/control"), &LivenessPolicy::DESKTOP, + None, ); assert!(!strings(&args).contains(&"-i")); } @@ -293,6 +345,7 @@ mod tests { &config(None, None, auth), Path::new("/tmp/control"), &LivenessPolicy::DESKTOP, + None, ); let preferred = strings(&args) .into_iter() diff --git a/linux/crates/ssh/src/first_hop.rs b/linux/crates/ssh/src/first_hop.rs index 4aeb3d8109..f6abd116e9 100644 --- a/linux/crates/ssh/src/first_hop.rs +++ b/linux/crates/ssh/src/first_hop.rs @@ -6,7 +6,7 @@ use std::time::Duration; use tablepro_core::NetworkEndpoint; use tokio::process::Command; -use crate::{SshConfig, SshDestination}; +use crate::SshDestination; /// `ssh -G` does no network I/O, but a `Match exec` line can run a /// command, so each lookup is still bounded. @@ -23,27 +23,23 @@ enum Resolved { ProxyCommand, } -/// The host and port the first TCP connection of a session goes to. +/// The host and port the first TCP connection of a session goes to, +/// from what `ssh -G` printed for the session's destination. /// /// An ssh_config alias is only a name until `ssh -G` resolves it, and a /// jump chain starts at its first hop, so this asks ssh rather than /// trusting what the user typed. `None` when the route is a /// ProxyCommand, whose first hop only that command knows. -pub(crate) async fn resolve_first_hop(ssh_program: &Path, config: &SshConfig) -> Option { - let mut destination = config.destination.clone(); - let mut jumps = config.jump_hosts.clone(); +pub(crate) async fn resolve_first_hop(ssh_program: &Path, destination: Option<&str>) -> Option { + let mut output = destination?.to_owned(); for _ in 0..MAX_HOPS { - let output = lookup(ssh_program, &destination, &jumps).await?; match parse_resolved(&output)? { Resolved::Direct(endpoint) => return Some(endpoint), Resolved::ProxyCommand => return None, - Resolved::Jump(first) => { - destination = first; - jumps = Vec::new(); - } + Resolved::Jump(first) => output = lookup(ssh_program, &first, &[]).await?, } } - tracing::debug!(destination = %config.destination, "the ssh jump chain did not end"); + tracing::debug!("the ssh jump chain did not end"); None } @@ -130,7 +126,6 @@ mod tests { use std::path::PathBuf; use super::*; - use crate::SshAuth; fn endpoint(host: &str, port: u16) -> NetworkEndpoint { NetworkEndpoint::new(host, port).unwrap() @@ -196,15 +191,14 @@ mod tests { program } - fn config(host: &str, jumps: &[&str]) -> SshConfig { - SshConfig { - destination: SshDestination::new(host, None, None).unwrap(), - jump_hosts: jumps - .iter() - .map(|jump| SshDestination::new(*jump, None, None).unwrap()) - .collect(), - auth: SshAuth::Agent, - } + async fn first_hop(program: &Path, host: &str, jumps: &[&str]) -> Option { + let destination = SshDestination::new(host, None, None).unwrap(); + let jumps: Vec = jumps + .iter() + .map(|jump| SshDestination::new(*jump, None, None).unwrap()) + .collect(); + let resolved = lookup(program, &destination, &jumps).await; + resolve_first_hop(program, resolved.as_deref()).await } /// The fake prints a jump for the database host and resolves the @@ -222,7 +216,7 @@ case "$last" in esac"#, ); - let first = resolve_first_hop(&program, &config("db", &["bastion"])).await; + let first = first_hop(&program, "db", &["bastion"]).await; assert_eq!(first, Some(endpoint("192.0.2.10", 2222))); } @@ -231,10 +225,10 @@ esac"#, async fn a_failing_lookup_and_a_jump_loop_resolve_to_nothing() { let temp = tempfile::tempdir().unwrap(); let failing = fake_ssh(temp.path(), "exit 255"); - assert_eq!(resolve_first_hop(&failing, &config("db", &[])).await, None); + assert_eq!(first_hop(&failing, "db", &[]).await, None); let looping = tempfile::tempdir().unwrap(); let program = fake_ssh(looping.path(), r"printf 'hostname x\nproxyjump loop\n'"); - assert_eq!(resolve_first_hop(&program, &config("loop", &[])).await, None); + assert_eq!(first_hop(&program, "loop", &[]).await, None); } } diff --git a/linux/crates/ssh/src/host_key_checking.rs b/linux/crates/ssh/src/host_key_checking.rs new file mode 100644 index 0000000000..3d301e6703 --- /dev/null +++ b/linux/crates/ssh/src/host_key_checking.rs @@ -0,0 +1,52 @@ +/// The StrictHostKeyChecking value the user's ssh config gives a +/// destination. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum HostKeyChecking { + Yes, + Ask, + AcceptNew, + No, +} + +impl HostKeyChecking { + /// `ssh -G` prints the value as `true`, `false`, `ask` or + /// `accept-new`, whichever spelling the config used. + pub(crate) fn from_ssh_g(ssh_g_output: &str) -> Option { + let value = ssh_g_output + .lines() + .find_map(|line| line.strip_prefix("stricthostkeychecking "))?; + match value.trim() { + "true" => Some(Self::Yes), + "ask" => Some(Self::Ask), + "accept-new" => Some(Self::AcceptNew), + "false" => Some(Self::No), + _ => None, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn every_value_ssh_g_prints_is_read() { + let cases = [ + ("true", HostKeyChecking::Yes), + ("ask", HostKeyChecking::Ask), + ("accept-new", HostKeyChecking::AcceptNew), + ("false", HostKeyChecking::No), + ]; + for (printed, expected) in cases { + let output = format!("user deploy\nhostname 10.0.0.5\nstricthostkeychecking {printed}\nport 22\n"); + + assert_eq!(HostKeyChecking::from_ssh_g(&output), Some(expected), "{printed}"); + } + } + + #[test] + fn a_missing_or_unknown_value_is_not_guessed() { + assert_eq!(HostKeyChecking::from_ssh_g("hostname 10.0.0.5\n"), None); + assert_eq!(HostKeyChecking::from_ssh_g("stricthostkeychecking sometimes\n"), None); + } +} diff --git a/linux/crates/ssh/src/lib.rs b/linux/crates/ssh/src/lib.rs index 038aa20dc5..deba84c29f 100644 --- a/linux/crates/ssh/src/lib.rs +++ b/linux/crates/ssh/src/lib.rs @@ -6,6 +6,7 @@ mod config; mod destination; mod first_hop; mod forward; +mod host_key_checking; mod known_hosts; mod password_owner; mod prompt; diff --git a/linux/crates/ssh/src/password_owner.rs b/linux/crates/ssh/src/password_owner.rs index 6dfa0d3ae0..8362705560 100644 --- a/linux/crates/ssh/src/password_owner.rs +++ b/linux/crates/ssh/src/password_owner.rs @@ -1,19 +1,9 @@ -use std::path::Path; - -use crate::SshConfig; -use crate::first_hop::lookup; - /// The `user@host` OpenSSH puts in the destination's password prompt, so /// a stored password goes to that prompt and never to a jump host's. /// /// sshconnect2.c prints `%.30s@%.128s's password: ` with the server user /// and HostKeyAlias, or the resolved HostName when there is no alias. -pub(crate) async fn resolve_password_owner(ssh_program: &Path, config: &SshConfig) -> Option { - let output = lookup(ssh_program, &config.destination, &config.jump_hosts).await?; - password_owner(&output) -} - -fn password_owner(ssh_g_output: &str) -> Option { +pub(crate) fn password_owner(ssh_g_output: &str) -> Option { let mut user = None; let mut hostname = None; let mut alias = None; diff --git a/linux/crates/ssh/src/session.rs b/linux/crates/ssh/src/session.rs index 9bc07dc1c5..e9993c00db 100644 --- a/linux/crates/ssh/src/session.rs +++ b/linux/crates/ssh/src/session.rs @@ -18,9 +18,10 @@ use tokio_util::sync::{CancellationToken, WaitForCancellationFutureOwned}; use crate::argv::{ControlOp, master_args}; use crate::askpass_bridge::AskpassBridge; -use crate::first_hop::resolve_first_hop; +use crate::first_hop::{lookup, resolve_first_hop}; +use crate::host_key_checking::HostKeyChecking; use crate::known_hosts::spawn_failure; -use crate::password_owner::resolve_password_owner; +use crate::password_owner::password_owner; use crate::stderr_classify::{ClassifyContext, classify}; use crate::supervisor::{ForwardCancel, Supervisor, TASKS, run_control}; use crate::{SshAuth, SshConfig, SshServices, lock}; @@ -72,12 +73,14 @@ impl SshSession { return Err(error); } }; + let resolved = lookup(&services.ssh_program, &config.destination, &config.jump_hosts).await; let password_owner = match config.auth { - SshAuth::Password { .. } => resolve_password_owner(&services.ssh_program, config).await, + SshAuth::Password { .. } => resolved.as_deref().and_then(password_owner), SshAuth::Agent | SshAuth::PrivateKey { .. } | SshAuth::KeyboardInteractive => None, }; + let host_keys = resolved.as_deref().and_then(HostKeyChecking::from_ssh_g); let spawned = Command::new(&services.ssh_program) - .args(master_args(config, &control, liveness)) + .args(master_args(config, &control, liveness, host_keys)) .env("SSH_ASKPASS", &services.askpass_program) .env("SSH_ASKPASS_REQUIRE", "force") .env_remove("SSH_ASKPASS_PROMPT") @@ -117,7 +120,7 @@ impl SshSession { liveness, &cancel, ), - resolve_first_hop(&services.ssh_program, config), + resolve_first_hop(&services.ssh_program, resolved.as_deref()), ); drop(listener); if let Err(error) = tokio::fs::remove_file(master_dir.join("askpass")).await { diff --git a/linux/crates/ssh/tests/openssh_session.rs b/linux/crates/ssh/tests/openssh_session.rs index 35516a64f7..d99c71af69 100644 --- a/linux/crates/ssh/tests/openssh_session.rs +++ b/linux/crates/ssh/tests/openssh_session.rs @@ -29,10 +29,21 @@ struct Fixture { _temp: tempfile::TempDir, host: String, port: u16, + config: PathBuf, known_hosts: PathBuf, services: SshServices, } +fn write_config(config: &Path, known_hosts: &Path, strict_host_key_checking: &str) -> std::io::Result<()> { + std::fs::write( + config, + format!( + "Host *\n UserKnownHostsFile {}\n GlobalKnownHostsFile /dev/null\n StrictHostKeyChecking {strict_host_key_checking}\n IdentityAgent none\n IdentitiesOnly yes\n PubkeyAuthentication no\n", + known_hosts.display() + ), + ) +} + async fn start() -> TestResult { let container = GenericImage::new("alpine", "3.22") .with_exposed_port(2222.tcp()) @@ -46,13 +57,7 @@ async fn start() -> TestResult { let temp = tempfile::tempdir()?; let known_hosts = temp.path().join("known_hosts"); let config = temp.path().join("ssh_config"); - std::fs::write( - &config, - format!( - "Host *\n UserKnownHostsFile {}\n GlobalKnownHostsFile /dev/null\n StrictHostKeyChecking ask\n IdentityAgent none\n IdentitiesOnly yes\n PubkeyAuthentication no\n", - known_hosts.display() - ), - )?; + write_config(&config, &known_hosts, "ask")?; let wrapper = temp.path().join("ssh"); std::fs::write( &wrapper, @@ -73,6 +78,7 @@ async fn start() -> TestResult { _temp: temp, host, port, + config, known_hosts, services, }) @@ -210,6 +216,56 @@ async fn unknown_host_declined_is_host_key_unknown_declined() { )); } +#[tokio::test] +#[ignore = "requires docker"] +async fn a_user_config_that_skips_the_host_key_check_still_asks() { + let fixture = start().await.unwrap(); + for strict in ["no", "accept-new"] { + write_config(&fixture.config, &fixture.known_hosts, strict).unwrap(); + if let Err(error) = std::fs::remove_file(&fixture.known_hosts) { + assert_eq!(error.kind(), std::io::ErrorKind::NotFound, "{error}"); + } + + let (session, prompter) = connect(&fixture, password_auth(), vec![PromptReply::Cancelled]).await; + + assert!( + matches!( + session, + Err(TransportError::Ssh(SshFailure::HostKeyUnknown { declined: true, .. })) + ), + "StrictHostKeyChecking {strict} trusted the key without asking" + ); + let purposes: Vec = prompter.requests().into_iter().map(|request| request.purpose).collect(); + assert!( + matches!(purposes.as_slice(), [PromptPurpose::SshHostKeyConfirmation { .. }]), + "{strict}: {purposes:?}" + ); + } +} + +#[tokio::test] +#[ignore = "requires docker"] +async fn a_user_config_that_requires_a_known_key_refuses_without_asking() { + let fixture = start().await.unwrap(); + write_config(&fixture.config, &fixture.known_hosts, "yes").unwrap(); + + let (session, prompter) = connect(&fixture, password_auth(), Vec::new()).await; + + assert!( + matches!( + session, + Err(TransportError::Ssh(SshFailure::HostKeyUnknown { + trust_allowed: false, + declined: false, + .. + })) + ), + "{session:?}" + ); + assert!(prompter.requests().is_empty(), "{:?}", prompter.requests()); + assert!(!fixture.known_hosts.exists(), "the key was trusted"); +} + #[tokio::test] #[ignore = "requires docker"] async fn wrong_password_is_authentication_rejected() { diff --git a/linux/crates/storage/src/connections.rs b/linux/crates/storage/src/connections.rs index 916eb719bf..918c5ad9b3 100644 --- a/linux/crates/storage/src/connections.rs +++ b/linux/crates/storage/src/connections.rs @@ -80,6 +80,20 @@ impl SavedConnection { } } + /// Whether the two differ at most in how the list shows them. A + /// field added later counts as part of the connection until it is + /// listed here. + fn connects_like(&self, other: &Self) -> bool { + let shown_alike = |saved: &Self| Self { + name: String::new(), + last_opened_at: None, + color: None, + group: None, + ..saved.clone() + }; + shown_alike(self) == shown_alike(other) + } + fn database_target(&self) -> String { let mut target = String::new(); if !self.username.is_empty() { @@ -177,16 +191,17 @@ pub struct ConnectionListSnapshot { /// What an import did to the list. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub struct ImportReport { - /// Connections the list did not have. + /// Connections the list did not have, including one whose id was + /// taken by a different connection and was given its own. pub added: usize, - /// Connections already in the list under the same id, overwritten - /// by the file's version. - pub replaced: usize, + /// Connections already in the list that the file only renamed, + /// recoloured or regrouped. + pub updated: usize, } impl ImportReport { pub fn total(self) -> usize { - self.added + self.replaced + self.added + self.updated } } @@ -308,6 +323,12 @@ impl ConnectionStore { /// rather than doubling them, and a file from another machine adds /// its own. `path` names the file in any problem reported, since /// the user picked it and needs to know which one was refused. + /// + /// The keyring keeps secrets by connection id, so an entry may only + /// be updated by one that sends them to the same place in the same + /// way. Anything else under a taken id is added under a new id and + /// starts with no secrets, so a crafted file cannot point a saved + /// connection's password at another host. pub fn import_blocking(&self, path: &Path, bytes: &[u8]) -> Result { let incoming = crate::document::decode_document::(path, bytes) .map_err(StorageError::DocumentUnavailable)? @@ -315,11 +336,24 @@ impl ConnectionStore { let mut state = self.lock(); let mut list = self.ensure_loaded(&mut state)?.to_vec(); let mut report = ImportReport::default(); - for connection in incoming { + for mut connection in incoming { match list.iter_mut().find(|saved| saved.id == connection.id) { - Some(existing) => { - *existing = connection; - report.replaced += 1; + Some(existing) if existing.connects_like(&connection) => { + existing.name = connection.name; + existing.color = connection.color; + existing.group = connection.group; + report.updated += 1; + } + Some(_) => { + connection.id = Uuid::new_v4(); + // Two rows of one name, one of them pointing somewhere + // else, would leave the user guessing which is theirs. + let taken: Vec<&str> = list.iter().map(|saved| saved.name.as_str()).collect(); + if taken.contains(&connection.name.as_str()) { + connection.name = suffixed_name(&connection.name, "imported", &taken); + } + list.push(connection); + report.added += 1; } None => { list.push(connection); @@ -444,7 +478,11 @@ fn read_document(path: &Path) -> Result, DocumentProblem> { pub fn duplicate(source: &SavedConnection, taken: &[SavedConnection]) -> SavedConnection { SavedConnection { id: Uuid::new_v4(), - name: copy_name(&source.name, &taken.iter().map(|c| c.name.as_str()).collect::>()), + name: suffixed_name( + &source.name, + "copy", + &taken.iter().map(|c| c.name.as_str()).collect::>(), + ), // A copy has never been opened, whatever the original did, so // it sorts by name rather than claiming the original's recency. // The colour and the group do carry: a copy points at the same @@ -455,15 +493,15 @@ pub fn duplicate(source: &SavedConnection, taken: &[SavedConnection]) -> SavedCo } /// `name (copy)`, then `name (copy 2)` and on, until one is free. -fn copy_name(name: &str, taken: &[&str]) -> String { - let first = format!("{name} (copy)"); +fn suffixed_name(name: &str, suffix: &str, taken: &[&str]) -> String { + let first = format!("{name} ({suffix})"); if !taken.contains(&first.as_str()) { return first; } // A name is short and the list is a person's own connections, so // counting up from two is bounded by how many copies they made. (2..) - .map(|n| format!("{name} (copy {n})")) + .map(|n| format!("{name} ({suffix} {n})")) .find(|candidate| !taken.contains(&candidate.as_str())) .unwrap_or(first) } @@ -1029,7 +1067,7 @@ mod tests { .import_blocking(Path::new("/tmp/connections.json"), &bytes) .expect("import"); - assert_eq!(report, ImportReport { added: 1, replaced: 0 }); + assert_eq!(report, ImportReport { added: 1, updated: 0 }); let list = target.load_blocking().expect("load"); assert_eq!(list.len(), 1); assert_eq!(list[0].name, "production"); @@ -1050,13 +1088,81 @@ mod tests { let first = store.import_blocking(Path::new("/tmp/c.json"), &bytes).expect("import"); let second = store.import_blocking(Path::new("/tmp/c.json"), &bytes).expect("import"); - assert_eq!(first, ImportReport { added: 0, replaced: 1 }); - assert_eq!(second, ImportReport { added: 0, replaced: 1 }); + assert_eq!(first, ImportReport { added: 0, updated: 1 }); + assert_eq!(second, ImportReport { added: 0, updated: 1 }); let list = store.load_blocking().expect("load"); assert_eq!(list.len(), 1); assert_eq!(list[0].name, "production (renamed)"); } + #[test] + fn an_import_never_points_a_saved_entry_or_its_secrets_somewhere_else() { + let mut elsewhere = saved("production"); + elsewhere.host = "db.attacker.example".to_owned(); + let mut in_the_clear = saved("production"); + in_the_clear.tls = TlsConfig::disabled(); + for (variant, crafted) in [("another host", elsewhere), ("TLS off", in_the_clear)] { + let root = TempDir::new().expect("tempdir"); + let store = store(&root); + let mine = SavedConnection { + id: crafted.id, + ..saved("production") + }; + store.upsert_blocking(mine.clone()).expect("upsert"); + let bytes = crate::document::encode_document(&ConnectionsDocument { + connections: vec![crafted.clone()], + }) + .expect("encode"); + + let report = store.import_blocking(Path::new("/tmp/c.json"), &bytes).expect("import"); + + assert_eq!(report, ImportReport { added: 1, updated: 0 }, "{variant}"); + let list = store.load_blocking().expect("load"); + assert_eq!(list.len(), 2, "{variant}"); + let kept = list.iter().find(|saved| saved.id == mine.id).expect("the saved entry"); + assert_eq!(*kept, mine, "{variant}: the import changed the saved entry"); + let imported = list.iter().find(|saved| saved.id != mine.id).expect("the import"); + assert_eq!(imported.name, "production (imported)", "{variant}"); + assert_eq!( + SavedConnection { + id: crafted.id, + name: crafted.name.clone(), + ..imported.clone() + }, + crafted, + "{variant}" + ); + } + } + + #[test] + fn an_import_added_under_a_new_id_keeps_a_name_nobody_holds() { + let root = TempDir::new().expect("tempdir"); + let store = store(&root); + let mine = saved("production"); + store.upsert_blocking(mine.clone()).expect("upsert"); + let theirs = SavedConnection { + id: mine.id, + host: "staging.internal".to_owned(), + ..saved("staging") + }; + let bytes = crate::document::encode_document(&ConnectionsDocument { + connections: vec![theirs], + }) + .expect("encode"); + + store.import_blocking(Path::new("/tmp/c.json"), &bytes).expect("import"); + + let mut names: Vec = store + .load_blocking() + .expect("load") + .iter() + .map(|saved| saved.name.clone()) + .collect(); + names.sort(); + assert_eq!(names, ["production", "staging"]); + } + #[test] fn an_import_leaves_the_connections_the_file_does_not_mention() { let root = TempDir::new().expect("tempdir"); @@ -1073,7 +1179,7 @@ mod tests { let report = store.import_blocking(Path::new("/tmp/c.json"), &bytes).expect("import"); - assert_eq!(report, ImportReport { added: 1, replaced: 0 }); + assert_eq!(report, ImportReport { added: 1, updated: 0 }); let names: Vec = store .load_blocking() .expect("load") diff --git a/linux/crates/test-support/src/close_race.rs b/linux/crates/test-support/src/close_race.rs new file mode 100644 index 0000000000..776beb4885 --- /dev/null +++ b/linux/crates/test-support/src/close_race.rs @@ -0,0 +1,46 @@ +use std::sync::{Arc, Barrier}; +use std::thread; + +use futures::FutureExt; +use futures::future::BoxFuture; + +/// Enough rounds that two closes land inside each other's check at +/// least once when the check is not atomic. +pub(crate) const ROUNDS: usize = 2_000; + +/// Per round, how many of two closes started on two threads at the +/// same moment were still pending after one poll. Under a hanging close +/// behaviour, only a close that ran the behaviour stays pending. +pub(crate) fn pending_closes( + rounds: usize, + make: impl Fn() -> T, + close: for<'a> fn(&'a T) -> BoxFuture<'a, ()>, +) -> Vec +where + T: Send + Sync + 'static, +{ + let targets: Arc> = Arc::new((0..rounds).map(|_| make()).collect()); + let barrier = Arc::new(Barrier::new(2)); + let racers: Vec<_> = (0..2) + .map(|_| { + let targets = targets.clone(); + let barrier = barrier.clone(); + thread::spawn(move || { + targets + .iter() + .map(|target| { + barrier.wait(); + close(target).now_or_never().is_none() + }) + .collect::>() + }) + }) + .collect(); + let pending: Vec> = racers + .into_iter() + .map(|racer| racer.join().unwrap_or_default()) + .collect(); + (0..rounds) + .map(|round| pending.iter().filter(|racer| racer.get(round) == Some(&true)).count()) + .collect() +} diff --git a/linux/crates/test-support/src/conformance/cases/cancellation.rs b/linux/crates/test-support/src/conformance/cases/cancellation.rs index 9f29d3130a..e95cb33a0e 100644 --- a/linux/crates/test-support/src/conformance/cases/cancellation.rs +++ b/linux/crates/test-support/src/conformance/cases/cancellation.rs @@ -20,9 +20,9 @@ const SEEN_LIMIT: Duration = Duration::from_secs(10); /// Case 3: cancelling an editor run ends it as cancelled within the /// cancel grace, and the session takes the next run. -pub async fn editor_cancel_reports_cancelled_and_session_stays_ready() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn editor_cancel_reports_cancelled_and_session_stays_ready( + engine: E, +) -> Result<(), ConformanceFailure> { let grace = test_liveness().cancel_grace + SCHEDULING_SLACK; let connection = connect(EDITOR_CASE, &engine, writable()).await?; let session = connection @@ -62,8 +62,9 @@ pub async fn editor_cancel_reports_cancelled_and_session_stays_ready() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn dropped_caller_stops_server_statement( + engine: E, +) -> Result<(), ConformanceFailure> { let bound = test_liveness().cancel_grace + Duration::from_secs(1); let connection = connect(DROPPED_CASE, &engine, writable()).await?; // An editor session owns its own connection, so the count never diff --git a/linux/crates/test-support/src/conformance/cases/close.rs b/linux/crates/test-support/src/conformance/cases/close.rs index 40d0bf09f6..903af70b79 100644 --- a/linux/crates/test-support/src/conformance/cases/close.rs +++ b/linux/crates/test-support/src/conformance/cases/close.rs @@ -7,8 +7,7 @@ const CASE: &str = "close_is_idempotent"; /// Case 14: closing twice is fine and quick, and a call after the close /// fails rather than reopening the pool or hanging. -pub async fn close_is_idempotent() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn close_is_idempotent(engine: E) -> Result<(), ConformanceFailure> { let liveness = test_liveness(); let connection = connect(CASE, &engine, writable()).await?; connection.probe(within(liveness.probe_timeout)).await?; diff --git a/linux/crates/test-support/src/conformance/cases/deadlines.rs b/linux/crates/test-support/src/conformance/cases/deadlines.rs index 719f09719f..155134308e 100644 --- a/linux/crates/test-support/src/conformance/cases/deadlines.rs +++ b/linux/crates/test-support/src/conformance/cases/deadlines.rs @@ -13,9 +13,9 @@ const DEADLINE: Duration = Duration::from_secs(1); /// Case 2: a read that never finishes returns a statement timeout within /// its deadline plus the cancel grace, and the connection still works. -pub async fn query_bound_times_out_within_deadline_plus_cancel_grace() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn query_bound_times_out_within_deadline_plus_cancel_grace( + engine: E, +) -> Result<(), ConformanceFailure> { let bound = DEADLINE + test_liveness().cancel_grace + SCHEDULING_SLACK; let connection = connect(CASE, &engine, writable()).await?; let unbounded = statement(&engine.unbounded_statement_sql(&unique("deadline"))); @@ -92,7 +92,7 @@ mod tests { #[tokio::test(start_paused = true)] async fn deadline_case_fails_when_bound_exceeded() { - let outcome = query_bound_times_out_within_deadline_plus_cancel_grace::>().await; + let outcome = query_bound_times_out_within_deadline_plus_cancel_grace(FakeEngine::::new()).await; assert!( matches!(&outcome, Err(ConformanceFailure::Expectation { case: CASE, detail }) if detail.contains("past the")), @@ -102,7 +102,7 @@ mod tests { #[tokio::test(start_paused = true)] async fn deadline_case_passes_when_the_driver_stops_at_the_deadline() { - query_bound_times_out_within_deadline_plus_cancel_grace::>() + query_bound_times_out_within_deadline_plus_cancel_grace(FakeEngine::::new()) .await .unwrap(); } diff --git a/linux/crates/test-support/src/conformance/cases/liveness.rs b/linux/crates/test-support/src/conformance/cases/liveness.rs index 1d9e373441..7f9e6f96bd 100644 --- a/linux/crates/test-support/src/conformance/cases/liveness.rs +++ b/linux/crates/test-support/src/conformance/cases/liveness.rs @@ -12,8 +12,9 @@ const DEADLINE: Duration = Duration::from_secs(1); /// Case 12: once the route goes quiet, a read reports the connection /// lost within its deadline plus the cancel grace, and the link says so /// to everything watching it. -pub async fn lost_transport_reports_connection_lost() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn lost_transport_reports_connection_lost( + engine: E, +) -> Result<(), ConformanceFailure> { let liveness = test_liveness(); let (options, fault) = engine.faulty_connect_options(writable(), liveness).await?; let connection = dial(CASE, E::driver(), options).await??; diff --git a/linux/crates/test-support/src/conformance/cases/metadata.rs b/linux/crates/test-support/src/conformance/cases/metadata.rs index 99f647552b..41d6d03440 100644 --- a/linux/crates/test-support/src/conformance/cases/metadata.rs +++ b/linux/crates/test-support/src/conformance/cases/metadata.rs @@ -7,9 +7,9 @@ const CASE: &str = "lists_created_table_with_primary_key_and_unique_identity"; /// Case 1: a table the suite creates shows up in the default schema, its /// key column is marked, and its rows can be named again by that key. -pub async fn lists_created_table_with_primary_key_and_unique_identity() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn lists_created_table_with_primary_key_and_unique_identity( + engine: E, +) -> Result<(), ConformanceFailure> { let connection = connect(CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, connection.as_ref(), "metadata").await?; @@ -50,8 +50,7 @@ const SHARED_CASE: &str = "calls_share_one_call_options"; /// One operation hands the same options to each of its calls, as a page /// does for its columns, its row identity and its rows: a call that /// finishes leaves them usable for the next. -pub async fn calls_share_one_call_options() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn calls_share_one_call_options(engine: E) -> Result<(), ConformanceFailure> { let connection = connect(SHARED_CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, connection.as_ref(), "shared").await?; let options = within(CALL_DEADLINE); diff --git a/linux/crates/test-support/src/conformance/cases/mod.rs b/linux/crates/test-support/src/conformance/cases/mod.rs index c8cbc8671b..e44f149368 100644 --- a/linux/crates/test-support/src/conformance/cases/mod.rs +++ b/linux/crates/test-support/src/conformance/cases/mod.rs @@ -1,5 +1,5 @@ -//! One file per area of the contract. Every case starts its own engine -//! and returns a `ConformanceFailure` rather than panicking. +//! One file per area of the contract. Every case gets an engine started +//! for it alone and returns a `ConformanceFailure` rather than panicking. pub mod cancellation; pub mod close; diff --git a/linux/crates/test-support/src/conformance/cases/pool.rs b/linux/crates/test-support/src/conformance/cases/pool.rs index 78447e50dd..e32b80cd77 100644 --- a/linux/crates/test-support/src/conformance/cases/pool.rs +++ b/linux/crates/test-support/src/conformance/cases/pool.rs @@ -18,19 +18,18 @@ const RETRY: Duration = Duration::from_millis(50); /// Case 5: with every pool slot running a statement, the probe still /// answers, because a busy pool is not a dead server. -pub async fn probe_succeeds_while_every_pool_slot_is_busy() -> Result<(), ConformanceFailure> -{ - let engine = E::start().await?; +pub async fn probe_succeeds_while_every_pool_slot_is_busy( + engine: E, +) -> Result<(), ConformanceFailure> { pool_case(&engine, None).await } /// Case 5 on a network engine, which also asks the server that exactly /// the pool's size of statements is running. -pub async fn probe_succeeds_while_every_network_pool_slot_is_busy() -> Result<(), ConformanceFailure> +pub async fn probe_succeeds_while_every_network_pool_slot_is_busy(engine: E) -> Result<(), ConformanceFailure> where E: PooledConformanceEngine + NetworkConformanceEngine, { - let engine = E::start().await?; let count_sql = |marker: &str| engine.running_statement_count_sql(marker); pool_case(&engine, Some(&count_sql)).await } @@ -171,7 +170,7 @@ mod tests { #[tokio::test(start_paused = true)] async fn pool_case_fails_when_saturation_never_appears() { - let outcome = probe_succeeds_while_every_pool_slot_is_busy::>().await; + let outcome = probe_succeeds_while_every_pool_slot_is_busy(FakeEngine::::new()).await; assert!( matches!(&outcome, Err(ConformanceFailure::Expectation { case: CASE, detail }) if detail.contains("still ran")), @@ -181,7 +180,7 @@ mod tests { #[tokio::test(start_paused = true)] async fn pool_case_passes_when_the_full_pool_reports_busy() { - probe_succeeds_while_every_pool_slot_is_busy::>() + probe_succeeds_while_every_pool_slot_is_busy(FakeEngine::::new()) .await .unwrap(); } diff --git a/linux/crates/test-support/src/conformance/cases/read_only.rs b/linux/crates/test-support/src/conformance/cases/read_only.rs index 2617442538..b0f1b38c10 100644 --- a/linux/crates/test-support/src/conformance/cases/read_only.rs +++ b/linux/crates/test-support/src/conformance/cases/read_only.rs @@ -10,8 +10,7 @@ const CASE: &str = "read_only_connection_refuses_writes"; /// Case 7: a read-only connection refuses a save, and its editor refuses /// both a row change and a schema change. -pub async fn read_only_connection_refuses_writes() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn read_only_connection_refuses_writes(engine: E) -> Result<(), ConformanceFailure> { let owner = connect(CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, owner.as_ref(), "read_only").await?; let reader = connect(CASE, &engine, WorkspaceSettings::workspace(true)).await?; diff --git a/linux/crates/test-support/src/conformance/cases/row_guards.rs b/linux/crates/test-support/src/conformance/cases/row_guards.rs index 6500ded42a..cee80fd3ff 100644 --- a/linux/crates/test-support/src/conformance/cases/row_guards.rs +++ b/linux/crates/test-support/src/conformance/cases/row_guards.rs @@ -1,6 +1,8 @@ +use tablepro_core::meta::RowIdentity; use tablepro_core::save::{ChangeSet, RowUpdate, plan_save}; +use tablepro_core::statement::RowGuard; use tablepro_core::value::Value; -use tablepro_core::{DriverError, WriteKind}; +use tablepro_core::{ColumnInfo, Connection, DriverError, TableRef, WriteBatch, WriteKind}; use crate::conformance::support::{ CALL_DEADLINE, apply, connect, count, create_scratch_table, expect, insert_sql, statement, unexpected, within, @@ -9,18 +11,20 @@ use crate::conformance::support::{ use crate::conformance::{ConformanceEngine, ConformanceFailure}; const CASE: &str = "row_guard_mismatch_applies_nothing"; +const KEPT_ID: i64 = 1; const MISSING_ID: i64 = 999; -/// Case 10: a save that edits a row which is no longer there matches no -/// row, and fails on its guard without changing anything. The save is -/// planned the way the grid plans one, so the guard is the one the app -/// would send to this engine. -pub async fn row_guard_mismatch_applies_nothing() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +/// Case 10: a save whose guard fails changes nothing. One update names +/// a row that is gone, so it matches nothing. The other names the row +/// that is there under a guard asking for one row more, so a driver +/// that applies the write despite its guard leaves the row changed. The +/// saves are planned the way the grid plans one, so the kind of guard +/// is the one the app would send to this engine. +pub async fn row_guard_mismatch_applies_nothing(engine: E) -> Result<(), ConformanceFailure> { let connection = connect(CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, connection.as_ref(), "row_guard").await?; let dialect = connection.dialect(); - let insert = insert_sql(dialect.as_ref(), &table, 1, "kept"); + let insert = insert_sql(dialect.as_ref(), &table, KEPT_ID, "kept"); apply(connection.as_ref(), &table, WriteKind::RowChanges, vec![insert]).await?; let columns = connection.fetch_columns(&table, within(CALL_DEADLINE)).await?; @@ -29,19 +33,26 @@ pub async fn row_guard_mismatch_applies_nothing() -> Resul .iter() .position(|column| column.name == "label") .ok_or_else(|| ConformanceFailure::setup(format!("the scratch table has no label column: {columns:?}")))?; - let changes = ChangeSet { - updates: vec![RowUpdate { - key: vec![Value::Int(MISSING_ID)], - assignments: vec![(label, Value::Text("changed".to_owned()))], - }], - ..ChangeSet::default() - }; - let plan = plan_save(dialect.as_ref(), &table, &columns, &identity, &changes).map_err(ConformanceFailure::setup)?; + let update = |id: i64| relabel(connection.as_ref(), &table, &columns, &identity, label, id); - let outcome = connection.write_batch(&plan.batch, within(CALL_DEADLINE)).await; + let missing = update(MISSING_ID)?; + let outcome = connection.write_batch(&missing, within(CALL_DEADLINE)).await; if !matches!(outcome, Err(DriverError::RowGuardFailed { actual: 0, .. })) { return Err(unexpected(CASE, format!("an update of a missing row gave {outcome:?}"))); } + + let mut present = update(KEPT_ID)?; + for step in &mut present.steps { + step.guard = one_row_more(&step.guard)?; + } + let outcome = connection.write_batch(&present, within(CALL_DEADLINE)).await; + if !matches!(outcome, Err(DriverError::RowGuardFailed { actual: 1, .. })) { + return Err(unexpected( + CASE, + format!("an update of a present row under a guard it cannot meet gave {outcome:?}"), + )); + } + let table_sql = dialect.qualify(&table); let kept = format!( "SELECT {} FROM {table_sql} WHERE {} = 'kept'", @@ -51,8 +62,126 @@ pub async fn row_guard_mismatch_applies_nothing() -> Resul let unchanged = connection.count_rows(&statement(&kept), within(CALL_DEADLINE)).await?; let rows = count(connection.as_ref(), &table).await?; expect(CASE, unchanged == 1 && rows == 1, || { - format!("after the refused update {unchanged} of {rows} rows still read 'kept'") + format!("after the refused updates {unchanged} of {rows} rows still read 'kept'") })?; connection.close().await?; Ok(()) } + +fn relabel( + connection: &dyn Connection, + table: &TableRef, + columns: &[ColumnInfo], + identity: &RowIdentity, + label: usize, + id: i64, +) -> Result { + let changes = ChangeSet { + updates: vec![RowUpdate { + key: vec![Value::Int(id)], + assignments: vec![(label, Value::Text("changed".to_owned()))], + }], + ..ChangeSet::default() + }; + let plan = plan_save(connection.dialect().as_ref(), table, columns, identity, &changes) + .map_err(ConformanceFailure::setup)?; + Ok(plan.batch) +} + +fn one_row_more(guard: &RowGuard) -> Result { + match guard { + RowGuard::AffectedExactly(rows) => Ok(RowGuard::AffectedExactly(rows.saturating_add(1))), + RowGuard::ProbeExactly { probe, rows } => Ok(RowGuard::ProbeExactly { + probe: probe.clone(), + rows: rows.saturating_add(1), + }), + RowGuard::None => Err(unexpected(CASE, "the grid planned an update with no row guard")), + } +} + +#[cfg(test)] +mod tests { + use tablepro_core::WriteReport; + use tablepro_core::column::{ColumnDefault, ColumnType}; + + use super::*; + use crate::conformance::fake_engine::{FakeEngine, FakeScenario}; + use crate::{ConnectionScript, FakeReply}; + + fn column(name: &str) -> ColumnInfo { + ColumnInfo { + name: name.to_owned(), + column_type: ColumnType::untyped_text("text"), + nullable: false, + primary_key: name == "id", + is_auto_increment: false, + is_generated: false, + default: ColumnDefault::None, + comment: None, + } + } + + fn guard_failed(actual: u64) -> FakeReply { + FakeReply::err(DriverError::RowGuardFailed { + statement_index: 0, + expected: actual.saturating_add(1), + actual, + }) + } + + /// A table holding the one 'kept' row. The update of the missing row + /// fails its guard; the update of the present row answers `present` + /// and leaves `kept_after` rows reading 'kept'. + fn script(present: FakeReply, kept_after: u64) -> ConnectionScript { + let mut script = FakeEngine::<()>::base_script(); + script + .write_batch + .when_sql_contains("INSERT", FakeReply::ok(WriteReport::new(vec![Some(1)]))) + .once(guard_failed(0)) + .once(present); + script + .fetch_columns + .always(FakeReply::ok(vec![column("id"), column("label")])); + script.row_identity.always(FakeReply::ok(RowIdentity::UniqueKey { + columns: vec!["id".to_owned()], + })); + script + .count_rows + .when_sql_contains("'kept'", FakeReply::ok(kept_after)) + .always(FakeReply::ok(1)); + script + } + + struct RollsBack; + + impl FakeScenario for RollsBack { + fn script() -> ConnectionScript { + script(guard_failed(1), 1) + } + } + + struct IgnoresTheGuard; + + impl FakeScenario for IgnoresTheGuard { + fn script() -> ConnectionScript { + script(FakeReply::ok(WriteReport::new(vec![Some(1)])), 0) + } + } + + #[tokio::test(start_paused = true)] + async fn row_guard_case_passes_when_both_updates_are_refused() { + row_guard_mismatch_applies_nothing(FakeEngine::::new()) + .await + .unwrap(); + } + + #[tokio::test(start_paused = true)] + async fn row_guard_case_fails_when_the_present_row_is_written_despite_its_guard() { + let outcome = row_guard_mismatch_applies_nothing(FakeEngine::::new()).await; + + assert!( + matches!(&outcome, Err(ConformanceFailure::Expectation { case: CASE, detail }) if detail.contains("present row")), + "{outcome:?}" + ); + } +} diff --git a/linux/crates/test-support/src/conformance/cases/row_limit.rs b/linux/crates/test-support/src/conformance/cases/row_limit.rs index 52b8b0d38f..9ed501da95 100644 --- a/linux/crates/test-support/src/conformance/cases/row_limit.rs +++ b/linux/crates/test-support/src/conformance/cases/row_limit.rs @@ -11,9 +11,9 @@ const KEPT_ROWS: u64 = 100; /// Case 6: a run that stops at its row limit inside a transaction keeps /// the transaction, and what runs after it still commits. -pub async fn row_limit_keeps_transaction_and_later_statements() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn row_limit_keeps_transaction_and_later_statements( + engine: E, +) -> Result<(), ConformanceFailure> { let connection = connect(CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, connection.as_ref(), "row_limit").await?; let dialect = connection.dialect(); diff --git a/linux/crates/test-support/src/conformance/cases/statement_timeout.rs b/linux/crates/test-support/src/conformance/cases/statement_timeout.rs index c528fcf479..fc78343651 100644 --- a/linux/crates/test-support/src/conformance/cases/statement_timeout.rs +++ b/linux/crates/test-support/src/conformance/cases/statement_timeout.rs @@ -12,9 +12,9 @@ const SCHEMA_CHANGE: Duration = Duration::from_secs(3); /// Case 11: the workspace statement timeout stops a slow row change, and /// the same work saved as a schema change runs to the end. -pub async fn schema_changes_batch_is_not_bounded_by_statement_timeout() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn schema_changes_batch_is_not_bounded_by_statement_timeout( + engine: E, +) -> Result<(), ConformanceFailure> { let settings = WorkspaceSettings { statement_timeout: STATEMENT_TIMEOUT, ..WorkspaceSettings::workspace(false) diff --git a/linux/crates/test-support/src/conformance/cases/streaming.rs b/linux/crates/test-support/src/conformance/cases/streaming.rs index fe4323c3cc..86dd0c17b8 100644 --- a/linux/crates/test-support/src/conformance/cases/streaming.rs +++ b/linux/crates/test-support/src/conformance/cases/streaming.rs @@ -30,9 +30,9 @@ const STREAM_DEADLINE: Duration = Duration::from_secs(120); /// sink in bounded batches, inside one started and finished result set, /// and a sink that stops reading ends the read without being reported /// as a whole one. The connection answers afterwards. -pub async fn stream_read_streams_every_row_in_bounded_batches() -> Result<(), ConformanceFailure> -{ - let engine = E::start().await?; +pub async fn stream_read_streams_every_row_in_bounded_batches( + engine: E, +) -> Result<(), ConformanceFailure> { let connection = connect(CASE, &engine, writable()).await?; let whole = Tally::default(); @@ -208,14 +208,14 @@ mod tests { #[tokio::test] async fn stream_case_passes_when_rows_arrive_in_batches() { - stream_read_streams_every_row_in_bounded_batches::>() + stream_read_streams_every_row_in_bounded_batches(FakeEngine::::new()) .await .unwrap(); } #[tokio::test] async fn stream_case_fails_when_rows_arrive_in_one_block() { - let outcome = stream_read_streams_every_row_in_bounded_batches::>().await; + let outcome = stream_read_streams_every_row_in_bounded_batches(FakeEngine::::new()).await; assert!( matches!(&outcome, Err(ConformanceFailure::Expectation { case: CASE, detail }) if detail.contains("one batch held")), diff --git a/linux/crates/test-support/src/conformance/cases/tls.rs b/linux/crates/test-support/src/conformance/cases/tls.rs index 11a07905f8..6f82e3b3ec 100644 --- a/linux/crates/test-support/src/conformance/cases/tls.rs +++ b/linux/crates/test-support/src/conformance/cases/tls.rs @@ -69,16 +69,17 @@ fn describe(outcome: &Result, DriverError>) -> String { } /// TLS case 1. -pub async fn verify_full_with_issuing_ca_connects() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn verify_full_with_issuing_ca_connects( + engine: E, +) -> Result<(), ConformanceFailure> { let tls = verified(&engine, engine.ca_file()); connects::(ISSUING_CASE, engine.connect_options(tls)).await } /// TLS case 2. -pub async fn verify_full_with_unrelated_ca_reports_unknown_issuer() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn verify_full_with_unrelated_ca_reports_unknown_issuer( + engine: E, +) -> Result<(), ConformanceFailure> { let tls = verified(&engine, engine.unrelated_ca_file()); let outcome = open::(UNRELATED_CASE, engine.connect_options(tls)).await?; expect( @@ -89,9 +90,9 @@ pub async fn verify_full_with_unrelated_ca_reports_unknown_issuer() --> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn verify_full_with_wrong_server_name_reports_name_mismatch( + engine: E, +) -> Result<(), ConformanceFailure> { let tls = verified(&engine, engine.ca_file()); let outcome = open::(NAME_CASE, engine.name_mismatch_connect_options(tls)).await?; expect( @@ -102,8 +103,7 @@ pub async fn verify_full_with_wrong_server_name_reports_name_mismatch() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn require_connects_without_ca(engine: E) -> Result<(), ConformanceFailure> { let tls = TlsConfig { mode: TlsMode::Require, ca_file: None, @@ -114,8 +114,9 @@ pub async fn require_connects_without_ca() -> Result<() } /// TLS case 5. -pub async fn disable_against_tls_only_server_is_refused() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn disable_against_tls_only_server_is_refused( + engine: E, +) -> Result<(), ConformanceFailure> { let expected = engine.plaintext_rejection_category(); let outcome = open::(DISABLE_CASE, engine.connect_options(TlsConfig::disabled())).await?; let refused = matches!(&outcome, Err(error) if error.category() == expected); @@ -126,8 +127,9 @@ pub async fn disable_against_tls_only_server_is_refused /// TLS case 6: the certificate principal gets in with its certificate /// and is refused without it. -pub async fn client_identity_is_required_by_the_server() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn client_identity_is_required_by_the_server( + engine: E, +) -> Result<(), ConformanceFailure> { let with_identity = TlsConfig { client_identity: Some(engine.client_identity()), ..verified(&engine, engine.ca_file()) diff --git a/linux/crates/test-support/src/conformance/cases/type_fallback.rs b/linux/crates/test-support/src/conformance/cases/type_fallback.rs index 44a3916362..af981881e0 100644 --- a/linux/crates/test-support/src/conformance/cases/type_fallback.rs +++ b/linux/crates/test-support/src/conformance/cases/type_fallback.rs @@ -9,8 +9,9 @@ const CASE: &str = "unsupported_types_never_decode_as_null"; /// Case 13: a value of a type the driver has no decoder for still reads /// back as something, never as a NULL that would hide it. -pub async fn unsupported_types_never_decode_as_null() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn unsupported_types_never_decode_as_null( + engine: E, +) -> Result<(), ConformanceFailure> { let cases = engine.unsupported_type_cases(); expect(CASE, !cases.is_empty(), || { "the engine declares unsupported types but lists none".to_owned() diff --git a/linux/crates/test-support/src/conformance/cases/writes.rs b/linux/crates/test-support/src/conformance/cases/writes.rs index 85b4ca39b0..20662a862e 100644 --- a/linux/crates/test-support/src/conformance/cases/writes.rs +++ b/linux/crates/test-support/src/conformance/cases/writes.rs @@ -9,8 +9,7 @@ const COMMITS_CASE: &str = "write_batch_commits_every_step"; const FAILED_CASE: &str = "failed_write_batch_outcome_matches_dml_atomicity"; /// Case 8: a save of three inserts reports each one and lands all three. -pub async fn write_batch_commits_every_step() -> Result<(), ConformanceFailure> { - let engine = E::start().await?; +pub async fn write_batch_commits_every_step(engine: E) -> Result<(), ConformanceFailure> { let connection = connect(COMMITS_CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, connection.as_ref(), "writes").await?; let dialect = connection.dialect(); @@ -32,9 +31,9 @@ pub async fn write_batch_commits_every_step() -> Result<() /// Case 9: a save whose second statement fails leaves what the engine's /// atomicity promises: nothing on a transactional engine, the first row /// on one that applies statement by statement. -pub async fn failed_write_batch_outcome_matches_dml_atomicity() -> Result<(), ConformanceFailure> -{ - let engine = E::start().await?; +pub async fn failed_write_batch_outcome_matches_dml_atomicity( + engine: E, +) -> Result<(), ConformanceFailure> { let atomicity = E::driver().capabilities().dml_writes; let connection = connect(FAILED_CASE, &engine, writable()).await?; let table = create_scratch_table(&engine, connection.as_ref(), "failed_write").await?; @@ -122,7 +121,8 @@ mod tests { #[tokio::test] async fn case_reports_expectation_on_wrong_outcome() { - let outcome = failed_write_batch_outcome_matches_dml_atomicity::>().await; + let outcome = + failed_write_batch_outcome_matches_dml_atomicity(FakeEngine::::new()).await; assert!( matches!(&outcome, Err(ConformanceFailure::Expectation { case: FAILED_CASE, detail }) if detail.contains("Transactional")), @@ -132,7 +132,7 @@ mod tests { #[tokio::test] async fn failed_write_case_passes_when_a_transactional_engine_rolls_back() { - failed_write_batch_outcome_matches_dml_atomicity::>() + failed_write_batch_outcome_matches_dml_atomicity(FakeEngine::::new()) .await .unwrap(); } diff --git a/linux/crates/test-support/src/conformance/connection_suite_macro.rs b/linux/crates/test-support/src/conformance/connection_suite_macro.rs index b361e33fea..94e6ab33b3 100644 --- a/linux/crates/test-support/src/conformance/connection_suite_macro.rs +++ b/linux/crates/test-support/src/conformance/connection_suite_macro.rs @@ -17,7 +17,12 @@ /// /// `suite_flags_match_driver_capabilities` is always emitted and never /// ignored: it fails when `endpoint` or `editor_transactions` disagrees -/// with the driver's capabilities. +/// with the driver's capabilities. `pool`, `unsupported_types` and +/// `bounded_ddl` have no capability to check against. +/// +/// A case that has not finished `CASE_LIMIT` after its engine started +/// fails with its name, so a driver that ignores a deadline or a cancel +/// cannot hold the test run. /// /// The example compiles every case against the test-support fakes; a /// driver's integration target supplies its own engine instead. @@ -147,51 +152,61 @@ macro_rules! connection_conformance_suite { ) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, lists_created_table_with_primary_key_and_unique_identity, $crate::conformance::cases::metadata::lists_created_table_with_primary_key_and_unique_identity::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, calls_share_one_call_options, $crate::conformance::cases::metadata::calls_share_one_call_options::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, query_bound_times_out_within_deadline_plus_cancel_grace, $crate::conformance::cases::deadlines::query_bound_times_out_within_deadline_plus_cancel_grace::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, editor_cancel_reports_cancelled_and_session_stays_ready, $crate::conformance::cases::cancellation::editor_cancel_reports_cancelled_and_session_stays_ready::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, read_only_connection_refuses_writes, $crate::conformance::cases::read_only::read_only_connection_refuses_writes::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, write_batch_commits_every_step, $crate::conformance::cases::writes::write_batch_commits_every_step::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, failed_write_batch_outcome_matches_dml_atomicity, $crate::conformance::cases::writes::failed_write_batch_outcome_matches_dml_atomicity::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, row_guard_mismatch_applies_nothing, $crate::conformance::cases::row_guards::row_guard_mismatch_applies_nothing::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, close_is_idempotent, $crate::conformance::cases::close::close_is_idempotent::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, stream_read_streams_every_row_in_bounded_batches, $crate::conformance::cases::streaming::stream_read_streams_every_row_in_bounded_batches::<$engine> } @@ -217,17 +232,27 @@ macro_rules! connection_conformance_suite { #[doc(hidden)] #[macro_export] macro_rules! __conformance_case { - (true, $name:ident, $case:expr) => { + (true, $engine:ty as $kind:ident, $name:ident, $case:expr) => { #[::tokio::test(flavor = "multi_thread", worker_threads = 2)] #[ignore = "requires docker"] async fn $name() -> ::core::result::Result<(), $crate::conformance::ConformanceFailure> { - ($case)().await + $crate::conformance::run_case( + ::core::stringify!($name), + <$engine as $crate::conformance::$kind>::start(), + $case, + ) + .await } }; - (false, $name:ident, $case:expr) => { + (false, $engine:ty as $kind:ident, $name:ident, $case:expr) => { #[::tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn $name() -> ::core::result::Result<(), $crate::conformance::ConformanceFailure> { - ($case)().await + $crate::conformance::run_case( + ::core::stringify!($name), + <$engine as $crate::conformance::$kind>::start(), + $case, + ) + .await } }; } @@ -260,11 +285,13 @@ macro_rules! __conformance_endpoint_cases { (network, $docker:tt, $engine:ty) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, dropped_caller_stops_server_statement, $crate::conformance::cases::cancellation::dropped_caller_stops_server_statement::<$engine> } $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, lost_transport_reports_connection_lost, $crate::conformance::cases::liveness::lost_transport_reports_connection_lost::<$engine> } @@ -278,6 +305,7 @@ macro_rules! __conformance_pool_case { (pooled, network, $docker:tt, $engine:ty) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, probe_succeeds_while_every_pool_slot_is_busy, $crate::conformance::cases::pool::probe_succeeds_while_every_network_pool_slot_is_busy::<$engine> } @@ -285,6 +313,7 @@ macro_rules! __conformance_pool_case { (pooled, file, $docker:tt, $engine:ty) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, probe_succeeds_while_every_pool_slot_is_busy, $crate::conformance::cases::pool::probe_succeeds_while_every_pool_slot_is_busy::<$engine> } @@ -299,6 +328,7 @@ macro_rules! __conformance_editor_case { (true, $docker:tt, $engine:ty) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, row_limit_keeps_transaction_and_later_statements, $crate::conformance::cases::row_limit::row_limit_keeps_transaction_and_later_statements::<$engine> } @@ -312,6 +342,7 @@ macro_rules! __conformance_types_case { (true, $docker:tt, $engine:ty) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, unsupported_types_never_decode_as_null, $crate::conformance::cases::type_fallback::unsupported_types_never_decode_as_null::<$engine> } @@ -325,6 +356,7 @@ macro_rules! __conformance_ddl_case { (true, $docker:tt, $engine:ty) => { $crate::__conformance_case! { $docker, + $engine as ConformanceEngine, schema_changes_batch_is_not_bounded_by_statement_timeout, $crate::conformance::cases::statement_timeout::schema_changes_batch_is_not_bounded_by_statement_timeout::<$engine> } diff --git a/linux/crates/test-support/src/conformance/fake_engine.rs b/linux/crates/test-support/src/conformance/fake_engine.rs index 5ac392cd0f..eb40caf832 100644 --- a/linux/crates/test-support/src/conformance/fake_engine.rs +++ b/linux/crates/test-support/src/conformance/fake_engine.rs @@ -26,6 +26,10 @@ pub(crate) trait FakeScenario: Send + Sync + 'static { pub(crate) struct FakeEngine(PhantomData S>); impl FakeEngine { + pub(crate) fn new() -> Self { + Self(PhantomData) + } + /// Answers the setup every case does, so a scenario scripts only the /// call it is about. pub(crate) fn base_script() -> ConnectionScript { @@ -45,7 +49,7 @@ impl ConformanceEngine for FakeEngine { } async fn start() -> Result { - Ok(Self(PhantomData)) + Ok(Self::new()) } fn connect_options(&self, settings: WorkspaceSettings, liveness: LivenessPolicy) -> ConnectOptions { diff --git a/linux/crates/test-support/src/conformance/mod.rs b/linux/crates/test-support/src/conformance/mod.rs index f7b84209f8..19d315bff9 100644 --- a/linux/crates/test-support/src/conformance/mod.rs +++ b/linux/crates/test-support/src/conformance/mod.rs @@ -3,8 +3,13 @@ //! A driver's integration target implements an engine adapter and //! invokes `connection_conformance_suite!` and `tls_conformance_suite!` //! with flags for what the engine supports. A flag that is off emits no -//! test, and the flags themselves are checked against the driver's -//! capability table, so a case cannot be skipped by declaring less. +//! test. The flags with a capability behind them (`endpoint`, +//! `editor_transactions` and the TLS `client_identity`) are checked +//! against the driver's capability table, so those cases cannot be +//! skipped by declaring less. `pool`, `unsupported_types` and +//! `bounded_ddl` describe the server rather than the driver (MySQL and +//! MariaDB differ on `bounded_ddl` under one driver), so the adapter +//! states them and no capability can contradict them. mod bounded_ddl_engine; pub mod cases; @@ -18,6 +23,7 @@ mod editor_transaction_engine; mod fake_engine; mod network_conformance_engine; mod pooled_conformance_engine; +mod run_case; mod suite_endpoint; mod support; mod test_liveness; @@ -34,6 +40,7 @@ pub use connection_suite_flags::ConnectionSuiteFlags; pub use editor_transaction_engine::EditorTransactionEngine; pub use network_conformance_engine::NetworkConformanceEngine; pub use pooled_conformance_engine::PooledConformanceEngine; +pub use run_case::{CASE_LIMIT, run_case}; pub use suite_endpoint::SuiteEndpoint; pub use test_liveness::test_liveness; pub use tls_conformance_engine::TlsConformanceEngine; diff --git a/linux/crates/test-support/src/conformance/run_case.rs b/linux/crates/test-support/src/conformance/run_case.rs new file mode 100644 index 0000000000..8a9ed504c5 --- /dev/null +++ b/linux/crates/test-support/src/conformance/run_case.rs @@ -0,0 +1,64 @@ +use std::future::Future; +use std::time::Duration; + +use crate::conformance::ConformanceFailure; +use crate::conformance::support::unexpected; + +/// How long a case may run once its engine is up. Against a healthy +/// server every case takes seconds; the limit only ends a case whose +/// driver ignores a deadline or a cancel, well before the `ci` nextest +/// profile kills the test without saying why. +pub const CASE_LIMIT: Duration = Duration::from_secs(120); + +/// Starting the engine is outside `CASE_LIMIT`: pulling and booting a +/// server image can take minutes and says nothing about the driver. +pub async fn run_case( + name: &'static str, + start: impl Future>, + case: C, +) -> Result<(), ConformanceFailure> +where + C: FnOnce(E) -> F, + F: Future>, +{ + let engine = start.await?; + tokio::time::timeout(CASE_LIMIT, case(engine)) + .await + .map_err(|_| unexpected(name, format!("did not finish within {CASE_LIMIT:?}")))? +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::conformance::ConformanceEngine; + use crate::conformance::cases::deadlines::query_bound_times_out_within_deadline_plus_cancel_grace; + use crate::conformance::fake_engine::{FakeEngine, FakeScenario}; + use crate::{ConnectionScript, FakeReply}; + + const CASE: &str = "query_bound_times_out_within_deadline_plus_cancel_grace"; + + struct NeverAnswers; + + impl FakeScenario for NeverAnswers { + fn script() -> ConnectionScript { + let mut script = FakeEngine::::base_script(); + script.query_bound.always(FakeReply::Hang); + script + } + } + + #[tokio::test(start_paused = true)] + async fn a_case_whose_driver_never_answers_fails_with_its_name() { + let outcome = run_case( + CASE, + FakeEngine::::start(), + query_bound_times_out_within_deadline_plus_cancel_grace, + ) + .await; + + assert!( + matches!(&outcome, Err(ConformanceFailure::Expectation { case: CASE, detail }) if detail.contains("did not finish")), + "{outcome:?}" + ); + } +} diff --git a/linux/crates/test-support/src/conformance/support.rs b/linux/crates/test-support/src/conformance/support.rs index 8194cb1f8a..a9033f1e8e 100644 --- a/linux/crates/test-support/src/conformance/support.rs +++ b/linux/crates/test-support/src/conformance/support.rs @@ -218,7 +218,8 @@ pub(crate) fn single_count(rows: &[Vec]) -> Option { } /// Ask the server, through an editor session, how many statements are -/// running, until `done` holds or `limit` passes. +/// running, until `done` holds or `limit` passes. `limit` is checked +/// between counts, so each count carries its own deadline. pub(crate) async fn wait_for_running( case: &'static str, monitor: &dyn EditorSession, @@ -228,7 +229,8 @@ pub(crate) async fn wait_for_running( ) -> Result { let give_up = Instant::now() + limit; loop { - let rows = run_completed(case, monitor, count_sql, run_options()).await?.rows(); + let counting = run_options().with_deadline(Some(CALL_DEADLINE)); + let rows = run_completed(case, monitor, count_sql, counting).await?.rows(); let Some(running) = single_count(&rows) else { return Err(ConformanceFailure::setup(format!( "the running-statement count returned {rows:?}, not one integer" @@ -246,3 +248,27 @@ pub(crate) async fn wait_for_running( tokio::time::sleep(Duration::from_millis(100)).await; } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::{FakeEditorSession, FakeRun, Replies}; + + #[tokio::test(start_paused = true)] + async fn wait_for_running_fails_when_a_count_never_answers() { + let mut runs = Replies::default(); + runs.always(FakeRun::until_stopped()); + let monitor = FakeEditorSession::new(runs); + + let waited = tokio::time::timeout( + CALL_DEADLINE * 2, + wait_for_running("case", &monitor, "SELECT count", |_| true, Duration::from_secs(1)), + ) + .await; + + assert!( + matches!(&waited, Ok(Err(ConformanceFailure::Expectation { case: "case", detail })) if detail.contains("TimedOut")), + "{waited:?}" + ); + } +} diff --git a/linux/crates/test-support/src/conformance/tls_suite_macro.rs b/linux/crates/test-support/src/conformance/tls_suite_macro.rs index b15d98bc4d..cfabc3c8a7 100644 --- a/linux/crates/test-support/src/conformance/tls_suite_macro.rs +++ b/linux/crates/test-support/src/conformance/tls_suite_macro.rs @@ -84,26 +84,31 @@ macro_rules! tls_conformance_suite { (engine: $engine:ty, client_identity: $identity:tt $(,)?) => { $crate::__conformance_case! { true, + $engine as TlsConformanceEngine, verify_full_with_issuing_ca_connects, $crate::conformance::cases::tls::verify_full_with_issuing_ca_connects::<$engine> } $crate::__conformance_case! { true, + $engine as TlsConformanceEngine, verify_full_with_unrelated_ca_reports_unknown_issuer, $crate::conformance::cases::tls::verify_full_with_unrelated_ca_reports_unknown_issuer::<$engine> } $crate::__conformance_case! { true, + $engine as TlsConformanceEngine, verify_full_with_wrong_server_name_reports_name_mismatch, $crate::conformance::cases::tls::verify_full_with_wrong_server_name_reports_name_mismatch::<$engine> } $crate::__conformance_case! { true, + $engine as TlsConformanceEngine, require_connects_without_ca, $crate::conformance::cases::tls::require_connects_without_ca::<$engine> } $crate::__conformance_case! { true, + $engine as TlsConformanceEngine, disable_against_tls_only_server_is_refused, $crate::conformance::cases::tls::disable_against_tls_only_server_is_refused::<$engine> } @@ -125,6 +130,7 @@ macro_rules! __tls_identity_case { (true, $engine:ty) => { $crate::__conformance_case! { true, + $engine as TlsConformanceEngine, client_identity_is_required_by_the_server, $crate::conformance::cases::tls::client_identity_is_required_by_the_server::<$engine> } diff --git a/linux/crates/test-support/src/fake_connection.rs b/linux/crates/test-support/src/fake_connection.rs index 0f589a4e19..b70706a73a 100644 --- a/linux/crates/test-support/src/fake_connection.rs +++ b/linux/crates/test-support/src/fake_connection.rs @@ -1,4 +1,4 @@ -use std::sync::atomic::AtomicUsize; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use async_trait::async_trait; @@ -28,6 +28,7 @@ pub struct FakeConnection { dialect: Arc, read_only: bool, close_behaviour: CloseBehaviour, + close_started: AtomicBool, link: LinkWatch, closed: CancellationToken, counters: CallCounters, @@ -40,6 +41,7 @@ impl FakeConnection { dialect: script.dialect.clone(), read_only: script.read_only, close_behaviour: script.close, + close_started: AtomicBool::new(false), script: Mutex::new(script), link: LinkWatch::new(), closed: CancellationToken::new(), @@ -375,7 +377,7 @@ impl Connection for FakeConnection { async fn close(&self) -> Result<(), DriverError> { CallCounters::record(&self.counters.close); - let first = !self.closed.is_cancelled(); + let first = !self.close_started.swap(true, Ordering::SeqCst); self.closed.cancel(); if first { self.close_behaviour.run().await; @@ -399,6 +401,7 @@ fn apply_limit(mut result: QueryResult, limit: RowLimit) -> QueryResult { mod tests { use std::time::Duration; + use futures::future::BoxFuture; use tablepro_core::LossPhase; use tablepro_core::meta::ObjectKind; use tablepro_core::statement::{Statement, StatementBuilder}; @@ -406,6 +409,7 @@ mod tests { use tokio::time::Instant; use super::*; + use crate::close_race::{ROUNDS, pending_closes}; fn select(sql: &str) -> Statement { let mut builder = StatementBuilder::new(); @@ -475,6 +479,26 @@ mod tests { assert_eq!(again.elapsed(), Duration::ZERO, "a second close waited again"); } + fn close_connection(connection: &FakeConnection) -> BoxFuture<'_, ()> { + Box::pin(async move { drop(connection.close().await) }) + } + + #[test] + fn concurrent_closes_run_the_close_behaviour_once() { + let pending = pending_closes( + ROUNDS, + || { + FakeConnection::new(ConnectionScript { + close: CloseBehaviour::Hang, + ..ConnectionScript::default() + }) + }, + close_connection, + ); + + assert!(pending.iter().all(|&hung| hung == 1), "{pending:?}"); + } + #[tokio::test(start_paused = true)] async fn a_bounded_reply_stops_at_the_deadline_and_a_late_one_does_not() { let mut script = ConnectionScript::default(); diff --git a/linux/crates/test-support/src/fake_editor_session.rs b/linux/crates/test-support/src/fake_editor_session.rs index 099c56727d..d51d1ec195 100644 --- a/linux/crates/test-support/src/fake_editor_session.rs +++ b/linux/crates/test-support/src/fake_editor_session.rs @@ -179,10 +179,10 @@ impl EditorSession for FakeEditorSession { } async fn close(&self) { - self.closes.fetch_add(1, Ordering::SeqCst); + let first = self.closes.fetch_add(1, Ordering::SeqCst) == 0; self.closing.cancel(); *lock(&self.state) = EditorSessionState::Lost; - if self.closes.load(Ordering::SeqCst) == 1 { + if first { self.close_behaviour.run().await; } self.closed.store(true, Ordering::SeqCst); @@ -195,10 +195,12 @@ mod tests { use std::time::Duration; use async_trait::async_trait; + use futures::future::BoxFuture; use tablepro_core::value::Value; use tablepro_core::{ResultEvent, RowLimit}; use super::*; + use crate::close_race::{ROUNDS, pending_closes}; #[derive(Default)] struct Discard; @@ -317,4 +319,19 @@ mod tests { assert_eq!(session.close_count(), 2); assert_eq!(session.state(), EditorSessionState::Lost); } + + fn close_session(session: &FakeEditorSession) -> BoxFuture<'_, ()> { + session.close() + } + + #[test] + fn concurrent_closes_run_the_close_behaviour_once() { + let pending = pending_closes( + ROUNDS, + || FakeEditorSession::with_close(Replies::default(), CloseBehaviour::Hang), + close_session, + ); + + assert!(pending.iter().all(|&hung| hung == 1), "{pending:?}"); + } } diff --git a/linux/crates/test-support/src/lib.rs b/linux/crates/test-support/src/lib.rs index e5b770ba83..f02954726a 100644 --- a/linux/crates/test-support/src/lib.rs +++ b/linux/crates/test-support/src/lib.rs @@ -8,6 +8,8 @@ use std::sync::{Mutex, MutexGuard, PoisonError}; mod call_counters; mod close_behaviour; +#[cfg(test)] +mod close_race; pub mod conformance; mod connection_script; mod fake_connection; diff --git a/linux/docs/adding-drivers.md b/linux/docs/adding-drivers.md index 6bb46903aa..4c4a95f034 100644 --- a/linux/docs/adding-drivers.md +++ b/linux/docs/adding-drivers.md @@ -1,239 +1,382 @@ # Adding a database driver -This is the canonical contributor task. Every database engine TablePro Linux supports has a driver crate under `crates/drivers/`. There is no plugin system: a driver is a Rust crate, statically linked, registered in one place at startup. See [decisions/0001-no-plugin-system.md](decisions/0001-no-plugin-system.md) for why. +Every engine is one crate under `crates/drivers/`, linked into the app at compile time and registered in one place. There is no plugin system ([ADR 0001](decisions/0001-no-plugin-system.md)). The contract a driver implements lives in `tablepro-core` and is described here; [ADR 0016](decisions/0016-driver-contract-v2.md) records why it has this shape. How each shipped engine meets it: [PostgreSQL](drivers/postgresql.md), [MySQL and MariaDB](drivers/mysql.md), [SQLite](drivers/sqlite.md), [SQL Server](drivers/sql-server.md), [ClickHouse](drivers/clickhouse.md). -End to end, adding a driver is six steps: +The work, in order: -1. Pick the underlying Rust library -2. Create a new crate -3. Implement `core::DatabaseDriver` and `core::Connection` -4. Add the crate to the workspace -5. Register the driver in `app::main` -6. Add tests +1. Choose the client library. +2. Create the crate. +3. Implement `DatabaseDriver` and its capabilities. +4. Implement `Connection` on `OwnedCalls`. +5. Write saves through a `WriteLedger`. +6. Implement `EditorSession`. +7. Implement `SqlDialect`, and `DdlDialect` where the app edits the engine's structure. +8. Register the driver. +9. Add a container fixture, the conformance suites and the engine's own tests. +10. Write the engine's page in `docs/drivers/`. -Each step is small. The whole task takes between half a day (PG-shaped engines) and a week (Oracle-shaped engines that need C FFI). +## 1. The client library -## 1. Pick the Rust library +The library must: -| Engine | Recommended crate | Notes | -|---|---|---| -| PostgreSQL | `sqlx` with `runtime-tokio` + `tls-rustls` + `postgres` | Fully async, prepared statements, streaming. | -| MySQL / MariaDB | `sqlx` with `mysql` feature | Same shape as PostgreSQL. | -| SQLite | `sqlx` with `sqlite` feature | File-based, no network. | -| MSSQL | `tiberius` | Pure Rust TDS. Watch governance — `praxiomlabs/rust-mssql-driver` is a credible alternative. | -| Oracle | `oracle` (rust-oracle, kubo) | Wraps ODPI-C. Requires Oracle Instant Client on the build host. | -| ClickHouse | official `clickhouse` crate | HTTP interface (8123). Dynamic results streamed via `FORMAT JSONCompactEachRowWithNamesAndTypes`. | -| Redis | `fred` | Modern tokio rewrite of redis-rs. | -| MongoDB | official `mongodb` | Mature, OpenTelemetry support. | -| DuckDB | `duckdb` (official) | Bundled native lib, edition 2024. | -| Cassandra / Scylla | `scylla` | Cassandra-compatible, shard-aware. | -| DynamoDB | `aws-sdk-dynamodb` | Type-safe AWS SDK. | -| BigQuery | `google-cloud-bigquery` (third-party) | No first-party Google SDK. | - -If the engine is not listed, open an issue first and discuss the crate choice before writing code. +- be maintained and published on crates.io, used without a fork ([ADR 0005](decisions/0005-supply-chain-policy.md)); `deny.toml` allows no git source; +- use rustls with aws-lc-rs, or no TLS of its own ([ADR 0009](decisions/0009-crypto-provider-and-sources.md)); +- either take a stream the driver hands it, or dial a TCP endpoint and a Unix socket path itself, because an SSH forward is a Unix socket ([Transports](../ARCHITECTURE.md#transports)); +- have a way to stop a statement already sent: a cancel request, a KILL, an attention, an interrupt; +- let the driver keep a connection clean for the next caller, with no server-side state it did not ask for. -## 2. Create the crate +What the shipped drivers use: -Convention: `crates/drivers//`. +| Engine | Client | Pool | Reaches the server through | +|---|---|---|---| +| PostgreSQL | tokio-postgres 0.7.18, tokio-postgres-rustls 0.14 | deadpool-postgres 0.14.2 | `Transport::open` | +| MySQL, MariaDB | sqlx 0.9 (`mysql`) | deadpool 0.13 | `Transport::route`, dialled by sqlx | +| SQLite | sqlx 0.9 (`sqlite-unbundled`) | sqlx, one connection | a file path | +| SQL Server | tiberius 0.13 | deadpool 0.13 | `Transport::open` | +| ClickHouse | clickhouse 0.15 over hyper-util and hyper-rustls | hyper's connection pool | `Transport::open` | -```bash -cd crates/drivers -cargo new --lib clickhouse -cd clickhouse -``` +Open an issue before starting an engine that is not listed, and say which library and why. -The crate is named `tablepro-driver-` in `Cargo.toml`. The library crate name is `drivers_` (Rust-conventional underscore). +## 2. The crate -Skeleton `Cargo.toml`: +`crates/drivers//`, package `tablepro-driver-`, library `drivers_`: ```toml [package] -name = "tablepro-driver-clickhouse" -version = "0.1.0" -edition = "2024" +name = "tablepro-driver-" +edition.workspace = true +rust-version.workspace = true +license.workspace = true publish = false [lib] -name = "drivers_clickhouse" +name = "drivers_" path = "src/lib.rs" [dependencies] tablepro-core = { path = "../../core" } -async-trait = "0.1" -clickhouse = { version = "0.15", default-features = false, features = ["rustls-tls"] } -tokio = { version = "1", features = ["rt", "macros", "net", "time"] } -thiserror = "2" +tablepro-net = { path = "../../net" } +async-trait.workspace = true +tokio = { workspace = true, features = ["macros", "rt", "sync", "time"] } +tokio-util = { workspace = true, features = ["rt"] } +tracing.workspace = true + +[dev-dependencies] +tablepro-test-fixtures = { path = "../../test-fixtures" } +tablepro-test-support = { path = "../../test-support" } +tokio = { workspace = true, features = ["macros", "rt-multi-thread", "test-util"] } + +[lints] +workspace = true ``` -Do not add unrelated dependencies. Do not depend on `gtk4`, `libadwaita`, or any other workspace crate except `tablepro-core`. +A driver depends on `tablepro-core`, and on `tablepro-net` when it speaks TLS or dials TCP. It never depends on GTK, on `tablepro-session`, on `tablepro-ssh` or on another driver. Add the crate to `members` in `linux/Cargo.toml`. -## 3. Implement the traits - -Two traits, both defined in `tablepro-core`: +## 3. `DatabaseDriver` and capabilities ```rust -#[async_trait::async_trait] +#[async_trait] pub trait DatabaseDriver: Send + Sync { fn id(&self) -> &'static str; fn display_name(&self) -> &'static str; - fn default_port(&self) -> u16; - async fn connect(&self, opts: ConnectOptions) -> Result, DriverError>; + fn capabilities(&self) -> &'static DriverCapabilities; + async fn connect( + &self, + options: ConnectOptions, + cancel: CancellationToken, + ) -> Result, DriverError>; } +``` -#[async_trait::async_trait] -pub trait Connection: Send + Sync { - async fn list_tables(&self) -> Result, DriverError>; - async fn fetch_columns(&self, table: &str) -> Result, DriverError>; - async fn fetch_rows(&self, table: &str, offset: u64, limit: u64) -> Result; - async fn execute(&self, sql: &str) -> Result; - async fn ping(&self) -> Result<(), DriverError>; - async fn close(self: Box) -> Result<(), DriverError>; -} +`id()` is written into saved connections and never changes once shipped. The ids in use are `postgres`, `mysql`, `sqlite`, `mssql` and `clickhouse`. + +### Capabilities + +`DriverCapabilities` is a `static` the driver returns before any connection exists. The connect dialog and `DriverConnector` read `endpoint`, `auth_modes`, `tls`, `read_only` and `database_required`. The conformance suites hold `endpoint`, `dml_writes` and the TLS client certificate flag to what the engine does. + +| Field | Meaning | +|---|---| +| `endpoint` | `Network { default_port }` or `File`. A file engine gets a path row instead of host, port, TLS and SSH. | +| `auth_modes` | The `AuthMode`s the driver can log in with. `Kerberos` only where `connect` maps it to a real integrated login. | +| `tls` | `TlsCapabilities`: the `TlsMode`s offered, whether a CA file and a client certificate are accepted, and where a server name override works (`ServerNameOverride`). `None` for an engine with no TLS. | +| `read_only` | How a read-only connection is held: `EngineEnforced` (the engine refuses every write), `SessionDefault` (a statement can lift it) or `LoginPrivileges` (only the login's grants hold). The connect dialog says which one the user gets. | +| `dml_writes`, `ddl_writes` | `WriteAtomicity` of row changes and of schema changes: `Transactional` or `StatementByStatement`. | +| `script_execution` | `NativeBatch` (the engine runs a script) or `SingleStatement` (one statement per run). | +| `commit_resolution` | How an unanswered commit can be resolved: `ServerTransactionStatus`, `QueryLog` or `Unavailable`. | +| `row_count_estimate` | `Statistics` ("about N"), `ExactMetadata` or `Unavailable`. | +| `object_kinds` | The `ObjectKind`s the sidebar can list. | +| `database_required` | Whether a connect without a database name is refused before anything is opened. | + +The shipped drivers: + +| | PostgreSQL | MySQL | SQLite | SQL Server | ClickHouse | +|---|---|---|---|---|---| +| `endpoint` | 5432 | 3306 | `File` | 1433 | 8123 | +| `auth_modes` | Password | Password | none | Password, Kerberos | Password | +| TLS modes | all three | all three | none | all three | all three | +| CA file | yes | yes | | yes | yes | +| Client certificate | yes | yes | | no | yes | +| Server name override | `AnyRoute` | `SshForwardOnly` | | `AnyRoute` | `AnyRoute` | +| `read_only` | `SessionDefault` | `SessionDefault` | `EngineEnforced` | `LoginPrivileges` | `EngineEnforced` | +| `dml_writes` | `Transactional` | `Transactional` | `Transactional` | `Transactional` | `StatementByStatement` | +| `ddl_writes` | `Transactional` | `StatementByStatement` | `Transactional` | `Transactional` | `StatementByStatement` | +| `script_execution` | `NativeBatch` | `NativeBatch` | `NativeBatch` | `NativeBatch` | `SingleStatement` | +| `commit_resolution` | `ServerTransactionStatus` | `Unavailable` | `Unavailable` | `Unavailable` | `QueryLog` | +| `row_count_estimate` | `Statistics` | `Statistics` | `Statistics` | `Statistics` | `ExactMetadata` | +| `database_required` | no | yes | no | no | no | + +SQL Server offers Kerberos only when built with the `kerberos` feature. MySQL's `dml_writes` is what InnoDB gives; the driver reads each table's storage engine before a save and writes statement by statement on one that cannot roll back. + +### Connecting + +`ConnectOptions` has no default, so a connection opened with a made-up host or timeout does not compile: + +- `target`: `ConnectTarget::Network(NetworkTarget)` or `ConnectTarget::File(FileEndpoint)`. A driver refuses the kind it does not serve with `DriverError::Unsupported`. +- `workspace_settings`: `WorkspaceSettings` with `read_only`, `statement_timeout` (60 s) and `lock_timeout` (10 s). +- `liveness`: `LivenessPolicy::DESKTOP` in the app; the conformance suite uses shorter values. +- `environment`: `ConnectEnvironment` with the app's version, sent next to `APPLICATION_NAME` where the protocol has room. + +A `NetworkTarget` carries `transport`, `credentials`, `database` and `tls`. `connect`: + +1. validates `tls` with `TlsConfig::validate(caps, transport.kind().class())` and refuses `Credentials` whose mode the driver does not support; +2. builds its TLS from `tls` (see [Trust](../ARCHITECTURE.md#trust)) and checks the certificate against `NetworkTarget::tls_server_name()`, never against the socket's address; +3. opens every stream with `transport.open()`, or dials `transport.route()` when the library dials itself; +4. logs in under `liveness.login_timeout`, with `APPLICATION_NAME` as the client name; +5. opens the first pool member before returning, so a refused password or a failed handshake is the connect's error, and gives up as soon as `cancel` fires; +6. reads `ServerInfo` and builds the dialect for that server version. + +A `ConnectionLost` during login goes through `Transport::explain_closed_stream`, which turns "the stream closed" into "the SSH master exited" where that is what happened. + +## 4. `Connection` + +The `Connection` is a workspace's link to one server: the pool behind the sidebar, the grid and every save. Every call takes `CallOptions`, a deadline and a `CancellationToken` that travel together: a deadline with no cancel leaves the server working after the app gave up, and a cancel with no deadline hangs on a server that never answers. + +| Method | Contract | +|---|---| +| `server_info`, `dialect`, `read_only` | Fixed at connect. | +| `link_state`, `lost` | A `LinkWatch` shared by every call. `lost()` resolves when the link is gone: a failed probe, or a failed call that a check outside the pool confirmed. One dead backend in the pool does not resolve it, so the session monitor can act on it. | +| `probe` | Whether the server answers. It never waits for a pool slot: with every slot busy it opens a connection of its own. | +| `list_schemas`, `list_objects`, `fetch_columns`, `fetch_indexes`, `fetch_foreign_keys`, `fetch_table_definition`, `row_identity`, `table_estimate` | Catalogue reads. `fetch_table_definition` reads columns, keys and dependents in one call so they come from the same moment. | +| `query_bound` | A generated read that keeps at most `limit` rows and reads one more to set `truncated`. | +| `count_rows` | A generated count. | +| `stream_read` | Every row of a read into a `ResultSink`, with no row limit, on a connection of its own, held read-only for its whole length. | +| `write_batch` | A save, refused before anything is sent on a read-only connection or when it asks to suspend foreign keys on an engine that cannot. | +| `resolve_commit` | What became of a commit whose answer never arrived. | +| `open_editor_session` | A tab's own connection, which outlives this one. | +| `close` | Stops every call and releases the pool. Idempotent, and never closes an editor session. | + +### `OwnedCalls` + +Dropping a future does not stop a server. Each call runs on a task the connection owns through `OwnedCalls::run_read` or `run_write`: + +```rust +self.calls + .run_read(options, PgCall::new(), stop.clone(), move |mut call, request| async move { + if let Err(error) = call.acquire(&pool, &timeouts, &request, &work_stop).await { + return (call, Err(error)); + } + let result = flight(&call, &begin, op).await; + (call, result) + }) + .await ``` -A driver crate exports two types: +- The work gets the `CallResource` and a `StopRequest`, and hands the resource back with its result. +- The supervisor races the work against the caller's cancel, the deadline and `close`. The first one to fire sets the `StopReason` (`Cancelled`, `DeadlineElapsed`, `Shutdown`) on the request. +- Dropping the caller's future is a cancel: the call keeps running on its task until the stop has settled. +- After a stop the work has `LivenessPolicy::cancel_grace` (5 s) to finish. Past it the call is abandoned: the resource's abandon flag is set, a read returns `ConnectionLost`, and a write returns what its `WriteLedger` saw. +- In `run_read`, an error the engine raised to acknowledge the stop reads as what asked for it: `Cancelled`, or `Timeout { phase: Statement, server_cancelled: true }`. `run_write` passes the work's result through, so a write maps an acknowledged stop itself, as `acknowledged_stop` does in the PostgreSQL driver's `write_batch.rs` and the SQL Server driver's `cancel.rs`. -- A zero-sized `*Driver` struct that implements `DatabaseDriver`. -- A connection struct (typically wrapping a connection pool from the underlying crate) that implements `Connection`. +Start the resource empty and fill it inside the work, as `PgCall::acquire` and `MysqlCall::acquire` do, so the deadline and the cancel cover the wait for a pool slot. Arm the engine stop once a connection is held, then check `StopRequest::is_requested` again: a stop asked for in between sent nothing. -Skeleton `src/lib.rs`: +### `EngineStop` and `StopDelivery` -```rust -use async_trait::async_trait; -use tablepro_core::{ - Connection, ConnectOptions, DatabaseDriver, DriverError, - ColumnInfo, ExecResult, QueryResult, TableInfo, -}; +`EngineStop` is the engine's own way of stopping work already sent: -pub struct ClickhouseDriver; +- `delivery()` says how the stop reaches the server. `OutOfBand(StopScope::Statement)` names the statement, so it cannot land on anything else. `OutOfBand(StopScope::Connection)` names the connection, so a late one can hit whatever that connection runs next. `InBand` means the running request carries it and the work sends it itself. +- `stop()` sends an out-of-band stop. `OwnedCalls` calls it at most once and polls it ahead of the work, since work waiting on the server cannot finish until the server has been told. +- `acknowledged(error)` says whether an error is the engine confirming the stop. -#[async_trait] -impl DatabaseDriver for ClickhouseDriver { - fn id(&self) -> &'static str { "clickhouse" } - fn display_name(&self) -> &'static str { "ClickHouse" } - fn default_port(&self) -> u16 { 8123 } - - async fn connect(&self, opts: ConnectOptions) -> Result, DriverError> { - let client = build_client(opts).await?; - Ok(Box::new(ClickhouseConnection { client })) - } -} +| Engine | Stop | Delivery | +|---|---|---| +| PostgreSQL | Cancel request over a new stream from the transport (`CancelToken::cancel_query_raw`) | `OutOfBand(Connection)` | +| MySQL | `KILL QUERY ` on a control connection over the same route | `OutOfBand(Connection)` | +| SQLite | The call's flag, read by a progress handler every 1,000 VM instructions | `OutOfBand(Statement)` | +| SQL Server | TDS attention on the request's own connection (`Client::cancel_query`) | `InBand` | +| ClickHouse | `KILL QUERY WHERE query_id = ? ASYNC` on a sessionless client | `OutOfBand(Statement)` | -struct ClickhouseConnection { - client: clickhouse::Client, -} +### `CallResource` release rules -#[async_trait] -impl Connection for ClickhouseConnection { - async fn list_tables(&self) -> Result, DriverError> { /* ... */ } - async fn fetch_columns(&self, table: &str) -> Result, DriverError> { /* ... */ } - async fn fetch_rows(&self, table: &str, offset: u64, limit: u64) -> Result { /* ... */ } - async fn execute(&self, sql: &str) -> Result { /* ... */ } - async fn ping(&self) -> Result<(), DriverError> { /* ... */ } - async fn close(self: Box) -> Result<(), DriverError> { /* ... */ } -} -``` +`CallResource` is the connection one call holds: `abandon_flag()` is what the pool's recycle and the resource's `Drop` read, and `discard()` takes the connection out of its pool and closes it. After the work returns: -Notes: +| Stop delivery | The call was stopped | Released to the pool when | Otherwise | +|---|---|---|---| +| `OutOfBand(Connection)` | yes | the engine acknowledged the stop and the stop itself settled | discarded | +| any | no, or not connection-scoped | the result is not `ConnectionLost` | discarded | +| any | abandoned past the grace | never | the abandon flag is set and the resource drops out of the pool | -- The `id()` is the stable string used in saved connection files. Once shipped, never change it. Pick something obvious and short (`postgres`, `mysql`, `clickhouse`). -- `default_port()` is what the connection dialog pre-fills. -- `DriverError` is a `thiserror` enum in `tablepro-core`. Map underlying crate errors into the variants. Add a new variant only after PR discussion. +A connection-scoped stop holds its connection until it settles because it names the connection, not the statement: released early, a cancel request still in flight would stop the next caller's statement on that connection. `OwnedCalls` waits for the stop up to the end of the grace. A stop that has not settled by then, or that the server never acknowledged, costs the connection rather than risking someone else's statement. -`DatabaseDriver` also has defaulted hooks for engines that break an assumption the app otherwise makes. Override one only when the default is wrong for your engine: +The driver adds its own rules on top. PostgreSQL taints a call whose transaction did not end with an acknowledged COMMIT or ROLLBACK, because deadpool's recycle check passes inside a half-finished transaction. MySQL taints a call stopped between statements, since a `KILL` may be on its way to an idle connection. SQL Server discards a connection whose attention was not acknowledged at a token boundary. -- `ddl_is_transactional()`: the structure editor batches DDL into one transaction when true. False for engines that commit implicitly on every DDL statement. -- `reports_rows_affected()`: the inline-edit Save path reads a zero `rows_affected` on an UPDATE or DELETE as another session having changed the row. Return false if the engine cannot produce a count, or every successful save warns about a lost update. -- `is_file_based()`: the connect dialog hides host, port, TLS, the Authentication group and SSH, and relabels Database to File path. True only for engines that open a local file. -- `supports_integrated_auth()`: the connect dialog shows the Method selector (Password / Windows (Kerberos)) only for drivers returning true, and while Kerberos is selected it hides the username and password rows and sends empty credentials. Return true only if `connect()` maps `AuthMode::Kerberos` onto a real integrated-auth path that reads the ambient Kerberos ticket cache; `connection_service::establish` refuses the mode for every other driver. +### Deciding that the link is lost -If your engine needs a different SQL spelling for a statement the app builds centrally, add the dialect branch in `core::sql_dialect` (`quote_ident`, `placeholder_for`, `build_update`, `build_order_and_pagination`) rather than rewriting the SQL inside the driver. ClickHouse takes `build_update`'s `ALTER TABLE … UPDATE` branch for this reason. +A call that times out or loses its connection may have met a dead route or only a dead backend. Each network driver decides in its `settle`: a stop that never got through means the route is gone; one the server answered means it is not; otherwise a connection of its own answers within the cancel grace. Only a confirmed loss marks the `LinkWatch` lost and turns a statement timeout into `ConnectionLost`. -## 4. Add the crate to the workspace +## 5. Writes and the `WriteLedger` -Edit `linux/Cargo.toml`: +A `WriteBatch` is a `WriteKind` (`RowChanges` or `SchemaChanges`), the `TableRef` it changes, the `GuardedStatement` steps and a `ForeignKeyMode`. A row change runs under the workspace statement timeout where the engine can stop one safely; a schema change never does. Both stay bounded by the call's deadline and cancel. -```toml -[workspace] -members = [ - "crates/app", - "crates/core", - "crates/storage", - "crates/drivers/postgres", - "crates/drivers/mysql", - "crates/drivers/sqlite", - "crates/drivers/clickhouse", # add this -] +Each step is a statement and a `RowGuard`: `None`, `AffectedExactly(n)`, or `ProbeExactly { probe, rows }`, which counts the rows with a `SELECT` before the statement runs. A mismatch means the row is not the one that was loaded, and the batch stops with `RowGuardFailed`. An engine that cannot report an affected count refuses `AffectedExactly` with `GuardUnsupported` before anything is sent. + +The work records each step as it goes: + +```rust +ledger.statement_sending(index); +let affected = execute(step).await?; +ledger.statement_confirmed(index); +// after the last step +ledger.commit_sending(Some(token)); ``` -Run `cargo check --workspace` from `linux/`. The new crate must compile in isolation against `core`. +`WriteLedger::stop_permitted` holds the engine's stop while it would turn a knowable outcome into an unknown one: on a `Transactional` engine once COMMIT is on the wire, on a `StatementByStatement` engine while a statement is in flight. + +When a write is interrupted, `WriteLedger::interrupted(cause, fate)` gives the outcome from what was sent and from `SessionFate` (`RolledBack`, `Discarded`, `RollbackIncomplete { executed }`): -## 5. Register the driver +| Atomicity | How far it got | Outcome | +|---|---|---| +| `Transactional` | COMMIT sent | `CommitOutcomeUnknown { token }` | +| `Transactional` | COMMIT not sent, session rolled back or discarded | `RolledBack` at the statement in flight or the next one | +| `Transactional` | COMMIT not sent, rollback incomplete | `PartiallyApplied { applied: executed }` | +| `StatementByStatement` | nothing in flight, nothing confirmed | `RolledBack` at statement 0 | +| `StatementByStatement` | nothing in flight, some confirmed | `PartiallyApplied { applied: confirmed }`, the next statement `Failed` | +| `StatementByStatement` | one in flight, and nothing confirmed or a commit token held | `CommitOutcomeUnknown { token }` | +| `StatementByStatement` | one in flight, some confirmed, no token | `PartiallyApplied { applied: confirmed }`, that statement `Unknown` | -Edit `crates/app/src/main.rs`: +A discarded session leaves nothing behind on a transactional engine, because a COMMIT that was never written cannot land once the session is gone. `WriteLedger::failed` and `WriteLedger::guard_failed` give the outcome of a statement the server refused and of a guard mismatch, with the same rules. The full table is the `outcome_table` test in `crates/core/src/write_ledger.rs`. -```rust -use tablepro_driver_clickhouse::ClickhouseDriver; +A transactional COMMIT that the server answers with an error rolled back; only a COMMIT with no answer is `CommitOutcomeUnknown`. + +### `resolve_commit` + +Where the engine can answer later, the ledger carries a `CommitToken`: + +- `PostgresXid { xid, legacy_txid }`: `pg_current_xact_id()` (or `txid_current()` before PostgreSQL 13), read inside the transaction before COMMIT, and asked back with `pg_xact_status` (or `txid_status`). +- `ClickHouseQuery { query_ids, first_unconfirmed }`: the query id of every statement, looked up in `system.query_log`. +`resolve_commit` answers with a `CommitResolution`: `Applied`, `NotApplied`, `PartiallyApplied`, `StillRunning` or `Unresolvable`. Where `commit_resolution` is `Unavailable`, SQL Server and SQLite answer `Unresolvable` and MySQL returns `Unsupported`. The app reports `CommitOutcomeUnknown` as unknown and asks the user to reload; it does not call `resolve_commit` yet. + +## 6. `EditorSession` + +An editor tab holds one session for as long as it is open, because the state a user builds there lives on that connection and nowhere else: an open transaction, a temporary table, a session setting. + +- It is never pooled and never reopened behind the user's back. A lost connection is `EditorSessionState::Lost`, and the tab opens a new session on its next run. +- One run at a time: a second `run` fails with `EditorSessionBusy`. +- `run(sql, options, sink)` takes `UserSql` as typed and `RunOptions`: `row_limit`, `deadline` and `cancel`. A caller stops a run through `RunOptions::cancel`. +- Only PostgreSQL and ClickHouse arm a drop guard on that token, so a dropped caller is a cancel there. MySQL, SQL Server and SQLite await the run inline: a dropped caller sends no `KILL`, attention or interrupt, and the session reports `Running` from then on. A new driver arms the guard, as `PgEditorSession::run` does. +- The app's editor splits a script with the engine's grammar and sends one statement per run, or one `GO` batch on SQL Server. A driver still runs whatever text it is given, a whole script included. +- A statement the server refused ends the run with `RunEnd::Failed` in the `RunSummary`; an `Err` from `run` means the session itself failed. +- `RunSummary` carries `elapsed`, the `RunEnd` (`Completed`, `Failed`, `Cancelled`, `TimedOut { server_cancelled }`) and the `TransactionState` after the run. `Unknown` is an answer of its own: the tab's close guard warns instead of guessing. +- `close` stops a running statement, then closes the connection within `close_timeout`. + +### `ResultSink` and backpressure + +A run streams `ResultEvent`s into a `ResultSink`: + +- `ResultSetStarted { columns }` opens a result set. +- `ColumnsResolved { set_index, columns }` replaces the columns of a set whose types the driver could only learn from its rows (SQLite). +- `Rows` carries a batch of rows, 256 in the shipped drivers. +- `RowLimitReached { retained }` comes once per set, before `ResultSetFinished`. +- `ResultSetFinished { retained, total_rows, truncation }` closes the set. `Truncation` is `Complete`, `ServerStopped` or `ReadPastLimit`, and `total_rows` is `None` when the driver did not read the rest. +- `CommandCompleted { rows_affected }` reports a statement that returned no rows; `None` is "not reported", not zero. +- `Notice` passes on a server notice or warning. + +`ResultSink::accept` is awaited before the next rows are read off the wire, so a grid that cannot keep up slows the read instead of growing a queue the size of the result. `SinkFlow::ConsumerGone` ends the run with `RunEnd::Cancelled`. PostgreSQL, SQL Server and ClickHouse then stop the statement on the server, and SQLite stops stepping it. MySQL sends no `KILL`: sqlx reads the rest of the result off the wire before the next statement. `stream_read` uses the same sink and events, without a row limit. + +## The row-limit rule + +`RowLimit` wraps a `NonZeroU64`; `RowLimit::EDITOR_DEFAULT` is 10,000. + +- `query_bound` keeps at most the limit and reads one more row to mark the result truncated. +- An editor run keeps at most the limit in each result set and sends `RowLimitReached` once. A set the server cut at one row past the limit ends with `Truncation::ServerStopped` and no `total_rows`; only MySQL reports it, when `sql_select_limit` stopped the set. A set read past the limit ends with `Truncation::ReadPastLimit`, on every engine. Where the engine allows, the rest of the set is stopped on the server rather than read: a `FETCH` of one row past the limit (PostgreSQL), `sql_select_limit` (MySQL), `KILL QUERY` (ClickHouse). SQLite stops stepping, so the rest is never produced. Otherwise the rows past the limit are counted without being decoded (SQL Server, and MySQL where `sql_select_limit` does not apply). +- The limit never ends the script, the user's transaction or the statements after the limited one. The conformance case `row_limit_keeps_transaction_and_later_statements` checks this on every engine with editor transactions. +- The driver never rewrites the user's SQL to add a `LIMIT`. +- `stream_read` has no limit. + +## 7. Dialects + +`Connection::dialect()` returns the engine's `SqlDialect`, built from `ServerInfo` at connect. Every builder in `tablepro-core` (pages, filters, row edits, exports) goes through it: quoting, placeholders, literals, the page window, keyset comparison, the engine's row address and type parsing. A dialect never reads session state. `SqlDialect::ddl()` returns the engine's `DdlDialect`, or `None` where the app does not edit the engine's structure. Never match on a driver id in core; add a dialect method. + +## 8. Registration + +Add the crate to `crates/app/Cargo.toml` and register it in `build_registry` in `crates/app/src/lib.rs`: + +```rust fn build_registry() -> DriverRegistry { let mut r = DriverRegistry::new(); - r.register(Arc::new(drivers_postgres::PgDriver)); + r.register(Arc::new(drivers_clickhouse::ClickhouseDriver)); + r.register(Arc::new(drivers_mssql::MssqlDriver)); r.register(Arc::new(drivers_mysql::MysqlDriver)); + r.register(Arc::new(drivers_postgres::PgDriver)); r.register(Arc::new(drivers_sqlite::SqliteDriver)); - r.register(Arc::new(ClickhouseDriver)); // add this r } ``` -Update `crates/app/Cargo.toml` to depend on the new driver crate. **This step is the one most often forgotten.** The driver crate compiles fine without it; the app simply does not know the driver exists. +Without this step the crate compiles and the app never offers it. -## 6. Tests +## 9. Tests -Two test layers, both required for merge: +The layout every driver follows: -**Unit tests** — in `src/lib.rs` `#[cfg(test)]` module. Exercise pure logic: SQL builders, type mappers, error mapping. Do not require a running database. +- Unit tests next to the code, for pure logic: the dialect, error mapping, decoders, statement classification. +- `tests/integration/main.rs` with one module per topic: `support.rs` (start a fixture, connect, run), `conformance_engine.rs`, `tls.rs` for a network engine, and the engine's own behaviour (`metadata.rs`, `write_batch.rs`, `cancellation.rs` and so on). A network engine's test starts its own container and is `#[ignore = "requires docker"]`. +- A container fixture in `crates/test-fixtures` for anything the default image cannot do: TLS from the generated PKI, a second principal that logs in with a client certificate. -**Integration tests** — in `tests/integration.rs`. Use [testcontainers-rs](https://crates.io/crates/testcontainers) to spin up a real instance: +The conformance suites are the contract's own tests. The driver's `conformance_engine.rs` implements `ConformanceEngine` and the engine traits its flags need, then invokes: ```rust -use testcontainers::clients::Cli; -use testcontainers::images::generic::GenericImage; - -#[tokio::test] -async fn list_tables_returns_seeded_tables() { - let docker = Cli::default(); - let image = GenericImage::new("clickhouse/clickhouse-server", "latest") - .with_exposed_port(8123); - let node = docker.run(image); - let port = node.get_host_port_ipv4(8123); - - let driver = ClickhouseDriver; - let conn = driver.connect(ConnectOptions { - host: "127.0.0.1".into(), - port, - username: "default".into(), - password: "".into(), - database: "default".into(), - ..Default::default() - }).await.unwrap(); - - conn.execute("CREATE TABLE foo (id Int32) ENGINE=Memory").await.unwrap(); - let tables = conn.list_tables().await.unwrap(); - assert!(tables.iter().any(|t| t.name == "foo")); +tablepro_test_support::connection_conformance_suite! { + engine: PgEngine, + docker: true, + endpoint: network, + pool: pooled, + editor_transactions: true, + unsupported_types: true, + bounded_ddl: true, } ``` -Integration tests run in CI on the Linux runner, gated behind `--ignored` so contributors without Docker can still run `cargo test`. +A network engine's `tls.rs` implements the TLS engine and invokes the TLS suite: -## Checklist for the PR +```rust +tablepro_test_support::tls_conformance_suite! { + engine: PgTlsEngine, + client_identity: true, +} +``` + +The cases cover listing and identity, deadlines within the cancel grace, a cancelled editor run, a dropped `query_bound` caller stopping the server's statement, a lost transport, a busy pool's probe, read-only refusal, write atomicity, row guards, the row limit, unbounded schema changes, streaming in bounded batches, type fallback and every TLS mode. [testing.md](testing.md#conformance) lists them with their flags. `suite_flags_match_driver_capabilities` runs without Docker and fails when a flag disagrees with the driver's capabilities, so a case cannot be skipped by declaring less. + +Behaviour the suites do not cover gets a test of its own against the real engine: the engine's stop, its error mapping, what a pooler does to it, each write outcome the ledger can produce. + +For a new package with Docker tests, add a test group and override in `.config/nextest.toml` and an entry in the `integration` matrix of `.github/workflows/build-linux.yml`. Run them one package at a time: + +```bash +cargo nextest run --locked --profile ci -p tablepro-driver- --run-ignored only +``` -- [ ] New crate at `crates/drivers//` compiles in isolation -- [ ] `DatabaseDriver` and `Connection` fully implemented (no `todo!()` in any method) -- [ ] Crate added to workspace `members` -- [ ] Driver registered in `app::build_registry` -- [ ] App `Cargo.toml` depends on the new driver crate -- [ ] At least one unit test for type / error mapping -- [ ] At least one integration test using testcontainers -- [ ] PR description includes the engine version tested against -- [ ] No new dependencies in `core` or `storage` crates -- [ ] `cargo clippy --all -- -D warnings` clean +## Checklist + +- [ ] The client library is on crates.io, unforked, and uses aws-lc-rs. +- [ ] `capabilities()` matches what the engine does; the suite flags agree. +- [ ] Every stream comes from the transport, and TLS checks `tls_server_name()`. +- [ ] Every call runs under `OwnedCalls` with the engine's stop, and a connection-scoped stop never releases an unsettled connection. +- [ ] `probe` never waits for a pool slot; `stream_read` never holds one. +- [ ] Saves record every step in a `WriteLedger`, and the stop is held while a commit is in flight. +- [ ] Editor sessions are one connection per tab, one run at a time, with `ResultSink` backpressure and the row-limit rule. +- [ ] Errors map to `DriverError` by the engine's code ([error-handling.md](error-handling.md)). +- [ ] Registered in `build_registry`; added to the workspace and to `crates/app/Cargo.toml`. +- [ ] Both conformance suites run; the CI matrix and nextest groups include the package. +- [ ] A page in `docs/drivers/` covers timeouts, stops, read-only, writes, trust and the engine's limits. +- [ ] `cargo clippy --all-targets -- -D warnings` passes with and without default features. diff --git a/linux/docs/decisions/0016-driver-contract-v2.md b/linux/docs/decisions/0016-driver-contract-v2.md new file mode 100644 index 0000000000..4934f167fb --- /dev/null +++ b/linux/docs/decisions/0016-driver-contract-v2.md @@ -0,0 +1,57 @@ +# 0016: Driver contract v2 + +- **Status**: Accepted +- **Date**: 2026-10-13 + +## Context + +Until the switch to this contract, drivers implemented the first `Connection` trait: `list_tables`, `fetch_rows`, `query`, `execute` and `execute_in_transaction`, with no deadline or cancel on any call. A cancel dropped the caller's future and left the statement running on the server, holding its locks. A save was one `execute_in_transaction` call that returned row counts or an error, never how far it had got. PostgreSQL ran on sqlx, which cannot send a cancel request, caches statements under names a transaction pooler hands to other clients, and dials its own sockets. The 2026-09-14 audit traced lost edits, runaway statements and pooler breakage to these gaps (F-001, F-008, F-012, F-013, F-041, F-047, F-058, F-109). + +## Decision + +Every driver implements the v2 contract in `tablepro-core`: a `DatabaseDriver` that states its `DriverCapabilities`, a `Connection` whose calls run under `OwnedCalls` with the engine's own stop, writes that record their progress in a `WriteLedger`, one `EditorSession` per editor tab, and a `RowLimit` the engine enforces where it can, each built on the engine's maintained client from crates.io with no fork. + +## Rationale + +**Owned calls.** Dropping a future does not stop a server. Each call runs on a task the connection owns, which carries a `StopRequest` into the work, sends the engine's own stop (`EngineStop`) and decides afterwards whether the connection is clean enough to reuse (`CallResource`). A stop that names a connection rather than a statement, as PostgreSQL's cancel request and MySQL's `KILL QUERY` do, can land late on whatever runs next, so that connection goes back to the pool only once the stop has settled and the server has acknowledged it. tiberius borrows its client for the whole request, so SQL Server's attention is sent in band by the task that owns the client. + +**Phase-aware write outcomes.** After a failed save the only question is what is in the database. The work records each step in a `WriteLedger` (statement sent, statement confirmed, commit sent), so an interrupted write becomes `RolledBack`, `PartiallyApplied` or `CommitOutcomeUnknown` from what was actually sent, not from a guess. The engine's stop is held while a commit, or a statement the engine cannot take back, is in flight. Where the server can answer later, the ledger keeps a `CommitToken` for `resolve_commit`: PostgreSQL's transaction id, ClickHouse's query ids. + +**Row limit.** `RowLimit` cannot be zero, and the editor keeps 10,000 rows by default. The limit stops the read on the server where the engine allows it without changing the statement: a cursor fetch on PostgreSQL, `sql_select_limit` on MySQL, `KILL QUERY` on ClickHouse once the limit is passed. SQL Server counts the rows past the limit without keeping them, and SQLite stops stepping the statement. The limit never ends the script or the user's transaction. + +**PostgreSQL on tokio-postgres.** sqlx-postgres keeps the backend's process id and secret key private, so it has no cancel request, and `pg_cancel_backend` from another session reaches the wrong backend behind PgBouncer. It also dials its own sockets and builds its own TLS configuration. tokio-postgres 0.7.18 logs in over any stream (`Config::connect_raw`), sends a cancel request over any stream (`CancelToken::cancel_query_raw`), supports SCRAM channel binding through tokio-postgres-rustls 0.14, and pipelines a transaction, its statement and its COMMIT into one round trip. deadpool-postgres 0.14.2 manages the pool. + +**PgBouncer safety.** Nothing is prepared under a name: generated SQL runs on the unnamed statement (`query_typed_raw`, `execute_typed`). No session-level setting is sent: workspace bounds are `SET LOCAL` inside each call's transaction, and editor sessions apply read-only per transaction and enforce deadlines with cancel requests. The driver therefore works behind transaction pooling with `max_prepared_statements = 0`. Read-only mode and timeouts as startup parameters were rejected because a default PgBouncer refuses startup parameters it does not track. + +**No forks.** The plan of 2026-09-14 carried forks of rust-postgres and tiberius. Both were dropped: tokio-postgres, postgres-types and postgres-protocol come from crates.io, and tiberius 0.13.0 is published with `Client::cancel_query`, `trust_cert_ca` and `hostname_in_certificate`. What upstream lacks is handled in the driver until the upstream change lands: + +| Upstream gap | Until it closes | +|---|---| +| tokio-postgres drops the transaction status of ReadyForQuery | The editor follows the status statement by statement, with a `SAVEPOINT` probe where the rules leave it open | +| tokio-postgres gives simple-query columns no type (rust-postgres #1358) | Editor result columns are text | +| tiberius takes no rustls `ClientConfig` | SQL Server TLS failures are classified from rustls's message text | +| tiberius stops reading at the first end of message, so an attention sent while the server is still waiting is never acknowledged (tiberius-rs/tiberius#459) | The connection is dropped after such a stop, and three cancel tests are held in `.config/nextest.toml` | + +The editor's row limit on PostgreSQL is a `DECLARE` and `FETCH` cursor. tokio-postgres offers a row-limited portal only inside a `Transaction` it opens itself, which would wrap every statement in BEGIN and COMMIT and refuse VACUUM. + +**MySQL without a relay.** sqlx dials a TCP endpoint or a Unix socket itself, and an SSH forward is a Unix socket OpenSSH listens on ([ADR 0013](0013-openssh-client-transport.md)), so the MySQL driver dials the transport's route and nothing relays bytes in between. + +**Trust per engine.** The engines' TLS stacks take trust anchors in different forms, so `NetworkTarget` carries the `TlsConfig` and each driver builds its own trust from it. PostgreSQL and ClickHouse share `tablepro_net::tls::build_client_config`; SQL Server and MySQL use what their clients accept. [ARCHITECTURE.md](../../ARCHITECTURE.md#trust) has the table. + +## Consequences + +- A new driver implements the whole contract and runs both conformance suites ([testing.md](../testing.md#conformance)). `suite_flags_match_driver_capabilities` keeps a driver from skipping a case by declaring less. +- PostgreSQL editor results are text until rust-postgres #1358 ships. The `SAVEPOINT` probe outside a transaction writes an ERROR line (25P01) to the server log. A cursor read plans for its first rows and never uses a parallel plan. Statement pooling is unsupported. +- Trust differs by engine until sqlx and tiberius take a rustls `ClientConfig`: MySQL trusts the Mozilla roots compiled into sqlx, not the system store. +- SQL Server TLS failures are classified more coarsely than the other engines', and three cancel tests stay held until tiberius-rs/tiberius#459 ships. +- [ADR 0009](0009-crypto-provider-and-sources.md) allowed `tiberius-rs/tiberius` as a git source until tiberius 0.13 was published. It is on crates.io now, and `allow-git` in `deny.toml` is empty. +- What each engine does with the contract is in `docs/drivers/`: [PostgreSQL](../drivers/postgresql.md), [MySQL and MariaDB](../drivers/mysql.md), [SQLite](../drivers/sqlite.md), [SQL Server](../drivers/sql-server.md), [ClickHouse](../drivers/clickhouse.md). + +## Alternatives considered + +- **Forks of rust-postgres and tiberius**: they closed the gaps above in the client, but each is a second codebase to keep current and patched ([ADR 0005](0005-supply-chain-policy.md)). +- **sqlx-postgres with `pg_cancel_backend`**: reaches the wrong backend behind PgBouncer, and sqlx dials its own sockets. +- **libpq through FFI**: a second TLS stack and a C dependency. +- **A private socket relay for MySQL**: no purpose once an SSH forward is a Unix socket. +- **Cancel by dropping the caller's future**: leaves the statement running on the server. +- **A `LIMIT` added to the user's SQL**: changes what the statement means, and does nothing for a statement that is not a `SELECT`. diff --git a/linux/docs/decisions/README.md b/linux/docs/decisions/README.md index 2083497c4a..6b2d4bb7d4 100644 --- a/linux/docs/decisions/README.md +++ b/linux/docs/decisions/README.md @@ -1,13 +1,13 @@ # Architecture Decision Records -These are short documents recording the **reasoning** behind major technical choices. They are not living documentation — once an ADR is accepted, it stays. If a later ADR supersedes it, link forward; do not edit history. +These are short documents recording the **reasoning** behind major technical choices. They are not living documentation: once an ADR is accepted, it stays. If a later ADR supersedes it, link forward; do not edit history. Every load-bearing decision in the Linux subproject has an ADR. If a contributor asks "why did we pick X", the answer should already be in this folder. If it is not, we missed an ADR. ## Format ```markdown -# 000N — Title +# 000N: Title - **Status**: Accepted | Superseded by 000M | Deprecated - **Date**: YYYY-MM-DD @@ -49,6 +49,7 @@ ADRs are short on purpose. If yours is more than a page, you are probably arguin | [0009](0009-crypto-provider-and-sources.md) | One crypto provider, upstream sources and dependency bans | Accepted | aws-lc-rs only; deny.toml enforces sources, licenses, bans and advisory ignores. | | [0013](0013-openssh-client-transport.md) | SSH through the system OpenSSH client | Accepted | ControlMaster per connection, Unix-socket forwards, prompts through an askpass helper. | | [0015](0015-gnome-50-platform-floor.md) | GNOME 50 platform floor | Accepted | GTK 4.22, libadwaita 1.9, GtkSourceView 5.18, GLib 2.88; AdwShortcutsDialog replaces the shortcuts window. | +| [0016](0016-driver-contract-v2.md) | Driver contract v2 | Accepted | Owned calls with engine stops, phase-aware write outcomes, server-side row limits; PostgreSQL on tokio-postgres; no forks. | ## Adding an ADR diff --git a/linux/docs/drivers/clickhouse.md b/linux/docs/drivers/clickhouse.md new file mode 100644 index 0000000000..a6796aae24 --- /dev/null +++ b/linux/docs/drivers/clickhouse.md @@ -0,0 +1,102 @@ +# ClickHouse + +Driver id `clickhouse`, crate `crates/drivers/clickhouse`. It speaks ClickHouse's HTTP interface through the official `clickhouse` crate 0.15 with its default features off, on a hyper-util client whose connector dials through the transport and whose TLS is hyper-rustls with the driver's own rustls config. The integration suite runs against ClickHouse 25.8. The contract it implements is in [adding-drivers.md](../adding-drivers.md). + +The default port is 8123, the plain HTTP port. With TLS on, use the server's HTTPS port (8443 by default); a TLS connect to the HTTP port fails. + +## Requests and connections + +HTTP has no login and no long-lived session, so there is no pool of logins: + +- hyper keeps idle HTTP connections for 2 s, shorter than the server's keep-alive (3 s before 23.11, 10 s after), so a reused connection is never one the server is about to close. +- Every connection is a stream from `Transport::open`. The URL names only the service, for the `Host` header and the TLS name, so an SSH forward never needs a DNS lookup of a name only the far side can resolve. +- The connect is the first request, `SELECT version(), currentDatabase()`, within the connect and login timeouts together. A refused password or a failed handshake shows up there. +- `probe` is one more request, `SELECT 1`, on the control client. + +Three clients share that connection pool: + +| Client | Used for | Settings beyond the common ones | +|---|---|---| +| Workspace | Catalogue reads, pages, counts, exports, write probes, editor sessions | `readonly=2` and `allow_ddl=0` on a read-only connection | +| Writes | The statements of a save | `wait_end_of_query=1`, `max_execution_time=0` | +| Control | `KILL QUERY`, `SYSTEM FLUSH LOGS`, commit resolution | None. It is not held read-only, because a `KILL` is not a read, and it runs only SQL the driver wrote. | + +Every request carries these settings: + +| Setting | Why | +|---|---| +| `output_format_decimal_trailing_zeros=1` | A Decimal keeps the scale it was stored with. | +| `http_write_exception_in_output_format=0` | An error after the first rows arrives as exception text, not as one more row. | +| `output_format_json_quote_denormals=1` | NaN and infinity arrive as words instead of null. | +| `output_format_json_validate_utf8=0` | A String that is not UTF-8 arrives byte for byte. | +| `mutations_sync=1` | A save waits for its mutation, so the grid refetches what the save wrote. | +| `cancel_http_readonly_queries_on_client_close=1` | A read whose HTTP connection the app dropped stops on the server even when no `KILL` got through. | + +The client names itself to the server as TablePro with the app's version (`with_product_info`). + +Results stream as `JSONCompactEachRowWithNamesAndTypes`, and each cell is read by the type the server named for it. Values go to the server as typed `{pN:Type}` parameters, never spliced into the SQL. SQL the user typed goes through `query_raw`, so a `?` in it stays a `?`. + +## TLS + +- `VerifyFull` trusts the system store plus the connection's CA file ([Trust](../../ARCHITECTURE.md#trust)) and checks the name from `NetworkTarget::tls_server_name()`. `Require` encrypts without checking the certificate. +- With a client certificate set, the driver sends `X-ClickHouse-SSL-Certificate-Auth: on` and no password, and the server logs the user in by the certificate's common name (`ssl_certificates` in the user's configuration). +- The crate's own TLS features stay off, so there is one TLS stack. + +## Read-only + +`ReadOnlyEnforcement::EngineEnforced`. A read-only connection sends `readonly=2` and `allow_ddl=0` with every workspace and editor request, and the server refuses a write whatever the SQL says (error 164, or 392 for DDL). `readonly=2` still lets a session change settings, which the editor needs. Every workspace read also carries `readonly=2`, on any connection, with its time bound. Saves on a read-only connection are refused before anything is sent. + +## Stops and timeouts + +A read carries `max_execution_time` (the nearer of the call's deadline and the workspace's 60 s, in whole seconds rounded up) with `timeout_overflow_mode=throw`, so the server stops it on time on its own (error 159). + +A stop is `KILL QUERY WHERE query_id = ? ASYNC` on the control client. Each query the driver sends carries a query id of its own, and the id is bound by the client, never written into the SQL. The `KILL` names that one query, so a late one cannot hit the next: `StopDelivery::OutOfBand(StopScope::Statement)`. The server answers the stopped query with error 394, which reads as `Cancelled`, or as `Timeout { server_cancelled: true }` when the deadline asked for it. + +| Server answer | Reported as | +|---|---| +| 394 after our `KILL` | `Cancelled` | +| 159 | `Timeout { phase: Statement, server_cancelled: true }` | +| 164, 392 | `ReadOnly(ServerRejected)` | +| 192, 193, 194, 516; 497 on the first request | `Auth` | +| 202, 203 | `Busy` | + +## Writes + +ClickHouse has no transactions: `WriteAtomicity::StatementByStatement` for row and schema changes. A row change is an `INSERT` or a mutation (`ALTER TABLE ... UPDATE` or `DELETE`), and a mutation is neither atomic nor reversible. + +1. Every probe runs first, so a row that moved stops the save before any mutation is sent. +2. Then each statement runs to its end on the writes client, under a query id of its own. `mutations_sync=1` makes it wait for the mutation, and `max_execution_time=0` means no server-side limit, because a mutation the server stops waiting for keeps running anyway. +3. A statement in flight is never killed: a `KILL` would not undo what the mutation already changed, only hide it. A stop is honoured between statements. + +The grid names a ClickHouse row by its sorting key (`RowIdentity::SortingKey`), which may match several rows. So a save proves its row with a probe before the mutation, and a batch with an exact affected-row guard is refused with `GuardUnsupported`: a mutation reports no row count. An `INSERT` reports its written rows; a mutation reports none, which is "not known", not zero. A table with no sorting key is read-only in the grid. The row-count estimate is `system.tables.total_rows`, which MergeTree keeps exact. + +A failed statement is sorted by whether it can have landed. A server error that refused it (a syntax error, an unknown column) did not apply. A cancel, a timeout, an unfinished mutation wait (341), a network error, or a body with no error code may have applied, so the outcome comes from the `WriteLedger`, with a `CommitToken::ClickHouseQuery` of every statement's query id. + +### Resolving an unknown outcome + +`CommitResolutionSupport::QueryLog`. `resolve_commit` asks about the first statement the save could not confirm: + +1. `SYSTEM FLUSH LOGS`, so the log has the latest entries. A login without the privilege skips it. +2. That query id in `system.query_log` from today and yesterday: + +| What the log says | `CommitResolution` | +|---|---| +| `QueryFinish` | `Applied` | +| `ExceptionBeforeStart`, not an `ALTER` | `NotApplied` | +| `ExceptionWhileProcessing` on an `INSERT` | `Unresolvable`: some blocks may have been written | +| An `ALTER` that failed while its mutation may live on (cancelled, timed out, the wait gave up, or failed while processing) | The mutation, below | +| Started, never ended | `StillRunning` if `system.processes` still lists it, else `Unresolvable` | +| No `system.query_log`, or no access to it | `Unresolvable` | + +3. For a mutation: `system.mutations` has no column that links a mutation to its `ALTER`, so the driver looks for mutations on the table the log names, created between the `ALTER`'s start and end. One that is done is `Applied`; one still running is `StillRunning`. None, because someone ran `KILL MUTATION` (which removes the row and keeps the parts already changed), or more than one, is `Unresolvable`. +4. The batch's answer: statements confirmed before the unconfirmed one, or not yet sent after it, turn a whole answer into `PartiallyApplied`. + +The query log and the process list are each node's own, and the driver asks the node its connection reaches; it does not query the cluster. Behind a load balancer that sends the next request to another node, that node's log has no entry for the statement and the answer is `Unresolvable`. + +## Editor sessions + +- An editor session is the workspace client with a `session_id` of its own and `session_timeout=3600`, so the server keeps the session's settings and temporary tables between runs. The server runs one query at a time per session, so a second run is `EditorSessionBusy`. +- One statement per run (`ScriptExecution::SingleStatement`), each with a new query id. +- The run's deadline, or the session's statement timeout, goes to the server as `max_execution_time` with `timeout_overflow_mode=throw`. A stop sends `KILL QUERY` and keeps reading, because the server answers the `KILL` by ending the response; a server that has not answered within the cancel grace is cut off. +- Past the row limit the driver sends `KILL QUERY` for the rest of the result rather than reading it. +- ClickHouse has no transaction an editor statement can leave open, so the transaction state is always `Idle`. diff --git a/linux/docs/drivers/mysql.md b/linux/docs/drivers/mysql.md new file mode 100644 index 0000000000..6620e98aa8 --- /dev/null +++ b/linux/docs/drivers/mysql.md @@ -0,0 +1,107 @@ +# MySQL and MariaDB + +Driver id `mysql`, crate `crates/drivers/mysql`. One driver serves both servers through sqlx 0.9 (`mysql`, `mysql-rsa`, `tls-rustls-aws-lc-rs`) with a deadpool 0.13 pool. `MySqlFlavour` tells them apart from `VERSION()` (MariaDB says so in it), because they name their timeouts, error numbers and catalogue columns differently. The integration suite runs against MySQL 8.4 and MariaDB 11.4. The contract it implements is in [adding-drivers.md](../adding-drivers.md). + +A connection needs a database name (`database_required`). + +## Connections + +| Connection | Used for | Session | +|---|---|---| +| Workspace pool, 4 connections (`WORKSPACE_POOL_MAX_SIZE`) | The sidebar, the grid, counts, saves | Workspace | +| One login per call | `stream_read` (exports), and `probe` when every pool slot is busy | Workspace without a statement timeout, or Control | +| One connection per editor tab | `EditorSession` | Editor | +| One control connection per workspace and per editor session | `KILL QUERY` | Control: nothing changed | + +A pool member is handed out again only after it answers a ping within the probe timeout. + +## Route + +sqlx dials its own sockets and takes no stream, so the driver passes it the transport's route: + +- A direct route is the host and port, dialled by sqlx. +- An SSH forward is the Unix socket OpenSSH listens on. sqlx connects to the socket, and its `host` carries the name TLS checks: the service's own, or the override. + +Nothing relays bytes in between. The whole login, dial included, is bounded by the connect and login timeouts together (25 s). sqlx 0.9 sets no `TCP_NODELAY` and no keepalive on its own dial, so on a direct route a dead peer is found by the probe and by each call's deadline, not by the kernel. + +## Session statements + +Every workspace and editor session starts the same way, so a statement means the same thing on each: + +```sql +SET NAMES utf8mb4 +SET SESSION TRANSACTION READ ONLY -- read-only connections only +SET SESSION max_execution_time = 60000 -- MySQL, milliseconds +SET SESSION max_statement_time = 60 -- MariaDB, seconds +SET SESSION lock_wait_timeout = 10 +SET SESSION innodb_lock_wait_timeout = 10 +``` + +- `SET NAMES utf8mb4` names no collation, so the server picks its default for the charset: MySQL's `default_collation_for_utf8mb4`, MariaDB's `character_set_collations`. That is what the mysql client gets too. +- The statement timeout is the workspace's 60 s. MySQL's `max_execution_time` bounds only read-only `SELECT`s; MariaDB's `max_statement_time` bounds every statement, which is why a schema change lifts it (below). An editor session sets it only when the session has a statement timeout of its own. +- Lock waits get the workspace's 10 s, in whole seconds rounded up. +- Numbers are sent as integers, never floats a server could read in another locale. + +A workspace session also pins two things the app's generated SQL depends on: `time_zone = '+00:00'`, so a `TIMESTAMP` reads as the instant it names, and `sql_mode` without `NO_BACKSLASH_ESCAPES`, because generated DDL writes literals such as `COMMENT '...'` that have no hex form. An editor session keeps the server's own zone and `sql_mode`, as the mysql client would. + +sqlx's own session statements (`PIPES_AS_CONCAT`, `NO_ENGINE_SUBSTITUTION`, its `SET NAMES`, its time zone) are turned off, its statement cache is 0 so no prepared statement outlives its call, and its statement logging is off. + +## TLS + +- `Disable` connects without TLS. A server with `require_secure_transport` answers with error 3159, reported as `ServerRequiresTls`. +- `Require` encrypts without checking the certificate. A server that does not offer TLS is reported as `ServerRefusedTls`. +- `VerifyFull` is sqlx's `VerifyIdentity`. It trusts the Mozilla roots compiled into sqlx plus the connection's CA file, and **not** the system's CA store ([Trust](../../ARCHITECTURE.md#trust)). A server certificate from a private CA always needs the CA file. +- The name checked is the host sqlx dials, so a server name override works only through an SSH forward, where the dial goes to a socket path (`ServerNameOverride::SshForwardOnly`). +- A client certificate and key are sent when set, for `REQUIRE X509` logins. +- With TLS off, `caching_sha2_password` sends the password encrypted with the server's RSA key, never in clear text, also over an SSH forward's socket. + +## Stops and timeouts + +A stop is `KILL QUERY ` on the control connection, which reaches the server over the same route as the connection it stops. The id comes from `CONNECTION_ID()` at login. It names the connection, not the statement: `StopDelivery::OutOfBand(StopScope::Connection)`. A pool connection goes back to the pool only when the server confirmed the stop (the statement ended with 1317, or with the server's own timeout after the `KILL` landed) and the `KILL` itself finished; a call stopped between statements also gives up its connection, since a `KILL` may be on its way to it. Error 1094 (no such thread) counts as delivered: there was nothing left to stop. + +| Server answer | Reported as | +|---|---| +| 1317 | `Cancelled`, or `Timeout { server_cancelled: true }` when the deadline asked for it | +| 3024 (MySQL), 1969 (MariaDB) | `Timeout { phase: Statement, server_cancelled: true }` | +| 1205 | `Timeout { phase: LockWait }` | +| 1792 | `ReadOnly(ServerRejected)` | +| 1045 | `Auth` | +| 1040 | `Busy` | +| 3159 | `Tls { failure: ServerRequiresTls }` | +| 1053, 1927 (MariaDB), 4031 (MySQL) | `ConnectionLost` | + +A call that timed out or lost its connection is checked before the link is declared lost: a `KILL` that never got through means the route is gone, one the server answered means it is not, and otherwise a login of its own answers within the cancel grace. + +### Through ProxySQL + +`KILL QUERY` stops the server thread `CONNECTION_ID()` named at login. Behind a proxy that multiplexes client connections over server connections, as ProxySQL does by default, the client may not be on that thread any more, and the `KILL` can miss or stop another client's statement. The driver has no proxy detection. What it does guarantee: a stop counts as confirmed only when the statement itself reported it, and a connection whose stop was not confirmed within the cancel grace is closed, so the app never claims a cancel that did not happen. Point TablePro at a ProxySQL user or query rule with multiplexing off, or at the server directly. + +## Read-only + +`ReadOnlyEnforcement::SessionDefault`: `SET SESSION TRANSACTION READ ONLY` on every session. Saves are refused before anything is sent, a write through generated SQL fails with 1792, and an export runs in `START TRANSACTION READ ONLY` and refuses SQL that does not start with `SELECT`, `WITH`, `SHOW`, `DESCRIBE`, `DESC`, `EXPLAIN` or `TABLE`. In the editor, the user's own `SET TRANSACTION READ WRITE` or `SET SESSION TRANSACTION READ WRITE` lifts it. For a guarantee, connect with a login that has no write privileges. + +## Writes + +Whether a write can be undone depends on the table's storage engine, not on the server. Before a row change the driver reads the table's engine and whether that engine supports transactions (`information_schema.TABLES` joined to `ENGINES`): + +| Change | Engine | How it runs | +|---|---|---| +| Row changes | Transactional (InnoDB) | One transaction. A failure rolls back. | +| Row changes | Not transactional (MyISAM, MEMORY, ARCHIVE, CSV, MariaDB's Aria), or a view | Every probe first, then statement by statement. Each statement lands on its own, and the `WriteLedger` reports how far it got. | +| Schema changes | Any | Statement by statement: each DDL statement commits on its own. | + +- An exact row guard relies on `CLIENT_FOUND_ROWS`, which sqlx turns on: an `UPDATE` reports the rows it matched, also the ones it left unchanged. +- On a non-transactional table, a statement whose connection drops before its answer may have run, so it is reported as unknown, not as refused. +- A trigger on an InnoDB table can write into a non-transactional one. A `ROLLBACK` then leaves those rows, and the server says so with warning 1196. The driver reads `SHOW WARNINGS` after every rollback, and a 1196 makes the outcome `PartiallyApplied` instead of `RolledBack`. +- MariaDB's `max_statement_time` would stop a long `ALTER` the way it stops a `SELECT`, so a schema change on MariaDB lifts it for the batch and puts it back after. A connection where it cannot be put back is closed. MySQL's bound covers only `SELECT`. +- A statement with parameters is prepared and closed after it runs; one without goes over the text protocol, which accepts statements MySQL cannot prepare, such as `CREATE TRIGGER`. +- MySQL cannot suspend foreign keys for one batch, so a batch that asks for it is refused before anything is sent. +- There is nothing to ask the server about a lost `COMMIT` (`CommitResolutionSupport::Unavailable`). `CommitOutcomeUnknown` carries no token, and the user reloads to see what landed. + +## Editor sessions + +- The editor sends one statement per run. Text with several statements runs as the server's own multi-statement batch over the text protocol, except that mysql-client `DELIMITER` lines split it where the client would, because the server does not know the directive. +- The row limit is `SET SESSION sql_select_limit = `, sent when the limit changes. The server stops a plain `SELECT` there, and the set ends with `Truncation::ServerStopped` and no total. A `SELECT` with a `LIMIT` of its own, or one inside a routine, is read to its end: the rows past the limit are counted, never decoded, and the set ends with `Truncation::ReadPastLimit` and its total. +- A stop sends `KILL QUERY` on the session's own control connection and gives the statement the cancel grace to end. A statement that has not ended by then ends the session, because its answer is still on the way. +- When the grid stops reading, the run ends as cancelled with no `KILL`, and sqlx reads the rest of the result off the wire before the next statement. +- The transaction state after each run comes from `@@in_transaction` on MariaDB and from `performance_schema.events_transactions_current` on MySQL. On MySQL before 8.0.16, with `performance_schema` off, or for a login that cannot read it, the state is `Unknown`, and the tab's close guard warns instead of guessing. diff --git a/linux/docs/drivers/postgresql.md b/linux/docs/drivers/postgresql.md new file mode 100644 index 0000000000..fdd14f6118 --- /dev/null +++ b/linux/docs/drivers/postgresql.md @@ -0,0 +1,144 @@ +# PostgreSQL + +Driver id `postgres`, crate `crates/drivers/postgres`. It runs on tokio-postgres 0.7.18 with postgres-types and postgres-protocol, deadpool-postgres 0.14.2 for the pool and tokio-postgres-rustls 0.14 for TLS, all from crates.io ([ADR 0016](../decisions/0016-driver-contract-v2.md)). The integration suite runs against PostgreSQL 17, with the catalogue also checked on 13 and 18. The contract it implements is in [adding-drivers.md](../adding-drivers.md). + +## Connections + +| Connection | Used for | Opened | +|---|---|---| +| Workspace pool, 4 connections (`WORKSPACE_POOL_MAX_SIZE`) | The sidebar, the grid, counts, saves | The first at connect, the rest on demand | +| One connection per call | `stream_read` (exports), and `probe` when every pool slot is busy | For that call, closed after it | +| One connection per editor tab | `EditorSession` | When the tab first runs SQL | +| One stream per stop | The cancel request | For that request | + +Every connection logs in with `Config::connect_raw` over a stream from the transport, so a direct route and an SSH forward behave the same. The login sends only the user, the password, the database, `application_name = TablePro` and `channel_binding = prefer`, under the 15 s login timeout. Startup options are never used: a transaction pooler refuses parameters it does not track. + +## TLS + +- `Disable` sends no SSLRequest. `Require` and `VerifyFull` both send it and refuse to continue in plaintext; they differ only in the certificate check. +- `VerifyFull` trusts the system store plus the connection's CA file ([Trust](../../ARCHITECTURE.md#trust)), and checks the name from `NetworkTarget::tls_server_name()`. +- A client certificate is offered when one is set, for `cert` authentication in `pg_hba.conf`. +- A server that answers the SSLRequest with `N` is reported as `ServerRefusedTls`, never as a certificate problem: the TLS connector records whether the handshake started. +- SCRAM channel binding (`tls-server-end-point`) is preferred, never required. It is used when the server's certificate is signed with RSA (SHA-1 to SHA-512) or ECDSA with SHA-256 or SHA-384. With any other signature, such as Ed25519 or ECDSA with SHA-512, tokio-postgres-rustls has no hash to bind with, and the login goes ahead without binding. + +## Workspace calls + +Each read is one pipelined flight on a pool connection: the transaction, the work and the COMMIT go out together and the answers come back in one round trip. + +```sql +START TRANSACTION READ ONLY; SET LOCAL statement_timeout = ; SET LOCAL lock_timeout = 10000 +-- the call's statement +COMMIT +``` + +- Reads always run in a `READ ONLY` transaction, also on a read-write connection. +- `statement_timeout` is the nearer of the call's deadline and the workspace's 60 s. `lock_timeout` is 10 s, so a blocked edit reports the lock instead of a generic timeout. +- `SET LOCAL` ends with the transaction, so nothing is left on a server connection a pooler shares. +- Generated SQL runs on the unnamed statement (`query_typed_raw`, `execute_typed`) with typed parameters. Nothing is prepared under a name. +- A connection whose flight did not end with an acknowledged COMMIT or ROLLBACK is taken out of the pool, because deadpool's recycle check (`RecyclingMethod::Verified`) passes inside a half-finished transaction. +- A read that returns no rows still has its columns: the statement runs once more as a subquery right-joined to one row on a false condition, in a read-only transaction of its own. +- A column of a type tokio-postgres does not know (an enum, `citext`, a domain) would make it prepare a type lookup under a name. Pages project such columns as text, so a read never does. + +`probe` takes a pool slot without waiting; when every slot is busy it opens a connection of its own and runs `SELECT 1`. `stream_read` opens a connection of its own, runs `START TRANSACTION READ ONLY` with no statement timeout, and refuses user SQL that would end that transaction (transaction control, `CALL`, `DO`) before sending anything. + +## Stops and timeouts + +A stop is PostgreSQL's cancel request (`CancelToken::cancel_query_raw`) over a new stream from the transport, so it goes through the SSH forward too. It names the backend, not the statement: `StopDelivery::OutOfBand(StopScope::Connection)`. The connection it chases goes back to the pool only when the server answered with SQLSTATE 57014 and the cancel request itself finished; otherwise the connection is closed, because a late cancel request would stop the next caller's statement. + +| Server answer | Reported as | +|---|---| +| 57014 after our cancel request | `Cancelled`, or `Timeout { server_cancelled: true }` when the deadline asked for it | +| 57014 with "statement timeout" | `Timeout { phase: Statement, server_cancelled: true }` | +| 55P03 | `Timeout { phase: LockWait }` | +| 25006 | `ReadOnly(ServerRejected)` | +| 28P01, 28000 | `Auth` | +| 53300 | `Busy` | +| 57P01, 57P02, 57P03 | `ConnectionLost` | + +A call that timed out or lost its connection is checked before the link is declared lost: a cancel request that never got through means the route is gone, one the server answered means it is not, and otherwise a connection of its own answers within the cancel grace. + +## Read-only + +`ReadOnlyEnforcement::SessionDefault`. Read-only is a property of each transaction the driver opens, never a session setting: + +- Workspace reads run in `READ ONLY` transactions, and saves are refused before anything is sent (`ReadOnly(ClientGuard)`). +- In the editor, a statement run outside a transaction is wrapped in `START TRANSACTION READ ONLY` and `COMMIT`. After the user's own `BEGIN`, the driver sends `SET TRANSACTION READ ONLY`. +- A statement that cannot run inside a transaction block, such as `VACUUM`, fails inside the wrapper with 25001 and is reported as refused by read-only mode. `PREPARE TRANSACTION`, `COMMIT PREPARED` and `ROLLBACK PREPARED` are refused before sending. +- `SET default_transaction_read_only = off` does not escape, because each transaction is opened read-only explicitly. +- A user's own `SET TRANSACTION READ WRITE` inside their transaction can still lift it. For a guarantee, connect with a role that cannot write. + +## Writes + +`WriteAtomicity::Transactional` for row and schema changes. A save is one transaction: + +1. `START TRANSACTION READ WRITE` with `SET LOCAL lock_timeout` of 10 s and `SET LOCAL statement_timeout` of 60 s for row changes, 0 (none) for schema changes. A schema change stays bounded by the call's deadline and cancel. +2. Each step: its probe, if any; then the statement with `execute_typed`; then the affected-row check. +3. The transaction id, `pg_current_xact_id()` (or `txid_current()` before PostgreSQL 13), read before COMMIT. +4. `COMMIT`. + +A COMMIT the server answers with an error (a deferred constraint, a serialization failure) rolled back. A COMMIT that gets no answer is `CommitOutcomeUnknown` with `CommitToken::PostgresXid`, and `resolve_commit` asks `pg_xact_status` (or `txid_status`): + +| Status | `CommitResolution` | +|---|---| +| `committed` | `Applied` | +| `aborted` | `NotApplied` | +| `in progress` | `StillRunning` | +| NULL, the record aged out of the commit log | `Unresolvable` | + +PostgreSQL cannot suspend foreign keys, so a batch that asks for it is refused before anything is sent. + +### Row identity + +The grid names a row by, in order: the primary key; a valid unique index with no predicate and no expressions over `NOT NULL` columns; or, for a table or a materialised view with neither, the row's physical address. The address is `ctid` with `xmin`, plus `tableoid` for a partitioned table, where a `ctid` is unique only within one partition. `xmin` is bound as `$n::xid`: after `VACUUM` a freed `ctid` can hold a new row, and `xmin` makes a save against the old row match nothing instead of the new one. Anything else is read-only in the grid. + +## Editor sessions + +An editor tab runs SQL the way psql does: + +- The script is split on the client with the PostgreSQL grammar and sent one statement per message, stopping at the first failure. When a statement changes `standard_conforming_strings`, the rest of the script is split again with the new rule. +- No session-level `SET` is ever sent. The run's deadline and the session's statement timeout are enforced with cancel requests, and nothing new goes to the server until an outstanding cancel request has finished. A statement that has not ended by the end of the cancel grace ends the session. +- Notices (`RAISE NOTICE`, warnings) arrive as `ResultEvent::Notice`. +- `COPY FROM STDIN` and `COPY TO STDOUT` are refused with `Unsupported`. +- Result columns are text: the simple protocol gives a column no type, so every value arrives as `Value::Text`. Typed columns wait on rust-postgres #1358. + +### The row limit + +A statement that returns rows (`SELECT`, `VALUES`, `TABLE`, `WITH`, or one that opens with a parenthesis) is read through a cursor, so a result larger than the limit stops on the server: + +```sql +BEGIN -- or START TRANSACTION READ ONLY, or SAVEPOINT tablepro_cursor_savepoint in the user's transaction +DECLARE tablepro_cursor NO SCROLL CURSOR FOR + +FETCH FORWARD FROM tablepro_cursor +CLOSE tablepro_cursor +COMMIT -- or RELEASE SAVEPOINT tablepro_cursor_savepoint +``` + +- One row past the limit tells a full result from a cut one. A cut result has `total_rows: None`. +- Each step goes as its own message, so a trailing `--` comment in the statement cannot swallow the `FETCH`. +- A `DECLARE` the server refuses because the statement cannot be a cursor (a data-modifying `WITH`: 0A000; `SELECT INTO`: 42601) is rolled back to the savepoint or out of the driver's transaction, and the statement runs again as a plain statement. Inside the user's transaction any other failure aborts their transaction, as it would without the limit. +- Error positions are corrected for the `DECLARE` prefix, so they count from the start of the user's statement. +- A cursor plans for its first rows (`cursor_tuple_fraction`, 0.1 by default) and never uses a parallel plan, so a statement can run with a different plan than in psql. + +### Transaction state + +tokio-postgres drops the transaction status the server sends with every ReadyForQuery, so the driver follows it from what each statement was and whether it succeeded, under PostgreSQL's own rules. Where the rules leave it open (`CALL` and `DO`, which can commit inside, `COMMIT AND CHAIN`, a failed `COMMIT`, and after any stop) it asks the server: + +```sql +SAVEPOINT tablepro_probe -- succeeds inside a transaction, then RELEASE SAVEPOINT tablepro_probe + -- 25P01 outside a transaction, 25P02 in an aborted one +``` + +Outside a transaction this probe fails by design, and the server logs it as an ERROR line (25P01). `RunSummary::transaction_state` is never `Unknown` on PostgreSQL. + +## Connection poolers + +The driver works behind PgBouncer in transaction pooling with `max_prepared_statements = 0`: it prepares nothing under a name and sets nothing on the session. The integration suite runs against `edoburu/pgbouncer` in all three modes (`tests/integration/pgbouncer.rs`). + +| PgBouncer `pool_mode` | Workspace | Editor | +|---|---|---| +| `session` | Works | Works, including the user's own `SET`, temporary tables and prepared statements across runs | +| `transaction` | Works | Each statement and each transaction works. A `SET`, a temporary table or a `PREPARE` lasts only until the user's transaction ends: the next statement may reach another server connection, and a session-level `SET` stays behind for whoever gets that one next. Use session pooling for those. | +| `statement` | Unsupported: reads and saves run in transaction blocks, which the pooler refuses, and a save fails with the pooler's message | Unsupported: the cursor flight is a transaction block | + +Cancel requests go to the pooler like any other connection, and PgBouncer forwards them to the server connection the client is using. diff --git a/linux/docs/drivers/sql-server.md b/linux/docs/drivers/sql-server.md new file mode 100644 index 0000000000..f3a03f9a53 --- /dev/null +++ b/linux/docs/drivers/sql-server.md @@ -0,0 +1,103 @@ +# SQL Server + +Driver id `mssql`, crate `crates/drivers/mssql`. It runs on tiberius 0.13.0 from crates.io (`tds73`, `rustls`, `chrono`, and `integrated-auth-gssapi` behind the crate's `kerberos` feature) with a deadpool 0.13 pool. The integration suite runs against SQL Server 2022. The contract it implements is in [adding-drivers.md](../adding-drivers.md). + +## Connections + +| Connection | Used for | Session | +|---|---|---| +| Workspace pool, 4 connections (`WORKSPACE_POOL_MAX_SIZE`) | The sidebar, the grid, counts, saves | `SET LOCK_TIMEOUT 10000; SET XACT_ABORT ON;` | +| One connection per call | `stream_read` (exports), and `probe` when every pool slot is busy | Workspace, or none for the probe | +| One connection per editor tab | `EditorSession` | The server's defaults: whatever the user sets is theirs | + +- `XACT_ABORT ON` rolls a transaction back on any error, so a failed statement never leaves one open on a pooled connection. `LOCK_TIMEOUT` makes a blocked edit report the lock (error 1222) instead of running into the statement deadline. +- A pool member is handed out again only when `@@TRANCOUNT` is 0. +- tiberius's own 30 s command timeout is off: each call is bounded by its own deadline, and an editor query may run longer. + +Every login gets its stream from the transport, so a direct route and an SSH forward behave the same. A stream must open within the connect timeout and the TDS login finish within the login timeout. + +## Routing + +An Azure SQL gateway or an availability group listener can answer a login with a routing token that names another server. The driver follows it once: it asks the transport for a route to the new host and port (`Transport::reroute`, a new forward on the same SSH master when the connection goes through SSH) and logs in there, checking TLS and building the Kerberos SPN against the new host. A second routing answer is refused, so a gateway that keeps redirecting cannot loop. + +## TLS + +TLS is negotiated inside the TDS pre-login exchange, after the server has said what encryption it supports. + +| `TlsMode` | tiberius setting | Certificate check | +|---|---|---| +| `Disable` | `EncryptionLevel::NotSupported` | None. A server that requires encryption is reported as `ServerRequiresTls`. | +| `Require` | `EncryptionLevel::Required` with `trust_cert` | None | +| `VerifyFull` | `EncryptionLevel::Required` | The system's CA store, loaded by tiberius, plus the connection's CA file (`trust_cert_ca`). The name is the host, or the override (`hostname_in_certificate`). | + +- A server that will not encrypt when asked to is reported as `ServerRefusedTls`. +- No client certificate: SQL Server logins are not authenticated by one, so a TLS setting that carries one is refused. +- tiberius takes no rustls `ClientConfig` and keeps only the text of a rustls error, so the driver classifies TLS failures from that text (`UnknownIssuer`, `NameMismatch`, `Expired` and so on). This is coarser than the other engines, which read the rustls error itself, and a phrase rustls changes in a later release falls back to `TlsFailure::Other`. +- The CA file is checked before the login, because tiberius reads it during the handshake and would report a bad file as an I/O error. + +## Authentication + +- **Password**: a SQL Server login. +- **Windows (Kerberos)**: only in a build with the `kerberos` feature, which links MIT Kerberos. The driver sends no username or password and uses the ticket the user already holds from `kinit`, for the SPN `MSSQLSvc/:` built from the host and port the user typed, never from a local SSH forward. The Kerberos calls block, so the login runs on Tokio's blocking pool and the login timeout ends it. The README's [SQL Server section](../../README.md#sql-server-with-windows-integrated-auth) has the realm and SPN rules. + +### Kerberos inside Flatpak + +The sandbox has no `/etc/krb5.conf` of the host, and its `/tmp` is private, so a FILE ticket cache from `kinit` on the host is invisible: + +- A build with `-Dflatpak=true` and Kerberos installs `build-aux/flatpak/krb5.conf` into the app's `sysconfdir`. It uses the KCM credential cache and finds the realm and the KDC through DNS. +- The development manifest grants the KCM socket, `/run/.heim_org.h5l.kcm-socket`, so a host that runs `sssd-kcm` shares its tickets with the app. +- A FILE cache works only when `KRB5CCNAME` points at a file the sandbox can read, such as one under the home folder. + +Running from source remains the supported path for Kerberos today. + +## Read-only + +`ReadOnlyEnforcement::LoginPrivileges`. SQL Server has no read-only session or transaction mode, so the login's permissions are the only guarantee, and the connect dialog says so. Use a login granted only `SELECT` for a read-only connection. The driver adds guards against accidents, not against intent: + +- Saves are refused before anything is sent (`ReadOnly(ClientGuard)`). +- A generated read on a read-only connection runs inside `BEGIN TRANSACTION` and `ROLLBACK TRANSACTION`, so a filter fragment that writes is undone. A fragment that ends the guard's transaction itself (errors 266, 3902, 3903) is reported as refused by read-only mode, never as a success. +- An export, on any connection, refuses user SQL that is not one batch of statements starting with `SELECT`, `WITH` or a parenthesis, or that contains a word that writes or runs code (`INSERT`, `EXEC`, `MERGE`, `INTO` and the like), and runs inside the same rolled-back transaction. +- An editor session has no guard: it runs whatever the login may run. + +## Stops and timeouts + +tiberius borrows its client for the whole of a request, so nothing outside the task that owns the client can stop one. The stop is a TDS attention, sent by that task on the request's own connection (`Client::cancel_query`): `StopDelivery::InBand`. When the stop fires, the task: + +1. gives the request a quarter of the cancel grace to reach a point between TDS tokens, because a request half read finishes cleanly if given a moment; +2. sends the attention and waits up to half of the grace for the server's acknowledgement; +3. on a workspace connection, rolls back any transaction the request left open (`IF @@TRANCOUNT > 0 ROLLBACK TRANSACTION`), because an attention does not end one. An editor session keeps the user's transaction. + +The connection is reused only when the attention was acknowledged at a token boundary. One stopped in the middle of a token, or whose attention got no acknowledgement or something else back, is closed. During a save the attention is never sent while `COMMIT TRANSACTION` is in flight. + +tiberius 0.13 stops reading at the first end of message, so an attention sent while SQL Server is still waiting on the request, as in a `WAITFOR`, is never acknowledged, and the connection is closed. A pooled connection is replaced; an editor session ends and the tab opens a new one on its next run. The fix is tiberius-rs/tiberius#459. Until a release has it, `.config/nextest.toml` holds the three tests that need it out of the default run: + +- `cancellation::editor_waitfor_cancel_then_select1_same_client` +- `cancellation::pooled_query_bound_waitfor_deadline_server_cancelled_client_reused` +- `conformance_engine::suite::editor_cancel_reports_cancelled_and_session_stays_ready` + +SQL Server has no session statement timeout. A workspace call's deadline is the nearer of the caller's and the workspace's 60 s, enforced by the attention; a schema change keeps only the caller's. + +| Server answer | Reported as | +|---|---| +| Attention acknowledged | `Cancelled`, or `Timeout { server_cancelled: true }` when the deadline asked for it | +| 1222 | `Timeout { phase: LockWait }` | +| 3906 | `ReadOnly(ServerRejected)` | +| 18456, 18452, 18486, 18487, 18488 | `Auth` | +| 17809 | `Busy` | +| A GSSAPI failure | `IntegratedAuth` | + +## Writes + +`WriteAtomicity::Transactional` for row and schema changes. A save is `BEGIN TRANSACTION`, each step, then `COMMIT TRANSACTION`: + +- A statement with parameters runs through tiberius's `execute`; one without runs through `sp_executesql` with the statement alone, because SQL Server refuses `CREATE VIEW` or `CREATE TRIGGER` next to an empty parameter list. +- The affected-row count includes rows a trigger touched, so a save proves it hit the right row with a probe before the statement rather than with that count. +- A `COMMIT` the server answers with an error rolled back; one whose connection dropped is `CommitOutcomeUnknown` with no token. There is nothing to ask the server afterwards (`CommitResolutionSupport::Unavailable`). +- SQL Server cannot suspend foreign keys for one batch, so a batch that asks for it is refused before anything is sent. + +## Editor sessions + +- A script runs one `GO` batch at a time, as sqlcmd and SSMS do, including `GO `. Each batch goes as a SQL batch, not through `sp_executesql`, so a `USE` or a `SET` in it lasts for the rest of the session. A failed batch does not stop the next one. Text the splitter cannot finish goes to the server whole, so the error is the server's own. +- Rows past the row limit are counted and dropped, never decoded. +- When the grid stops reading, the driver sends an attention instead of reading millions of unwanted rows. +- The transaction state comes from `@@TRANCOUNT` and `XACT_STATE()`, so a transaction an error has doomed reads as `Failed`. diff --git a/linux/docs/drivers/sqlite.md b/linux/docs/drivers/sqlite.md new file mode 100644 index 0000000000..c7417cb847 --- /dev/null +++ b/linux/docs/drivers/sqlite.md @@ -0,0 +1,76 @@ +# SQLite + +Driver id `sqlite`, crate `crates/drivers/sqlite`. It runs on sqlx 0.9 with `sqlite-unbundled`, so it links the system's libsqlite3 rather than a second copy, and calls `libsqlite3-sys` directly once, for `sqlite3_get_autocommit`, until sqlx exposes a connection's transaction state. The library must be SQLite 3.37.0 or newer (`SQLITE_MIN_VERSION`), which the driver checks when it opens the file. The contract it implements is in [adding-drivers.md](../adding-drivers.md). + +There is no network, login or TLS: the target is a `FileEndpoint`, an absolute path with a `FileOpenMode`. + +## Opening a file + +Every connection to the file opens with the same options: + +- The path is a file name, never a URI, so `?` and `%41` in it are part of the name. +- A missing file is `FileNotFound`; the driver never creates one by accident (`create_if_missing` is off). +- A read-only connection opens the file with `SQLITE_OPEN_READONLY`. +- `busy_timeout` is the workspace's lock timeout, 10 s: how long SQLite waits for another connection's lock before it reports `Busy`. +- `foreign_keys = ON`, and `trusted_schema = OFF`, so a view or trigger in a file from elsewhere cannot run SQL functions with side effects. +- The statement cache is off, because sqlx keeps a statement's columns from its first prepare and SQLite prepares it again with more columns after an `ALTER TABLE`. +- sqlx's statement logging is off. + +The connect reads `sqlite_version()` from the file once, so a file that is not a database fails the connect (`NotADatabase`) rather than the first page. SQLite only says it could not open a file; the driver tells a moved file (`FileNotFound`) from one the user may not read (`FileAccessDenied`) by looking at the path. + +### New databases + +`FileOpenMode::CreateNew` is what the New Database flow sends after the save dialog has asked about replacing a file. The driver writes an empty file next to the target with mode 0600, syncs it, renames it over the path, removes any `-wal`, `-shm` or `-journal` file left beside the old database (it would be replayed into the new one), and syncs the folder so the rename is durable. A reader sees the old file or the new one, never half of one. An existing file is replaced, not refused. + +A reconnect uses `FileEndpoint::for_reopen`, which is always `OpenExisting`, so reconnecting never empties a database the user just created. + +### Where the file lives + +SQLite relies on POSIX file locks and, for a database in WAL mode, on shared memory in the `-shm` file beside it. Both work on a local file system. On NFS or SMB, locks can fail without an error, and two writers can then corrupt the file. A FUSE mount may not implement the locks at all. A file chosen through the Flatpak document portal sits on the portal's own FUSE file system, which exposes the chosen file rather than its folder, so the journal and WAL files SQLite keeps beside it may not work there. + +The driver does not detect any of these. Open such a file read-only, or copy it to a local folder first. Under the development Flatpak, which grants `--filesystem=home`, a file under the home folder is opened at its real path, not through the portal. + +## Connections + +| Connection | Used for | +|---|---| +| One workspace connection (`WORKSPACE_POOL_MAX_SIZE` is 1) | The sidebar, the grid, counts, saves | +| One read-only connection per call | `stream_read` (exports) | +| One connection per editor tab | `EditorSession` | + +One workspace connection is enough because SQLite takes one writer at a time anyway, and it keeps a temporary table or an attached database where the next call can see it. Calls wait for it for up to the pool wait (5 s). + +## Stops and timeouts + +SQLite has no statement timeout and no out-of-band cancel. Each call installs a fresh flag, and a progress handler checks it every 1,000 virtual-machine instructions; when the flag is up the statement ends with `SQLITE_INTERRUPT`. The flag names the call, not the connection, so a stop that arrives late cannot reach the next call: `StopDelivery::OutOfBand(StopScope::Statement)`. `sqlite3_interrupt` is not used, because sqlx's handle to the connection waits for the worker thread to finish its current command, which is the statement the stop was meant for. + +The workspace statement timeout (60 s) becomes a deadline the flag enforces. The flag is lowered before a rollback runs, so the rollback is never interrupted by the stop that caused it. + +## Read-only + +`ReadOnlyEnforcement::EngineEnforced`: the file is opened read-only, so SQLite refuses every write, whatever the SQL (`SQLITE_READONLY`, reported as `ReadOnly(ServerRejected)`). Saves on a read-only connection are refused before anything is sent. Exports always open their connection read-only. + +## Writes + +`WriteAtomicity::Transactional` for row and schema changes. A save is one transaction that starts with `BEGIN IMMEDIATE`, which takes the write lock before the first probe reads anything: contention shows up once, as `Busy` after the lock timeout, before anything has run. + +SQLite is the one engine that can suspend foreign keys for a batch (`ForeignKeyMode::SuspendedThenVerified`), which a table rebuild needs for its copy: + +1. `PRAGMA foreign_keys = OFF` before `BEGIN`, since the pragma does nothing inside a transaction. +2. `PRAGMA foreign_key_check` on the named tables before the batch and after it. +3. Only violations the batch introduced fail it, and the batch rolls back. A file can hold violations from long before, written while foreign keys were off, and those do not count. +4. `PRAGMA foreign_keys = ON` after the `COMMIT` or `ROLLBACK`. + +A `COMMIT` never crosses a network, so nothing is left to resolve afterwards (`CommitResolutionSupport::Unavailable`). + +### Row identity + +The grid names a row by its primary key, then by a unique index whose columns are all `NOT NULL`, then by `rowid`. A `WITHOUT ROWID` table with neither key, or a table with a column named `rowid` that hides the real one, is read-only in the grid. + +## Editor sessions + +- A script is split where SQLite itself would end a statement, so the semicolons inside a trigger body do not end it. Text the splitter cannot finish, such as an unterminated string, goes to SQLite whole, so the error is SQLite's own. The splitter is tested against `sqlite3_complete` (`tests/script_plan_matches_sqlite3_complete.rs`). +- Each statement is prepared first, so a `SELECT` that finds nothing still has its columns. A column with no declared type gets its type from the first row, sent as `ResultEvent::ColumnsResolved`. +- The row limit stops the read one row past the limit, and the set ends with `Truncation::ReadPastLimit`. SQLite computes a row only when it is stepped, so the rest of the result is never produced. +- A stop raises the run's flag and gives the statement the cancel grace to notice it; past that its read is dropped. +- The transaction state after a run comes from `sqlite3_get_autocommit`. diff --git a/linux/docs/error-handling.md b/linux/docs/error-handling.md index 7e960ab412..ae18e8e053 100644 --- a/linux/docs/error-handling.md +++ b/linux/docs/error-handling.md @@ -1,131 +1,169 @@ # Error handling -Two error styles, applied per layer. Mixing them is a review red flag. +Errors are typed enums, built with `thiserror` wherever they cross a crate boundary. There is no `anyhow` in the workspace: the caller matches on the enum, and the app decides what to say from it, never from the message. -## Rule +## Errors by layer -| Layer | Error type | Why | +| Crate | Error types | Notes | |---|---|---| -| `core` (traits, contracts) | `thiserror` enums | Stable variants; consumers match on them. | -| `core::DriverError`, `storage::StorageError` | `thiserror` enums | Cross crate boundaries; need exhaustive matching. | -| `drivers/` | `thiserror` enum mapping the underlying crate's error into `core::DriverError` | Underlying crate's errors do not leak. | -| `storage` (internal) | `thiserror` enum, `StorageError` | Same reasoning. | -| `app` (UI handlers, services, internal glue) | `anyhow::Result` | Composition. Errors are mostly displayed and dropped. | -| Tests | `anyhow::Result` or `?` against domain errors | Whatever is shortest. | +| `tablepro-core` | `DriverError`, `ConfigError`, `TransportError`, `SshFailure`, `CertificateFileProblem` | The contract every driver and transport speaks. | +| `tablepro-net` | `tls::TlsSetupError` | Wraps a `ConfigError` or a rustls error while a TLS config is built; drivers turn it into `DriverError`. | +| `tablepro-ssh` | `SshFailure`, inside `TransportError::Ssh` | Classified from OpenSSH's exit status and stderr. | +| `drivers/` | `DriverError` only | The client library's error never leaves the crate. | +| `tablepro-storage` | `StorageError`, `SettingsError`, `KeyringError`, `CredentialError` | | +| `tablepro-session` | `ConnectError`, `OperationError`, `ClaimError` | `ConnectError::Driver` and `OperationError::Driver` carry the driver's error unchanged. | +| `tablepro` (the app) | Small enums per module (`StartupError`, `LoggingError`, `AcquireError`, ...) | User-facing text comes from `crates/app/src/ui/error_text/`. | + +## `DriverError` + +`DriverError` is `#[non_exhaustive]`, `Clone` and `PartialEq`, so a result can be stored, compared in tests and sent across threads. Each variant has one meaning: + +| Variant | Meaning | +|---|---| +| `Transport(TransportError)` | The route failed: name resolution, refused, unreachable, a connect or SSH timeout, or an `SshFailure`. | +| `Auth { diagnostics }` | The server refused the credentials, with its own explanation when it gave one. | +| `IntegratedAuth(String)` | A Kerberos login failed before the server saw it. | +| `Tls { failure, detail }` | The handshake failed; `TlsFailure` says why. | +| `Server(ServerDiagnostics)` | The server refused the statement. | +| `ReadOnly(ReadOnlyRefusal)` | A write on a read-only connection: `ClientGuard` (the driver refused before sending) or `ServerRejected` (the server refused). | +| `Cancelled` | The caller stopped it. | +| `Timeout { phase, server_cancelled }` | A bound passed. `server_cancelled` says the server confirmed it stopped the statement. | +| `ConnectionLost { during }` | The connection went away; `LossPhase` says while doing what. | +| `RolledBack`, `RowGuardFailed`, `CommitOutcomeUnknown`, `PartiallyApplied` | What a failed save left in the database ([adding-drivers.md](adding-drivers.md#5-writes-and-the-writeledger)). | +| `GuardUnsupported` | The engine reports no row count, so a guarded write cannot run. | +| `FileNotFound`, `FileAccessDenied`, `NotADatabase`, `FileIo` | A file engine's file. | +| `Busy` | The server has no room for another connection, or the pool had no free slot in time. | +| `EditorSessionBusy` | The tab's session is already running a statement. | +| `Config(ConfigError)` | The settings cannot be used: a bad host, port, path, TLS option, auth mode or missing database name. | +| `Decode { column, detail }` | The driver could not read a value. A bug on our side, not the server's. | +| `Protocol(String)` | The server or the client library said something the driver did not expect. | +| `Unsupported { feature }` | The engine has no such feature. | -`anyhow` is fine for a function that wraps several different error sources and forwards them to a UI dialog or a log line. It is wrong for a public API that callers must reason about. +Rules: -## `thiserror` patterns +- Add a variant only when the app has to act on it differently from every existing one. A new message is not a reason. +- Never wrap a `Box` or the client library's error. Keep what the app needs: a code, a phase, a path. +- The `#[error]` text is for logs. The UI writes its own sentence from the variant. -Domain errors are exhaustive enums: +## `ErrorCategory` -```rust -#[derive(Debug, thiserror::Error)] -pub enum DriverError { - #[error("connection refused")] - ConnectionRefused, +`DriverError::category()` (and `TransportError::category()`, `SshFailure::category()`) sorts every error into the kind of problem it is. The app branches on the category, so an error a driver adds later still lands in a branch that says what to do next. - #[error("authentication failed")] - AuthFailed, +| Category | Variants | What the app does | +|---|---|---| +| `Configuration` | `Config`, `Unsupported`; SSH setup failures | Names the setting to change. The session stops retrying until the user changes something. | +| `Network` | `Transport` (resolution, refused, unreachable); a rejected SSH forward | Names the host and port. A session retries on a backoff. | +| `Tls` | `Tls` | One sentence per `TlsFailure`, with a remedy button where one exists. The session stops retrying. | +| `Authentication` | `Auth`, `IntegratedAuth`; SSH auth and host-key failures | Says the login or the host key was refused, with the server's reason. Never echoes what was typed. The session stops retrying. | +| `ReadOnly` | `ReadOnly` | Says who refused: this connection's setting, or the server. | +| `Server` | `Server` | The server's own words, code, detail and hint. | +| `Interrupted` | `Cancelled` | "Stopped." | +| `Timeout` | `Timeout`, `TransportError::Timeout` | One sentence per `TimeoutPhase`. | +| `ConnectionLost` | `ConnectionLost` | Says the connection closed. The session reconnects on a backoff. | +| `WriteOutcome` | `RolledBack`, `RowGuardFailed`, `CommitOutcomeUnknown`, `PartiallyApplied` | Says what is in the database now. Only `RolledBack` and `RowGuardFailed` may say nothing changed. | +| `File` | the four file variants | Names the path. A connect that failed on the file is not retried. | +| `Busy` | `Busy`, `EditorSessionBusy` | Says to try again shortly, or to wait for the tab's statement. | +| `Internal` | `GuardUnsupported`, `Decode`, `Protocol`; `SshFailure::Protocol` | Says what the driver could not do: check a save, read a value, read the reply. | + +## `ServerDiagnostics` + +A server error keeps the shape the server sent it in. `ServerCode` holds each engine's own numbering, because the number is what tells a missing password apart from a missing table: + +| `ServerCode` | Engine | Shown as | +|---|---|---| +| `SqlState(String)` | PostgreSQL | SQLSTATE 23505 | +| `MySql { number, sqlstate }` | MySQL, MariaDB | MySQL error 1062 | +| `Tds { number, state, class }` | SQL Server | SQL Server error 2627 | +| `ClickHouse { code }` | ClickHouse | ClickHouse code 62 | +| `Sqlite { primary, extended }` | SQLite | SQLite error 2067 | - #[error("TLS handshake failed: {0}")] - Tls(String), +`ServerDiagnostics` also carries `severity`, `detail`, `hint`, `position`, `where_context`, `schema`, `table`, `column` and `constraint`. PostgreSQL fills most of them; the other engines mostly leave them empty. `position` counts characters from the start of the statement the user wrote: a driver that wraps the statement corrects it (the PostgreSQL cursor flight subtracts its `DECLARE` prefix). - #[error("query failed: {message}")] - Query { message: String, sqlstate: Option }, +A driver maps the codes the app acts on to their own variants and passes everything else through as `Server`: - #[error("connection closed unexpectedly")] - Disconnected, +| Meaning | PostgreSQL | MySQL, MariaDB | SQL Server | ClickHouse | SQLite | +|---|---|---|---|---|---| +| `Auth` | 28P01, 28000 | 1045 | 18456, 18452, 18486-18488 | 192, 193, 194, 516; 497 at login | | +| `ReadOnly` (server) | 25006 | 1792 | 3906 | 164, 392 | 8 (`SQLITE_READONLY`) | +| Statement `Timeout` | 57014 with "statement timeout" | 3024 (MySQL), 1969 (MariaDB) | | 159 | | +| `LockWait` timeout | 55P03 | 1205 | 1222 | | | +| `Cancelled` | 57014 after our cancel request | 1317 | attention acknowledged | 394 after our KILL | 9 (`SQLITE_INTERRUPT`) after our stop | +| `Busy` | 53300 | 1040 | 17809 | 202, 203 | 5, 6 (`SQLITE_BUSY`, `SQLITE_LOCKED`) | +| `ConnectionLost` | 57P01, 57P02, 57P03 | 1053, 1927, 4031 | | | | - #[error("driver internal error: {0}")] - Internal(String), -} -``` +The engine pages ([PostgreSQL](drivers/postgresql.md), [MySQL and MariaDB](drivers/mysql.md), [SQLite](drivers/sqlite.md), [SQL Server](drivers/sql-server.md), [ClickHouse](drivers/clickhouse.md)) give each driver's mapping, and the places where a driver can only read the message, as SQL Server's TLS failures through tiberius. -Rules: +## `TimeoutPhase` -- Variants are stable. Once shipped, do not rename or remove. Add new variants at the end. -- Avoid wrapping arbitrary `Box` inside variants. Map underlying errors into specific variants. The `Internal(String)` variant is the escape hatch for cases that genuinely cannot be classified — use it sparingly. -- The `#[error]` message is for logs and developer-facing surfaces. The UI builds its own message based on the variant. - -## Driver-side error mapping - -Each driver crate maps the underlying crate's errors: - -```rust -fn map_sqlx_error(err: sqlx::Error) -> DriverError { - use sqlx::Error::*; - match err { - Database(e) => DriverError::Query { - message: e.message().to_string(), - sqlstate: e.code().map(|c| c.to_string()), - }, - Io(e) if e.kind() == std::io::ErrorKind::ConnectionRefused => DriverError::ConnectionRefused, - Tls(e) => DriverError::Tls(e.to_string()), - PoolClosed | PoolTimedOut => DriverError::Disconnected, - other => DriverError::Internal(format!("{other}")), - } -} -``` +`Timeout` and `TransportError::Timeout` name the phase that ran out, because each phase has a different cause and a different fix: -The driver does not pass through `sqlx::Error` to callers. Callers see only `DriverError`. +| Phase | Bound | Meaning | +|---|---|---| +| `Connect` | `LivenessPolicy::connect_timeout`, 10 s | No TCP connection, or the socket did not open. | +| `Login` | `login_timeout`, 15 s | Connected, but the login or TLS handshake did not finish. | +| `SshHandshake` | `ssh_handshake_timeout`, 20 s, paused while a prompt is open | OpenSSH did not authenticate in time. | +| `SshChannelOpen` | `ssh_channel_open_timeout`, 10 s | `ssh -O forward` did not answer in time. | +| `LockWait` | `WorkspaceSettings::lock_timeout`, 10 s | Another session holds a lock this statement needs. | +| `Statement` | The call's deadline, or the workspace statement timeout of 60 s | The statement ran too long. `server_cancelled` says whether the server confirmed the stop. | +| `Probe` | `probe_timeout`, 5 s | The server did not answer a liveness check. | +| `CancelAcknowledgement` | `cancel_grace`, 5 s | The server did not confirm a stop. No shipped driver raises it: a stop the server never confirms ends as an abandoned call, `ConnectionLost`. | -## UI display +## `TlsFailure` and its remedies -`app` translates `DriverError` and `StorageError` into user-facing messages with full context. The mapping lives in `app::ui::error_message`: +Drivers classify a handshake failure with `tablepro_net::tls::classify`, which walks the error's sources to the rustls error. SQL Server reads rustls's message text instead, because tiberius keeps only the text. Where the driver can tell, a failure before the server agreed to TLS is `ServerRefusedTls`, never a certificate problem. -```rust -fn message_for(err: &DriverError) -> String { - match err { - DriverError::ConnectionRefused => "Could not reach the database. Is it running?".into(), - DriverError::AuthFailed => "Username or password is wrong.".into(), - DriverError::Tls(detail) => format!("TLS handshake failed: {detail}"), - DriverError::Query { message, sqlstate: Some(s) } => format!("Query failed (SQLSTATE {s}): {message}"), - DriverError::Query { message, .. } => format!("Query failed: {message}"), - DriverError::Disconnected => "The connection was closed. Try reconnecting.".into(), - DriverError::Internal(detail) => format!("Internal driver error: {detail}"), - } -} -``` +| `TlsFailure` | Cause | What the app offers | +|---|---|---| +| `UnknownIssuer` | The certificate's issuer is not in the trust set. | **Choose CA Certificate…**, which opens the connect dialog at the CA file row. | +| `NameMismatch` | The certificate is for another name. | **Set TLS Server Name**, which opens the connect dialog at the server name row. | +| `Expired`, `NotYetValid`, `Revoked` | The certificate itself is bad. | Nothing: the server's certificate has to be replaced. | +| `ServerRefusedTls` | The server does not speak TLS on this port. | The message; turn TLS off or use the TLS port. | +| `ServerRequiresTls` | TLS was off and the server insists on it. | The message; turn TLS on. | +| `HandshakeRejected` | The server ended the handshake, for example over the client certificate. | The server's reason. | +| `CaFileInvalid`, `ClientIdentityInvalid` | A file the user chose cannot be used. | The message names the file. | +| `Other` | Anything else. | The rustls message. | + +The buttons are `TlsRemedy` in `crates/app/src/ui/error_text/tls_remedy.rs`, shown on the connect failure's toast. Which certificates each engine trusts, and how to choose a CA file under Flatpak, is in [ARCHITECTURE.md](../ARCHITECTURE.md#trust). + +## Mapping errors in a driver + +- Map by the engine's code, not by message text. Where the client library keeps nothing but the text, say so in a comment and pin the text with a test. +- Keep the server's diagnostics: `DriverError::reported(ServerDiagnostics)`, and `DriverError::auth(Some(diagnostics))` for a refused login. +- An error the engine raised to acknowledge our stop is the stop: `Cancelled`, or `Timeout { server_cancelled: true }`. `EngineStop::acknowledged` is where `OwnedCalls::run_read` learns which errors those are; `run_write` passes the result through, so a write maps its own. +- A `ConnectionLost` goes through `Transport::explain_closed_stream`, so an SSH master that exited is reported as that. +- A dead route is reported only once a check outside the pool confirms it ([adding-drivers.md](adding-drivers.md#deciding-that-the-link-is-lost)). -Do not display raw `Debug` or `Display` output for domain errors. Always go through this layer. +## User-facing text + +`crates/app/src/ui/error_text/` turns errors into sentences: `driver_message` for a `DriverError`, `connect_error_message` and `operation_error_message` for the session's errors, and an SSH module for `SshFailure`. Every string goes through `crate::i18n`. Never show `Debug` or the `#[error]` text of an error on its own. ## Logging -Use the `tracing` crate. `app::logging::init` installs a `tracing_subscriber` that writes to **stderr**, which the GNOME session journals for both Flatpak and system installs. Panic output goes to the same place, so a panic and the log lines around it stay in order. There is no `tracing-journald` writer and no custom panic hook. +Logging goes through `tracing`. `crates/app/src/logging.rs` installs a `tracing_subscriber` fmt layer that writes to **stderr**, which the GNOME session journals for both Flatpak and system installs. Panics land in the same stream, so a panic and the lines around it stay in order. There is no journald writer and no panic hook. -The filter comes from `RUST_LOG`. Without it the level follows the build profile: `debug` for a development build, `info` otherwise. A `RUST_LOG` that does not parse falls back to the profile default and logs a warning saying so, rather than silently dropping to the default. +- The filter comes from `RUST_LOG`. Without it, a development build logs at `debug` and an installed one at `info`. A `RUST_LOG` that does not parse falls back to that default and logs a warning saying so. +- `TABLEPRO_LOG_FORMAT=json` writes one JSON object per line; `text`, the default, writes plain lines. ```sh RUST_LOG=debug ./_build/crates/app/tablepro -RUST_LOG=tablepro_app::ui=trace,info ./_build/crates/app/tablepro +RUST_LOG=tablepro=trace,info TABLEPRO_LOG_FORMAT=json ./_build/crates/app/tablepro journalctl --user -f -t tablepro ``` -Levels: - -- `error!` — something the user must see, or a contract was violated. -- `warn!` — recoverable but suspicious. -- `info!` — significant lifecycle events: app start, driver registered, connection opened. -- `debug!` — verbose internal flow. -- `trace!` — query bodies, network frames. Off by default. - -Never log passwords, secret tokens, or full query parameters at any level above `trace!`. The lint enforces this in CI by grepping for known sensitive identifiers. +Levels: `error!` for what the user must see or a broken contract, `warn!` for recoverable but suspicious, `info!` for lifecycle events, `debug!` for internal flow, `trace!` for query text and wire detail. -## `unwrap` and `expect` +Never log a password, a secret, a key or a bound value. sqlx's own statement logging is turned off in the MySQL and SQLite drivers, and a test in each crate (`tests/statement_logging.rs`) proves no statement text reaches the subscriber. -Banned in production paths. The only legitimate uses: +tokio-postgres is not filtered yet. It logs the text of every simple query, which is all editor SQL, through `log` at `debug`, and `tracing-subscriber` forwards `log` records to the same output. A development build logs at `debug` by default, so an `ALTER ROLE ... PASSWORD '...'` run in the editor reaches the journal. Until the default filter covers it, run with `RUST_LOG=debug,tokio_postgres=info`. -- `OnceLock::get_or_init` initialisers that genuinely cannot fail. -- Test code. -- Single-call type conversions on values whose validity is locally provable (e.g. `"5432".parse::().expect("constant literal")`). +## `unwrap`, `expect` and `panic!` -In every other case, propagate the error. If a function "cannot fail", make it `infallible` by typing. +The workspace lints deny `unwrap_used`, `expect_used`, `panic`, `todo` and `unimplemented`; `clippy.toml` allows them in tests only. Return a real error instead. The one accepted use outside tests is in a `const`, where the compiler proves the value, as `RowLimit::EDITOR_DEFAULT` does. -## Anti-patterns flagged in review +## Patterns refused in review -- `Result>` in a public function. Use a `thiserror` enum. -- `anyhow::Error` returned from `core` or `storage`. Those crates expose typed errors only. -- `unwrap()` after a `Result` from a fallible operation. Always handle or propagate. -- `match err { _ => "Something went wrong" }`. Always exhaustive. -- A `String` error type. We have one shipped product; use the proper enum. +- A `String` or `Box` as an error type in a public function. +- A client library's error in a public signature of a driver crate. +- `match error { _ => "Something went wrong" }`. Match the category, then the variant. +- A retry loop around an error the category says retrying cannot fix: `Configuration`, `Authentication`, `File`, `WriteOutcome`. +- Treating `CommitOutcomeUnknown` or `PartiallyApplied` as a failure that changed nothing. diff --git a/linux/docs/testing.md b/linux/docs/testing.md index 90ef0d7f67..3a4a693c7d 100644 --- a/linux/docs/testing.md +++ b/linux/docs/testing.md @@ -117,13 +117,15 @@ Each flag adds cases and needs its engine trait: | Flag | Adds | Needs | |---|---|---| -| `endpoint: network` | a dropped caller stops the server's statement; a lost transport reports `ConnectionLost` | `NetworkConformanceEngine` | +| `endpoint: network` | a dropped `query_bound` caller stops the server's statement; a lost transport reports `ConnectionLost` | `NetworkConformanceEngine` | | `pool: pooled` | a probe succeeds while every pool slot is busy | `PooledConformanceEngine` | | `editor_transactions: true` | the row limit keeps the transaction and the statements after it | `EditorTransactionEngine` | | `unsupported_types: true` | an unsupported type never decodes as NULL | `TypedConformanceEngine` | | `bounded_ddl: true` | a schema-change batch is not cut off by the statement timeout | `BoundedDdlEngine` | -A flag that is off emits no test. `suite_flags_match_driver_capabilities` is always emitted and never ignored: it fails when `endpoint` or `editor_transactions` disagrees with the driver's `DriverCapabilities`, so a driver cannot skip a case by declaring less. `docker: true` marks every other case `#[ignore = "requires docker"]`; SQLite passes `docker: false` and `endpoint: file`, so its suite runs in the unit suite. +A flag that is off emits no test. `suite_flags_match_driver_capabilities` is always emitted and never ignored: it fails when `endpoint` or `editor_transactions` disagrees with the driver's `DriverCapabilities`, so those cases cannot be skipped by declaring less. `pool`, `unsupported_types` and `bounded_ddl` have no capability behind them: they describe the server, and MySQL and MariaDB differ on `bounded_ddl` under one driver, so the adapter states them and review checks them. `docker: true` marks every other case `#[ignore = "requires docker"]`; SQLite passes `docker: false` and `endpoint: file`, so its suite runs in the unit suite. + +Each case starts its own engine, then has two minutes (`CASE_LIMIT`) to finish. A driver that ignores a deadline or a cancel fails the case with its name instead of holding the test run. ### TLS conformance diff --git a/linux/po/POTFILES.in b/linux/po/POTFILES.in index 94d85cf1b4..8628960946 100644 --- a/linux/po/POTFILES.in +++ b/linux/po/POTFILES.in @@ -14,7 +14,9 @@ crates/app/src/runtime/deliver.rs crates/app/src/runtime/gio_process_hold.rs crates/app/src/runtime/mod.rs crates/app/src/services/active_session.rs +crates/app/src/services/cell_overlay.rs crates/app/src/services/change_tracker.rs +crates/app/src/services/change_tracker_registry.rs crates/app/src/services/history_availability.rs crates/app/src/services/history_service.rs crates/app/src/services/mod.rs @@ -23,6 +25,12 @@ crates/app/src/services/secret_save_report.rs crates/app/src/services/session_context.rs crates/app/src/services/single_instance.rs crates/app/src/services/structure_tracker.rs +crates/app/src/services/structure_tracker_handle.rs +crates/app/src/services/structure_tracker_registry.rs +crates/app/src/services/tab_registry.rs +crates/app/src/services/tab_tracker_handle.rs +crates/app/src/services/tracker_delta.rs +crates/app/src/services/tracker_edit.rs crates/app/src/services/workspace_state.rs crates/app/src/storage.rs crates/app/src/test_support.rs @@ -35,6 +43,7 @@ crates/app/src/test_support/unlabelled_widget.rs crates/app/src/test_support/wait_timed_out.rs crates/app/src/test_support/widget_lookup.rs crates/app/src/ui/app/browse.rs +crates/app/src/ui/app/connect_flow.rs crates/app/src/ui/app/connection.rs crates/app/src/ui/app/import.rs crates/app/src/ui/app/mod.rs @@ -80,6 +89,7 @@ crates/app/src/ui/editor/run_scope.rs crates/app/src/ui/editor/script_runner.rs crates/app/src/ui/editor/sidebar_policy.rs crates/app/src/ui/editor/significant_tokens.rs +crates/app/src/ui/editor/sql_file.rs crates/app/src/ui/editor/statement_cursor.rs crates/app/src/ui/editor/tests.rs crates/app/src/ui/editor/transaction_dirty.rs