test(linux): run the million-row case on SQLite in its own binary

linux/ROADMAP.md

@@ -181,7 +181,7 @@ What has to hold before the first release, for a user on GNOME 50 with Ubuntu 26
 | 1 | Installs `tablepro` from the PPA or the COPR project, and gets updates from the system's updater | CI builds both packages offline the way Launchpad and COPR do, and starts them in fresh containers | The release setup, the first tag, and an install then an upgrade from the PPA and COPR |
 | 2 | Connects to their everyday PostgreSQL, MySQL, MariaDB, SQL Server, ClickHouse or SQLite database | The supported releases and why others are out, in the README's [supported servers](README.md#supported-servers). CI runs every driver's Docker suite on each tested release, MySQL 8.0 and SQL Server 2017 and 2019 among them | Nothing |
 | 3 | Browses tables of real-world data (dates, decimals, JSON, UUIDs, NULLs) and sees correct values | Per-engine decoding with the type fallback and unsupported-type conformance cases ([ADR 0014](docs/decisions/0014-lossless-values-and-dialects.md)). A value corpus on every engine, MariaDB too: dates and times at their extremes and in other zones, decimals, JSON, UUIDs, Unicode and binary values, read back through a table tab, Export All and an editor run | Nothing |
-| 4 | Runs typical queries against tables of a million rows or more without running out of memory or freezing | The 10,000-row editor limit with Export All, and driver tests over 2,000,000 rows. Pages seek past the edge row's key, or walk PostgreSQL heap block windows for a table with no key; OFFSET only under a user sort, a restored deep offset or a partitioned keyless table. Docker tests page 60k and 25k-row tables both ways and check the plan. A 1,000,000-row table is paged to its end and exported on PostgreSQL, MySQL, MariaDB, SQL Server and ClickHouse, with bounds on time and on resident peak growth | The same test on SQLite, in a process of its own: its suites run in the unit suite, where the peak would count every test's |
+| 4 | Runs typical queries against tables of a million rows or more without running out of memory or freezing | The 10,000-row editor limit with Export All, and driver tests over 2,000,000 rows. Pages seek past the edge row's key, or walk PostgreSQL heap block windows for a table with no key; OFFSET only under a user sort, a restored deep offset or a partitioned keyless table. Docker tests page 60k and 25k-row tables both ways and check the plan. A 1,000,000-row table is paged to its end and exported on PostgreSQL, MySQL, MariaDB, SQL Server, ClickHouse and SQLite, with bounds on time and on resident peak growth. SQLite's case runs in the unit suite in a test binary of its own, so the peak is its alone | Nothing |
 | 5 | Edits data in place, and the changes commit correctly even when the connection or the app misbehaves | Write, row guard and commit resolution conformance on every engine, with the fault fakes and the grid save Docker test. `kill_during_save` kills TablePro halfway through a SQLite save: the file keeps none of it, and the next launch restores the session with its history, settings and saved tabs | A save into a SQLite file another program holds locked fails as `DriverError::Busy`, whose message says the server has no room for another connection |
 | 6 | Trusts the app with credentials: TLS, SSH tunnels, no password in plain text | Verify full by default, the TLS conformance suite, the Secret Service, and statement logging silenced | An outside security review |
 | 7 | Recovers when the network drops or the database restarts | Health monitoring, backoff and replay rules with their tests, a stopped container in the liveness tests, and a session test that restarts PostgreSQL: the read in flight runs again, the save in flight does not and leaves no row | Nothing |

linux/crates/drivers/clickhouse/tests/integration/conformance_engine.rs

@@ -195,5 +195,6 @@ mod suite {
 
     tablepro_test_support::large_table_conformance_suite! {
         engine: ClickHouseEngine,
+        docker: true,
     }
 }

linux/crates/drivers/mssql/tests/integration/conformance_engine.rs

@@ -192,5 +192,6 @@ mod suite {
 
     tablepro_test_support::large_table_conformance_suite! {
         engine: MssqlEngine,
+        docker: true,
     }
 }

linux/crates/drivers/mysql/tests/integration/conformance_engine.rs

@@ -256,6 +256,7 @@ mod mysql_suite {
 
     tablepro_test_support::large_table_conformance_suite! {
         engine: MySqlEngine<MysqlServer>,
+        docker: true,
     }
 }
 
@@ -279,5 +280,6 @@ mod mariadb_suite {
 
     tablepro_test_support::large_table_conformance_suite! {
         engine: MySqlEngine<MariadbServer>,
+        docker: true,
     }
 }

linux/crates/drivers/postgres/tests/integration/conformance_engine.rs

@@ -207,5 +207,6 @@ mod suite {
 
     tablepro_test_support::large_table_conformance_suite! {
         engine: PgEngine,
+        docker: true,
     }
 }

linux/crates/drivers/sqlite/tests/integration/conformance_engine.rs

@@ -1,135 +1,6 @@
-//! The connection conformance suite on SQLite.
+//! The connection and DDL conformance suites on SQLite.
 
-use std::path::PathBuf;
-use std::sync::Arc;
-use std::time::Duration;
-
-use async_trait::async_trait;
-use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection};
-use sqlx::{Connection as _, Executor as _};
-use tablepro_core::{
-    ConnectEnvironment, ConnectOptions, ConnectTarget, DatabaseDriver, FileEndpoint, LivenessPolicy, TableRef,
-    WorkspaceSettings,
-};
-use tablepro_driver_sqlite::SqliteDriver;
-use tablepro_test_support::conformance::{
-    ConformanceDdlEngine, ConformanceEngine, ConformanceFailure, EditorTransactionEngine, PooledConformanceEngine,
-};
-use tempfile::TempDir;
-
-use crate::support::create_database;
-
-/// A database file of the case's own, which exists before the driver
-/// opens it: the suite never relies on the driver creating one.
-pub struct SqliteEngine {
-    _directory: TempDir,
-    path: PathBuf,
-    endpoint: FileEndpoint,
-}
-
-fn quote(name: &str) -> String {
-    format!("\"{}\"", name.replace('"', "\"\""))
-}
-
-#[async_trait]
-impl ConformanceEngine for SqliteEngine {
-    fn driver() -> Arc<dyn DatabaseDriver> {
-        Arc::new(SqliteDriver)
-    }
-
-    async fn start() -> Result<Self, ConformanceFailure> {
-        let directory = tempfile::tempdir().map_err(ConformanceFailure::setup)?;
-        let path = directory.path().join("conformance.sqlite3");
-        create_database(&path).await.map_err(ConformanceFailure::setup)?;
-        let endpoint = FileEndpoint::open_existing(path.clone()).map_err(ConformanceFailure::setup)?;
-        Ok(Self {
-            _directory: directory,
-            path,
-            endpoint,
-        })
-    }
-
-    fn connect_options(&self, settings: WorkspaceSettings, liveness: LivenessPolicy) -> ConnectOptions {
-        ConnectOptions {
-            target: ConnectTarget::File(self.endpoint.clone()),
-            workspace_settings: settings,
-            liveness,
-            environment: ConnectEnvironment::new("conformance"),
-        }
-    }
-
-    fn scratch_table(&self, name: &str) -> TableRef {
-        TableRef::unqualified(name)
-    }
-
-    fn create_scratch_table_sql(&self, table: &TableRef) -> String {
-        format!(
-            "CREATE TABLE {} (id integer primary key, label text not null)",
-            quote(&table.name)
-        )
-    }
-
-    /// SQLite has no sleep, but a recursive CTE without a stop never
-    /// produces its count.
-    fn unbounded_statement_sql(&self, marker: &str) -> String {
-        format!("WITH RECURSIVE r(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM r) SELECT count(*), '{marker}' FROM r")
-    }
-
-    /// Hold an exclusive lock on a connection of the engine's own for
-    /// `hold`, and return a read that waits for it in the busy handler.
-    async fn blocking_statement_sql(&self, marker: &str, hold: Duration) -> Result<String, ConformanceFailure> {
-        let options = SqliteConnectOptions::new().filename(&self.path);
-        let mut holder = SqliteConnection::connect_with(&options)
-            .await
-            .map_err(ConformanceFailure::setup)?;
-        holder
-            .execute("BEGIN EXCLUSIVE")
-            .await
-            .map_err(ConformanceFailure::setup)?;
-        tokio::spawn(async move {
-            tokio::time::sleep(hold).await;
-            let _ = holder.execute("ROLLBACK").await;
-            let _ = holder.close().await;
-        });
-        Ok(format!("SELECT count(*), '{marker}' FROM sqlite_schema"))
-    }
-
-    fn row_source_sql(&self, rows: u64) -> String {
-        format!("WITH RECURSIVE r(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM r WHERE n < {rows}) SELECT n FROM r")
-    }
-}
-
-impl PooledConformanceEngine for SqliteEngine {
-    const WORKSPACE_POOL_MAX_SIZE: usize = tablepro_driver_sqlite::WORKSPACE_POOL_MAX_SIZE;
-}
-
-impl EditorTransactionEngine for SqliteEngine {
-    fn begin_sql(&self) -> String {
-        "BEGIN".to_owned()
-    }
-
-    fn commit_sql(&self) -> String {
-        "COMMIT".to_owned()
-    }
-}
-
-impl ConformanceDdlEngine for SqliteEngine {
-    fn integer_type(&self) -> &'static str {
-        "INTEGER"
-    }
-
-    fn text_type(&self) -> &'static str {
-        "TEXT"
-    }
-
-    fn timestamp_type(&self) -> &'static str {
-        "TEXT"
-    }
-
-    fn current_timestamp_expression(&self) -> &'static str {
-        "CURRENT_TIMESTAMP"
-    }
-}
+use crate::sqlite_engine::SqliteEngine;
 
 tablepro_test_support::ddl_conformance_suite! {
     engine: SqliteEngine,

linux/crates/drivers/sqlite/tests/integration/main.rs

@@ -23,6 +23,8 @@ mod open_semantics;
 #[cfg(test)]
 mod sessions;
 #[cfg(test)]
+mod sqlite_engine;
+#[cfg(test)]
 mod support;
 mod value_corpus;
 #[cfg(test)]

linux/crates/drivers/sqlite/tests/integration/metadata.rs

@@ -6,7 +6,8 @@ use tablepro_core::meta::{
 };
 use tablepro_core::{Connection, RowLimit, Value};
 
-use crate::support::{TestFile, apply, call, create_database, query, statement};
+use crate::sqlite_engine::create_database;
+use crate::support::{TestFile, apply, call, query, statement};
 
 fn table(name: &str) -> TableRef {
     TableRef::unqualified(name)

linux/crates/drivers/sqlite/tests/integration/open_semantics.rs

@@ -7,7 +7,8 @@ use std::path::PathBuf;
 use tablepro_core::{DriverError, ErrorCategory, FileEndpoint, Value, WorkspaceSettings};
 use tokio_util::sync::CancellationToken;
 
-use crate::support::{TestFile, apply, connect, create_database, query, writable};
+use crate::sqlite_engine::create_database;
+use crate::support::{TestFile, apply, connect, query, writable};
 
 #[tokio::test]
 async fn open_existing_missing_path_file_not_found_no_file_created() {

linux/crates/drivers/sqlite/tests/integration/sqlite_engine.rs

@@ -0,0 +1,149 @@
+//! The conformance engine adapter on SQLite. The `large_table` test
+//! target shares it, so it uses nothing else from this target.
+
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+use std::time::Duration;
+
+use async_trait::async_trait;
+use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection};
+use sqlx::{Connection as _, Executor as _};
+use tablepro_core::{
+    ConnectEnvironment, ConnectOptions, ConnectTarget, DatabaseDriver, FileEndpoint, LivenessPolicy, TableRef,
+    WorkspaceSettings,
+};
+use tablepro_driver_sqlite::SqliteDriver;
+use tablepro_test_support::conformance::{
+    ConformanceDdlEngine, ConformanceEngine, ConformanceFailure, EditorTransactionEngine, LargeTableEngine,
+    PooledConformanceEngine,
+};
+use tempfile::TempDir;
+
+/// Make an empty database the way another program would, so the
+/// driver only ever opens a file that exists.
+pub async fn create_database(path: &Path) -> Result<(), sqlx::Error> {
+    let options = SqliteConnectOptions::new().filename(path).create_if_missing(true);
+    SqliteConnection::connect_with(&options).await?.close().await
+}
+
+/// A database file of the case's own, which exists before the driver
+/// opens it: the suite never relies on the driver creating one.
+pub struct SqliteEngine {
+    _directory: TempDir,
+    path: PathBuf,
+    endpoint: FileEndpoint,
+}
+
+fn quote(name: &str) -> String {
+    format!("\"{}\"", name.replace('"', "\"\""))
+}
+
+#[async_trait]
+impl ConformanceEngine for SqliteEngine {
+    fn driver() -> Arc<dyn DatabaseDriver> {
+        Arc::new(SqliteDriver)
+    }
+
+    async fn start() -> Result<Self, ConformanceFailure> {
+        let directory = tempfile::tempdir().map_err(ConformanceFailure::setup)?;
+        let path = directory.path().join("conformance.sqlite3");
+        create_database(&path).await.map_err(ConformanceFailure::setup)?;
+        let endpoint = FileEndpoint::open_existing(path.clone()).map_err(ConformanceFailure::setup)?;
+        Ok(Self {
+            _directory: directory,
+            path,
+            endpoint,
+        })
+    }
+
+    fn connect_options(&self, settings: WorkspaceSettings, liveness: LivenessPolicy) -> ConnectOptions {
+        ConnectOptions {
+            target: ConnectTarget::File(self.endpoint.clone()),
+            workspace_settings: settings,
+            liveness,
+            environment: ConnectEnvironment::new("conformance"),
+        }
+    }
+
+    fn scratch_table(&self, name: &str) -> TableRef {
+        TableRef::unqualified(name)
+    }
+
+    fn create_scratch_table_sql(&self, table: &TableRef) -> String {
+        format!(
+            "CREATE TABLE {} (id integer primary key, label text not null)",
+            quote(&table.name)
+        )
+    }
+
+    /// SQLite has no sleep, but a recursive CTE without a stop never
+    /// produces its count.
+    fn unbounded_statement_sql(&self, marker: &str) -> String {
+        format!("WITH RECURSIVE r(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM r) SELECT count(*), '{marker}' FROM r")
+    }
+
+    /// Hold an exclusive lock on a connection of the engine's own for
+    /// `hold`, and return a read that waits for it in the busy handler.
+    async fn blocking_statement_sql(&self, marker: &str, hold: Duration) -> Result<String, ConformanceFailure> {
+        let options = SqliteConnectOptions::new().filename(&self.path);
+        let mut holder = SqliteConnection::connect_with(&options)
+            .await
+            .map_err(ConformanceFailure::setup)?;
+        holder
+            .execute("BEGIN EXCLUSIVE")
+            .await
+            .map_err(ConformanceFailure::setup)?;
+        tokio::spawn(async move {
+            tokio::time::sleep(hold).await;
+            let _ = holder.execute("ROLLBACK").await;
+            let _ = holder.close().await;
+        });
+        Ok(format!("SELECT count(*), '{marker}' FROM sqlite_schema"))
+    }
+
+    fn row_source_sql(&self, rows: u64) -> String {
+        format!("WITH RECURSIVE r(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM r WHERE n < {rows}) SELECT n FROM r")
+    }
+}
+
+impl PooledConformanceEngine for SqliteEngine {
+    const WORKSPACE_POOL_MAX_SIZE: usize = tablepro_driver_sqlite::WORKSPACE_POOL_MAX_SIZE;
+}
+
+impl EditorTransactionEngine for SqliteEngine {
+    fn begin_sql(&self) -> String {
+        "BEGIN".to_owned()
+    }
+
+    fn commit_sql(&self) -> String {
+        "COMMIT".to_owned()
+    }
+}
+
+impl ConformanceDdlEngine for SqliteEngine {
+    fn integer_type(&self) -> &'static str {
+        "INTEGER"
+    }
+
+    fn text_type(&self) -> &'static str {
+        "TEXT"
+    }
+
+    fn timestamp_type(&self) -> &'static str {
+        "TEXT"
+    }
+
+    fn current_timestamp_expression(&self) -> &'static str {
+        "CURRENT_TIMESTAMP"
+    }
+}
+
+impl LargeTableEngine for SqliteEngine {
+    fn fill_scratch_table_sql(&self, table: &TableRef, rows: u64, label_width: usize) -> Vec<String> {
+        vec![format!(
+            "WITH RECURSIVE numbers(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM numbers WHERE n < {rows}) \
+             INSERT INTO {} (id, label) SELECT n, printf('%0{label_width}d', n) FROM numbers",
+            quote(&table.name)
+        )]
+    }
+}

linux/crates/drivers/sqlite/tests/integration/support.rs

@@ -6,8 +6,6 @@ use std::sync::{Arc, Mutex, PoisonError};
 use std::time::Duration;
 
 use async_trait::async_trait;
-use sqlx::Connection as _;
-use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection};
 use tablepro_core::meta::TableRef;
 use tablepro_core::statement::{GuardedStatement, Statement, StatementBuilder};
 use tablepro_core::{
@@ -21,6 +19,8 @@ use tempfile::TempDir;
 use tokio::time::Instant;
 use tokio_util::sync::CancellationToken;
 
+use crate::sqlite_engine::create_database;
+
 /// A database file in a directory of its own, removed with it.
 pub struct TestFile {
     directory: TempDir,
@@ -48,13 +48,6 @@ impl TestFile {
     }
 }
 
-/// Make an empty database the way another program would, so the
-/// driver only ever opens a file that exists.
-pub async fn create_database(path: &Path) -> Result<(), sqlx::Error> {
-    let options = SqliteConnectOptions::new().filename(path).create_if_missing(true);
-    SqliteConnection::connect_with(&options).await?.close().await
-}
-
 pub fn writable() -> WorkspaceSettings {
     WorkspaceSettings::workspace(false)
 }