cluster-e2e — clustered SQLite on ePHPm, two ways

Two-node ePHPm cluster fixture with side-by-side comparison of the two clustered-SQLite paths ePHPm ships:

• mode-turso-cdc — [db.sqlite] engine = "turso" + [db.sqlite.replication] cdc_experimental = true + cluster.channel v1. The primary tails its local Turso engine's turso_cdc table and ships per-txn batches over an authenticated yamux stream; replicas apply. Turso Database is Beta upstream; this entire path is EXPERIMENTAL-labeled.
• mode-sqld — the shipping-production default. rusqlite in-process for reads, sqld sidecar spawned as a child process for writes, WAL frames streamed between primary and replica over sqld's gRPC channel.

Same topology, same static container IPs, same WordPress+wp-cli workload, same cluster-secret. Engine swap is the only change.

Everything here is upstream except five load-bearing temporary patches that all carry the marker UPSTREAM_PENDING_7855 so we can grep them out cleanly once the upstream fix lands. See BUILDING.md and evidence/pin-lines.txt.

Topology

┌────────────────────────────── podman network cluster-e2e-net (10.89.42.0/24)
│
│   ┌──────────── ephpm-a (10.89.42.10) ────────────┐
│   │  HTTP  :8080  →  host :18100                    │
│   │  MySQL :3306  →  host :13306                    │
│   │  gossip :8093  (cluster-internal)               │
│   │  channel :8094 (cluster-internal, mode-turso-cdc)│
│   │  sqld gRPC :5001 (cluster-internal, mode-sqld)  │
│   │  KV data :7947 (cluster-internal)               │
│   └─────────────────────────────────────────────────┘
│                             ↑↓  gossip + KV replication
│                             ↑↓  turso-cdc: yamux stream on :8094
│                             ↑↓  sqld:      gRPC on :5001
│   ┌──────────── ephpm-b (10.89.42.11) ────────────┐
│   │  HTTP  :8080  →  host :18200                    │
│   │  MySQL :3306  →  host :13307                    │
│   │  (same ports internally, dedicated container IP) │
│   └─────────────────────────────────────────────────┘
│
└─────── wp-runner / bench-runner (docker.io/library/wordpress:cli, ghcr.io/hatoo/oha)
                                    also on the same network to eliminate host-network variance

Host-side ports (18100/18200/13306/13307) avoid 8080/9090/18443/9944/ 9950/9951/9960 per the ephemerd runner-fleet conventions.

How to reproduce

./scripts/build-binary.sh                     # ~5-40 min (WSL Rust build)
./scripts/down.sh --wipe-data                 # clean slate
./scripts/up.sh turso-cdc                     # or:  scripts/up.sh sqld
./scripts/wp-install.sh                       # wp core install through ephpm-a
./scripts/replicate-check.sh turso-cdc        # siteurl match + post-lag histogram
./scripts/bench-read.sh turso-cdc 1           # oha home-page load (c=1,4,16,64)
./scripts/bench-write.sh turso-cdc 1          # parallel wp post create clients
./scripts/failover.sh turso-cdc               # kill primary, measure promotion

scripts/bench-run-all.sh chains the full sweep (both modes × three iterations × three workloads); expect ~1 hour end-to-end on this laptop when everything works.

Both modes now reach the "replication proven" step end-to-end. The mode-sqld leg required a one-line ephpm-sqld fix (PR #189, commit 93e920c) that omits --grpc-listen-addr when spawning a Replica sqld — see evidence/sqld-phase-c-transcript.txt for the post-fix bring-up and evidence/sqld-mode-startup-failure.txt for the pre-fix runtime error that surfaced it.

Phase A — does upstream turso#7855 close the ePHPm incident?

The ePHPm side has been carrying a heal in litewire-turso that force-restarts autocommit after every pragma-TVF read (SELECT ... FROM pragma_table_info(...)) because the first such read leaves the connection in a phantom-transaction state: subsequent writes appear locally but never commit. Without the heal, dbDelta() (WordPress upgrade path) silently drops schema.

Upstream root-caused it (PragmaVirtualTableCursor outlives Halt so is_nested_stmt() misclassifies the outer statement) and shipped turso#7855 with a regression test suite mirroring the exact incident shape.

Verification method: point litewire's [patch.crates-io] at luthermonson/turso@fix/pragma-vtab-txn-state, replace sql_uses_pragma_tvf with false (heal inert), run the regression suite plus WordPress wp core install end-to-end.

Result: PASS. All 256 tests across litewire backend/translate/mysql /turso pass with the heal off; wp core install completes on the primary through both modes (see evidence/phase-a-transcript.txt and evidence/phase-c-transcript.txt §2).

Update 2026-07-16: #7855 merged upstream. The pin has moved from the fork branch to the immutable upstream main merge commit tursodatabase/turso@f2ba5a2, and the litewire-turso suite (44 tests, incl. pragma_tvf_read_does_not_poison_session) re-passes on that rev with the heal still off. litewire#11 also merged (699e634). Remaining unwind waits only on a turso crates.io release containing the fix — see BUILDING.md.

Phase B — bring up the cluster

scripts/up.sh <mode> creates a /24 podman network at 10.89.42.0/24, assigns each ephpm container a static IP (10.89.42.10 / .11), and starts them 8 seconds apart. Static IPs are non-negotiable in both modes:

• turso-cdc — the ephpm@bc24b2a plumbing fix computes the cluster channel's advertise address by falling back to gossip's listen IP when the channel is bound on a wildcard. If both are wildcards, the ephpm process refuses to start with a clear config error pointing at [cluster] bind / [cluster.channel] listen.
• sqld — sqlite_election publishes --grpc-listen-addr verbatim to peers as the "dial the primary here" URL. If the primary bound 0.0.0.0, the replica dials http://0.0.0.0:5001 on its own local stack and gets refused. Same class of bug, no fix yet in the sqld path.

The 8-second stagger between node-a and node-b start is deliberate. The election is gossip-KV lowest-ordinal-wins; both nodes bootstrap as Primary in the ~1s before they can see each other via gossip. If they race, the second writer wins the last-write-wins gossip KV sqlite:primary slot and the first-started node has to demote. Not a consensus algorithm — see Phase D item #6.

Phase C — prove replication

mode-turso-cdc: PASS (end-to-end, this is the story)

scripts/replicate-check.sh turso-cdc on a fresh cluster + wp-install returns:

Test	Result	Detail
wp option get siteurl on both nodes	PASS	matches on node-a and node-b
Single post, node-a → node-b	PASS	~14 s wall clock
50 posts sequential, replica lag distribution	PASS	n=50, p50=13.1 s, p99=17.6 s

Machine-readable line: REPLCHECK mode=turso-cdc siteurl_ab=PASS post_lag_ms=13976 p50_ms=13068 p99_ms=17561 samples=50. Full transcript: evidence/phase-c-transcript.txt §3. Pre-plumbing-fix transcript (before ephpm@bc24b2a) preserved at evidence/archive-pre-plumbing-fix/ — that's the run that surfaced the URL-form parse bug and the wildcard-advertise bug in the first place.

mode-sqld: PASS (post ephpm/ephpm#189)

scripts/up.sh sqld boots the cluster, both nodes bootstrap as Primary in the ~4s before gossip converges, then the last-write-wins KV slot resolves and one node demotes to Replica. The just-demoted node re-spawns its sqld child in Replica mode; before ephpm/ephpm#189 (commit 93e920c) this failed with:

error: the argument '--grpc-listen-addr <GRPC_LISTEN_ADDR>' cannot be
       used with '--primary-grpc-url <PRIMARY_GRPC_URL>'

The fix in ephpm-sqld/src/lib.rs::build_command composes the sqld CLI conditionally — Primary gets --grpc-listen-addr, Replica gets --primary-grpc-url, never both — and adds regression tests. With that binary in bin/ephpm, mode-sqld now runs the same three checks as mode-turso-cdc:

scripts/replicate-check.sh sqld on a fresh cluster + wp-install returns:

Test	Result	Detail
wp option get siteurl on both nodes	PASS	matches on node-a and node-b
Single post, node-a → node-b	PASS	~15.3 s wall clock
50 posts sequential, replica lag distribution	PASS	n=50, p50=14.0 s, p99=97.0 s

Machine-readable line: REPLCHECK mode=sqld siteurl_ab=PASS post_lag_ms=15268 p50_ms=14031 p99_ms=96988 samples=50. Full transcript: evidence/sqld-phase-c-transcript.txt. Pre-fix runtime error preserved at evidence/sqld-mode-startup-failure.txt.

Phase C — side-by-side comparison

Same fixture, same rig, engine swap only. mode-sqld numbers below are from this run; mode-turso-cdc from the previous chapter (bench/results/mode-turso-cdc/).

Read-heavy (oha 20 s, WP home page rendered through node-a):

c	turso-cdc rps	sqld rps	turso-cdc p50	sqld p50	turso-cdc p99	sqld p99	success
1	1.60	1.60	628 ms	657 ms	718 ms	730 ms	100%
4	12.40	12.05	336 ms	287 ms	423 ms	546 ms	100%
16	26.25	27.55	576 ms	542 ms	1735 ms	1713 ms	100%
64	34.44	39.60	1640 ms	1439 ms	6622 ms	5635 ms	100%

Write-heavy (serial round-robin wp post create, 5 per client — see Phase D #9):

c	turso-cdc write rps	sqld write rps	turso-cdc catchup	sqld catchup	replicated (both)
1	0.20	0.18	5695 ms	7299 ms	5/5
4	0.20	0.20	7198 ms	4793 ms	20/20
16	0.19	0.20	4608 ms	4144 ms	80/80

Install-time (wp core install wall clock through the primary, bench/results/mode-*/install/iter-1.txt):

Mode	Install wall (ms)	Homepage HTTP
turso-cdc	64 530	200
sqld	64 683	200

Replication-lag distribution (50 sequential posts):

Mode	post_lag (single)	p50	p99	samples	passed
turso-cdc	13 976 ms	13 068 ms	17 561 ms	50	50/50
sqld	15 268 ms	14 031 ms	96 988 ms	50	50/50

Read the A/B:

• Reads scale identically across c=1–16; sqld pulls +15% rps at c=64. Both saturate around 35–40 rps at c=64 — the ceiling is the PHP+MySQL-wire path on the primary, not the storage engine.
• Writes are near-parity in throughput (both ~0.2 rps, floor bound by wp-cli wall time — Phase D #9). Catchup on the replica is faster on sqld at c=4/16 (WAL-frame streaming is tighter than the turso CDC batch tail), slower at c=1 (cold-path setup dominates).
• Zero writes dropped in either mode across all concurrencies.
• p99 replication lag is sqld's outlier — 97 s vs turso-cdc's 17.6 s. Median is comparable (14 s vs 13 s). The p99 blowout came from one iteration in the 50-post loop; the single-post Test 2 didn't reproduce it. Worth revisiting if this bench ever gates a release; for a demonstration fixture, functional correctness end-to-end is proven.

Same fixture, no rate-limiter taint, no cluster-side flag flips. Engine swap is the only variable.

Phase D — honesty checklist

Discovered by this exercise (all real bugs, all filed against the built ephpm binary of feat/turso-cdc-replication @ 41a3726):

0. ephpm-sqld/src/lib.rs::spawn_child passes --grpc-listen-addr even for Replica roles → sqld exits immediately → mode-sqld can't replicate at all. FIXED in ephpm/ephpm#189 (commit 93e920c), cherry-picked into the fixture build. See Phase C mode-sqld section above for the post-fix bring-up. 0.5. The turso-cdc tail loop starts before any wire session enables capture_data_changes_conn, so both nodes spam ERROR ephpm_server::turso_cdc: CDC tail poll error: SQLite error: Parse error: no such table: turso_cdc at ~10 Hz until the first PHP request lands. Loop recovers cleanly; user-hostile in the interim.

Split-brain-at-boot — election is lowest-ordinal wins over gossip KV, but if both nodes bootstrap before gossip finishes exchanging state each writes itself into the sqlite:primary KV slot and the second writer wins by last-write-wins. Workaround: scripts/up.sh stages 8s between node starts. Consensus (Raft/Paxos) would eliminate this.

Design gaps still applicable (from the original readme):

1. Fresh replica bootstrap is not implemented — a brand-new node-b started against an already-populated node-a only sees writes from the moment its cdc/default stream opens. Operator ships the initial DB file out-of-band or accepts forward-only replication. Snapshot-over-transport is reserved in ephpm_cluster::stream_type::SNAPSHOT_PREFIX but not built. (The bench works around this by starting both nodes empty and letting wp-install run through the primary.)
2. Kill-primary loses unshipped batches — a committed transaction on the primary whose CDC row hadn't yet been polled + broadcast + written by the subscriber is gone. Same divergence class as sqld async replication. No sync/quorum knob.
3. Watermark resets on restart — CdcTailer::new(&mgmt, 0) starts from cursor 0; on a primary restart it re-scans and re-broadcasts already-replayed batches. apply_batch is idempotent so replicas aren't corrupted, but bandwidth is wasted.
4. Turso engine limitations — VACUUM is rejected. Empty result sets from unexecuted prepared SELECTs need column_decltype to fill in types; when Turso 0.7 can't provide it, the wire frontend falls back to a LIMIT 0 probe. Some pragma paths still surprising. Also: wp core install's transient-cleanup step uses MySQL multi-table DELETE a, b FROM t a, t b WHERE ... — not translated by litewire, logged as a WordPress database error but install still succeeds.
5. TLS between nodes is deferred. [cluster.channel] handshakes with a shared secret (HKDF-derived ChaCha20-Poly1305 for the KV plane); the cluster channel data plane is opaque yamux over TCP — authenticated by the channel handshake, not TLS-wrapped.
6. Election is gossip-KV lowest-ordinal-wins. Stable and simple, no Raft/Paxos underneath. Split brains during partitions avoided by TTL-heartbeat expiry, not consensus.
7. PHP mode = "fpm" (request-per-thread). WordPress runs stock; the ePHPm worker adapter is a separate story and not tested here.
8. This is a loopback-network single-host bench, not a bare-metal-cluster bench. Numbers below reflect a single Windows 11 laptop running podman-machine (WSL2) with both containers on the same podman bridge — network latency is zero-ish, contention is CPU-only. A real cluster on real network would be different (probably worse on lag, similar on rps ceilings).
9. Write-heavy bench is serial-interleaved, not true-parallel. On this box's git-bash + podman-remote combination, ( inner ) & background subshells hit a container-launch race (podman-run inside the backgrounded subshell never spawned). The bench-write.sh workaround is to round-robin the wp-cli calls one client at a time, so c=N measures N clients each firing sequentially in round-robin order rather than N truly simultaneous. Absolute throughput is a lower bound; relative comparison across c is still meaningful for saturation / lag behaviour.

Numbers

Rig: Windows 11 laptop, podman-machine (WSL2 Ubuntu, 16 vCPU / 2 GB guest), ephemerd fleet OFF during bench (no runner CPU competition). Both modes measured; see the Phase C side-by-side comparison above and bench/CHART.txt for the two-column ASCII rendering.

Because the write bench is serial (see Phase D #9), the "write rps" is essentially "1 / per-write wall time" ≈ 0.2 rps = 5 s per commit through wp-cli → MySQL wire → litewire → (Turso | rusqlite → sqld). The important number is replica catchup — after N × 5 writes on the primary, node-b sees the FULL SET in 4.1–7.3 s across both modes. Zero writes were dropped (replicated = total_posts for every c and every mode).

Numbers here are for one iteration; scripts/bench-run-all.sh chains three iterations and scripts/bench-summarise.py computes medians. For headline numbers: both modes reach ~35–40 rps at c=64 with 100% success at ~1.5 s p50 latency (sqld slightly faster); both replicate every write with zero drops; cross-node write visibility is limited by the ~14 s wp-cli/podman outer-loop cost — not by the storage engine, which streams under 15 s at the median in both modes.

Repo layout

bin/                 built ephpm binary (glibc-dynamic Linux)
configs/mode-*/      per-mode node-a.toml and node-b.toml
docker/              Dockerfile.quick — slim runtime image (binary + tini)
scripts/             build-binary.sh, up.sh, down.sh, wp-install.sh,
                     replicate-check.sh, failover.sh,
                     bench-install.sh, bench-read.sh, bench-write.sh,
                     bench-run-all.sh, bench-summarise.py,
                     lib-wp.sh (shared wp-cli helper)
bench/results/       raw bench output per mode/workload/iteration
evidence/            phase-a-transcript.txt (Phase A verification),
                     phase-c-transcript.txt (post-plumbing-fix
                       cluster + replication run for mode-turso-cdc,
                       plus the original mode-sqld failure that
                       motivated ephpm/ephpm#189)
                     sqld-phase-c-transcript.txt (post-fix mode-sqld
                       cluster + replication + bench run — this
                       chapter's deliverable)
                     replcheck-turso-cdc.txt (raw replicate-check output)
                     sqld-mode-startup-failure.txt (pre-fix runtime log
                       — kept for the ephpm#189 backstory)
                     archive-pre-plumbing-fix/ (the original Phase C
                       run that surfaced the bugs since fixed)
                     pin-lines.txt (every UPSTREAM_PENDING_7855 site
                       plus the ephpm/ephpm#189 marker)
BUILDING.md          how the temp patches are applied and how to unwind

License

MIT. This is a demonstration fixture; the interesting code is upstream in ephpm/ephpm and ephpm/litewire.

---

Assembled by Claude Fable.