Skip to content

fix(linux): start the ClickHouse fixture without a blind HTTP wait - #3426

Merged
datlechin merged 3 commits into
linuxfrom
fix/linux-clickhouse-fixture-start
Oct 15, 2026
Merged

datlechin merged 3 commits into
linuxfrom
fix/linux-clickhouse-fixture-start

Conversation

@datlechin

Copy link
Copy Markdown
Member

Why

The ClickHouse docker job keeps failing on Linux PRs with container is not ready: container startup timeout. A start either answers within about 4 s or hangs for the whole timeout, and only in the multi-thread suites. Raising the timeout from 60 s to 180 s only made the failure slower. The wait polled /ping with no per-request timeout and never checked whether the container was still running. CLICKHOUSE_DB also made the image start, stop and restart the server before it answered.

Change

  • The fixture no longer sets CLICKHOUSE_DB. Readiness is a CREATE DATABASE IF NOT EXISTS tablepro over HTTP, each attempt capped at 5 s, retried within a 60 s deadline.
  • A failed start reports the reason, the last reply, the container state (exit code, OOM kill) and the last 40 log lines, and a container that exits fails within a couple of seconds.
  • Drops testcontainers' unused HTTP wait feature (reqwest and three other crates).

Verified

A new docker test checks the server starts once; it fails on the old fixture (2 starts). The full ClickHouse docker suite passed 8 local runs. Median start went from 3.4 s to 1.0 s.

Not in this PR

  • The CI hang itself never reproduced locally, so the stalling call inside reqwest or docker-proxy is not identified.

All 12 CI startup timeouts in 14 runs used the full timeout, each in a
multi-thread test, while testcontainers polled /ping with no per-request
timeout. The fixture now creates the database over HTTP in bounded
attempts, so the server starts once and a stuck attempt is retried.
The server now also logs to the console. A start that fails says why:
container exited or no answer by the deadline, the last reply, the
docker state with exit code and OOM kill, and the last log lines.
@datlechin
datlechin merged commit 0216f0f into linux Oct 15, 2026
18 checks passed
@datlechin
datlechin deleted the fix/linux-clickhouse-fixture-start branch October 15, 2026 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant