Skip to content

ci: split the api IT shard into two hash partitions so it fits its 90-minute budget - #7695

Merged
iliyan-velichkov merged 1 commit into
masterfrom
ci-split-api-shard
Oct 6, 2026
Merged

iliyan-velichkov merged 1 commit into
masterfrom
ci-split-api-shard

Conversation

@iliyan-velichkov

@iliyan-velichkov iliyan-velichkov commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Cause

Since 2026-10-01, the integration-tests-{h2,postgresql} (api, !ui & !slow) jobs have been cancelled at their timeout-minutes: 90 on nearly every master push and nightly. It happened on about 20 jobs, including today's 37275439965, 37275382075 and nightly 37255397525. This is not a hang. In the cancelled h2 job of 37275439965, 133 classes had completed, all green, and the JVM was still starting a new class every ~40 s until the cancel. The sum of their test times is 83.2 min, plus the ~3 min build. Even the last green h2 run took 89 of its 90 minutes. The shard grew from 83 classes (57 / 62 min, the measurement behind #7283) to ~135. Its classes still cost a near-uniform 40-60 s each, almost all of it the per-method Spring context, so, as ci.md already said, @Tag("slow") cannot rebalance it. Splitting it needs a partition mechanism.

Change

  • ClassPartitionFilter (tests-integrations, support/) is a JUnit Platform PostDiscoveryFilter, registered through META-INF/services and applied after the -Dit.groups tag filter. With -Dit.partition=k/n it keeps the classes whose top-level class name hashes into partition k of n. The partitions are disjoint and complete by construction: a new IT lands in exactly one of them without being tagged, and nested classes (MessagingFacadeIT$TopicTest) stay with their enclosing class. A name-pattern split (-Dit.test) could silently drop classes; this one cannot. Without the property every class is kept, so the PR smoke gate and local runs are unchanged. A malformed value fails the run instead of selecting nothing.
  • it.partition is a root-pom property (empty by default), passed to the failsafe forks as the dirigible.it.partition system property.
  • build.yml and nightly.yml: on both DB legs, api becomes api-1 / api-2 (partition: "1/2" / "2/2", same tag expression !ui & !slow & !upgrade, keeping the upgrade exclusion from tests: an UpgradeFromPreviousReleaseIT boots the build on a deployment the previous release wrote (#7641) #7685). The other shards pass an empty partition.
  • ci.md and the workflow comments describe the split, and say to go to /3 when a partition nears the cap again.

Verification

  • Dry runs of -Dit.groups="!ui & !slow & !upgrade" with -Djunit.platform.execution.dryRun.enabled=true, re-run after rebasing on master 21e7d4942d:
    • no partition: 140 classes
    • 1/2: 64 classes
    • 2/2: 76 classes
    • No class appears in both partitions, their union is exactly the unpartitioned set, and UpgradeFromPreviousReleaseIT is in none of them.
    • Before the rebase, against !ui & !slow, the split was 135 = 64 + 71.
  • Expected balance: replaying the partition on today's per-class times from the cancelled h2 api job (133 classes) gives 38.5 / 44.7 min, and from the PostgreSQL job (119 classes reached) 39.5 / 43.4 min. Both are well under the cap, and the job count per leg goes from 4 to 5.
  • ClassPartitionFilterTest: 14/14 green. It covers no or blank partition, exactly one partition per class for n = 1..3, nested classes and methods following their top-level class, descriptors without a class source, and malformed values refused.
  • formatter:validate on tests-integrations, with the cache wiped. The workflow YAML parses.
  • Not verified: an actual api-1 / api-2 run on GitHub Actions. That happens on the first master push or a workflow_dispatch of the nightly after merge.

Separately, the PostgreSQL slow shard failure (Liquibase changelog lock) is fixed in #7693.

🤖 Generated with Claude Code

…-minute budget

The HTTP-level "api" shard (!ui & !slow) has grown to ~135 classes and ~83 min
of test time, and since 2026-10-01 it has been cancelled at the 90-minute job
cap on nearly every master push and nightly, on both databases, with every
class it reached green. Its classes cost a near-uniform 40-60 s each, so no
tag can move a long pole out of it.

-Dit.partition=k/n now keeps the selected classes whose top-level class name
hashes into partition k of n: ClassPartitionFilter, a JUnit PostDiscoveryFilter
registered through META-INF/services and applied after the tag filter. The
partitions are disjoint and complete by construction - a new IT lands in
exactly one of them without being tagged - and nested classes stay with their
enclosing class. Without the property every class is kept, so the PR smoke gate
and local runs are unchanged. build.yml and nightly.yml run "api" as api-1 and
api-2 (1/2, 2/2) on both the H2 and the PostgreSQL leg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iliyan-velichkov iliyan-velichkov self-assigned this Oct 5, 2026
@iliyan-velichkov
iliyan-velichkov merged commit aefa3ef into master Oct 6, 2026
10 checks passed
@iliyan-velichkov
iliyan-velichkov deleted the ci-split-api-shard branch October 6, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant