cortexproject · CharlieTLe · Apr 12, 2026 · Apr 12, 2026 · Apr 12, 2026 · Apr 12, 2026
diff --git a/flaky-tests/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod.md b/flaky-tests/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod.md
@@ -0,0 +1,30 @@
+# Flaky Test: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Multiple subtests failed on arm64 without `-race`. Errors at `blocks_cleaner_test.go:699` ("Not equal") and `blocks_cleaner_test.go:829`/`:866` ("Received unexpected error"). Likely timing-dependent block cleanup assertions that assume synchronous completion.
+
+## Occurrences
+
+### 2026-04-13T21:00:49Z
+- **Job**: [ci / test-no-race (arm64)](https://github.com/cortexproject/cortex/actions/runs/24366476213)
+- **Package**: `github.com/cortexproject/cortex/pkg/compactor`
+- **File**: `pkg/compactor/blocks_cleaner_test.go:699, :829, :866`
+- **Notes**: Failed on arm64 without `-race`. Passed on amd64 (both race and no-race) and arm64 with `-race`.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (0.47s)
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:699
+        Error:      	Not equal:
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:829
+        Error:      	Received unexpected error:
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:699
+        Error:      	Not equal:
+        Error Trace:	/__w/cortex/cortex/pkg/compactor/blocks_cleaner_test.go:866
+        Error:      	Received unexpected error:
+FAIL	github.com/cortexproject/cortex/pkg/compactor	90.399s
+```
+
+</details>
diff --git a/flaky-tests/TestDistributorQuerier_QueryIngestersWithinBoundary.md b/flaky-tests/TestDistributorQuerier_QueryIngestersWithinBoundary.md
@@ -0,0 +1,53 @@
+# Flaky Test: TestDistributorQuerier_QueryIngestersWithinBoundary
+
+**Status**: Fixed
+**Occurrences**: 2
+**Root Cause**: Timing-sensitive test. `now` is captured at test setup (line 561) via `time.Now()`, but the code under test in `newDistributorQueryable` also calls `time.Now()` internally to compute the lookback boundary. Under the race detector (slower execution), enough wall-clock time elapses between setup and execution that the internal boundary shifts. The "maxT well after lookback boundary" subtest uses only a 10-second margin (`now.Add(-lookback + 10*time.Second)`), which is not enough buffer when the race detector slows execution. The result is that `queryMaxT` falls before the internally-computed boundary, the query is skipped, and `distributor.Calls` is empty instead of having 1 call.
+
+## Occurrences
+
+### 2026-04-12T19:09:52Z
+- **Job**: [ci / test (amd64)](https://github.com/cortexproject/cortex/actions/runs/24314068155)
+- **Package**: `github.com/cortexproject/cortex/pkg/querier`
+- **File**: `pkg/querier/distributor_queryable_test.go:638`
+- **Subtest**: `maxT_well_after_lookback_boundary`
+- **Notes**: Passed on arm64, passed on test-no-race (amd64), failed only under `-race` on amd64.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary (0.00s)
+    --- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary (0.01s)
+        distributor_queryable_test.go:638:
+            	Error Trace:	/__w/cortex/cortex/pkg/querier/distributor_queryable_test.go:638
+            	Error:      	"[]" should have 1 item(s), but has 0
+            	Test:       	TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary
+            	Messages:   	should manipulate when maxT is well after boundary
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/querier	34.451s
+```
+
+</details>
+
+### 2026-04-12T20:32:32Z
+- **Job**: [ci / test (amd64)](https://github.com/cortexproject/cortex/actions/runs/24315645679)
+- **Package**: `github.com/cortexproject/cortex/pkg/querier`
+- **File**: `pkg/querier/distributor_queryable_test.go:638`
+- **Subtest**: `maxT_well_after_lookback_boundary`
+- **Notes**: Same failure, again on amd64 with `-race`. Passed on arm64 and test-no-race.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary (0.00s)
+    --- FAIL: TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary (0.00s)
+        distributor_queryable_test.go:638:
+            	Error Trace:	/__w/cortex/cortex/pkg/querier/distributor_queryable_test.go:638
+            	Error:      	"[]" should have 1 item(s), but has 0
+            	Test:       	TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary
+            	Messages:   	should manipulate when maxT is well after boundary
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/querier	33.505s
+```
+
+</details>
diff --git a/flaky-tests/TestMinimizeSpreadTokenGenerator.md b/flaky-tests/TestMinimizeSpreadTokenGenerator.md
@@ -0,0 +1,29 @@
+# Flaky Test: TestMinimizeSpreadTokenGenerator
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Floating point precision boundary. The test asserts that token spread distance error is <= 0.01, but the result was 0.01097 — barely over the threshold. The token generation algorithm involves randomness and the assertion threshold is too tight for non-deterministic outcomes. Failed on amd64 without `-race`.
+
+## Occurrences
+
+### 2026-04-16T21:27:42Z
+- **Job**: [ci / test-no-race (amd64)](https://github.com/cortexproject/cortex/actions/runs/24534699120)
+- **Package**: `github.com/cortexproject/cortex/pkg/ring`
+- **File**: `pkg/ring/token_generator_test.go:202` (called from `:106`)
+- **Notes**: Error was 0.01097, threshold is 0.01. Passed on arm64 and amd64 with `-race`.
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestMinimizeSpreadTokenGenerator (2.03s)
+    token_generator_test.go:202:
+        Error Trace:	/__w/cortex/cortex/pkg/ring/token_generator_test.go:202
+                    	/__w/cortex/cortex/pkg/ring/token_generator_test.go:106
+        Error:      	Condition failed!
+        Test:       	TestMinimizeSpreadTokenGenerator
+        Messages:   	[minimize-42-zone1] expected and real distance error is greater than 0.01 -> 0.010970902851567321[8.2595524e+07/8.3511723e+07]
+FAIL
+FAIL	github.com/cortexproject/cortex/pkg/ring	67.285s
+```
+
+</details>
diff --git a/flaky-tests/TestParquetFuzz.md b/flaky-tests/TestParquetFuzz.md
@@ -0,0 +1,26 @@
+# Flaky Test: TestParquetFuzz
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Fuzz test with non-deterministic query generation. The test generates random PromQL queries and compares Cortex results against Prometheus. 1 out of N randomly generated test cases failed, likely due to floating point precision differences or edge cases in query evaluation. This is inherently non-deterministic.
+
+## Occurrences
+
+### 2026-04-13T20:44:25Z
+- **Job**: [ci / integration (ubuntu-24.04, amd64, integration_query_fuzz)](https://github.com/cortexproject/cortex/actions/runs/24365433548)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/parquet_querier_test.go:176`
+- **Notes**: "1 test cases failed" out of the fuzz suite. Non-deterministic by nature.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/home/runner/work/cortex/cortex/integration/parquet_querier_test.go:176
+        Error:      	finished query fuzzing tests
+        Test:       	TestParquetFuzz
+        Messages:   	1 test cases failed
+--- FAIL: TestParquetFuzz (24.12s)
+FAIL	github.com/cortexproject/cortex/integration	200.599s
+```
+
+</details>
diff --git a/flaky-tests/TestQuerierWithBlocksStorageLimits.md b/flaky-tests/TestQuerierWithBlocksStorageLimits.md
@@ -0,0 +1,27 @@
+# Flaky Test: TestQuerierWithBlocksStorageLimits
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: The test expects HTTP 422 (Unprocessable Entity) for a query exceeding limits but received HTTP 500 (Internal Server Error) instead. This is likely a race where the store-gateway or querier hasn't fully initialized its limit-checking path, causing the error to surface as a generic 500 rather than the expected 422. Failed on arm64.
+
+## Occurrences
+
+### 2026-04-16T18:18:08Z
+- **Job**: [ci / integration (ubuntu-24.04-arm, arm64, integration_querier)](https://github.com/cortexproject/cortex/actions/runs/24526116820)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/querier_test.go:468`
+- **Notes**: Expected 422, got 500. Passed on amd64 in the same run.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/home/runner/work/cortex/cortex/integration/querier_test.go:468
+        Error:      	Not equal:
+                    	expected: 422
+                    	actual  : 500
+        Test:       	TestQuerierWithBlocksStorageLimits
+--- FAIL: TestQuerierWithBlocksStorageLimits (11.00s)
+FAIL	github.com/cortexproject/cortex/integration	193.371s
+```
+
+</details>
diff --git a/flaky-tests/TestQuerierWithBlocksStorageRunningInSingleBinaryMode.md b/flaky-tests/TestQuerierWithBlocksStorageRunningInSingleBinaryMode.md
@@ -0,0 +1,28 @@
+# Flaky Test: TestQuerierWithBlocksStorageRunningInSingleBinaryMode
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Transient Docker infrastructure failure. The Consul container (`e2e-cortex-test-consul`) disappeared mid-test, causing `Error response from daemon: No such container: e2e-cortex-test-consul`. This is a CI environment issue, not a code bug. The test passed on amd64 in the same run and passed on arm64 in run #6.
+
+## Occurrences
+
+### 2026-04-12T21:00:08Z
+- **Job**: [ci / integration (ubuntu-24.04-arm, arm64, integration_querier)](https://github.com/cortexproject/cortex/actions/runs/24316060467)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/querier_test.go:217`
+- **Subtest**: `[TSDB]_blocks_sharding_enabled,_redis_index_cache,_bucket_index_enabled,thanosEngine=false`
+- **Notes**: Passed on amd64 in the same run. Docker container `e2e-cortex-test-consul` vanished during test execution.
+
+<details><summary>Build logs</summary>
+
+```
+20:57:39 Error response from daemon: No such container: e2e-cortex-test-consul
+        Error Trace:	/home/runner/work/cortex/cortex/integration/querier_test.go:217
+        Error:      	Received unexpected error:
+--- FAIL: TestQuerierWithBlocksStorageRunningInSingleBinaryMode (69.50s)
+    --- FAIL: TestQuerierWithBlocksStorageRunningInSingleBinaryMode/[TSDB]_blocks_sharding_enabled,_redis_index_cache,_bucket_index_enabled,thanosEngine=false (34.33s)
+FAIL
+FAIL	github.com/cortexproject/cortex/integration	279.595s
+```
+
+</details>
diff --git a/flaky-tests/TestQuerierWithStoreGatewayDataBytesLimits.md b/flaky-tests/TestQuerierWithStoreGatewayDataBytesLimits.md
@@ -0,0 +1,30 @@
+# Flaky Test: TestQuerierWithStoreGatewayDataBytesLimits
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Transient Docker infrastructure failure. Multiple containers vanished mid-test (`e2e-cortex-test-consul`, `e2e-cortex-test-distributor`, `e2e-cortex-test-minio-9000`, `e2e-cortex-test-store-gateway`, `e2e-cortex-test-ingester`, `e2e-cortex-test-querier`). Same infrastructure instability pattern as TestQuerierWithBlocksStorageRunningInSingleBinaryMode. The test passed on arm64 in the same run and on both arches in runs #1-#4.
+
+## Occurrences
+
+### 2026-04-12T21:37:26Z
+- **Job**: [ci / integration (ubuntu-24.04, amd64, integration_querier)](https://github.com/cortexproject/cortex/actions/runs/24316771541)
+- **Package**: `github.com/cortexproject/cortex/integration`
+- **File**: `integration/querier_test.go:565`
+- **Notes**: Widespread Docker container disappearance. All `e2e-cortex-test-*` containers returned "No such container" errors.
+
+<details><summary>Build logs</summary>
+
+```
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-distributor
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-minio-9000
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-store-gateway
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-ingester
+21:34:11 Error response from daemon: No such container: e2e-cortex-test-querier
+21:34:19 Error response from daemon: No such container: e2e-cortex-test-consul
+        Error Trace:	/home/runner/work/cortex/cortex/integration/querier_test.go:565
+        Error:      	Not equal:
+--- FAIL: TestQuerierWithStoreGatewayDataBytesLimits (10.69s)
+FAIL	github.com/cortexproject/cortex/integration	202.827s
+```
+
+</details>
diff --git a/flaky-tests/TestQueueConcurrency.md b/flaky-tests/TestQueueConcurrency.md
@@ -0,0 +1,31 @@
+# Flaky Test: TestQueueConcurrency
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Deadlock/hang in concurrent test. The test spawns 30 goroutines that concurrently enqueue, dequeue, and delete from a queue. Goroutines where `cnt%5 == 0` (and odd) call `dequeueRequest(0, false)` which blocks on a channel waiting for an item. If other goroutines have already drained the queue or called `deleteQueue`, the dequeue goroutine blocks forever — there will never be another enqueue to unblock it. This causes the entire test to hang until the 30-minute timeout. The issue manifests more on arm64 with `-race` due to slower execution and different goroutine scheduling.
+
+## Occurrences
+
+### 2026-04-12T20:23:09Z
+- **Job**: [ci / test (arm64)](https://github.com/cortexproject/cortex/actions/runs/24314927948)
+- **Package**: `github.com/cortexproject/cortex/pkg/scheduler/queue`
+- **File**: `pkg/scheduler/queue/user_queues_test.go:461`
+- **Notes**: Timed out after 30m on arm64 with `-race`. Passed on amd64 with `-race`, passed on both arches without `-race`. Goroutine dump shows `dequeueRequest` stuck waiting on channel at `user_request_queue.go:35`.
+
+<details><summary>Build logs</summary>
+
+```
+panic: test timed out after 30m0s
+	running tests:
+		TestQueueConcurrency (29m51s)
+
+goroutine 100 [chan receive, 29 minutes]:
+github.com/cortexproject/cortex/pkg/scheduler/queue.(*FIFORequestQueue).dequeueRequest(0xc000410080, 0xc8ea31?, 0x6?)
+	/__w/cortex/cortex/pkg/scheduler/queue/user_request_queue.go:35 +0x48
+github.com/cortexproject/cortex/pkg/scheduler/queue.TestQueueConcurrency.func1(0xf)
+	/__w/cortex/cortex/pkg/scheduler/queue/user_queues_test.go:477 +0x280
+
+FAIL	github.com/cortexproject/cortex/pkg/scheduler/queue	1800.101s
+```
+
+</details>
diff --git a/flaky-tests/TestRuler_rules.md b/flaky-tests/TestRuler_rules.md
@@ -0,0 +1,25 @@
+# Flaky Test: TestRuler_rules
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Same alert state race as TestRuler_rules_limit. The test expects `"state":"unknown"` for an alerting rule but gets `"state":"inactive"`. The ruler evaluates rules asynchronously after startup, and the alert state transitions before the HTTP response is captured. This manifested in the `integration-configs-db` job which runs the ruler tests with a database backend.
+
+## Occurrences
+
+### 2026-04-13T21:05:42Z
+- **Job**: [ci / integration-configs-db (ubuntu-24.04, amd64)](https://github.com/cortexproject/cortex/actions/runs/24366476213)
+- **Package**: `github.com/cortexproject/cortex/pkg/ruler`
+- **File**: `pkg/ruler/api_test.go:306`
+- **Notes**: Same `"state":"unknown"` vs `"state":"inactive"` pattern as TestRuler_rules_limit.
+
+<details><summary>Build logs</summary>
+
+```
+        Error Trace:	/go/src/github.com/cortexproject/cortex/pkg/ruler/api_test.go:306
+        Error:      	Not equal:
+                    	expected: ..."state":"unknown"...
+                    	actual  : ..."state":"inactive"...
+--- FAIL: TestRuler_rules (0.05s)
+```
+
+</details>
diff --git a/flaky-tests/TestRuler_rules_limit.md b/flaky-tests/TestRuler_rules_limit.md
@@ -0,0 +1,28 @@
+# Flaky Test: TestRuler_rules_limit
+
+**Status**: Skipped
+**Occurrences**: 1
+**Root Cause**: Race condition in alert state. The test expects `"state":"unknown"` for an alerting rule but gets `"state":"inactive"`. The ruler evaluates rules asynchronously after startup, and under the race detector the alert state transitions from "unknown" to "inactive" before the HTTP response is captured. This is a timing-dependent assertion.
+
+## Occurrences
+
+### 2026-04-13T20:43:11Z
+- **Job**: [ci / test (amd64)](https://github.com/cortexproject/cortex/actions/runs/24365433548)
+- **Package**: `github.com/cortexproject/cortex/pkg/ruler`
+- **File**: `pkg/ruler/api_test.go:422`
+- **Notes**: Failed on amd64 with `-race`. The diff is `"state":"unknown"` (expected) vs `"state":"inactive"` (actual).
+
+<details><summary>Build logs</summary>
+
+```
+--- FAIL: TestRuler_rules_limit (0.06s)
+    api_test.go:422:
+        Error Trace:	/__w/cortex/cortex/pkg/ruler/api_test.go:422
+        Error:      	Not equal:
+                    	expected: ..."state":"unknown"...
+                    	actual  : ..."state":"inactive"...
+        Test:       	TestRuler_rules_limit
+FAIL	github.com/cortexproject/cortex/pkg/ruler	43.279s
+```
+
+</details>
diff --git a/flaky-tests/audit-log.md b/flaky-tests/audit-log.md
@@ -0,0 +1,25 @@
+# Flaky Test Audit Log
+
+This file tracks every CI run on the `flaky-test-audit` branch. Any test failure on this branch is a flaky test since no test logic has been modified from `master`.
+
+| Timestamp | Result | Details | CI Job |
+|-----------|--------|---------|--------|
+| 2026-04-12T19:09:52Z | FLAKY | TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary (pkg/querier) | [ci run 24314068155](https://github.com/cortexproject/cortex/actions/runs/24314068155) |
+| 2026-04-12T19:47:00Z | PASS | No flaky tests detected | [ci run 24314518781](https://github.com/cortexproject/cortex/actions/runs/24314518781) |
+| 2026-04-12T20:23:09Z | FLAKY | TestQueueConcurrency (pkg/scheduler/queue) — timed out after 30m on arm64 | [ci run 24314927948](https://github.com/cortexproject/cortex/actions/runs/24314927948) |
+| 2026-04-12T20:32:32Z | FLAKY | TestDistributorQuerier_QueryIngestersWithinBoundary/maxT_well_after_lookback_boundary (pkg/querier) — occurrence #2 | [ci run 24315645679](https://github.com/cortexproject/cortex/actions/runs/24315645679) |
+| 2026-04-12T20:49:56Z | FLAKY | TestQuerierWithBlocksStorageRunningInSingleBinaryMode (integration/querier, arm64) — Docker container vanished; TestQueueConcurrency timeout (arm64) — already skipped | [ci run 24316060467](https://github.com/cortexproject/cortex/actions/runs/24316060467) |
+| 2026-04-12T21:22:00Z | PASS | No flaky tests detected (with skips applied) | [ci run 24316099145](https://github.com/cortexproject/cortex/actions/runs/24316099145) |
+| 2026-04-12T21:37:26Z | FLAKY | TestQuerierWithStoreGatewayDataBytesLimits (integration/querier, amd64) — Docker containers vanished | [ci run 24316771541](https://github.com/cortexproject/cortex/actions/runs/24316771541) |
+| 2026-04-13T20:43:11Z | FLAKY | TestRuler_rules_limit (pkg/ruler) — alert state race; TestParquetFuzz (integration/query_fuzz) — fuzz non-determinism; requires_docker — Docker install failure | [ci run 24365433548](https://github.com/cortexproject/cortex/actions/runs/24365433548) |
+| 2026-04-13T21:00:49Z | FLAKY | TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (pkg/compactor, arm64); TestRuler_rules (pkg/ruler, configs-db) — alert state race | [ci run 24366476213](https://github.com/cortexproject/cortex/actions/runs/24366476213) |
+| 2026-04-16T17:27:44Z | INFRA | integration_overrides Docker install failed — Docker Hub rate limit (toomanyrequests). All tests passed. | [ci run 24524134384](https://github.com/cortexproject/cortex/actions/runs/24524134384) |
+| 2026-04-16T17:50:32Z | INFRA | requires_docker Docker install failed — Docker Hub rate limit (toomanyrequests). All tests passed. | [ci run 24525138727](https://github.com/cortexproject/cortex/actions/runs/24525138727) |
+| 2026-04-16T18:18:08Z | FLAKY | TestQuerierWithBlocksStorageLimits (integration/querier, arm64) — expected 422 got 500 | [ci run 24526116820](https://github.com/cortexproject/cortex/actions/runs/24526116820) |
+| 2026-04-16T18:52:00Z | PASS | All jobs passed, no flaky tests detected | [ci run 24527178425](https://github.com/cortexproject/cortex/actions/runs/24527178425) |
+| 2026-04-16T19:15:00Z | PASS | All jobs passed, no flaky tests detected (2nd consecutive clean run) | [ci run 24528187016](https://github.com/cortexproject/cortex/actions/runs/24528187016) |
+| 2026-04-16T19:38:00Z | PASS | All jobs passed (3rd consecutive clean run) | [ci run 24529191249](https://github.com/cortexproject/cortex/actions/runs/24529191249) |
+| 2026-04-16T20:00:00Z | PASS | All jobs passed (4th consecutive clean run) | [ci run 24530173026](https://github.com/cortexproject/cortex/actions/runs/24530173026) |
+| 2026-04-16T20:58:00Z | PASS | All jobs passed (5th consecutive clean run) | [ci run 24532713349](https://github.com/cortexproject/cortex/actions/runs/24532713349) |
+| 2026-04-16T21:22:00Z | PASS | All jobs passed (6th consecutive clean run) | [ci run 24533695679](https://github.com/cortexproject/cortex/actions/runs/24533695679) |
+| 2026-04-16T21:27:42Z | FLAKY | TestMinimizeSpreadTokenGenerator (pkg/ring) — floating point precision boundary (0.01097 > 0.01 threshold) | [ci run 24534699120](https://github.com/cortexproject/cortex/actions/runs/24534699120) |