Preserve overlapping input rows in locked append and consolidation

docs/THREADING.md

@@ -277,7 +277,7 @@ The 64-byte padding between `tail` and `head`
 cache-line ping-pong between producer and consumer collapses
 throughput by 2-10x.
 
-### 5.3 Non-explicit atomic APIs — init and relation identity (5 rows)
+### 5.3 Init and relation identity/nonce allocators (8 rows)
 
 | Anchor (`file:function[#N]`) | Field | Op | Order | Justification |
 |---|---|---|---|---|
@@ -286,14 +286,18 @@ throughput by 2-10x.
 | `relation.c:col_rel_new_identity` | `wl_next_relation_identity` | `atomic_load_explicit` | `relaxed` | Read the candidate identity before the non-wrapping CAS reservation loop; the counter only has to hand out distinct values, no other memory is published through it |
 | `relation.c:col_rel_new_identity#2` | `wl_next_relation_identity` | `atomic_compare_exchange_weak_explicit` | `relaxed`/`relaxed` | Reserve a unique relation identity and retry with the observed value after a lost race; uniqueness comes from the RMW, not from ordering |
 | `relation.c:col_rel_test_set_next_identity` | `wl_next_relation_identity` | `atomic_store_explicit` | `relaxed` | Test-only seam for selecting the terminal allocator state; production allocation is not concurrent with this reset |
+| `relation.c:col_rel_mutation_set_nonce_allocate` | `wl_next_mutation_set_nonce` | `atomic_load_explicit` | `relaxed` | Read the candidate nonce before the nonwrapping CAS reservation loop; the counter only supplies unique admission provenance and publishes no other state |
+| `relation.c:col_rel_mutation_set_nonce_allocate#2` | `wl_next_mutation_set_nonce` | `atomic_compare_exchange_weak_explicit` | `relaxed`/`relaxed` | Reserve a unique nonzero mutation-set nonce and retry with the observed value after a lost race; the RMW provides uniqueness without publishing payload state |
+| `relation.c:wl_columnar_relation_test_set_mutation_nonce` | `wl_next_mutation_set_nonce` | `atomic_store_explicit` | `relaxed` | Test-only seam selects allocator exhaustion or retry states before test admissions; tests do not race this reset with nonce allocation |
 
-These are the sites in `wirelog/` that use the **non-explicit** atomic APIs
-(`atomic_load`/`atomic_store`); they default to `memory_order_seq_cst`.
-The identity allocator uses the same default ordering because the CAS loop
-must reserve each relation identity without reuse; the test-only store is
-only used to exercise allocator exhaustion.
+Only the two `io_adapter.c` rows use non-explicit atomic APIs, which default
+to `memory_order_seq_cst`. The relation identity and mutation-set nonce
+allocator rows use the explicit relaxed order shown in the table: their CAS
+operations reserve unique nonwrapping values and do not publish payload state.
+The test-only stores reset allocator state before tests and do not race with
+allocation.
 
-### 5.4 `wirelog/columnar/join.c` — keyed-join cancel/budget and typed output (20 rows)
+### 5.4 `wirelog/columnar/join.c` — keyed-join cancel/budget and typed output (19 rows)
 
 | Anchor (`file:function[#N]`) | Field | Op | Order | Justification |
 |---|---|---|---|---|
@@ -462,7 +466,7 @@ measured by `bench/bench_intern.c`; baselines are in `docs/INTERN_PERF.md`
 
 ### 5.12 Existing inventory total
 
-21 + 4 + 5 + 19 + 1 + 1 + 1 + 37 + 5 + 7 + 3 = **104 atomic call sites**
+21 + 4 + 8 + 19 + 2 + 2 + 3 + 37 + 5 + 7 + 3 = **111 atomic call sites**
 before the source-access contract below.
 
 ### 5.13 `wirelog/columnar/source_access.h` — relation source gate (21 rows)
@@ -514,7 +518,7 @@ those slots and arena allocations are quiescent.
 | `source_access.h:wl_columnar_source_access_writer_release#2` | `gate->state` | `atomic_compare_exchange_weak_explicit` | release/relaxed | Publish writer payload completion and retry spurious failure |
 | `source_access.h:wl_columnar_source_access_writer_move` | `src->owner->state` | `atomic_load_explicit` | acquire | Confirm that the source token still holds WRITER before moving its address-bound ownership to another token |
 
-### 5.14 `wirelog/columnar/relation.c` and `session.c` — alias ownership and pool promotion (28 rows)
+### 5.14 `wirelog/columnar/relation.c` and `session.c` — alias ownership and pool promotion (30 rows)
 
 The canonical owner's flattened alias count uses `wl_atomic_u64` because a
 quiesced worker can retire its alias while unrelated readers still hold the
@@ -540,6 +544,8 @@ concurrent alias removals cannot underflow the count.
 | `relation.c:col_rel_destroy_checked#2` | `r->source_access.state` | `atomic_store_explicit` | release | Keep the retired pool slot closed until allocator reset or reuse |
 | `relation.c:col_rel_destroy_checked#3` | `r->descriptor_access.state` | `atomic_store_explicit` | release | Keep the retired descriptor closed until allocator reset or reuse |
 | `relation.c:col_rel_install_shared_view_unprotected` | `dst->storage_alias_borrows` | `atomic_store_explicit` | relaxed | A newly installed alias descriptor has no child aliases of its own |
+| `relation.c:wl_columnar_memory_reservation_inert` | `reservation->owner_bits` | `atomic_load_explicit` | relaxed | Confirm a prepared radix workspace reservation has no owner before reusing its caller-owned token storage; the helper is called while the exact mutation lease stabilizes the relation and workspace |
+| `relation.c:wl_columnar_memory_reservation_inert#2` | `reservation->state` | `atomic_load_explicit` | relaxed | Confirm the token is inert before workspace preparation; this is an initialization check, not a concurrent ownership decision, under the caller's mutation lease |
 | `relation.c:wl_columnar_relation_rebind_permit_valid` | `lease->owner->state` | `atomic_load_explicit` | acquire | Validate that the upgraded canonical source gate holds WRITER before publication |
 | `relation.c:wl_columnar_relation_rebind_permit_valid#2` | `lease->secondary_owner->state` | `atomic_load_explicit` | acquire | Validate that the upgraded destination descriptor gate holds WRITER before publication |
 | `relation.c:wl_columnar_relation_install_shared_view_with_lease` | `dst->descriptor_access.state` | `atomic_compare_exchange_weak_explicit` | acquire/relaxed | Upgrade the sole transferable descriptor reader to exclusive descriptor admission and retry spurious failure |
@@ -555,7 +561,7 @@ concurrent alias removals cannot underflow the count.
 | `session.c:session_pool_rel_promote#2` | `src->retained_reservation.owner_bits` | `atomic_load_explicit` | acquire | Promote a committed reservation only when the pool slot still owns it |
 | `session.c:session_pool_rel_promote#3` | `src->storage_alias_borrows` | `atomic_store_explicit` | relaxed | Leave the closed pool tombstone with no child aliases |
 
-104 + 21 + 28 = **153 atomic call sites**.
+111 + 21 + 30 = **162 atomic call sites**.
 
 The `#N` suffix counts all atomic sites in a symbol, regardless of operation;
 the first site remains unsuffixed. `scripts/ci/check-threading-doc.sh` uses
@@ -615,7 +621,7 @@ restores them after moving the image's current reservations to the source.
 | `relation.c:col_rel_mutable_image_commit#5` | `old.storage_alias_borrows` | `atomic_store_explicit` | relaxed | Clear the stack retirement copy's alias count before physical cleanup; the copy is private |
 
 ### 5.16 `wirelog/columnar/memory_governor.c` and `relation.c` — atomic
-replacement admission and compaction (19 rows)
+replacement admission and compaction (27 rows)
 
 Replacement admission temporarily accounts for the new footprint while the
 old reservation remains committed. The overlap CAS is the admission
@@ -680,7 +686,7 @@ the committed token after publication and before a growth transaction.
 | `eval_dedup.c:wl_columnar_eval_dedup_test_fail_next_growth_alloc` | test-only fault flag | `atomic_store_explicit` | release | Arm one allocation refusal before a test invokes dedup growth; excluded from the production library |
 | `eval_dedup.c:wl_columnar_eval_dedup_set_grow` | test-only fault flag | `atomic_exchange_explicit` | acquire-release | Consume the one-shot fault safely when test workers grow dedup tables; excluded from the production library |
 
-The complete source audit now contains **214 atomic call sites**.
+The complete source audit now contains **219 atomic call sites**.
 
 ---
 

scripts/ci/check-threading-doc.sh

@@ -19,7 +19,7 @@ fi
 rows="$tmp_dir/rows"
 sed -nE 's/^\| `([^`]+:[A-Za-z_][A-Za-z0-9_]*(#[0-9]+)?)` \| [^|]* \| `([^`]*)` \|.*/\1\t\3/p' "$doc" >"$rows"
 row_count=$(wc -l <"$rows")
-expected_rows="${WIRELOG_THREADING_EXPECTED_ROWS:-214}"
+expected_rows="${WIRELOG_THREADING_EXPECTED_ROWS:-219}"
 [ "$row_count" -eq "$expected_rows" ] || {
     echo "check-threading-doc: FAIL: expected $expected_rows audit rows, found $row_count" >&2
     exit 1

tests/meson.build

@@ -4272,6 +4272,7 @@ relation_mutation_set_exe = executable(
   backend_src, intern_src, arena_src, thread_src, workqueue_src,
   include_directories: [wirelog_inc, wirelog_src_inc],
   dependencies: [nanoarrow_dep, threads_dep, xxhash_dep, mbedtls_dep, math_dep],
+  c_args: ['-DWL_TEST_MUTATION_SET_HOOK=1'],
 )
 test('relation_mutation_set', relation_mutation_set_exe,
   suite: ['unit', 'tsan'], timeout: 60)

tests/test_consolidate_incremental_delta.c

@@ -84,11 +84,11 @@ consolidation_pause_hook(col_rel_t *relation,
         return;
     state->hook_state_ok = stage == state->expected_stage
         && relation == state->expected_relation
-        && relation->storage_owner == state->expected_owner
+        && relation->storage_owner == relation
         && relation->col_shared == NULL
         && relation->columns[0] != state->old_columns
         && relation->nrows == state->expected_nrows
-        && state->expected_owner->storage_alias_borrows > 0;
+        && state->expected_owner->storage_owner == state->expected_owner;
     for (uint32_t i = 0; state->hook_state_ok && i < relation->nrows; i++)
         state->hook_state_ok = relation->columns[0][i]
             == state->expected_values[i];
@@ -2688,6 +2688,95 @@ test_shared_view_storage_exhaustion(void)
     PASS();
 }
 
+static void
+test_source_exhaustion_precedes_delta_reservation(void)
+{
+    TEST("source exhaustion precedes shared delta COW and clears stale denial");
+
+    col_rel_t *rel = test_rel_alloc(1);
+    col_rel_t *delta_owner = test_rel_alloc(1);
+    col_rel_t *delta_out = test_rel_alloc(1);
+    int64_t source_rows[] = { 10, 20 };
+    int64_t owner_row = 99;
+    ASSERT(rel && delta_owner && delta_out, "source preflight relations");
+    ASSERT(test_rel_append_row(rel, &source_rows[0]) == 0
+        && test_rel_append_row(rel, &source_rows[1]) == 0
+        && test_rel_append_row(delta_owner, &owner_row) == 0
+        && col_rel_install_shared_view(delta_out, delta_owner) == 0,
+        "source and shared delta fixtures");
+
+    uint64_t last = WL_COLUMNAR_REL_GENERATION_INVALID - 1u;
+    rel->storage_generation = last;
+    rel->storage_owner_generation = last;
+    rel->memory_budget_denial_pending = true;
+    delta_out->memory_budget_denial_pending = true;
+    int64_t *delta_columns = delta_out->columns[0];
+    uint64_t delta_generation = delta_out->storage_generation;
+    uint64_t delta_view_generation = delta_out->view_generation;
+    uint64_t delta_borrows = col_rel_storage_alias_borrow_count(delta_owner);
+    uint64_t owner_generation = delta_owner->storage_generation;
+    int fast_path = -1;
+
+    ASSERT(col_op_consolidate_incremental_delta(rel, 1, delta_out,
+        &fast_path) == EOVERFLOW && fast_path == -1,
+        "exhausted source is refused before reserving output");
+    ASSERT(!rel->memory_budget_denial_pending
+        && !delta_out->memory_budget_denial_pending,
+        "admitted non-budget failure clears stale denial evidence");
+    ASSERT(rel->nrows == 2 && rel->columns[0][0] == 10
+        && rel->columns[0][1] == 20
+        && rel->storage_generation == last
+        && rel->storage_owner_generation == last,
+        "source remains unchanged at exhausted generation");
+    ASSERT(delta_out->col_shared && delta_out->col_shared[0]
+        && delta_out->storage_owner == delta_owner
+        && delta_out->columns[0] == delta_columns
+        && delta_out->storage_generation == delta_generation
+        && delta_out->view_generation == delta_view_generation
+        && col_rel_storage_alias_borrow_count(delta_owner) == delta_borrows
+        && delta_owner->storage_generation == owner_generation
+        && delta_out->nrows == 1 && delta_out->columns[0][0] == owner_row,
+        "shared delta remains borrowed without COW or generation changes");
+
+    test_rel_free(delta_out);
+    test_rel_free(delta_owner);
+    test_rel_free(rel);
+    PASS();
+}
+
+static void
+test_admission_failure_preserves_stale_denial(void)
+{
+    TEST("mutation admission failure preserves stale denial evidence");
+
+    col_rel_t *rel = test_rel_alloc(1);
+    col_rel_t *delta_out = test_rel_alloc(1);
+    int64_t source_row = 10, delta_row = 20;
+    wl_columnar_source_access_writer_t held = { 0 };
+    ASSERT(rel && delta_out
+        && test_rel_append_row(rel, &source_row) == 0
+        && test_rel_append_row(delta_out, &delta_row) == 0,
+        "admission failure relations");
+    rel->memory_budget_denial_pending = true;
+    delta_out->memory_budget_denial_pending = true;
+    ASSERT(col_rel_source_writer_acquire(rel, &held) == 0,
+        "hold source writer to refuse mutation-set admission");
+    int fast_path = -1;
+    int rc = col_op_consolidate_incremental_delta(rel, 0, delta_out,
+            &fast_path);
+    ASSERT(wl_columnar_source_access_writer_release(&held) == 0,
+        "release held source writer");
+    ASSERT(rc == EBUSY && fast_path == -1
+        && rel->memory_budget_denial_pending
+        && delta_out->memory_budget_denial_pending
+        && rel->nrows == 1 && delta_out->nrows == 1,
+        "failed admission preserves prior evidence and rows");
+
+    test_rel_free(delta_out);
+    test_rel_free(rel);
+    PASS();
+}
+
 /* ================================================================
  * Issue #2049: every view-generation advance an incremental consolidation
  * may make is reserved before the delta sort, as #2046 does for storage.
@@ -2906,7 +2995,7 @@ test_view_generation_headroom_reserved(void)
 
 /* ================================================================
  * Issue #2057: delta_out's view generation advances once per emitted row
- * (col_rel_append_row_locked) and once more when a failure rolls the
+ * (col_rel_append_row_with_lease) and once more when a failure rolls the
  * emission back.  Every such advance is reserved before delta_out is first
  * mutated, so exhaustion is refused with EOVERFLOW instead of saturating
  * delta_out's view generation behind a successful return.
@@ -3106,6 +3195,8 @@ main(void)
     test_binary_retirement_uses_last_generation();
     test_fallback_exhaustion_without_timestamps();
     test_shared_view_storage_exhaustion();
+    test_source_exhaustion_precedes_delta_reservation();
+    test_admission_failure_preserves_stale_denial();
 
     /* View-generation headroom (Issue #2049) */
     test_view_generation_headroom_reserved();