fix: a retrain dropped rows from probed vector searches

[vyruss]

Clustering assigns every cold row to the cluster of its nearest centroid, and a probed search reads only the clusters nearest the query. A retrain computed new centroids and pointed searches at them but left every row assigned under the previous centroids, so a row's cluster number now named a different centroid, or none, and probed searches silently left rows out: a table clustered into two groups and retrained into one returned three of its eight rows to a search that reads every cluster.

vector_train now ends by assigning every cold row to its nearest centroid of the set it just wrote, in the same transaction and under the table's claim, so the centroids and the assignments change together; only a row whose cluster changed is rewritten, and a retrain at the same nlist starts from the current centroids so each keeps its identity and most rows stay put. vector_assign is removed, since training covers the rows that predate it. The claim is taken once per transaction, before the sample, and released at commit, as the compactor holds it, so the modelled protocol is unchanged.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: pgEdge/coldfront/.coderabbit.yaml

Review profile: CHILL

Plan: Essentials

Run ID: 88843dcd-5b7f-4920-80dd-4f570d2f9754

📥 Commits

Reviewing files that changed from the base of the PR and between ed8195f and 391da9b.

⛔ Files ignored due to path filters (3)

• extension/coldfront/test/expected/adopt_iceberg_table.out is excluded by !**/*.out
• extension/coldfront/test/expected/vector_assign.out is excluded by !**/*.out
• extension/coldfront/test/expected/vector_multicolumn.out is excluded by !**/*.out

📒 Files selected for processing (10)

• ci/journey.sh
• docs/architecture_decoupled.md
• docs/architecture_vectors.md
• docs/changelog.md
• docs/usage_vectors.md
• extension/coldfront/Makefile
• extension/coldfront/coldfront--1.0.sql
• extension/coldfront/test/sql/adopt_iceberg_table.sql
• extension/coldfront/test/sql/vector_assign.sql
• extension/coldfront/test/sql/vector_multicolumn.sql

💤 Files with no reviewable changes (1)

• extension/coldfront/test/sql/vector_assign.sql

Included review availability: This review used your included allowance. 4 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

---

📝 Walkthrough

Walkthrough

Vector training now handles centroid initialization, Lloyd iterations, and cold-row assignment. Retraining at the configured cluster count reuses live centroid identities; changed counts use k-means++ seeding. The shared nearest-centroid expression supports training assignment and scoring. Documentation and journey tests reflect these changes. The vector_assign regression script and related documentation were removed.

Changes

Vector Training

Layer / File(s)	Summary
Training and retraining pipeline extension/coldfront/coldfront--1.0.sql, docs/architecture_vectors.md, docs/changelog.md, docs/usage_vectors.md	Training uses DuckDB temporary tables for sampling and Lloyd iterations. It reuses live centroids when the requested count matches the configured count; otherwise, it uses k-means++ seeding. It persists a new generation and documents the training behavior.
Cold-row assignment and nearest-centroid scoring extension/coldfront/coldfront--1.0.sql, docs/architecture_vectors.md, docs/architecture_decoupled.md, docs/usage_vectors.md, extension/coldfront/Makefile, extension/coldfront/test/sql/adopt_iceberg_table.sql, extension/coldfront/test/sql/vector_assign.sql, extension/coldfront/test/sql/vector_multicolumn.sql	Training updates cold rows whose stored assignment differs from the nearest centroid. _vec_nearest_expr provides shared nearest-centroid scoring. Documentation and tests reflect the assignment workflow; the vector_assign regression script and related references were removed.
Retraining and probe validation ci/journey.sh	Journey checks cover nearest-centroid assignments, exact-scan probe results, centroid ID retention at the same cluster count, and delete-file counts.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~40 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 391da

The documented retraining workflow has no established blocker to merging after normal checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: fixing row loss during retraining in probed vector searches.
Description check	✅ Passed	The description directly explains the retraining bug, the reassignment fix, centroid identity preservation, and removal of vector_assign.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 1 files. (8 skipped: 8 …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}