From 2485a99bcf96f7d5b24adaaf3b748f7d16aaa14a Mon Sep 17 00:00:00 2001 From: Scott Converse Date: Fri, 22 May 2026 10:27:58 -0600 Subject: [PATCH 1/2] docs: clarify Longmont proof chunk parameters --- CHANGELOG.md | 4 ++-- README.md | 8 ++++++-- docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md | 6 ++++-- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d131837..d94a989 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,8 +24,8 @@ MINOR; bug fixes ship as PATCH. CivicCore-owned pipeline instead of rebuilding module-local parser and vector storage paths. - Added a Longmont Code of Ordinances PDF proof showing a 12.4 MB municipal - code corpus parsed into 1,789 persisted chunks with 1,789 768-dimensional - Ollama embeddings. + code corpus parsed with `chunk_size=900` / `chunk_overlap=90` into 1,789 + persisted chunks with 1,789 768-dimensional Ollama embeddings. ## [1.1.0] - 2026-05-11 diff --git a/README.md b/README.md index ebc21ef..daabbe2 100644 --- a/README.md +++ b/README.md @@ -381,8 +381,12 @@ document = await ingest_file( The local Longmont proof in [`docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md`](docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md) -parsed the 12.4 MB Longmont Code of Ordinances PDF into 1,789 persisted chunks -and wrote 1,789 768-dimensional `nomic-embed-text` embeddings. +parsed the 12.4 MB Longmont Code of Ordinances PDF with `chunk_size=900` / +`chunk_overlap=90` into 1,789 persisted chunks and wrote 1,789 +768-dimensional `nomic-embed-text` embeddings. Downstream modules may choose +different chunking parameters; for example CivicCode's active PR #61 proof +uses `chunk_size=500` / `chunk_overlap=50` and therefore produces a different +chunk count from the same PDF. ## Live connector sync primitives diff --git a/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md b/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md index dca8faa..18040fc 100644 --- a/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md +++ b/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md @@ -4,6 +4,8 @@ Purpose: prove the shared CivicCore ingestion pipeline parses a real municipal P Command class: local proof script using `testcontainers.postgres.PostgresContainer("pgvector/pgvector:pg17")`, `civiccore.migrations.runner.upgrade_to_head()`, and `civiccore.ingest.ingest_file()`. +Chunking parameters for this CivicCore proof: `chunk_size=900`, `chunk_overlap=90`. + Corpus: - `C:\Users\scott\OneDrive\Desktop\Claude\longmont-code-corpus\Longmont, CO Code of Ordinances.pdf` @@ -34,8 +36,8 @@ sample_chunk_text=SUPPLEMENT NO. 8 March 2026 CODE OF ORDINANCES City of LONGMON Acceptance facts: - Real PDF parser path used: yes. -- Sentence-aware chunk rows persisted: `1789`. +- Sentence-aware chunk rows persisted: `1789` with `chunk_size=900` / `chunk_overlap=90`. - Ollama `nomic-embed-text` vectors persisted: `1789`. - Vector dimensionality: `768`. - Database schema: CivicCore baseline migration `documents` / `document_chunks` on pgvector PostgreSQL. - +- Note: CivicCode PR #61 intentionally uses `chunk_size=500` / `chunk_overlap=50` for its active Longmont proof, so its 2,931 chunk count is not expected to match this CivicCore proof's 1,789 chunk count. From 19356ecd142d25b702610a032a47a12480076f5c Mon Sep 17 00:00:00 2001 From: Scott Converse Date: Fri, 22 May 2026 13:10:29 -0600 Subject: [PATCH 2/2] docs: reference dual Longmont chunk proof --- CHANGELOG.md | 4 +++- README.md | 4 +++- docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md | 2 +- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d94a989..f5cebf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,7 +25,9 @@ MINOR; bug fixes ship as PATCH. storage paths. - Added a Longmont Code of Ordinances PDF proof showing a 12.4 MB municipal code corpus parsed with `chunk_size=900` / `chunk_overlap=90` into 1,789 - persisted chunks with 1,789 768-dimensional Ollama embeddings. + persisted chunks with 1,789 768-dimensional Ollama embeddings; CivicCode PR + #61 adds a committed dual-run proof script that reproduces this count and the + CivicCode `500/50` count through the same CivicCore `ingest_file()` path. ## [1.1.0] - 2026-05-11 diff --git a/README.md b/README.md index daabbe2..5008283 100644 --- a/README.md +++ b/README.md @@ -386,7 +386,9 @@ parsed the 12.4 MB Longmont Code of Ordinances PDF with `chunk_size=900` / 768-dimensional `nomic-embed-text` embeddings. Downstream modules may choose different chunking parameters; for example CivicCode's active PR #61 proof uses `chunk_size=500` / `chunk_overlap=50` and therefore produces a different -chunk count from the same PDF. +chunk count from the same PDF. CivicCode PR #61 commits +`scripts/prove-longmont-civiccore-chunk-params.py`, a dual-run proof that +reproduces both counts through `civiccore.ingest.ingest_file`. ## Live connector sync primitives diff --git a/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md b/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md index 18040fc..e7d84c2 100644 --- a/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md +++ b/docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md @@ -40,4 +40,4 @@ Acceptance facts: - Ollama `nomic-embed-text` vectors persisted: `1789`. - Vector dimensionality: `768`. - Database schema: CivicCore baseline migration `documents` / `document_chunks` on pgvector PostgreSQL. -- Note: CivicCode PR #61 intentionally uses `chunk_size=500` / `chunk_overlap=50` for its active Longmont proof, so its 2,931 chunk count is not expected to match this CivicCore proof's 1,789 chunk count. +- Note: CivicCode PR #61 intentionally uses `chunk_size=500` / `chunk_overlap=50` for its active Longmont proof, so its 2,931 chunk count is not expected to match this CivicCore proof's 1,789 chunk count. CivicCode PR #61 commits `scripts/prove-longmont-civiccore-chunk-params.py`, which ingests the same PDF through `civiccore.ingest.ingest_file()` twice in one run and reproduces both counts.