Skip to content

docs: clarify Longmont proof chunk parameters #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
scottconverse wants to merge 2 commits into
base: main
Choose a base branch
from
Open

docs: clarify Longmont proof chunk parameters #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2485a99
docs: clarify Longmont proof chunk parameters
May 22, 2026
19356ec
docs: reference dual Longmont chunk proof
May 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,10 @@ MINOR; bug fixes ship as PATCH.
CivicCore-owned pipeline instead of rebuilding module-local parser and vector
storage paths.
- Added a Longmont Code of Ordinances PDF proof showing a 12.4 MB municipal
code corpus parsed into 1,789 persisted chunks with 1,789 768-dimensional
Ollama embeddings.
code corpus parsed with `chunk_size=900` / `chunk_overlap=90` into 1,789
persisted chunks with 1,789 768-dimensional Ollama embeddings; CivicCode PR
#61 adds a committed dual-run proof script that reproduces this count and the
CivicCode `500/50` count through the same CivicCore `ingest_file()` path.

## [1.1.0] - 2026-05-11

Expand Down
10 changes: 8 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,8 +381,14 @@ document = await ingest_file(

The local Longmont proof in
[`docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md`](docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md)
parsed the 12.4 MB Longmont Code of Ordinances PDF into 1,789 persisted chunks
and wrote 1,789 768-dimensional `nomic-embed-text` embeddings.
parsed the 12.4 MB Longmont Code of Ordinances PDF with `chunk_size=900` /
`chunk_overlap=90` into 1,789 persisted chunks and wrote 1,789
768-dimensional `nomic-embed-text` embeddings. Downstream modules may choose
different chunking parameters; for example CivicCode's active PR #61 proof
uses `chunk_size=500` / `chunk_overlap=50` and therefore produces a different
chunk count from the same PDF. CivicCode PR #61 commits
`scripts/prove-longmont-civiccore-chunk-params.py`, a dual-run proof that
reproduces both counts through `civiccore.ingest.ingest_file`.

## Live connector sync primitives

Expand Down
6 changes: 4 additions & 2 deletions docs/qa/civiccore-longmont-ingest-proof-2026-05-21.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Purpose: prove the shared CivicCore ingestion pipeline parses a real municipal P

Command class: local proof script using `testcontainers.postgres.PostgresContainer("pgvector/pgvector:pg17")`, `civiccore.migrations.runner.upgrade_to_head()`, and `civiccore.ingest.ingest_file()`.

Chunking parameters for this CivicCore proof: `chunk_size=900`, `chunk_overlap=90`.

Corpus:

- `C:\Users\scott\OneDrive\Desktop\Claude\longmont-code-corpus\Longmont, CO Code of Ordinances.pdf`
Expand Down Expand Up @@ -34,8 +36,8 @@ sample_chunk_text=SUPPLEMENT NO. 8 March 2026 CODE OF ORDINANCES City of LONGMON
Acceptance facts:

- Real PDF parser path used: yes.
- Sentence-aware chunk rows persisted: `1789`.
- Sentence-aware chunk rows persisted: `1789` with `chunk_size=900` / `chunk_overlap=90`.
- Ollama `nomic-embed-text` vectors persisted: `1789`.
- Vector dimensionality: `768`.
- Database schema: CivicCore baseline migration `documents` / `document_chunks` on pgvector PostgreSQL.

- Note: CivicCode PR #61 intentionally uses `chunk_size=500` / `chunk_overlap=50` for its active Longmont proof, so its 2,931 chunk count is not expected to match this CivicCore proof's 1,789 chunk count. CivicCode PR #61 commits `scripts/prove-longmont-civiccore-chunk-params.py`, which ingests the same PDF through `civiccore.ingest.ingest_file()` twice in one run and reproduces both counts.