Repository navigation
refactor(graph_buffer,store,zova): port the storage chain from C to Zig - #165
Merged
Merged
Conversation
This was referenced Nov 11, 2026
graph_buffer.c (2,184 lines) becomes graph_buffer.zig (1,831 lines). graph_buffer.h stays the ABI: the 42 exported functions keep their names and signatures, and checkExports verifies them against the header. Layout follows the C file section by section: the id/QN/label/name indexes, edge dedup and secondary indexes, cascade deletes, the dump builders, flush/merge into the store, and the vector and token-vector side tables. CBM_DYN_ARRAY is mirrored by a generic extern struct so the struct layout is unchanged. cbm_gbuf_new_shared_ids is exported without `pub`: translate-c cannot resolve its `_Atomic int64_t *` prototype, so checkExports cannot compare it. Its signature is checked by the C callers at link time only. Behavior differences: - Allocation failure in the dump, flush and merge paths returns CBM_NOT_FOUND where the C code dereferenced a NULL array. Flush and merge now allocate their id map before touching the store. - Negative node ids are rejected before they index an id map; C read before the start of the array. - The dump builder no longer allocates the `src_nodes` array (never read) and copies each edge's url_path/local_name once instead of twice. Wiring: Makefile.cbm drops GRAPH_BUFFER_SRCS from PROD_SRCS and LINT_SRCS and adds the Zig file and header to CBM_ZIG_SRCS; build.zig drops the C file; cbm_zig_c.h adds graph_buffer.h, semantic.h and the zova delta and publish-model headers the module calls. Signed-off-by: ata-sesli <atasesli@protonmail.com>
store.c (8,553 lines) becomes store.zig (6,913 lines). store.h stays the ABI: all 108 exported functions keep their names and signatures, and checkExports verifies them against the header. Layout follows the C file section by section: open/close and schema, projects, nodes, edges, search, BFS and trail traversal, file hashes, the architecture aspects, Louvain/Leiden clustering, and vector search (SQLite and Zova sessions). Row collection, statement preparation and string duplication go through a few small helpers (Rows, prepareOne, columnDup, ArchScope) instead of the repeated macro/realloc blocks. qsort/bsearch are still the libc ones with C comparators, so result ordering is unchanged. vs_fill_sparse_random uses std.hash.XxHash3, which matches the vendored XXH3_64bits_withSeed (checked against the C implementation). Behavior differences: - WHERE-clause building clamps to the buffer; C accumulated the untruncated snprintf length. - Exclude-label binds stop at ST_SEARCH_MAX_BINDS; C wrote past a 32-entry stack array. - NULL store in cbm_store_get_project, find_nodes_by_file_overlap, find_edges_by_url_path, count_edges_by_type, cbm_store_find_architecture_docs and cbm_store_get_architecture returns CBM_STORE_ERR; C dereferenced it. - cbm_glob_to_like returns NULL on allocation failure. - The typed Zova BFS trim loop stops at 0 hops for max_results < 0; C read hops[-1]. - cbm_store_bfs_trail returns CBM_STORE_ERR when its stack allocation fails; the trail row limit uses saturating 64-bit math. - cbm_deduplicate_hops returns OK for hop_count <= 0 and ERR on allocation failure. - cbm_store_normalize_arch_path returns false for a NULL or zero-size buffer. NULL edge_types with a positive count means "no types". - Leiden and cluster building: allocation failures that C left unchecked (weights, cluster arrays, top_nodes/packages/edge_types) now yield no clusters or singleton communities. Cluster package names are moved into the result instead of copied and freed. - fetch_limit = limit * 5 in vector search saturates. - The SQL branch after `if (s->zova_workspace_query) return ERR` in cbm_store_vector_search_ex was unreachable and is gone. - cbm_store_architecture_free tolerates NULL arrays with non-zero counts. Not verified: vs_normalize_and_quantize sums v*v without FMA contraction. If clang fused that sum on aarch64, the last bit of the magnitude (and rarely one int8 component) can differ from the C build. Wiring: Makefile.cbm drops STORE_SRCS from PROD_SRCS and LINT_SRCS and adds the Zig file and header to CBM_ZIG_SRCS; build.zig drops the C file; checkExports raises its comptime branch quota to 50000 for the 108 store exports. tests/test_github_workflows.sh names store.zig in its lstat check. scripts/security-audit.sh drops the store.c entry from the fopen known-safe list: store.c had no fopen call, so the entry was already stale, and store.zig has none either. Audit counts are unchanged. Signed-off-by: ata-sesli <atasesli@protonmail.com>
Ports seven src/zova files that do not call the Zova SDK. The headers stay the ABI; each .c is replaced by a Zig module with the same exported symbols and checked by checkExports. cbm_zova_route.c -> cbm_zova_route.zig cbm_zova_bridge_stub.c -> cbm_zova_bridge_stub.zig cbm_zova_writer_gate.c -> cbm_zova_writer_gate.zig cbm_zova_edge_payload.c -> cbm_zova_edge_payload.zig cbm_zova_coordination.c -> cbm_zova_coordination.zig cbm_zova_v5_snapshot.c -> cbm_zova_v5_snapshot.zig cbm_zova_delta.c -> cbm_zova_delta.zig 1,862 lines of C removed, 1,806 lines of Zig added. Build wiring: - translate-c now receives -DCBM_WITH_ZOVA=0|1 and, with Zova, the SDK include directory (Makefile.cbm, build.zig, the Windows prebuilt step in _build.yml). The Zig side reads c.CBM_WITH_ZOVA. cbm_zig_c.h includes zova.h when the macro is 1. - The bridge stub is compiled only when CBM_WITH_ZOVA is 0, as before (the C file was an empty translation unit otherwise). - The writer gate uses cbm_once from compat_thread instead of calling InitOnceExecuteOnce / pthread_once directly. Behavior differences: - cbm_zova_workspace_delta_build: the endpoints of a deleted edge are read through the same bounds-checked helper as every other edge. A `before` model whose edge has a source_id or target_id outside its node array now fails with -1 instead of reading outside node_stable_ids. The edge count sum before + after saturates instead of wrapping. - cbm_zova_v5_snapshot_open: when an allocation fails after a row count was stored, the cleanup path no longer walks a NULL array. - Coordination deadlines use saturating addition, so a very large wait_ms cannot overflow int64. Not changed, reported for review: the canonical-varint overflow guard in the edge payload reader (shift == bits - 7) can never be true on a 64-bit size_t (shift is a multiple of 7, 57 is not). It is kept exactly as in C. A tenth byte with bits above the top bit is therefore truncated rather than rejected, in both the C and the Zig version. Verified: zova 39, zova_operations 59, zova_incremental_native 1, zova_plugin 8, zova_bridge 4, zova_c_sql_functions 3, mcp 176, cli 132 (all equal to the baseline on main); libcbm_zig.a builds natively with and without Zova; zig fmt --check, check-build-sources.sh, security-audit.sh and tests/test_github_workflows.sh pass. Signed-off-by: ata-sesli <atasesli@protonmail.com>
src/zova/cbm_zova_plugin.c (1,296 lines) becomes src/zova/cbm_zova_plugin.zig (1,206 lines). The header stays the ABI and cbm_zova_plugin_descriptor() keeps its symbol and signature. The plugin implements the Zova plugin SDK callbacks, so zova_plugin.h is now translated together with zova.h when CBM_WITH_ZOVA=1. Without Zova the descriptor export returns NULL, as the C file did. Layout follows the C file: session and trace helpers, the cbm_trace, cbm_edges and cbm_cypher table functions, then registration. Differences in form, not behavior: - Trace rows are sorted with std.mem.sortUnstable instead of qsort. The comparator is a total order (the node id breaks every tie), so the result is the same. - The three table-function registrations share one tableOperation helper instead of three near-identical struct literals. - cypher run cleanup is one defer block instead of `goto done`. No behavior differences intended. Verified with the pinned Zova: zova_plugin 8, zova_bridge 4, zova 39, zova_operations 59, zova_incremental_native 1, zova_c_sql_functions 3, mcp 176, cli 132 (all equal to the baseline on main). Signed-off-by: ata-sesli <atasesli@protonmail.com>
cbm_zova_publish_model.c (1,472 lines) becomes cbm_zova_publish_model.zig (1,309 lines). The header is unchanged and stays the ABI; cbm_zig.zig checks every export against it. Wiring: - Makefile.cbm: the C file leaves ZOVA_SRCS; the Zig file and pipeline/worker_pool.h join CBM_ZIG_SRCS. - build.zig: the C file is removed from the native sources. - cbm_zig_c.h: pipeline/worker_pool.h is now translated, because the node build runs through cbm_parallel_for. - tests/test_github_workflows.sh: the Makefile check names the Zig file. Form differences from the C file: - Internal steps return an error union; the exports map it to -1. - The single-field wrapper structs around edges, hashes and vectors are gone; the model holds the public row types directly. - Sorting still goes through qsort with the same comparators, so the order of equal rows is unchanged. - The dead check `unique_topology <= 0` is dropped. - cbm_zova_publish_model_reuse_vector_digests measures the hex digests inside their 65-byte arrays instead of with an unbounded strlen. - The node chunk count is computed in usize; the C `count + 1023` could overflow int near INT_MAX. No behavior differences intended. Verified with the pinned Zova: zova 39, zova_operations 59, zova_incremental_native 1, zova_plugin 8, zova_bridge 4, zova_c_sql_functions 3, mcp 176, cli 132 (same as main). scripts/security-audit.sh, scripts/check-build-sources.sh, tests/test_github_workflows.sh and zig fmt --check pass. Signed-off-by: ata-sesli <atasesli@protonmail.com>
src/zova/cbm_zova_operations.c (2,810 lines) is replaced by
src/zova/cbm_zova_operations.zig. cbm_zova_operations.h stays the ABI; the
15 exported functions keep their names and signatures and are checked by
checkExports in src/cbm_zig.zig.
Form differences, no behavior difference intended:
- internal helpers return bool or small enums instead of 0/-1/-2/-3/1
- goto cleanup is replaced by defer
- the database and workspace archive verifiers share one prelude
- manifest header and entries are formatted into stack buffers and written
with fwrite instead of fprintf
- the regular-file checks use the one cbm_lstat result; the redundant
!S_ISLNK clause and the separate Windows stat branch are gone (cbm_lstat
is stat on Windows)
- status and the free-space check share one statvfs/GetDiskFreeSpaceExA
helper
Build wiring:
- src/cbm_zig_c.h includes <sys/statvfs.h> on POSIX
- the Windows form of cbm_mkdir_mode is a comma expression that
translate-c cannot express, so the Zig file calls cbm_mkdir there (the
same _mkdir call)
- tests/test_github_workflows.sh checks the Zig file for raw lstat/mkdir
scripts/security-audit.sh: the two fopen("w") REVIEW lines for the manifest
writers now name cbm_zova_operations.zig; the count is unchanged.
Signed-off-by: ata-sesli <atasesli@protonmail.com>
cbm_zova_repository.c (3,553 lines) becomes cbm_zova_repository.zig (3,619 lines). The header stays the ABI; every export keeps its name, signature and no-Zova stub behaviour. All SQL text is unchanged. Form differences: - the seven degree arrays of a search page are one allocation block - node and token vector reads share snapshotVectorBytes/snapshotVectorCount - the unused out_stable_ids parameter of the snapshot node reader is gone - snapshot metrics are not updated on failure paths (the snapshot is discarded there) Behavior differences: - Zova SDK results are always released (defer). C skipped the release when a keyed read returned a different count and when the payload get-many call failed. - a partial FTS catalog row is freed when validation fails - search_fts frees the key array on step and realloc errors - search frees the page, and the rows moved into it, on the data-error path - a statement returned together with a failure status is finalized - a failed open closes a returned handle - cbm_zova_workspace_snapshot_free does not walk an array that failed to allocate while its count was already set (OOM only) - the vector byte buffer is freed when the id or project copy fails (OOM only) Build wiring: Makefile.cbm ZOVA_SRCS/CBM_ZIG_SRCS, build.zig, checkExports in src/cbm_zig.zig. Signed-off-by: ata-sesli <atasesli@protonmail.com>
src/zova/cbm_zova.c (6,125 lines) becomes src/zova/cbm_zova.zig (5,033 lines). cbm_zova.h stays the ABI: every function keeps its name and signature, and src/cbm_zig.zig checks each export against the header. This was the last C file of src/zova; ZOVA_SRCS in Makefile.cbm is now empty. Form differences from the C file: - The C file had two bodies per function, a no-Zova stub and the real one. The Zig file has one export per function that starts with the stub behaviour (`if (!with_zova) ...`). - The seven functions that only a Zova build defines sit in the `zova_only` struct, which is empty without Zova: cbm_zova_database_format_status, cbm_zova_repack_phase_name, cbm_zova_register_sql_functions, cbm_zova_user_database_verify_workspace_schema, cbm_zova_user_database_schema_is_current, cbm_zova_user_database_publish_workspace_tx and cbm_zova_user_database_publish_model_tx. - Internal helpers return bool or an optional instead of 0/-1 plus an out parameter. Functions that record profile phases or write outputs after a failure keep the C control flow (`ok = ok and step()`). - user_database_publish_model_at_generation and cbm_zova_user_database_publish_delta were two copies of the same guard/open/WAL/transaction/timing code. Both now call userDatabasePublish, which takes the transaction body as a tagged union. - publishModelBuild is the model-building prelude that publish_workspace_tx and publish_workspace_at_generation shared. - sessionOpen opens both the vector and the graph session. - The three identical binary searches over file, node and token keys share namedKeyFind. - cbm_zova_user_database_delete_workspace builds its four "DELETE FROM <table> WHERE workspace_key=?1" statements at compile time instead of with snprintf. - Parameters and outputs that no caller used are gone (`model` in the delta node upsert, `workspace_key` and `out_edge_keys` in the delta native put, `db` in the statement reset helpers, `project` in write_nodes), and so are two unused struct types. - All SQL text is unchanged. A script compared every string literal of the two files; the only differences are printf formats written as Zig formats and phase names assembled at compile time. Behavior differences (all on failure paths): - cbm_zova_graph_session_adjacency freed `count + 1` rows when a neighbor query failed, which read one row past the array when it was full. It frees `count` rows. - user_database_write_edges leaked the edge payload buffer when encoding the payload failed. The buffer is freed. - user_delta_delete_native_tx: when a topology delete row was NULL the C code set rc = -1 but still called zova_graph_edge_delete_many on the partly filled array, and that call's status replaced the error. It now fails before the call. Reported, not changed: - cbm_zova_user_database_delete_workspace ignores a failure to prepare or bind the index state lookup. It then treats the workspace as having no vector collections, deletes its rows and leaves the collections behind. The port keeps that. Build wiring: Makefile.cbm (ZOVA_SRCS emptied, cbm_zova.zig added to CBM_ZIG_SRCS), build.zig (C file removed from native_sources), src/cbm_zig.zig (checkExports for the module and for zova_only). Signed-off-by: ata-sesli <atasesli@protonmail.com>
ata-sesli
force-pushed
the
refactor/store-zova-zig
branch
from
November 11, 2026 10:17
273011e to
0687076
Compare
GNU ld takes from a static library only the symbols that earlier inputs still need. Since the Zova callers moved from C sources into libcbm_zig.a, libzova_c.a came before its only user and the Linux release link failed with undefined zova_* references. List the Zig library first in the release, test-runner and test-repro-runner links. Signed-off-by: ata-sesli <atasesli@protonmail.com>
1 of 4 tasks
ata-sesli
added a commit
that referenced
this pull request
Nov 11, 2026
The C semantic code never asked for fused multiply-add, but compilers decided for it: Apple clang fused `a * b + c` on arm64, while the Linux release (gcc -std=c11) and the baseline-CPU x86_64 builds did not. The semantic scores therefore differed in the last bit between platforms, which can flip a SEMANTICALLY_RELATED edge whose score sits exactly at the threshold. The Zig port (#166) copied clang's choice through fp.mulAdd, which fuses wherever the CPU has FMA. That kept macOS arm64 unchanged but changed Linux arm64, and the store quantizer (#165) never fused at all. Use one rule everywhere instead: never fuse. Zig does not fuse on its own, so `a * b + c` is written as-is and every platform computes the same scores. fp.zig is removed; its call sites in semantic.zig, rotsq.zig and pass_semantic_edges.zig become plain expressions with the same operand order. Fusing everywhere was not an option: release builds target the baseline x86_64 CPU, which has no FMA instruction, so @mulAdd would become a software fma call in the hot loops. Signed-off-by: ata-sesli <atasesli@protonmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Ports
src/graph_buffer,src/storeand all ofsrc/zovafrom C to Zig. Stacked on thesrc/foundationport (refactor/foundation-zig); review that one first.The C headers do not change. Each Zig module exports the same symbols, and
cbm_zig.zigchecks every exportedcbm_*function against its header declaration at compile time. No C file and no shim remains in the three directories;ZOVA_SRCSinMakefile.cbmis empty. All SQL text is unchanged.One commit per unit, in dependency order:
d088cea8src/graph_buffer32831937src/store2b73a952src/zovamodulese0bb9882cbm_zova_pluginb41eb5c8cbm_zova_publish_model365ba98dcbm_zova_operations7605d7f7cbm_zova_repository273011e3cbm_zovaNet change: 27,855 lines of C removed, 24,120 lines of Zig added. Each commit message lists that unit's form differences in full.
Functions that exist only in a Zova build sit in a
zova_onlystruct that is empty without Zova. Every other export starts with the no-Zova stub behaviour (if (!with_zova) ...), replacing the two bodies per function the C file had.Behavior differences
All are on failure or out-of-range paths.
graph_buffer
CBM_NOT_FOUND. The C code dereferenced a NULL array.store
snprintflength.ST_SEARCH_MAX_BINDS. C wrote past a 32-entry stack array.CBM_STORE_ERRin six functions that dereferenced it.cbm_glob_to_likereturns NULL on allocation failure.max_results < 0. C readhops[-1].32831937.zova
cbm_zova_workspace_delta_build: deleted-edge endpoints are bounds-checked; the edge count sum saturates.cbm_zova_v5_snapshot_openandcbm_zova_workspace_snapshot_freeno longer walk a NULL array after a failed allocation.cbm_zova_graph_session_adjacencyfreedcount + 1rows on a failed neighbor query. It freescount.user_database_write_edgesfrees the payload buffer when encoding fails.user_delta_delete_native_tx: a NULL topology-delete row now fails beforezova_graph_edge_delete_many. In C that call still ran and its status replaced the error.Reported, not changed
size_t. Kept as in C.cbm_zova_user_database_delete_workspaceignores a failed prepare or bind of the index-state lookup, then deletes the workspace rows and leaves its vector collections behind. Kept as in C.docs/adr/0001-coordination-daemon.mdcitescbm_zova.candstore.cline numbers that were already stale on main. Not touched.Build and audit wiring
Makefile.cbmandbuild.zigdrop the C sources and list the Zig files;scripts/check-build-sources.shpasses.cbm_zig_c.hadds the headers the ported code calls, including<sys/statvfs.h>on POSIX.scripts/security-audit.sh: the stalestore.centry is removed from the fopen known-safe list (neitherstore.cnorstore.zigcallsfopen). The two fopen("w") REVIEW lines for the manifest writers now namecbm_zova_operations.zig. No audit count changes.tests/test_github_workflows.shchecks the Zig operations file.Verification
Run on macOS arm64 at
273011e3, throughscripts/zova-run-tests.shwith the pinned Zova:a592846a): the 14 foundation suites, cli 132, store_nodes 54, store_edges 25, store_search 69, store_arch 28, cypher 221, watcher 63, discover 132, semantic 33, graph_buffer 59, worker_pool 19, zova 39, zova_operations 59, zova_c_sql_functions 3, zova_bridge 4, zova_plugin 8, zova_incremental_native 1, pipeline 276 (13 not run), mcp 176.libcbm_zig.abuilds for native, x86_64-linux-gnu, aarch64-linux-gnu, x86_64-windows-gnu and aarch64-macos, each with and without Zova.zig fmt --check,scripts/check-build-sources.sh,scripts/security-audit.shandtests/test_github_workflows.shpass.mcpfailedindex_recovery_parallel_quarantines_crasherin one run (175/176) and passed in the next two. The same test fails intermittently on main.make -f Makefile.cbm testwas run on this branch and on main. Both give 4,843 passed, 870 failed, 624 not run, with identical failure lines. The target builds without Zova (CBM_WITH_ZOVA ?= 0), so every test that indexes a repository fails at publish. This PR does not change that.Not run / not verified:
scripts/zova-run-tests.shwith no arguments); only the 33 suites above.lint-ciandsmoke-test.sh.vs_normalize_and_quantizein the store contracts floating-point operations (fused multiply-add) the same way as the C build. The store suites pass; quantized output was not compared bit for bit.Checklist
git commit -s)make -f Makefile.cbm test): fails identically on main, see above; the Zova-enabled suites passmake -f Makefile.cbm lint-ci): not run