Repository navigation
refactor(pipeline): port src/pipeline from C to Zig - #167
Merged
ata-sesli merged 17 commits intoNov 11, 2026
Merged
Conversation
ata-sesli
force-pushed
the
refactor/store-zova-zig
branch
from
November 11, 2026 10:17
273011e to
0687076
Compare
Replace src/semantic/{semantic,ast_profile,rotsq}.c with Zig modules in
libcbm_zig. The C headers (semantic.h, ast_profile.h, rotsq.h) stay the
ABI: each Zig module exports the same symbols, and cbm_zig.zig checks
every pub cbm_* export against its translated header declaration.
- semantic.zig: tokenizer, corpus (IDF, random indexing, co-occurrence),
vector math and the combined score. It reads the pretrained nomic
table through nomic/code_vectors.h, as the C code did.
- ast_profile.zig: the AST structural profile and its string form.
- rotsq.zig: the rotational scalar quantizer.
- fp.zig: mulAdd. The C build contracts `a * b + c` into one fused
multiply-add where the CPU has FMA (clang's -ffp-contract=on); Zig
never fuses on its own. mulAdd fuses at the same places, so the
results match the C build on the same CPU.
Behavior differences from the C code:
- rotsq builds its +/-1 diagonal at compile time instead of on first use
under cbm_once. The values are the same (XXH3 of the dimension index
with the same seed).
Build wiring: Makefile.cbm and build.zig drop the C sources and list the
Zig files; cbm_zig_c.h gains nomic/code_vectors.h, pipeline/worker_pool.h
and the two semantic headers. scripts/extract_nomic_vectors.py names
cbm_zig_c.h as the place that includes the vector table.
Net change: -80 lines.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
cbm_zig_c.h gains the headers the pipeline port calls: the internal pipeline headers, the graph buffer, the extraction and LSP headers, the service patterns and a few foundation headers. translate-c has no C11 atomic types, so it drops every declaration that names one, and with it the pipeline context (its cancel flag is an atomic_int *) and the shared id counter. cbm_zig_c.h therefore shows translate-c a view in which an atomic type is its plain type and the atomic operations are the compiler builtins. Only that translation unit sees the view; the C sources keep <stdatomic.h>. Zig code reads and writes such fields with @atomicLoad / @AtomicStore / @atomicrmw. No code changes: nothing in Zig uses these declarations yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
cbm_parallel_for keeps its threading model: nworkers - 1 threads with 8 MB stacks plus the calling thread, all pulling indices from one shared atomic counter with relaxed ordering, and the serial fallback for one worker or one item. Net change: +1 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
- fqn.zig: qualified-name computation and project-name derivation. - path_alias.zig: tsconfig/jsconfig path aliases. Entries are still sorted with libc qsort, so aliases of equal specificity keep the order the C code gave them. - registry.zig: the symbol registry and the resolution strategies. The per-thread resolve, import-map and reachability caches stay thread-local. - util.zig: C-interop helpers these modules share (dyn_array.h and profile.h macros as functions, C string helpers, element addresses of C pointers, a once guard, the cancel check). They belong in src/foundation/interop.zig and live here while the foundation port is on another branch. Behavior difference: util.ix, which turns a C int count into a usize for loops and allocations, maps a negative value to 0. A C loop `for (i = 0; i < n; i++)` does not run for a negative n either. Net change: -18 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The spill file format is unchanged. The field lists that the C code expanded from X-macros are comptime tuples walked with inline for, and comptime assertions pin the struct sizes the format depends on. security-audit.sh: the one write-mode fopen under src/pipeline (the spill file) is now in result_spill.zig, so the known-safe list names src/pipeline/*.zig. No remaining pipeline C file opens a file for writing. Net change: -33 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
…to Zig - laravel_routing.zig: Laravel 11+ route mounts read from bootstrap/app.php with the PHP grammar. - service_bindings.zig: HTTP client instances (axios.create) and calls that an import binds to an external client library. Net change: -39 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
…nt passes from C to Zig - pass_semantic.zig: Go implicit interface satisfaction. - pass_tests.zig: TESTS edges. - pass_configures.zig: env-var name and config-key helpers. - pass_enrichment.zig: decorator tokenization and the decorator_tags pass. Net change: -66 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
…emantic-edge passes from C to Zig - pass_complexity.zig: interprocedural complexity propagation. - pass_importance.zig: the per-symbol importance score. - pass_similarity.zig: SIMILAR_TO edges from MinHash fingerprints. - pass_semantic_edges.zig: SEMANTICALLY_RELATED edges. The parallel phases keep their work splitting (one atomic index per phase). The LSH hyperplane dot product uses fp.mulAdd so it rounds as the C build does on a CPU with FMA. These passes still sort with libc qsort and format numbers with snprintf, so ties and rounding are as before. Net change: -264 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
- pass_gitdiff.zig: parsers for git diff output. - pass_githistory.zig: change coupling and file history from git log. git is still run through cbm_popen_git with the same validated command line. security-allowlist.txt names pass_githistory.zig for that call in place of pass_githistory.c; the number of allowed calls is unchanged. Net change: -23 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
… passes from C to Zig - pass_compile_commands.zig: compile_commands.json parsing. - pass_envscan.zig: environment URL scanner. - pass_configlink.zig: config-to-code linking. Behavior difference: pass_envscan compiles its regexes once through util.Once instead of cbm_once. cbm_once needs CBM_ONCE_INIT, a C initializer macro that translate-c cannot express on every platform. Like cbm_once, a caller that arrives while another thread compiles the regexes waits until it is done; it spins instead of blocking. Net change: -145 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
Dockerfile, compose, Terraform, shell, .env, Helm, Kubernetes and Kustomize scanning. The line reader and the fixed-size result buffers truncate at the same lengths as the C code. Net change: -347 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
Route nodes for HTTP_CALLS/ASYNC_CALLS edges, handler matching and DATA_FLOWS edges. Net change: -133 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
Manifest parsing (package.json, go.mod, Cargo.toml, pyproject.toml, composer.json, pubspec.yaml, pom.xml, build.gradle, mix.exs, gemspec, build.zig and build.zig.zon), the package map, module resolution, the import-target resolver and the namespace map. Behavior difference (Windows only): the manifest walk skips reparse points before it descends. The C code asked GetFileAttributesW for FILE_ATTRIBUTE_REPARSE_POINT; the Zig code asks cbm_stat_no_follow, which fails for a reparse point, on a path that is already known to stat successfully. The depth bound that guarantees termination is unchanged. Net change: -268 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The cross-file LSP glue: collecting project-wide defs, the per-module def index and filter, the per-language dispatch, and the Zig anytype argument collection. The filter counters stay relaxed atomics and the Rust manifest pointer stays thread-local. Net change: -127 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The extract and resolve phases keep their threading model and memory ownership: files sorted by size and pulled from one atomic index, per-worker graph buffers on cache-line-aligned worker states merged sequentially, the back-pressure and futility latches, result spill with per-worker ownership, source retention under the same caps, per-file slab reclaim, and the lazily built shared Rust registry behind its mutex. All counters keep their memory orders. lsp_resolve_shim.c: lsp_resolve.h defines its helpers as static inline, and Zig reads them through translate-c. cbm_pipeline_lsp_target_node counts its tail-match lookups with C11 atomics, which translate-c cannot express, so Zig sees only its declaration. The shim compiles the header's own body into a linkable function, which keeps lsp_resolve.h the single source for the C code that includes it (internal/cbm/lsp and the tests). Net change: -122 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The incremental route keeps its order: classify, snapshot inbound edges, purge, seed the registry, add structure, extract and resolve, restore edges, the derived passes behind their digests, publish. Net change: +18 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The orchestrator: lifecycle, file errors and coverage stamps, the structure pass, the parallel passes, the predump passes in their order, and publication. struct cbm_pipeline stays private to this module; the headers only name it as an opaque type. cbm_pipeline_run's cleanup label became a helper for the full-index part, called before the same cleanup sequence. With this commit no pipeline pass is built from C. Makefile.cbm and build.zig list only lsp_resolve_shim.c under src/pipeline. Net change: -35 lines. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com> Co-authored-by: Ata Sesli <atasesli@protonmail.com>
ata-sesli
force-pushed
the
refactor/pipeline-zig
branch
from
November 11, 2026 11:23
45ff8e1 to
89fe759
Compare
1 of 4 tasks
Contributor
Author
Release-build timing on redisThe timings in the description compare binaries from Both commits built on Linux x86_64 with gcc and the pinned Zova: Neither binary contains
In a release build the ports are neither a speedup nor a regression: the differences are 0.1–0.4 s, about the run-to-run spread for the full index. Counts:
🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Ports
src/pipelinefrom C to Zig, following the pattern of #163. Stacked on thesrc/semanticport (#166); review #164, #165 and #166 first.Stacking note: the head branch lives in a fork, so this PR cannot use #166's branch as its base. It targets
refactor/store-zova-zigand therefore also shows #166's single commit (0bbaa373). The 16 commits after it are this PR.All 28 C files in
src/pipelinebecome Zig modules inlibcbm_zig. It is a faithful port: the threading model, work splitting, memory ownership and pass order are unchanged.The C headers do not change.
cbm_zig.zigchecks every exportedcbm_*function against its header declaration at compile time.Net change for these 16 commits: 23,828 lines removed, 22,283 added (−1,545).
One commit per pass group, in dependency order:
5598eb47cbm_zig_c.hreads the pipeline headers144d22659c74db0f592ac2da52349de4a3b32e1d4bd5d9b436829628341a8ec501d41baf1210a1784f333f174fab05640ba34b306982636a45ff8e1cWhat stays C, and why
src/pipeline/lsp_resolve_shim.c(new, 21 lines):lsp_resolve.hdefinescbm_pipeline_lsp_target_nodeas astatic inlinethat uses C11 atomics, which translate-c cannot express. The shim compiles the header's own body into a linkable function, so the header stays the single source for the C code that includes it.src/cbm_zig_c.h: translate-c drops every declaration that names an atomic type, including the pipeline context. For that one translation unit an atomic type is shown as its plain type; the Zig side reads and writes those fields with@atomicLoad/@atomicStore/@atomicRmw. The C sources keep the real<stdatomic.h>.Helpers that belong in
foundation/interop.zigsrc/pipeline/util.zig(160 lines) holds C-interop helpers that were kept out ofsrc/foundationso this PR does not collide with #164:CStr,MutStr,streq,ptrDiff,ix,at,DynArraywithdaPush/daPushPtr/daFree/daReserve,profStart/profEnd,Once,isDirMode,checkCancel. They can move in a follow-up, together with the small log and time helpers repeated inpipeline.zig,pipeline_incremental.zigandpass_parallel.zig.Behavior differences
util.ix: a negative C count becomes 0 when used as an index bound, so loops run as often as the Cfordid.pass_envscan: the regexes are compiled once throughutil.Once, which spins while another thread initializes instead of blocking incbm_once.CBM_ONCE_INITdoes not translate on every platform.pass_pkgmap, Windows only: the reparse-point check usescbm_stat_no_followinstead ofGetFileAttributesW. Compiled for Windows, not run there.cbm_pipeline_run: thegoto cleanuppath is a helper function with the same cleanup order.No C bug was fixed in this PR.
Build and audit wiring
Makefile.cbmandbuild.zigdrop the C sources (PIPELINE_SRCSis nowlsp_resolve_shim.c) and list the Zig files and pipeline headers inCBM_ZIG_SRCS;scripts/check-build-sources.shpasses.cbm_zig_c.hadds the extraction, LSP, xxhash and pipeline headers the ported code calls.security-allowlist.txtnamespass_githistory.zigfor thecbm_popen_gitentry;security-audit.sh's fopen("w") known-safe list namessrc/pipeline/*.zig. No audit count changes.Verification
Run on Linux x86_64 with the pinned Zova.
At this branch's tip (
45ff8e1c, on top of #165 and #166):The production binary builds with Zova.
18 suites through
scripts/zova-run-tests.shpass with the same counts as main (a592846a): pipeline 276 (13 not run), parallel 34, incremental 149, semantic 33, ast_profile 10, mcp 176, graph_buffer 59, registry 63, worker_pool 19, fqn 91, path_alias 12, infrascan 14 (1 not run), configlink 9, importance 10, index_resilience 2, call_reference 24, zova_incremental_native 1, extraction 206 (93 not run).The full
scripts/zova-run-tests.shpasses: 5,840 passed, 624 not run (language tier).Redis, against current main (
f02b64d3, which includes refactor(foundation): port src/foundation from C to Zig #164), both built withmake -f Makefile.cbm cbmwith Zova, three rounds of three runs per step:f02b64d3The port is faster because
zig build cbmcompiles the C sources without-Doptimize, which leaves undefined-behaviour checks in the C code, while the Zig library isReleaseFast.Redis full index: node and edge counts for all 35 labels and edge types are identical to main in every round.
Redis incremental re-index: every count is identical to main except
CALLS, which varies from round to round on both binaries (main 40,814 / 40,813 / 40,821; this PR 40,809 / 40,820 / 40,817). Main does not reproduce its own number, so equality cannot be asserted for it.libcbm_zig.abuilds for native, x86_64-linux-gnu, aarch64-linux-gnu, x86_64-windows-gnu and aarch64-macos (without Zova).zig fmt --check,scripts/check-build-sources.sh,scripts/security-audit.shandscripts/check-dco.shpass.Before the rebase onto #165 (same pipeline files, on top of main):
a592846a).CALLSedges with confidence, strategy, callee and line, and the endpoints of the 16 other edge types are identical to main.Not run / not verified:
make -f Makefile.cbm test,lint-ciandsmoke-test.sh.Checklist
git commit -s)make -f Makefile.cbm test): not run; the Zova-enabled suites listed above passmake -f Makefile.cbm lint-ci): not run🤖 Generated with Claude Code