Skip to content

refactor(pipeline): port src/pipeline from C to Zig - #167

Merged
ata-sesli merged 17 commits into
octacity-org:mainfrom
professional-tester:refactor/pipeline-zig
Nov 11, 2026
Merged

ata-sesli merged 17 commits into
octacity-org:mainfrom
professional-tester:refactor/pipeline-zig

Conversation

@professional-tester

@professional-tester professional-tester commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Ports src/pipeline from C to Zig, following the pattern of #163. Stacked on the src/semantic port (#166); review #164, #165 and #166 first.

Stacking note: the head branch lives in a fork, so this PR cannot use #166's branch as its base. It targets refactor/store-zova-zig and therefore also shows #166's single commit (0bbaa373). The 16 commits after it are this PR.

All 28 C files in src/pipeline become Zig modules in libcbm_zig. It is a faithful port: the threading model, work splitting, memory ownership and pass order are unchanged.

The C headers do not change. cbm_zig.zig checks every exported cbm_* function against its header declaration at compile time.

Net change for these 16 commits: 23,828 lines removed, 22,283 added (−1,545).

One commit per pass group, in dependency order:

Commit Ported
5598eb47 build wiring only: cbm_zig_c.h reads the pipeline headers
144d2265 worker_pool
9c74db0f fqn, path_alias and registry
592ac2da result_spill
52349de4 laravel_routing and service_bindings
a3b32e1d the semantic, tests, configures and enrichment passes
4bd5d9b4 the complexity, importance, similarity and semantic-edge passes
36829628 the gitdiff and githistory passes
341a8ec5 the compile_commands, envscan and configlink passes
01d41baf the infrascan and k8s passes
1210a178 pass_route_nodes
4f333f17 pass_pkgmap
4fab0564 pass_lsp_cross
0ba34b30 pass_parallel
6982636a pipeline_incremental
45ff8e1c pipeline

What stays C, and why

  • src/pipeline/lsp_resolve_shim.c (new, 21 lines): lsp_resolve.h defines cbm_pipeline_lsp_target_node as a static inline that uses C11 atomics, which translate-c cannot express. The shim compiles the header's own body into a linkable function, so the header stays the single source for the C code that includes it.
  • Atomics view in src/cbm_zig_c.h: translate-c drops every declaration that names an atomic type, including the pipeline context. For that one translation unit an atomic type is shown as its plain type; the Zig side reads and writes those fields with @atomicLoad / @atomicStore / @atomicRmw. The C sources keep the real <stdatomic.h>.

Helpers that belong in foundation/interop.zig

src/pipeline/util.zig (160 lines) holds C-interop helpers that were kept out of src/foundation so this PR does not collide with #164: CStr, MutStr, streq, ptrDiff, ix, at, DynArray with daPush/daPushPtr/daFree/daReserve, profStart/profEnd, Once, isDirMode, checkCancel. They can move in a follow-up, together with the small log and time helpers repeated in pipeline.zig, pipeline_incremental.zig and pass_parallel.zig.

Behavior differences

  • util.ix: a negative C count becomes 0 when used as an index bound, so loops run as often as the C for did.
  • pass_envscan: the regexes are compiled once through util.Once, which spins while another thread initializes instead of blocking in cbm_once. CBM_ONCE_INIT does not translate on every platform.
  • pass_pkgmap, Windows only: the reparse-point check uses cbm_stat_no_follow instead of GetFileAttributesW. Compiled for Windows, not run there.
  • cbm_pipeline_run: the goto cleanup path is a helper function with the same cleanup order.

No C bug was fixed in this PR.

Build and audit wiring

  • Makefile.cbm and build.zig drop the C sources (PIPELINE_SRCS is now lsp_resolve_shim.c) and list the Zig files and pipeline headers in CBM_ZIG_SRCS; scripts/check-build-sources.sh passes.
  • cbm_zig_c.h adds the extraction, LSP, xxhash and pipeline headers the ported code calls.
  • security-allowlist.txt names pass_githistory.zig for the cbm_popen_git entry; security-audit.sh's fopen("w") known-safe list names src/pipeline/*.zig. No audit count changes.

Verification

Run on Linux x86_64 with the pinned Zova.

At this branch's tip (45ff8e1c, on top of #165 and #166):

  • The production binary builds with Zova.

  • 18 suites through scripts/zova-run-tests.sh pass with the same counts as main (a592846a): pipeline 276 (13 not run), parallel 34, incremental 149, semantic 33, ast_profile 10, mcp 176, graph_buffer 59, registry 63, worker_pool 19, fqn 91, path_alias 12, infrascan 14 (1 not run), configlink 9, importance 10, index_resilience 2, call_reference 24, zova_incremental_native 1, extraction 206 (93 not run).

  • The full scripts/zova-run-tests.sh passes: 5,840 passed, 624 not run (language tier).

  • Redis, against current main (f02b64d3, which includes refactor(foundation): port src/foundation from C to Zig #164), both built with make -f Makefile.cbm cbm with Zova, three rounds of three runs per step:

    Step main f02b64d3 this PR
    Full index (9 runs each) 32.3–33.6 s, median 32.4 25.0–25.4 s, median 25.0
    First incremental re-index (3 runs each) 17.7–17.8 s 10.6–10.9 s
    Later incremental re-indexes (6 runs each) 7.2–7.3 s 6.9–7.0 s

    The port is faster because zig build cbm compiles the C sources without -Doptimize, which leaves undefined-behaviour checks in the C code, while the Zig library is ReleaseFast.

  • Redis full index: node and edge counts for all 35 labels and edge types are identical to main in every round.

  • Redis incremental re-index: every count is identical to main except CALLS, which varies from round to round on both binaries (main 40,814 / 40,813 / 40,821; this PR 40,809 / 40,820 / 40,817). Main does not reproduce its own number, so equality cannot be asserted for it.

  • libcbm_zig.a builds for native, x86_64-linux-gnu, aarch64-linux-gnu, x86_64-windows-gnu and aarch64-macos (without Zova).

  • zig fmt --check, scripts/check-build-sources.sh, scripts/security-audit.sh and scripts/check-dco.sh pass.

Before the rebase onto #165 (same pipeline files, on top of main):

  • All 43 affected suites match main (a592846a).
  • Redis content, full index: all 43,329 nodes, all 13,323 Function nodes with their properties, all 40,807 CALLS edges with confidence, strategy, callee and line, and the endpoints of the 16 other edge types are identical to main.
  • The production binary builds at each of the 16 commits; redis counts are identical at the 6 commits checked.

Not run / not verified:

  • After the rebase: the per-node and per-edge content comparison, and builds of the 15 intermediate commits.
  • cpython and kubernetes were not indexed with this branch.
  • make -f Makefile.cbm test, lint-ci and smoke-test.sh.
  • Any runtime test on Windows or macOS. Those paths are compile-checked only.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test): not run; the Zova-enabled suites listed above pass
  • Lint passes (make -f Makefile.cbm lint-ci): not run
  • New behavior is covered by a test: no new behavior; the existing suites cover the ported code

🤖 Generated with Claude Code

@ata-sesli
ata-sesli force-pushed the refactor/store-zova-zig branch from 273011e to 0687076 Compare November 11, 2026 10:17
professional-tester and others added 17 commits November 11, 2026 14:22
Replace src/semantic/{semantic,ast_profile,rotsq}.c with Zig modules in
libcbm_zig. The C headers (semantic.h, ast_profile.h, rotsq.h) stay the
ABI: each Zig module exports the same symbols, and cbm_zig.zig checks
every pub cbm_* export against its translated header declaration.

- semantic.zig: tokenizer, corpus (IDF, random indexing, co-occurrence),
  vector math and the combined score. It reads the pretrained nomic
  table through nomic/code_vectors.h, as the C code did.
- ast_profile.zig: the AST structural profile and its string form.
- rotsq.zig: the rotational scalar quantizer.
- fp.zig: mulAdd. The C build contracts `a * b + c` into one fused
  multiply-add where the CPU has FMA (clang's -ffp-contract=on); Zig
  never fuses on its own. mulAdd fuses at the same places, so the
  results match the C build on the same CPU.

Behavior differences from the C code:

- rotsq builds its +/-1 diagonal at compile time instead of on first use
  under cbm_once. The values are the same (XXH3 of the dimension index
  with the same seed).

Build wiring: Makefile.cbm and build.zig drop the C sources and list the
Zig files; cbm_zig_c.h gains nomic/code_vectors.h, pipeline/worker_pool.h
and the two semantic headers. scripts/extract_nomic_vectors.py names
cbm_zig_c.h as the place that includes the vector table.

Net change: -80 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>

Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
cbm_zig_c.h gains the headers the pipeline port calls: the internal
pipeline headers, the graph buffer, the extraction and LSP headers, the
service patterns and a few foundation headers.

translate-c has no C11 atomic types, so it drops every declaration that
names one, and with it the pipeline context (its cancel flag is an
atomic_int *) and the shared id counter. cbm_zig_c.h therefore shows
translate-c a view in which an atomic type is its plain type and the
atomic operations are the compiler builtins. Only that translation unit
sees the view; the C sources keep <stdatomic.h>. Zig code reads and
writes such fields with @atomicLoad / @AtomicStore / @atomicrmw.

No code changes: nothing in Zig uses these declarations yet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
cbm_parallel_for keeps its threading model: nworkers - 1 threads with
8 MB stacks plus the calling thread, all pulling indices from one shared
atomic counter with relaxed ordering, and the serial fallback for one
worker or one item.

Net change: +1 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
- fqn.zig: qualified-name computation and project-name derivation.
- path_alias.zig: tsconfig/jsconfig path aliases. Entries are still
  sorted with libc qsort, so aliases of equal specificity keep the order
  the C code gave them.
- registry.zig: the symbol registry and the resolution strategies. The
  per-thread resolve, import-map and reachability caches stay
  thread-local.
- util.zig: C-interop helpers these modules share (dyn_array.h and
  profile.h macros as functions, C string helpers, element addresses of
  C pointers, a once guard, the cancel check). They belong in
  src/foundation/interop.zig and live here while the foundation port is
  on another branch.

Behavior difference: util.ix, which turns a C int count into a usize
for loops and allocations, maps a negative value to 0. A C loop
`for (i = 0; i < n; i++)` does not run for a negative n either.

Net change: -18 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The spill file format is unchanged. The field lists that the C code
expanded from X-macros are comptime tuples walked with inline for, and
comptime assertions pin the struct sizes the format depends on.

security-audit.sh: the one write-mode fopen under src/pipeline (the
spill file) is now in result_spill.zig, so the known-safe list names
src/pipeline/*.zig. No remaining pipeline C file opens a file for
writing.

Net change: -33 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
…to Zig

- laravel_routing.zig: Laravel 11+ route mounts read from
  bootstrap/app.php with the PHP grammar.
- service_bindings.zig: HTTP client instances (axios.create) and calls
  that an import binds to an external client library.

Net change: -39 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
…nt passes from C to Zig

- pass_semantic.zig: Go implicit interface satisfaction.
- pass_tests.zig: TESTS edges.
- pass_configures.zig: env-var name and config-key helpers.
- pass_enrichment.zig: decorator tokenization and the decorator_tags
  pass.

Net change: -66 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
…emantic-edge passes from C to Zig

- pass_complexity.zig: interprocedural complexity propagation.
- pass_importance.zig: the per-symbol importance score.
- pass_similarity.zig: SIMILAR_TO edges from MinHash fingerprints.
- pass_semantic_edges.zig: SEMANTICALLY_RELATED edges. The parallel
  phases keep their work splitting (one atomic index per phase). The
  LSH hyperplane dot product uses fp.mulAdd so it rounds as the C build
  does on a CPU with FMA.

These passes still sort with libc qsort and format numbers with
snprintf, so ties and rounding are as before.

Net change: -264 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
- pass_gitdiff.zig: parsers for git diff output.
- pass_githistory.zig: change coupling and file history from git log.

git is still run through cbm_popen_git with the same validated command
line. security-allowlist.txt names pass_githistory.zig for that call in
place of pass_githistory.c; the number of allowed calls is unchanged.

Net change: -23 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
… passes from C to Zig

- pass_compile_commands.zig: compile_commands.json parsing.
- pass_envscan.zig: environment URL scanner.
- pass_configlink.zig: config-to-code linking.

Behavior difference: pass_envscan compiles its regexes once through
util.Once instead of cbm_once. cbm_once needs CBM_ONCE_INIT, a C
initializer macro that translate-c cannot express on every platform.
Like cbm_once, a caller that arrives while another thread compiles the
regexes waits until it is done; it spins instead of blocking.

Net change: -145 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
Dockerfile, compose, Terraform, shell, .env, Helm, Kubernetes and
Kustomize scanning. The line reader and the fixed-size result buffers
truncate at the same lengths as the C code.

Net change: -347 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
Route nodes for HTTP_CALLS/ASYNC_CALLS edges, handler matching and
DATA_FLOWS edges.

Net change: -133 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
Manifest parsing (package.json, go.mod, Cargo.toml, pyproject.toml,
composer.json, pubspec.yaml, pom.xml, build.gradle, mix.exs, gemspec,
build.zig and build.zig.zon), the package map, module resolution, the
import-target resolver and the namespace map.

Behavior difference (Windows only): the manifest walk skips reparse
points before it descends. The C code asked GetFileAttributesW for
FILE_ATTRIBUTE_REPARSE_POINT; the Zig code asks cbm_stat_no_follow,
which fails for a reparse point, on a path that is already known to
stat successfully. The depth bound that guarantees termination is
unchanged.

Net change: -268 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The cross-file LSP glue: collecting project-wide defs, the per-module
def index and filter, the per-language dispatch, and the Zig anytype
argument collection. The filter counters stay relaxed atomics and the
Rust manifest pointer stays thread-local.

Net change: -127 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The extract and resolve phases keep their threading model and memory
ownership: files sorted by size and pulled from one atomic index,
per-worker graph buffers on cache-line-aligned worker states merged
sequentially, the back-pressure and futility latches, result spill with
per-worker ownership, source retention under the same caps, per-file
slab reclaim, and the lazily built shared Rust registry behind its
mutex. All counters keep their memory orders.

lsp_resolve_shim.c: lsp_resolve.h defines its helpers as static inline,
and Zig reads them through translate-c. cbm_pipeline_lsp_target_node
counts its tail-match lookups with C11 atomics, which translate-c
cannot express, so Zig sees only its declaration. The shim compiles the
header's own body into a linkable function, which keeps lsp_resolve.h
the single source for the C code that includes it (internal/cbm/lsp and
the tests).

Net change: -122 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The incremental route keeps its order: classify, snapshot inbound
edges, purge, seed the registry, add structure, extract and resolve,
restore edges, the derived passes behind their digests, publish.

Net change: +18 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
The orchestrator: lifecycle, file errors and coverage stamps, the
structure pass, the parallel passes, the predump passes in their order,
and publication. struct cbm_pipeline stays private to this module; the
headers only name it as an opaque type.

cbm_pipeline_run's cleanup label became a helper for the full-index
part, called before the same cleanup sequence.

With this commit no pipeline pass is built from C. Makefile.cbm and
build.zig list only lsp_resolve_shim.c under src/pipeline.

Net change: -35 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
@ata-sesli
ata-sesli force-pushed the refactor/pipeline-zig branch from 45ff8e1 to 89fe759 Compare November 11, 2026 11:23
@ata-sesli
ata-sesli changed the base branch from refactor/store-zova-zig to main November 11, 2026 11:24
@ata-sesli
ata-sesli merged commit e274c1e into octacity-org:main Nov 11, 2026
@professional-tester

Copy link
Copy Markdown
Contributor Author

Release-build timing on redis

The timings in the description compare binaries from make -f Makefile.cbm cbm, which compiles the C sources in debug mode with undefined-behaviour checks. They overstate the effect of the port. This repeats the measurement through the release path.

Both commits built on Linux x86_64 with gcc and the pinned Zova:

make -j8 -f Makefile.cbm build/c/codebase-memory-mcp CBM_WITH_ZOVA=1 ZOVA_ROOT=<zova pin> CC=gcc CXX=g++

Neither binary contains __ubsan_handle_* symbols. Three rounds of three runs per step:

Step f02b64d3 (before) e274c1eb (main after #165, #166, #167)
Full index (9 runs each), median 20.0 s (19.9–21.0) 19.7 s (19.5–21.8)
First incremental re-index (3 runs each) 10.5–10.6 s 10.1–10.2 s
Later incremental re-indexes (6 runs each) 6.4–6.5 s 6.3–6.4 s

In a release build the ports are neither a speedup nor a regression: the differences are 0.1–0.4 s, about the run-to-run spread for the full index. e274c1eb contains #165, #166 and #167 together, so this compares all three ports against f02b64d3, not the pipeline port alone.

Counts:

  • Full index: all 35 label and edge-type counts are identical between the two binaries in every round.
  • Incremental re-index: every count is identical except CALLS: 40,814 / 40,820 / 40,818 on f02b64d3, 40,816 in all three rounds on e274c1eb. The run-to-run variation exists before the ports.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants