Skip to content

refactor(semantic): port src/semantic from C to Zig - #166

Merged
ata-sesli merged 1 commit into
octacity-org:mainfrom
professional-tester:refactor/semantic-zig
Nov 11, 2026
Merged

ata-sesli merged 1 commit into
octacity-org:mainfrom
professional-tester:refactor/semantic-zig

Conversation

@professional-tester

Copy link
Copy Markdown
Contributor

What does this PR do?

Ports src/semantic from C to Zig, following the pattern of #163. Stacked on the storage-chain port (#165, refactor/store-zova-zig); review #164 and #165 first.

semantic.c, ast_profile.c and rotsq.c become semantic.zig, ast_profile.zig and rotsq.zig in libcbm_zig, plus a 28-line fp.zig. No C file and no shim remains in src/semantic.

The C headers do not change. cbm_zig.zig checks every exported cbm_* function against its header declaration at compile time.

Net change: 2,236 lines removed, 2,154 added (−82).

Behavior differences

  • rotsq: the ±1 diagonal is built at compile time instead of on first use through cbm_once. Same values.
  • Floating point: fp.mulAdd fuses multiply-add at the places where clang's default -ffp-contract=on fused it in the C build, so scores stay bit-identical (see Verification).

Build and audit wiring

  • Makefile.cbm and build.zig drop the three C sources and list the Zig files and headers in CBM_ZIG_SRCS; scripts/check-build-sources.sh passes.
  • cbm_zig_c.h adds nomic/code_vectors.h and semantic/ast_profile.h, and moves semantic/semantic.h to the ported-modules list.
  • scripts/extract_nomic_vectors.py: the hint it prints named semantic.c; it now names src/cbm_zig_c.h.
  • No audit count changes.

Verification

Run on Linux x86_64 with the pinned Zova.

At this branch's commit (on top of #165):

  • The production binary builds with Zova.
  • libcbm_zig.a builds for native, x86_64-linux-gnu, aarch64-linux-gnu, x86_64-windows-gnu and aarch64-macos (without Zova).
  • zig fmt --check, scripts/check-build-sources.sh, scripts/security-audit.sh and scripts/check-dco.sh pass.

Before the rebase onto #165 (same semantic files, on top of main a592846a):

  • 43 suites through scripts/zova-run-tests.sh match main, including semantic 33, ast_profile 10, pipeline 276, incremental 149 and mcp 176.
  • Full index of redis, cpython and kubernetes: node and edge counts per label and type are identical to main. On redis, all 480 SEMANTICALLY_RELATED edges and their scores are byte-identical.
  • Full-index wall time, median of 3 (main / port): redis 32.9 s / 27.1 s, cpython 155.4 s / 145.4 s, kubernetes 285.9 s / 266.1 s. The port is faster because zig build cbm compiles the C sources without -Doptimize, which leaves undefined-behaviour checks in the C code, while the Zig library is ReleaseFast. On redis only the semantic_edges pass changed (9.07 s → 3.23 s).

Not run / not verified:

  • The suites and the graph comparison were not repeated at this exact commit. They were repeated on the stacked pipeline branch (see the follow-up PR).
  • Incremental re-index counts are not compared: the same main binary gives different counts from run to run (redis CALLS 40,815 or 40,813).
  • make -f Makefile.cbm test, lint-ci and smoke-test.sh.
  • Any runtime test on Windows or macOS. Those paths are compile-checked only.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test): not run; the Zova-enabled suites listed above pass
  • Lint passes (make -f Makefile.cbm lint-ci): not run
  • New behavior is covered by a test: no new behavior; the existing suites cover the ported code

🤖 Generated with Claude Code

Replace src/semantic/{semantic,ast_profile,rotsq}.c with Zig modules in
libcbm_zig. The C headers (semantic.h, ast_profile.h, rotsq.h) stay the
ABI: each Zig module exports the same symbols, and cbm_zig.zig checks
every pub cbm_* export against its translated header declaration.

- semantic.zig: tokenizer, corpus (IDF, random indexing, co-occurrence),
  vector math and the combined score. It reads the pretrained nomic
  table through nomic/code_vectors.h, as the C code did.
- ast_profile.zig: the AST structural profile and its string form.
- rotsq.zig: the rotational scalar quantizer.
- fp.zig: mulAdd. The C build contracts `a * b + c` into one fused
  multiply-add where the CPU has FMA (clang's -ffp-contract=on); Zig
  never fuses on its own. mulAdd fuses at the same places, so the
  results match the C build on the same CPU.

Behavior differences from the C code:

- rotsq builds its +/-1 diagonal at compile time instead of on first use
  under cbm_once. The values are the same (XXH3 of the dimension index
  with the same seed).

Build wiring: Makefile.cbm and build.zig drop the C sources and list the
Zig files; cbm_zig_c.h gains nomic/code_vectors.h, pipeline/worker_pool.h
and the two semantic headers. scripts/extract_nomic_vectors.py names
cbm_zig_c.h as the place that includes the vector table.

Net change: -80 lines.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>

Signed-off-by: professional-tester <250027118+professional-tester@users.noreply.github.com>
Co-authored-by: Ata Sesli <atasesli@protonmail.com>
@ata-sesli
ata-sesli force-pushed the refactor/semantic-zig branch from 0bbaa37 to c0413f8 Compare November 11, 2026 11:24
@ata-sesli
ata-sesli changed the base branch from refactor/store-zova-zig to main November 11, 2026 11:24
@ata-sesli
ata-sesli merged commit e0301ab into octacity-org:main Nov 11, 2026
ata-sesli added a commit that referenced this pull request Nov 11, 2026
The C semantic code never asked for fused multiply-add, but compilers
decided for it: Apple clang fused `a * b + c` on arm64, while the Linux
release (gcc -std=c11) and the baseline-CPU x86_64 builds did not. The
semantic scores therefore differed in the last bit between platforms,
which can flip a SEMANTICALLY_RELATED edge whose score sits exactly at
the threshold.

The Zig port (#166) copied clang's choice through fp.mulAdd, which fuses
wherever the CPU has FMA. That kept macOS arm64 unchanged but changed
Linux arm64, and the store quantizer (#165) never fused at all.

Use one rule everywhere instead: never fuse. Zig does not fuse on its
own, so `a * b + c` is written as-is and every platform computes the
same scores. fp.zig is removed; its call sites in semantic.zig,
rotsq.zig and pass_semantic_edges.zig become plain expressions with the
same operand order. Fusing everywhere was not an option: release builds
target the baseline x86_64 CPU, which has no FMA instruction, so @mulAdd
would become a software fma call in the hot loops.

Signed-off-by: ata-sesli <atasesli@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants