Skip to content

fix(python): grow buffers instead of failing, guard optional ops, stop redundant FFI work - #814

Open
Python11235 wants to merge 4 commits into
cactus-compute:mainfrom
Python11235:main
Open

Python11235 wants to merge 4 commits into
cactus-compute:mainfrom
Python11235:main

Conversation

@Python11235

@Python11235 Python11235 commented Sep 22, 2026 •

Copy link
Copy Markdown

Went through the Python bindings while wiring them into a project and found a handful of
things that bite at runtime. All fixes are on the Python side only — no FFI declarations,
signatures, or argtypes were touched, so this shouldn't affect any other binding.

Buffer-too-small is currently reported as a generic failure

cactus_tokenize, cactus_embed, cactus_image_embed, cactus_audio_embed, and
cactus_preprocess_audio_features all use a fixed scratch buffer and only check rc < 0.
But the C side returns -2 for "buffer too small" and doesn't set last_error_message
(complete.cpp, embed.cpp), and _err() only reads cactus_get_last_error(). So any
text over 8192 tokens or any model with embedding_dim > 4096 dies with
RuntimeError: Tokenization failed / Embedding failed and no hint about the real cause.

These now detect -2 and retry with a grown buffer. Worth noting cactus_embed does not
write the dimension on the -2 path, so that one doubles rather than trusting the out-param;
cactus_tokenize does write the count, so it uses that as a hint.

-2 handling in the index helpers

  • cactus_index_get: early-returns {"results": []} for an empty id list — the C function
    rejects count == 0 with -1, so an empty lookup used to raise.
  • cactus_index_query / cactus_index_get: reject empty embeddings before building a
    zero-length ctypes array.
  • cactus_index_add: same for an empty batch (it indexed embeddings[0] unconditionally).
  • Embeddings in index_get are now read back through the returned size with a bounds check.

Optional ops raise a raw AttributeError

Ops registered via _bind_optional are bound to None when a build doesn't export them, but
the call sites invoked them directly, so you'd get a bare AttributeError instead of
anything actionable. These now go through _call_optional and raise a clear
"rebuild with cactus build --python" message. gather's hasattr check became a plain
is not None test on the already-bound handle.

Graph.input didn't validate dynamic_dims

len(dynamic_dims) was forwarded as the rank without comparing it to shape. Now raises a
ValueError on a mismatch instead of passing an inconsistent rank to C.

Tensor metadata made construction cost one FFI call per node

_tensor_from_node called _get_output_info (a round trip into graph.get_output_buffer)
for every node created, so building an N-node graph cost N extra FFI calls plus the objects
to hold them. shape and dtype are now lazy properties that resolve on first access and
cache, and Graph.input passes what it already knows. Tensor.numpy() reuses the same
resolution.

set_input forced a copy and validated twice

_coerce_input_array always rebuilt the array, copying even when the input was already
contiguous and correctly typed. It now returns the array as-is in that case. Also dropped the
element-count check, which is implied by the shape + byte-size checks. (Byte-swapped dtypes
still fall through to ascontiguousarray, since numpy dtype equality correctly rejects them.)

Per-call buffer churn

cactus_complete, cactus_prefill, cactus_transcribe, the streaming functions,
cactus_render_prompt, cactus_score_window and cactus_rag_query each did a fresh
create_string_buffer(1 << 20) / 64 KiB per call, which is a zeroed allocation and memset
every time. Added a small reusable-buffer helper. This is safe because all of those C
functions NUL-terminate into a char* buffer and _from_json stops at the NUL.

Also removed the per-call getattr(_lib, fn_name) lookups in _binary, _scalar, _reduce
and _conv_with_optional_bias (hottest path in the bindings), and made _prepare_pcm avoid
copying contiguous uint8 input.

Verification

Ran a differential check of every argtypes / _bind_optional declaration against
cactus_engine.h, cactus_graph.h, graph_ffi.cpp and index_ffi.cpp — no signature
changes, all counts and types still match. Behaviour was exercised against a stub library
(buffer growth, optional-op errors, lazy metadata, malformed JSON, buffer reuse, PCM
marshalling). Happy to clean that harness up into a real test suite if that's useful.

Not included

Tensor.numpy() doesn't retain the node before reading its output pointer, which looks like a
real ordering gap against graph.execute(). Fixing it properly means deciding when execute()
retains nodes, so I left it out of this PR rather than guessing at the intended API.

Signed-off-by: themailbot94@gmail.com

Refactor C function calls to handle optional functions and improve error handling. Update buffer management for various functions to use a reusable buffer pattern. Modify tensor handling to support shape and dtype resolution.
Python11235 and others added 3 commits October 15, 2026 17:41
- Allocate response buffers per call again. The shared module-level
  buffers let concurrent calls (e.g. the server running two warm models
  in executor threads) write into the same memory, so one request could
  receive another request's response.
- Drop the global callback cache. It kept every token callback (and its
  closure) alive forever, leaking one entry per streaming server request,
  and raised TypeError for unhashable callables. The local `cb` already
  keeps the trampoline alive for the synchronous C call.
- Look up optional symbols with getattr(_lib, name, None). Accessing a
  missing symbol as `_lib.name` raises AttributeError, which broke the
  gather -> cactus_graph_gather fallback and hid the "rebuild with
  cactus build --python" message from _call_optional.
- Restore the (ptr, size) return of _prepare_pcm, keeping the faster
  buffer copy. The returned pointer holds a reference to its data, which
  fixes the TestPreparePcm tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hf1fEfGn1rnodpQdjope6c
Fix regressions in Python bindings refactor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants