Repository navigation
fix(python): grow buffers instead of failing, guard optional ops, stop redundant FFI work - #814
Open
Python11235 wants to merge 4 commits into
Open
Python11235 wants to merge 4 commits into
Python11235 wants to merge 4 commits into
Conversation
Refactor C function calls to handle optional functions and improve error handling. Update buffer management for various functions to use a reusable buffer pattern. Modify tensor handling to support shape and dtype resolution.
- Allocate response buffers per call again. The shared module-level buffers let concurrent calls (e.g. the server running two warm models in executor threads) write into the same memory, so one request could receive another request's response. - Drop the global callback cache. It kept every token callback (and its closure) alive forever, leaking one entry per streaming server request, and raised TypeError for unhashable callables. The local `cb` already keeps the trampoline alive for the synchronous C call. - Look up optional symbols with getattr(_lib, name, None). Accessing a missing symbol as `_lib.name` raises AttributeError, which broke the gather -> cactus_graph_gather fallback and hid the "rebuild with cactus build --python" message from _call_optional. - Restore the (ptr, size) return of _prepare_pcm, keeping the faster buffer copy. The returned pointer holds a reference to its data, which fixes the TestPreparePcm tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hf1fEfGn1rnodpQdjope6c
Fix regressions in Python bindings refactor
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Went through the Python bindings while wiring them into a project and found a handful of
things that bite at runtime. All fixes are on the Python side only — no FFI declarations,
signatures, or argtypes were touched, so this shouldn't affect any other binding.
Buffer-too-small is currently reported as a generic failure
cactus_tokenize,cactus_embed,cactus_image_embed,cactus_audio_embed, andcactus_preprocess_audio_featuresall use a fixed scratch buffer and only checkrc < 0.But the C side returns
-2for "buffer too small" and doesn't setlast_error_message(
complete.cpp,embed.cpp), and_err()only readscactus_get_last_error(). So anytext over 8192 tokens or any model with embedding_dim > 4096 dies with
RuntimeError: Tokenization failed/Embedding failedand no hint about the real cause.These now detect
-2and retry with a grown buffer. Worth notingcactus_embeddoes notwrite the dimension on the
-2path, so that one doubles rather than trusting the out-param;cactus_tokenizedoes write the count, so it uses that as a hint.-2handling in the index helperscactus_index_get: early-returns{"results": []}for an empty id list — the C functionrejects
count == 0with-1, so an empty lookup used to raise.cactus_index_query/cactus_index_get: reject empty embeddings before building azero-length ctypes array.
cactus_index_add: same for an empty batch (it indexedembeddings[0]unconditionally).index_getare now read back through the returned size with a bounds check.Optional ops raise a raw
AttributeErrorOps registered via
_bind_optionalare bound toNonewhen a build doesn't export them, butthe call sites invoked them directly, so you'd get a bare
AttributeErrorinstead ofanything actionable. These now go through
_call_optionaland raise a clear"rebuild with
cactus build --python" message.gather'shasattrcheck became a plainis not Nonetest on the already-bound handle.Graph.inputdidn't validatedynamic_dimslen(dynamic_dims)was forwarded as the rank without comparing it toshape. Now raises aValueErroron a mismatch instead of passing an inconsistent rank to C.Tensormetadata made construction cost one FFI call per node_tensor_from_nodecalled_get_output_info(a round trip intograph.get_output_buffer)for every node created, so building an N-node graph cost N extra FFI calls plus the objects
to hold them.
shapeanddtypeare now lazy properties that resolve on first access andcache, and
Graph.inputpasses what it already knows.Tensor.numpy()reuses the sameresolution.
set_inputforced a copy and validated twice_coerce_input_arrayalways rebuilt the array, copying even when the input was alreadycontiguous and correctly typed. It now returns the array as-is in that case. Also dropped the
element-count check, which is implied by the shape + byte-size checks. (Byte-swapped dtypes
still fall through to
ascontiguousarray, since numpy dtype equality correctly rejects them.)Per-call buffer churn
cactus_complete,cactus_prefill,cactus_transcribe, the streaming functions,cactus_render_prompt,cactus_score_windowandcactus_rag_queryeach did a freshcreate_string_buffer(1 << 20)/64 KiBper call, which is a zeroed allocation and memsetevery time. Added a small reusable-buffer helper. This is safe because all of those C
functions NUL-terminate into a
char*buffer and_from_jsonstops at the NUL.Also removed the per-call
getattr(_lib, fn_name)lookups in_binary,_scalar,_reduceand
_conv_with_optional_bias(hottest path in the bindings), and made_prepare_pcmavoidcopying contiguous
uint8input.Verification
Ran a differential check of every
argtypes/_bind_optionaldeclaration againstcactus_engine.h,cactus_graph.h,graph_ffi.cppandindex_ffi.cpp— no signaturechanges, all counts and types still match. Behaviour was exercised against a stub library
(buffer growth, optional-op errors, lazy metadata, malformed JSON, buffer reuse, PCM
marshalling). Happy to clean that harness up into a real test suite if that's useful.
Not included
Tensor.numpy()doesn't retain the node before reading its output pointer, which looks like areal ordering gap against
graph.execute(). Fixing it properly means deciding whenexecute()retains nodes, so I left it out of this PR rather than guessing at the intended API.
Signed-off-by: themailbot94@gmail.com