Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ jobs:
license "Cactus Compute License"
include Language::Python::Virtualenv
depends_on "python@3.12"
depends_on macos: :ventura
depends_on macos: :sonoma
depends_on arch: :arm64

livecheck do
Expand Down
40 changes: 25 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,42 +196,48 @@ graph.hard_reset();
│ │
│ cactus run [model|path] run a model (downloads if needed) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target accelerator (default: auto) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --image <path> image file for VLM inference │
│ --audio <path> audio file for audio chat │
│ --system <prompt> system prompt │
│ --prompt <text> send prompt immediately │
│ --thinking enable thinking/reasoning mode │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local convert+transpile fallback │
│ --reconvert force local rebuild from source │
│ │
│ cactus transcribe [model] live microphone transcription with a model│
│ --file <audio.wav> audio file to transcribe (WAV) │
│ --language <code> language code (default: en) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force reconversion from source │
│ --reconvert force local rebuild from source │
│ │
│ cactus download [model] download a pre-built bundle │
│ cactus download [model] get a bundle (prebuilt, else build) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target accelerator (default: auto) │
│ --token <token> HuggingFace token │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ │
│ cactus convert <model> [dir] convert HuggingFace weights to CQ │
│ cactus convert <model> [dir] build a runnable bundle locally │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --token <token> HuggingFace token │
│ --reconvert force build from source │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --lora <path> merge a LoRA adapter before converting │
│ │
│ cactus transpile <model> build a runnable bundle from CQ weights │
│ --weights-dir <path> path to CQ weights (default: lookup) │
│ --task <auto|...> force task type (default: auto) │
│ --artifact-dir <path> bundle output (default: weights/<model>) │
│ │
│ cactus serve [model] OpenAI-compatible local HTTP server │
│ --host <addr> bind address (default: 127.0.0.1) │
│ --port <port> port (default: 8080) │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild from source │
│ --no-cloud-handoff disable automatic cloud handoff │
│ --confidence-threshold <0..1> handoff to cloud below this confidence │
│ --cloud-timeout-ms <n> max wait for cloud handoff │
│ │
│ cactus list list local converted weights and bundles │
│ cactus list list downloaded models │
│ │
│ cactus build build cactus libraries │
│ --apple Apple (iOS/macOS) │
Expand All @@ -243,6 +249,10 @@ graph.hard_reset();
│ (default: all) │
│ --model <hf-id> default: LiquidAI/LFM2-VL-450M │
│ --transcription-model <hf-id> default: openai/whisper-base │
│ --bits 1|2|3|4 CQ quantization (default: 4) │
│ --platform auto|cpu|apple target platform (default: auto) │
│ --token <token> HuggingFace token (gated models) │
│ --reconvert force local rebuild of test models │
│ --suite <name> run a single test suite by name │
│ (resolved across components, │
│ e.g. llm → engine) │
Expand Down
2 changes: 1 addition & 1 deletion bindings/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ ctypes FFI to `cactus_engine.h`. The binding module lives at
It loads `libcactus_engine.{so,dylib}` from `/python/cactus/bindings/lib/` (bundled)
or `cactus-engine/build/` (dev build), populated by `cactus build --python`.

For installation, the high-level API, the CLI, the server, and the transpiler,
For installation, the high-level API, the CLI, and the server,
see [`/python/README.md`](/python/).
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,8 @@ static bool test_stream_matches_oneshot() {

std::vector<int16_t> pcm = load_wav(std::string(g_assets) + "/test.wav");
if (pcm.size() > 20 * 16000) pcm.resize(20 * 16000);
auto golden = words_of(transcribe_oneshot(model, pcm));
std::string oneshot_text = transcribe_oneshot(model, pcm);
auto golden = words_of(oneshot_text);
if (golden.size() < 4) { cactus_destroy(model); std::cout << " SKIP (test.wav has no usable speech)\n"; return true; }

cactus_stream_transcribe_t stream = cactus_stream_transcribe_start(model, nullptr);
Expand Down Expand Up @@ -135,6 +136,8 @@ static bool test_stream_matches_oneshot() {
cactus_destroy(model);

auto sw = words_of(streamed);
std::cout << " ONE-SHOT: " << oneshot_text << "\n";
std::cout << " STREAM: " << streamed << "\n";
double rc = recall(golden, sw), pr = recall(sw, golden);
bool ok = rc >= 0.85 && pr >= 0.85;
std::cout << " recall=" << std::fixed << std::setprecision(3) << rc << " precision=" << pr
Expand All @@ -143,8 +146,8 @@ static bool test_stream_matches_oneshot() {
}

int main() {
TestUtils::TestRunner runner("Stream Tests");
runner.run_test("stream_matches_oneshot", test_stream_matches_oneshot());
TestUtils::TestRunner runner("Stream Transcribe Tests");
runner.run_test("stream_transcribe_matches_oneshot", test_stream_matches_oneshot());
runner.print_summary();
return runner.all_passed() ? 0 : 1;
}
2 changes: 1 addition & 1 deletion docs/cactus_engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ best build for your host, e.g. Apple Silicon on macOS); pass `--platform cpu`
for the generic build. The result can be loaded directly via `cactus_init()`.

For models not on Cactus-Compute, build a bundle from source with
`cactus convert <model>` followed by `cactus transpile <model>`.
`cactus convert <model>`.

## Types

Expand Down
24 changes: 7 additions & 17 deletions docs/cactus_quants.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,28 +183,18 @@ Three patterns are visible:

## Using CQ Weights

CQ weights are produced by `cactus convert`. The runtime bundle (graph + manifest)
is then built from those weights with `cactus transpile`, and run via
`cactus run`:
`cactus convert <model>` builds a runnable bundle locally; `cactus run` runs a
bundle path or a model id.

```bash
# 1. Convert at 4-bit (default)
cactus convert google/gemma-4-E2B-it ./gemma4-weights

# 1'. Convert at 2-bit
cactus convert google/gemma-4-E2B-it ./gemma4-weights --bits 2

# 2. Build the runtime bundle from the CQ weights
cactus transpile google/gemma-4-E2B-it \
--weights-dir ./gemma4-weights --artifact-dir ./gemma4-bundle

# 3. Run (pass either a HF model id or a bundle path)
cactus convert google/gemma-4-E2B-it ./gemma4-bundle
cactus convert google/gemma-4-E2B-it ./gemma4-bundle --bits 2
cactus run ./gemma4-bundle
```

For models on huggingface.co/Cactus-Compute, you can skip the convert+transpile
steps entirely: `cactus run <model-id>` will fetch the pre-built bundle (or fall
back to local convert+transpile if no bundle is published), then run.
For models on huggingface.co/Cactus-Compute, `cactus run <model-id>` (or `cactus
download <model-id>`) fetches the pre-built bundle, building locally if none is
published.

The CQ matmul kernels live in `cactus-kernels/src/matmul.cpp`. At inference,
the kernel applies a Walsh-Hadamard transform to the FP16 activation (not the weight),
Expand Down
4 changes: 2 additions & 2 deletions docs/cactus_transpiler.md
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ huggingface.co/Cactus-Compute) or a local path to a bundle directory.
| Option | Description |
|--------|-------------|
| `--bits 1\|2\|3\|4` | CQ quantization bits when downloading (default: 4) |
| `--platform auto\|cpu\|apple` | Target accelerator (default: auto — best for the host) |
| `--platform auto\|cpu\|apple` | Target platform (default: auto — best for the host) |
| `--token <token>` | HuggingFace token (gated models) |
| `--prompt <text>` | Input prompt |
| `--input-ids <ids>` | Comma-separated token IDs |
Expand All @@ -473,7 +473,7 @@ huggingface.co/Cactus-Compute) or a local path to a bundle directory.
| `--confidence-threshold <f>` | Confidence threshold below which to hand off to cloud |
| `--cloud-timeout-ms <N>` | Max wait time for cloud handoff |
| `--no-cloud-tele` | Disable cloud telemetry (write to cache only) |
| `--reconvert` | Force local convert+transpile fallback |
| `--reconvert` | Force local rebuild from source |

---

Expand Down
2 changes: 1 addition & 1 deletion docs/choose-bindings.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Pick the right one for your platform and use case:
- **React Native** -- already have a React Native project? Just `npm install` and go
- **Flutter** -- cross-platform mobile + Mac with full native bindings
- **Kotlin** -- native Android apps or Kotlin Multiplatform
- **Swift** -- native iOS/macOS apps with Apple NPU acceleration
- **Swift** -- native iOS/macOS apps with the Apple Neural Engine

!!! tip "Server-side or scripting?"
- **Python** -- server-side inference, batch processing, or rapid prototyping
Expand Down
5 changes: 0 additions & 5 deletions docs/finetuning.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,7 @@ cactus convert Qwen/Qwen3-0.6B ./my-qwen3-0.6b --lora username/my-lora-adapter

### 4. Run

Build the native library, transpile the weights into a runtime bundle, then run:

```bash
cactus build
cactus transpile Qwen/Qwen3-0.6B \
--weights-dir ./my-qwen3-0.6b --artifact-dir ./my-qwen3-0.6b
cactus run ./my-qwen3-0.6b
```

Expand Down
Loading
Loading