[siilats]

Apple M6 Macs are rejected by Rec, and the bundled INT4 profile targets machines with at least 48 GiB. This adds M6 support, a pinned mixed 3/4-bit checkpoint for 32–47 GiB Macs, selectable oQ3e weights, and measured prefill throughput. Both small-memory profiles default to 262,144 total input/output tokens.

Select a quant

rec serve --quant 3/4
rec serve --quant oq3e

Stop the existing server before switching. --quant also works with pull and doctor; auto preserves RAM-based selection. An explicit profile cannot be combined with --manifest. Machines with 48 GiB or more retain the official INT4 default.

• oQ3e is pinned to mlx-community/flint-oQ3e revision 274084481e0cafc1850408b315bb3375ec46a280, with SHA-256 verification for every downloaded file.
• Preserve the checkpoint's per-module 3/6/8-bit affine weights and group sizes 64/128. Normalize the language_model. prefix and routing-bias names without dropping bias values.
• Use native MLX operations for oQ3e rather than the resident kernels' INT4 projection assumptions. Mixed 3/4-bit keeps the existing optimized path.
• Reuse a matching native prompt prefix using isolated BF16 KV snapshots, including sliding-window metadata. Recompute safely on a mismatch. Decoding does not mutate the saved prefix. This is one in-memory prefix, not a disk or branching cache.
• Automatically include the sibling oQ3e descriptor when packaging the small-memory profile and replace built executables atomically to avoid stale macOS code-signature pages when rebuilding a previously running binary.

oQ3e cache validation

On the tested 32 GiB M6, real weights loaded with approximately 18.5 GB peak MLX allocation. These are short smoke measurements, not an agent-quality benchmark.

Same 252-token prompt	Uncached, after warm-up	Cached
Reused tokens	0	251
Prefill	0.255 s	0.029 s

Cached and uncached greedy output token IDs matched exactly. The packaged HTTP server separately reused 250 of 251 prompt tokens and returned identical answers. Both profiles advertise 262,144 tokens; a fully occupied 256K prompt was not memory-stress-tested on the 32 GiB Mac. oQ3e is a native decode path with different performance from the optimized mixed-bit path; switching quants has not been established to solve agent loops.

Runtime and prefill t/s

The runtime moves to CPython 3.14.10, nanobind 3.1.0, and MLX pinned to fc344c986c4169d99f76b5d95ae6b4a58a359a63, including thread-local random-state compatibility and native ABI packaging. Prefill diagnostics append throughput computed from newly processed tokens; no completed rate is claimed for cancellation.

Short runtime benchmark

Apple M6, 32 GiB, identical mixed 3/4-bit weights, temperature 1, top-p .95, top-k 20, 128 generated tokens. Medians of two measured runs after warm-up, alternating old/new/new/old; fast-sync enabled, prompt cache reset. Startup and HTTP overhead excluded. This compares the runtime stack, not quant quality.

Metric	Python 3.12.12 / MLX 0.32.0	Python 3.14.10 / MLX HEAD	Change
Decode, 34-token prompt	85.1 tok/s	84.2 tok/s	−1.1%
Decode, 4,095-token prompt	76.5 tok/s	75.8 tok/s	−0.9%
First token, 34-token prompt	185 ms	149 ms	19.5% faster
First token, 4,095-token prompt	2.585 s	2.288 s	11.5% faster

The dedicated M6 profile retains the measured default schedule: 36/32/32 workers, 2048-token prefill, generic mixed experts. The 4096-token/M5-specialization combination did not improve the tested 8192-token prompt. Exploratory schedules, hardware references and limitations are in engine/M6_TUNING.md. Architecture assertions include the tested applegpu_g18g and applegpu_g18s.

Anthropic cache reporting

Preserve actual prefix-cache reads in both JSON and streaming Messages responses so Claude Code sessions and RSIWatch can report reuse. Chat counts cached tokens inside prompt_tokens; Messages now splits those into new input_tokens and cache_read_input_tokens without double counting. For example, 100 prompt tokens with 80 reused tokens reports 20 new input tokens and 80 cache reads. Cache creation remains zero because Rec uses automatic local KV-prefix reuse rather than billed cache-creation tiers.

Streaming sends the final cumulative usage split in message_delta, replacing provisional counts from message_start. Existing transcripts that recorded zero cache reads cannot be reconstructed by this fix.

• 12 Anthropic protocol tests and 17 HTTP serving tests passed, covering cached JSON responses and final streamed usage.
• An isolated real Claude Code 2.1.319 probe saved 20 new input tokens and 80 cache reads in its native session JSONL, confirming compatibility with RSIWatch's session reader.
• Rebuilt the local binary as 0.1.0-dev-anthropic-cache; restart Rec to load the updated runtime.

Validation

• 78 engine tests passed, including native decoder sampling, EOS, cancellation, context bounds and KV snapshot isolation after sliding-window wraparound.
• CLI Go race tests and vet passed; full CLI, Node/client wiring, installer and packaging checks passed for this update.
• Built 0.1.0-dev-oq3e-cache; verified quant selection, unknown-profile rejection, 262,144-token status and real packaged HTTP generation/cache reuse.
• Earlier mixed-bit validation covered tool replay, streaming, scalar/packet sampling, EOS, cancellation, protocol, and real-model prefix rollback.

Known limitation — keep draft

Independent native-versus-resident parity for the existing optimized mixed-bit path remains unresolved: 72 exact comparisons failed, with maximum absolute logit difference 41. The prior MLX 0.32.0 runtime also reproduced that failure on M6. No tolerances were relaxed. A matched-history audit agreed on the top token at 44/48 positions for new native versus new Rec and 46/48 for old versus new Rec. These are numerical consistency measurements, not task-accuracy results. oQ3e uses native MLX instead of that resident path.