MicroLLM

A compiler for LLMs.

Take a HuggingFace model (weights + config) → emit a single, self-contained binary that runs the model on a target hardware. No Python. No CUDA. No libtorch. No libllama. No runtime framework — just the OS and its GPU driver.

$ hf download mlx-community/Qwen3-8B-4bit --local-dir ./Qwen3-8B-4bit

$ microllm compile ./Qwen3-8B-4bit --quant int4 -o compiled/qwen3-8b
done in 34s. artifact: 4.3 GB (graph.json + weights.bin + tokenizer.json).

$ microllm bake compiled/qwen3-8b \
    --tokenizer compiled/qwen3-8b/tokenizer.json \
    --stub target/release/microllm-stub \
    --output ./qwen3
wrote ./qwen3 (4.62 GB: stub + graph + weights + tokenizer).

$ ./qwen3 'Explain a transformer in one sentence.'
A transformer is a neural-network architecture that processes
sequences using self-attention to weigh relationships between
tokens in parallel.

[31 tok/s decode · M1 Pro 16 GB · one file, no Python/CUDA/framework]

./qwen3 is the whole product: runtime, Metal kernels, int4 weights and tokenizer in a single executable. Nothing to install, nothing to download at run time. (microllm forward compiled/qwen3-8b --chat --prompt '…' runs the same model straight from the artifact directory — handy while iterating, before you bake.)

Status

v0.2.0 — Qwen3-8B on Apple Silicon, MLX-native asymmetric int4 (I4G64Bias pass-through: bit-exact MLX weights, no requantization loss).

• Decode 31.7 tok/s short-context / ~30 tok/s at 3.4K context on an M1 Pro 16 GB — within 2.5% of MLX on the same machine, ahead at long context. Greedy decode runs as event-pipelined GPU chunks with on-GPU argmax token feedback (no per-token host roundtrip).
• Single-binary output verified end to end (bake → verify → run) at both Qwen3-0.6B (528 MB) and Qwen3-8B (4.62 GB).
• 4.3 GB artifact (byte-parity with MLX's own file), ~6.5 GB resident.
• Long-context coherence validated through 3.4K tokens.
• Also in-tree: legacy symmetric int4 Qwen3 path.

Why

Today's LLMs ship as: model weights + Python + framework + driver + container. Most of that stack is overhead. MicroLLM treats an LLM exactly like a C program: weights are source, the compiler does graph fusion + kernel selection + memory planning + quantization, the output is an executable.

A binary's hash is its identity. Sign it. Notarize it. Email it. Put it on a USB. Distribute it like any other software.

Architecture

HuggingFace files                                    Apple Silicon binary
─────────────────                                    ────────────────────
config.json         ─┐                            ┌─ ./qwen3 (one file)
model.safetensors   ─┼─▶ Frontend ─▶ IR ─▶ Optim ─┼─ • Mach-O arm64
tokenizer.json      ─┤    (loader)               │  • metallib (kernels)
generation_config   ─┘                           │  • int4 weights + scales
                                                 └─ • static tokenizer table
                       ▲             ▲           
                       │             │           
                  microllm-       microllm-      
                  frontend        ir + optim     

Crates

Crate	Purpose
microllm-ir	Static computation-graph IR + dtypes + ops + fusion passes
microllm-frontend	Loaders for HuggingFace/MLX formats; per-model graph builders
microllm-codegen	Quantizers + MLX int4 import/pass-through
microllm-linker	Deterministic weight-blob format (64-byte-aligned tensors, mmap zero-copy)
microllm-runtime	Metal forward paths + hand-tuned kernels (kernels/*.metal)
microllm-bake	Self-extracting signed single-binary packaging
microllm-cli	microllm compile / forward / bake / verify driver

Build

Prerequisites:

• Rust 1.78+ (stable)
• Xcode + Metal Toolchain — xcodebuild -downloadComponent MetalToolchain
• huggingface-cli (only for just download-qwen3)

just check       # cargo check --workspace
just test        # all crates (IR + frontend pass today)
just metallib    # compile Metal kernels into a metallib
just download-qwen3   # pulls Qwen/Qwen3-0.6B into ./models/Qwen3-0.6B
just inspect-qwen3    # parse config + emit graph stats (no codegen yet)

Verifiable builds

Binaries produced with --reproducible-build --sign-identity <key> are byte-reproducible and cryptographically signed. The runtime self-checks its own bytes at every startup and refuses to run on tampering. An auditor can rebuild the same binary from source + model and prove byte-for-byte equality with the shipped artifact.

Sign

# Generate a 32-byte Ed25519 private key (raw bytes; PEM not yet supported in v1).
head -c 32 /dev/urandom > signer.key

microllm bake compiled/qwen3-8b-mlx \
  --tokenizer compiled/qwen3-8b-mlx/tokenizer.json \
  --stub target/release/microllm-stub \
  --output qwen3-signed.bin \
  --reproducible-build \
  --sign-identity signer.key \
  --signer-id "acme-prod-2026Q2"

Output is a v2 baked binary with:

• a CBOR provenance manifest binding model + runtime + build environment by content hash,
• a 64-byte Ed25519 signature over BLAKE3(file with sig + hash regions zeroed),
• the BLAKE3 content hash embedded in the footer.

Verify

# Runtime-style integrity + signature check.
microllm verify qwen3-signed.bin
# → ok: qwen3-signed.bin integrity + signature verified

# Same check + dump the embedded manifest.
microllm verify qwen3-signed.bin --show-manifest

A v2 binary calls the same runtime_verify path at every startup. Override with MLLM_ALLOW_UNVERIFIED=1 for dev use (loud warning).

Reproduce from source (auditor)

git clone https://github.com/<org>/microllm
cd microllm && git checkout <code-revision-from-manifest>
# Pull the model the manifest pins (or use a locally-trusted copy):
hf download <model-source-uri>

SOURCE_DATE_EPOCH=<from-manifest> \
  cargo run --release -p microllm-cli -- compile <model-dir> -o compiled/<name>

cargo run --release -p microllm-cli -- bake compiled/<name> \
  --tokenizer <model-dir>/tokenizer.json \
  --stub target/release/microllm-stub \
  --output rebuilt.bin \
  --reproducible-build \
  --sign-identity signer.key \
  --signer-id <key-id> \
  --source-date-epoch <from-manifest>

# Compare embedded content hashes:
microllm verify rebuilt.bin --show-manifest > rebuilt.txt
microllm verify shipped.bin --show-manifest > shipped.txt
diff rebuilt.txt shipped.txt

If the manifests match (including the content_hash field), the auditor has proven byte-for-byte equality without trusting the build farm.

Trust model

Protects against:

• Binary substitution in distribution (MITM, mirror compromise, registry takeover).
• Tampering with a deployed binary — any single-byte modification is detected at startup and by microllm verify.
• Drift between "what was reviewed" and "what was deployed" — auditor re-derives byte-for-byte.
• Vendor opacity — provenance manifest binds model identity (weights hash, tokenizer hash, source URI) to runtime identity (stub hash, graph hash, code revision).

Does NOT protect against:

• Compromise of the signing key (key custody is out of scope; v1 reads a raw 32-byte file).
• A malicious build farm signing the right key over malicious contents.
• Memory tampering during inference (only verified at startup, not continuously).
• Side-channel attacks on inference (timing, power, EM).
• Dependency-chain attacks (compromised Rust crate, compromised metallib). Use cargo-vet or SLSA build attestation for that layer.
• Anything about the model's behavior — this proves WHAT was shipped, not WHETHER its behavior is safe to use.

For regulated environments (FDA SOUP, SOX-equivalent model risk, gov/defense supply-chain attestation), the reproducibility claim is the primary value: an auditor can independently verify that a deployed binary corresponds exactly to a reviewed source revision + reviewed model, without trusting the vendor's build process.

License

Apache-2.0.

Portions of the Metal kernels and runtime synchronization code are derived from Apple MLX (MIT). See NOTICE for the full attribution; each affected source file carries a "PORTIONS DERIVED FROM APPLE MLX" header.