Skip to content

About

LLM Inference Engine for Metal, CUDA and Vulkan.

Topics

Resources

Contributing

Security policy

Stars

2.3k stars

Watchers

12 watching

Forks

Repository files navigation

Help Support TensorFold · @tensorfold_dev on X

TensorFold logo

TensorFold

TensorFold serves language models from a Zig binary on Apple Silicon and NVIDIA GPUs. The engine reads checkpoints, tokenizes requests and runs Metal or CUDA kernels directly. Serving needs no Python or MLX installation.

Building with a coding agent? Have it load the contributor skill and follow CONTRIBUTING.md.

Quick start

Start with Qwen3.5-2B, a small 4-bit checkpoint. The commands below download its pinned revision, then serve it as local-model. Model weights have their own licenses.

Mac with Apple Silicon

Install with Homebrew, then start the server:

brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Linux with an NVIDIA GPU

Needs glibc 2.28 or newer and NVIDIA driver R580 or newer (CUDA 13.0). The run guide lists the archive's requirements; Windows has its own guide.

V=1.3.0
case "$(uname -m)" in
  aarch64|arm64) PLATFORM=linux-aarch64 ;;
  x86_64) PLATFORM=linux-x86_64 ;;
  *) echo 'No release archive for this CPU' >&2; exit 1 ;;
esac
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz"
curl -fLO "https://github.com/ashhart/TensorFold/releases/download/v$V/tensorfold-$V-$PLATFORM.tar.gz.sha256"
sha256sum -c "tensorfold-$V-$PLATFORM.tar.gz.sha256"
tar -xzf "tensorfold-$V-$PLATFORM.tar.gz"
export PATH="$PWD/tensorfold-$V-$PLATFORM/bin:$PATH"
tensorfold() { command tensorfold-native "$@"; }
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Send a request

Leave the server running. In another terminal, on either platform:

curl -fsS http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":128,"temperature":0}'

The API also serves completions, Responses and Anthropic Messages. /v1/models lists the served name.

To talk to a model in the terminal in one command, tensorfold chat MODEL starts the server itself, streams each reply and stops the server when the chat ends; /help lists the chat's commands, and /bye or Ctrl-D ends it:

tensorfold chat mlx-community/Qwen3.5-2B-MLX-4bit

With the server above still running, tensorfold chat --url http://127.0.0.1:8080 chats with it instead, and echo "Say hello." | tensorfold chat --url http://127.0.0.1:8080 prints one reply for a script. See the chat command for its flags and the serve flags it passes on.

Docs

Contributors

Thank you to everyone who has sent TensorFold a pull request, a measurement or a bug report:

@0mao0, @321sssrt-bit, @AbrahamCete, @aditya1503, @AdrianBinDC, @ajensenwaud, @akol1, @alenperic, @Alexbob0, @alexderz, @anvilsong, @Arminova, @b-ostrov, @BadAd84, @barelyworkingcode, @Benjamin-Wegener, @benthecarman, @benwilson, @BHCC2025, @Bizuayeu, @BlivionIaG, @BobClawblaw, @borodach23, @Boscoeuk, @boxabirds, @brandonmmusic-max, @bunnyfu, @CanadaApollo6, @CerebralCoding, @cesarswong, @chadhurley25075-png, @chaog992, @Charlie-Louis, @Chedrian07, @chris247474, @christrade215, @crescit, @cshintov, @cwschroeder, @DakotaTexas, @danyo1399, @Deesha08, @Defilan, @DevRico003, @di37, @diwakergupta, @drowzeys, @easthambuilds, @ecohash-co, @edurdias, @eleqtrizit, @ericlsimplifi, @EugeneClaw, @fche, @feni6, @florianbrede-ayet, @gbgbgbg, @GDACONSULT, @gecobattya, @gilby, @Gogo6969, @gprot42, @GraithSecurity, @grantoverton, @grearjake-star, @greatyingzi, @GustavBlack, @harrisonfriia, @haxudev, @heitke, @hichaiuse, @ivanfioravanti, @jasontitus, @javierbits, @jayleaton, @jeffpeng3, @jeidbugs404, @Jenpo, @jerryfane, @jetnet, @jishnuvenugopal, @jkuepker, @joeynyc, @johnymoo, @JordiPosthumus, @joshuaswarren, @JRaxworthy, @jrb-ai, @jschmied, @juliankang4, @justincasey, @kelchm, @kieranscotney, @kingjamez, @kky42, @laudney, @lcgutierrez, @LECYWZA, @lijian1999, @liumorrisclaw, @LXD-8, @m-naoki-m, @mapamalu, @mcclanahanaman, @MESevenJourney, @mgoldwasser, @MiaAI-Lab, @mikolaj92, @millaguie, @Mirrdhyn, @Moutonc, @MovieMaker93, @mrpmorris, @mrv777, @MV10, @NeoAiLabs, @Nipale-ai, @nmcbride, @nood-co1, @nullburn, @olexale, @omar16100, @optimisme, @otarkhan, @outcastofmusic, @paragontasx, @peacockesq, @philip-pentatonic, @plotarmordev, @pmeenan, @ppardus, @pulseandthread, @quigles1977, @rafafortes, @raymondkpwong, @robertpitt, @rootsbymenda, @RoscoeTT, @RS91Dev, @russellmm, @salmanarshad321, @samwang0041-star, @sanjaibalajee, @satindergrewal, @scottleimroth, @sethforprivacy, @sfxnz, @shantanugoel, @simon-lin88, @simonmd, @spenchey, @squarrier, @ss-cong, @Stephane-Eden, @styles01, @SvangenStudios, @swbiggart, @SxMShaDoW, @sxuff, @taussoe, @tfolkman, @ThinkOffApp, @ThomasHalwax, @Thotheris, @tinyapps, @tolewis, @tomByrer, @tonydehnke, @tournierjc, @tovereast, @tpischke, @urtho, @valacroix, @vcruz305, @vinicius-symetrix, @wojo, @xjqx2z, @yguanrong, @Yuepixel, @YvesLaRose, @ZackSample, @zweck.

docs/credits.md names whose Python engine work each part of the native engine carries.

About

LLM Inference Engine for Metal, CUDA and Vulkan.

Topics

Resources

Contributing

Security policy

Stars

2.3k stars

Watchers

12 watching

Forks

Releases

Sponsor this project

Packages

Contributors

Languages