AI BriefingTensorFold: Serve LLMs with a Zig Binary, No Python Required
ashhart/TensorFold
About the project
TensorFold is an inference engine that serves language models on Apple Silicon and NVIDIA GPUs. A single binary written in Zig reads checkpoints and directly executes Metal or CUDA kernels. It allows you to run models locally without installing Python or MLX.
It provides OpenAI and Anthropic-compatible APIs, making it easy to integrate with existing clients. It supports 4-bit checkpoints for various models, including Qwen3.5-2B and Nemotron 3.5 Lightning, and works on both macOS and Linux environments. It can manage multiple streams simultaneously through parallel request processing.
The experimental Living Weights feature allows you to teach new facts to the weights of a running model. Additionally, it offers an enterprise environment through TensorFold Starlight for managing model access permissions and token usage at the organization level.
It is suitable for developers who want to set up a local LLM inference environment while avoiding complex dependency management or Python runtime installation. It is released under the Apache-2.0 license, and model execution and chat can be visually managed through a desktop app and web interface.
ashhart/TensorFold
LLM Inference Engine for Metal, CUDA and Vulkan.
Zig
This introduction was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report errors, attribution issues, or removal requests via Contact.