needle: 45M LLM Calling Tools with 28MB RAM

Needle 2 is a 45M parameter small language model developed by Cactus Compute. The entire model is distributed as a single 14MB binary and uses only 28MB of RAM during inference. It employs CQ2-bit quantization technology and a custom engine to operate on-device without network connectivity.

Core features include tool calling, device control, and structured data extraction. By registering Python functions as tools, the model selects the appropriate function, fills in the arguments, and executes it, returning results in JSON format. Specifying a Pydantic model allows the extraction of only the necessary fields from text, returning type-safe objects.

Differentiation from existing small models lies in memory efficiency and reliability control. By applying a 256-token sliding window and KV sink techniques, memory usage remains at the 28MB level regardless of conversation length. Additionally, a learned head assigns a confidence score to each response, designed to escalate if the score falls below a configured threshold.

LoRA fine-tuning is supported to optimize the model for specific tools or domains. Training is possible on both NVIDIA GPUs and Apple Silicon via JAX-based learning, and tuned models are exported as a single .cact file for immediate execution on the same engine. It is suitable for developers aiming to integrate LLM capabilities into resource-constrained edge devices such as smartphones, wearables, smart home devices, and robots.