Needle Unveils 26M-Parameter Tool-Calling Model

Needle has open-sourced a function-calling model with 26 million parameters.

• The goal is tool calling for agents running on low-power devices such as smartphones, watches, and glasses.
• The architecture is Simple Attention Networks, using only attention and gating, with no MLP.
• The authors view tool calling as a problem closer to retrieval-and-assembly than reasoning, and argue that cross-attention is the key element in this domain.

Training and performance are as follows.

• Pretraining on 200B tokens was done in 27 hours on 16 TPU v6e.
• Additional training was done for 45 minutes on 2B tokens of synthetic function-calling data.
• The synthetic data was generated with Gemini and covers 15 tool categories (timers, messaging, navigation, smart home, etc.).
• They claim 6,000 tok/s prefill and 1,200 tok/s decode on consumer devices.
• They state that it outperforms FunctionGemma-270M, Qwen-0.6B, Granite-350M, and LFM2.5-350M on single-shot function calling.

The code and architecture documentation have been released on GitHub, and they state that testing and fine-tuning are possible on Mac/PC.