tool-prune Released, Tool Schema Selection Speed at 0.4ms

tool-prune has been released to address the issue where including more than 50 tool schemas in the context of local 7B/8B models causes attention degradation and hallucinations. This tool performs pre-filtering on the client side before model invocation to remove unnecessary schemas.

Key Performance and Features

• Ultra-fast Processing: Utilizes the FWHT (Fast Walsh-Hadamard Transform) algorithm to complete filtering in 0.4ms offline. Applying SIMD reduces this to 23µs.
• Token Reduction: Reduces prompt tokens by 92%, significantly lowering the burden on the model's context window.
• No Additional Cost: Unlike existing in-band retrieval methods, it requires no additional LLM generation turns or round trips.
• Lightweight: Implemented in pure JS and Python with no external dependencies.

Usage

Users call router.filter(userPrompt) to obtain a top tool list before invoking the model. For deterministic queries, an option is provided to dispatch directly in under 1ms. It can be tested directly via the GitHub repository and online playground.