tunelab: Cutting LLM Costs with Small Models

For repetitive, simple LLM tasks such as Classification, Routing, and Extraction, it is more efficient to use fine-tuned small models instead of expensive Frontier Models.

tunelab provides a workflow that migrates these tasks to local small models and verifies whether the resulting model outperforms the existing API. Its key features are as follows.

• Hierarchical approach: It automatically explores the optimal stage based on cost and accuracy, ranging from embedding similarity (Centroids) to small classifiers, LoRA fine-tuning, and Continued Pretraining.
• Rigorous performance verification: It uses a Champion/Challenger mechanism that sets an accuracy bar in advance and deploys only models verified against held-out data into the actual system.
• Optimized for local training: On Apple Silicon (M1+), MLX/LoRA enables fast local fine-tuning without GPU rental or API costs.
• Cost and performance case study: In tests on the Banking77 dataset, applying a 3-stage cascade structure improved accuracy to 94%, with about 88% of traffic handled locally, cutting costs by 8x.