Toss Reduces LLM Costs with Selective Neural Approximation (SNAP)

Details

Selective Neural Approximation (SNAP), developed by Toss, processes simple requests with a small neural approximation model and falls back to the existing LLM for difficult requests.
In the first application target, text-based identical product identification, the system achieved a Coverage of 34.74% by adopting 34,963 out of 100,648 validation requests under a 99.9% target match rate condition.
Operational results over approximately 6 days showed that the neural approximation model handled 33–37% of daily requests, saving approximately 7.6 billion input tokens and approximately 45.2 million output tokens.
In serving performance tests, the system recorded a p50 response time of 9.02ms and p99 of 15.23ms at a throughput of 1,203 requests per second, with no failed or dropped requests.
The system classifies tasks using the hash of the beginning of the System Prompt and automates the process from candidate discovery to model training via Airflow daily batches, securing a model within a day.
Instead of switching all requests at once, partial application starting from validated segments reduces development burden and allows cost savings to begin quickly.

Key point