KB Securities Cuts Tokens by 35.9% and Costs by 15.8% Using Amazon Bedrock-Based Context Tier-Routing

KB Securities introduced Amazon Bedrock-based Context Tier-Routing to its internal agentic AI, 'Kkebi AI', reducing total token usage by 35.9%.
By transitioning to a method that selects models based on task difficulty at the time of LLM invocation, it maintained an average quality score of 85.2 while shortening overall E2E response time by 25.0%.
An Elevate-only policy, which prohibits downgrading after promotion to a higher Tier, was applied to prevent Prompt Cache invalidation and ensure cost efficiency.
Using an XGBoost classifier, a routing accuracy of 85.83% was achieved, with the training and deployment process automated via the Amazon SageMaker AI pipeline.

Technical Implementation and Policy

An XGBoost binary classifier was trained using a 2,052-dimensional vector, combining intfloat/e5-large embeddings with structural features, as input.
The STANDARD Tier was configured to use the self-hosted GPU serving gemma4-31B-it, while the ADVANCED Tier used Claude Sonnet 4.6.
A Fail-open policy, which retains the previous model in case of router failure, was applied to ensure service stability.

The Medium difficulty range recorded the highest efficiency, with a 29% reduction in response time and a 42.1% token savings rate.
In the Hard range, 41 cases where the router misclassified tasks as STANDARD saw an average score drop of 4.4 points, revealing the limitations of lightweight models.
Due to Cache Write costs incurred during model switching, the actual cloud API cost reduction rate (15.8%) was lower than the token savings rate (35.9%).