SK Planet Optimizes SQL Assistant Performance with AWS Bedrock Prompt Caching

In the Chat DIC project — an in-house Text-based SQL Assistant and Table meta search system developed by SK Planet's Data Infrastructure team — the team leveraged AWS Bedrock's Prompt Caching feature to optimize performance and cost. This feature stores repeatedly used prompt context such as system and tools in a cache, reducing model recomputation and cutting response latency and token costs.

In the initial AWS Lambda-based architecture, Gateway Timeout limits (maximum 29 seconds), lack of SSE support, and Throttling issues created limitations in handling streaming responses and large-scale requests. To resolve this, the team switched to an ALB and EC2-based architecture and applied Prompt Caching by designating cache points for the unchanging system and tools prompts.

When caching is applied, caching is possible starting from a minimum of 1,024 tokens for supported models such as Claude 3.7 Sonnet, with a TTL of 5 minutes. Since the messages field changes with every conversation, it was excluded from caching, and only the fixed system and tools were cached to maximize efficiency. This helped mitigate the Throttling issue and improve response speed.