Deep Insight Chatbot Through Inner Loop Engineering – 4 Design Decisions for a Conversational Analytics Chatbot

To go beyond a simple chart-generating chatbot and make it a professional tool that analysts use every day, four challenges must be addressed: data protection, speed, reliability, and cost.

The analysis process is divided into the Outer Loop, which produces automated deliverables, and the Inner Loop, where analysts directly explore data. The core of this design is building a separate architecture for the Inner Loop to solve the problems of response latency and cost.

The key design decisions are as follows:

• Data protection: The LLM only generates SQL, while the actual execution is handled by a sandboxed DuckDB, preventing data leakage and hallucination.
• Speed: An OLAP engine (DuckDB) is kept resident per session, reducing query response times to the hundreds-of-ms range.
• Reliability: The generated SQL is exposed to the user, ensuring transparency so analysts can directly verify and modify it.
• Cost: Prompt cache hit rates are maintained above 90% to control LLM costs incurred during multi-turn conversations.

This system improves stability by isolating the Outer Loop and Inner Loop into separate ECS service instances so they do not encroach on each other's resources.