TOrder's Text2SQL Agent 'TSpoon' Case Study

Built on data accumulated from over 300,000 tablets and more than 25 million monthly users, TOrder created the Text2SQL agent 'TSpoon' so that all company members can directly query order, store, and advertising data.

The core goal was to convert natural language questions into SQL, but instead of executing immediately, ensure safety through Human-in-the-Loop dual approval. Users mention the bot with @ in Slack to ask questions, and the agent first understands the intent of the question, then shows the plan for first-stage approval. It then shows the generated SQL again and confirms whether to execute at second-stage approval, with buttons controlled via Block Kit to also prevent duplicate clicks.

The architecture was built as a workflow based on LangGraph StateGraph. Reasoning and embedding are handled by Claude models on Amazon Bedrock, SQL validation and execution use Amazon Athena, vector search uses Amazon S3 Vectors, and result storage and large-scale exports use Amazon S3. S3 Vectors achieved sufficient precision with metadata filter search alone, allowing operation at a cost of under $5 per month.

The search design was split into a 4-Stage pipeline instead of simple similarity search.

• Stage 1: Find schemas similar to the question to determine relevant tables
• Stage 2: Within the scope of the determined tables, search for query patterns, JSON fields, and cross-table patterns
• Stage 3: Backfill missing table schemas and query patterns referenced by patterns
• Stage 4: Supplement domain definitions and enum values using a business term dictionary

This approach focused on "accurately filling missing dependencies" rather than "fetching more." In particular, in cross-table queries, if one table's schema is missing, the LLM cannot utilize that pattern, so reference relationships are tracked to supplement the context.

In the SQL pipeline, Athena's EXPLAIN is used for syntax validation, and if it fails, the error is analyzed, additional RAG search is performed, and a regeneration loop runs up to 3 times. If the same error repeats, it is immediately escalated, and when a new turn begins, state is reset according to hierarchical lifetimes (RETRY, TURN, QUERY, SESSION) so that past failures do not affect the next query.

The trial and error was also concrete. For intent classification, adding few-shot examples raised consistency from 70% to 90%, and date inference errors were resolved by having the system fixedly inject current_date based on KST. Also, in calculations like dwell time, even with cross-table few-shot examples, the model couldn't use them if the schema was missing, so Stage 3 backfill supplemented the missing table schemas and patterns.

Ultimately, the lesson from this project is that it is not the prompt itself but context supply and execution control that determine Text2SQL quality. Rather than expecting the model to handle things well on its own, structuring the necessary knowledge to be searchable and allowing humans to verify what was seen and judged is what makes it possible to put into production.