Product Engineer: A Data Agent Handling 500 Analysis Requests a Day — From AI That Helps With Work to AI That Does the Work

MyRealTrip's Data Agent received over 2,000 analysis requests within less than a week of going live on Slack, and on some days handled 500-600 requests a day. Previously, a single analyst could only handle about 4-7 requests a day, but now any team member can tag the agent on Slack and instantly get results with conditions attached, such as yesterday's accommodation transaction volume, excluding cancellations, or compared to the previous week.

The key was making the agent not just a simple tool but an AI colleague that can be entrusted with control. To do this, Harness handles workflows, permissions, and safety nets, while Knowledge fixes the definitions of transaction volume and table meanings. For example, metrics like GMV_total are defined in YAML specifying a SUM(<payment_amount>) basis, and whether cancellations are reflected branches depending on the request context.

Slack messages are received by the Listener, then the Dispatcher separates chit-chat, duplicates, and analysis requests, after which the Parent Worker organizes the context and hands it off to sub-agents. Workers are divided into simple analysis, complex analysis, search, reports, visualization, and Google Sheets, and are run by spinning up multiple Claude Code Max accounts in a Docker cluster. The classification model was switched from Haiku to Sonnet, achieving stable classification at 1/8 to 1/10 the tokens compared to Opus. The numbered emoji attached to responses shows which worker handled that request.

In a headless architecture, rules have to be re-inserted every time a new session is spun up, so the Parent Worker was switched to a Lazy Load approach, holding only addresses instead of full content. As a result, it reduced the waste of 230K tokens that used to disappear per session, and by using Claude Setup Token and headless mode, costs were also lower than operating with API keys. The knowledge system operates in the order Metric Registry -> DBT metadata -> direct exploration, and Auto Memory is kept off, using only explicit Knowledge.

The safety net has 4 layers: Role / Gate / Rule / Hook.

• Role: Permissions are divided by worker, so the report worker cannot directly touch BigQuery, and the analysis worker cannot write to Confluence.
• Gate: If the request scope is empty or ambiguous, no worker is spun up and a clarifying question is asked instead.
• Rule: Natural language policies are written into the Metric Registry and worker-specific task specifications.
• Hook: Right before SQL execution, read-only status, scan limits, and PII masking are given a final check.

Before deployment, over 100 shell tests and evals verify accuracy, behavioral paths, stability, and format. When a failure case appears, the cause is analyzed to fix the Metric Registry or rules and add a new eval, and if a worker hits the 5-hour/weekly limit, a standby backup worker immediately takes over. The message that runs through to the end is that repeating workflows should be designed as a Closure-loop, where humans can be left out, rather than a Human-in-the-loop, where humans intervene every time.