Deep Insight Through Harness Engineering: A Design Journey from Local Development to Production Operations

Bringing Agentic AI to production requires not just reasoning logic, but harness engineering that covers long-running execution, tool calls, session isolation, security, and monitoring. Deep Insight has organized the design decisions behind its transition from local development to stable production operation on AWS.

The key background is that agents are far more non-deterministic than traditional software, sessions run longer, and multiple users must be isolated simultaneously. As a result, simple prompt optimization alone proved insufficient, and the challenge of designing the very environment in which agents run came to the forefront.

Deep Insight's first choice was Amazon Bedrock AgentCore Runtime. This execution environment isolates each user session in a microVM, supports long-running sessions of up to 8 hours each, and provides Active CPU-based billing and auto-scaling suited to I/O-wait-heavy workloads. Combined with VPC mode and AWS PrivateLink, it also allows restricting network paths, making it well-suited for analytics services handling sensitive data.

The most important design decision was the complete separation of code generation and code execution. In the initial self-hosted approach, Python code was executed directly via subprocess.run(), but as multiple requests arrived concurrently, agent reasoning and code execution competed for the same runtime resources, creating bottlenecks. As a result, a structure became necessary where AgentCore Runtime handles only code generation, while execution is offloaded to a separate compute layer.

AWS Fargate was chosen as the execution layer. The reasons are that it allows execution without time limits and without the burden of server management, and system packages and Python dependencies can be pinned at build-time via a Dockerfile for deployment. In particular, fonts-nanum for Korean chart labels, document conversion tools, and even matplotlib font caches were pre-baked into the image, reducing the initialization overhead for each session.

Code execution is further split into two stages.

• First, the code generated by the LLM is Base64-encoded to avoid HTTP and shell escaping issues, and only the file is saved inside the container.
• Then, the saved file is executed via subprocess, with timeout used as needed to control long-running executions.

This structure absorbs encoding issues during the file-writing process and infinite loops/errors during the execution stage at separate boundaries. The agent still calls a single write_and_execute_tool(), but internally, the harness is separated to safely handle uncertain LLM-generated code.

An ALB sits between AgentCore Runtime and Fargate, distributing requests across multiple Fargate Tasks and using Health Check to ensure only ready containers receive traffic. Sticky session is also used so that subsequent requests within the same analysis session go to the same Task, maintaining continuity of intermediate outputs and state. A direction for aggregating and storing results in S3 after a session ends is also proposed.