Building a Production-Grade Agent Loop

In late 2025, Liquid AI conducted an experiment to verify whether coding agents could autonomously solve production-grade problems. The output of this experiment is the open-source BPE tokenizer trainer toktoktok.

Existing libraries were limited by memory shortages or missing features, necessitating a new trainer capable of processing trillions of tokens on a single machine. The subjects of the experiment were Claude Opus 4.5 and Codex based on GPT-5.2, which were the strongest coding models at the time.

Three criteria were set for designing a successful agent loop.

• Production-grade: Aims for actual deployment rather than a prototype, and must involve new problems not present in pre-training data.
• Multi-domain expertise: Requires a complex tech stack that is difficult to solve with single-domain expertise alone.
• External verifiability: Results must be verifiable using third-party software (e.g., tiktoken) that the agent cannot modify.

Through real-world problems meeting these conditions, it was confirmed whether agents could complete tasks to a reliable level without human supervision.