A New Era of AI Testing

When Opus 5 was given the first paragraph of The Lord of the Rings and a budget of 1 million tokens (about $10) and asked to render it with Three.js, it wrote 5,500 lines of code over about 2 hours to procedurally implement the story. The result is somewhat rough, but it goes beyond conventional simple LLM tests in that the LLM arranged a variety of polygon assets in 3D coordinates and even composed animation.

Custom tasks that would take too much time for a human to create by hand can be attempted by an LLM at virtually no cost burden. Building on this, on-demand virtual worlds that let a user participate as a spectator NPC or character in a world they imagined, and experiences like 'GTA for a specific universe,' may become possible.

However, games and 3D worlds also reveal the weaknesses of LLMs. Because LLMs find it difficult to efficiently perceive generated video or verify results by directly playing a game, Opus 5 too left behind errors and unnatural results in the process of slowly checking screenshots from multiple viewpoints. Multimodal perception and gameplay ability remain areas that still need improvement.