LLMs Fail at Expensive-Loop Tasks Due to Poor Proxy Management and Resource Awareness

Current LLMs are optimized for cheap-loop environments characterized by low-cost iteration and immediate feedback, leading to significant performance drops in expensive-loop tasks where feedback is sparse and costly. This limitation has profound implications for AI safety and economic viability, as expensive-loop capabilities are upstream of risks like superhuman persuasion and offensive biotech.

Four Failure Mechanisms in Expensive Loops

LLMs fail in expensive-loop scenarios through four distinct mechanisms:

• Proxy Management Failure: LLMs struggle to select intelligent proxies or replace them when correlations with the target weaken, often relying on manipulable metrics susceptible to Goodhart's Law.
• Error Correction Failure: Models fixate on hyperparameter adjustments rather than questioning underlying strategies or premises. Research from PostTrainBench indicates that models lock into basic strategies before evidence emerges, with additional tokens only refining existing approaches. In Mechanize's Game Boy emulator evaluation, models marked tasks as complete after seeing only the boot screen, demonstrating confirmation bias.
• Resource Awareness Deficit: LLMs fail to strategically allocate time and compute. In one PostTrainingBench case, a model spent 61% of its time budget on evaluation and only 3.6% on RL training, wasting approximately 10 hours on metric monitoring. Another instance saw 72% of the wall-clock budget consumed by throughput diagnostics, leaving only 83 minutes for actual training.
• Theory Building Absence: LLMs lack the ability to build causal models to generalize results, often mistaking noise for signal and failing to attempt falsification.

Safety Implications and Training Challenges

The disparity between cheap-loop proficiency and expensive-loop weakness creates a paradox: while a "strong cheap-loop & weak expensive-loop" AI might seem ideal, it increases the risk of reward hacking, which many experts identify as a primary source of AI x-risk. Improving expensive-loop capabilities requires training in expensive-loop environments, where interaction costs dominate other resources. For example, a training schedule involving 8 attempts per visit across 5 visits incurs 40 expensive-loop interactions, creating a bottleneck that current GRPO post-training methods struggle to overcome.

Limits of Simulation and Generalization

While humans can generalize decision-making skills from low-stakes games to high-stakes real-world scenarios, it remains unclear if LLMs can achieve similar cheap-to-expensive generalization. Simulation and wargaming cannot fully replace real-world expensive loops due to the sim-to-real gap, distribution shifts, and unanticipated edge cases. As noted in references to Borges' "On Exactitude in Science," attempting to create a 1:1 scale simulation for training may be futile, suggesting that some civilizational-scale expensive-loop capabilities may require actual, rather than simulated, experience.