Achieving 44% on ARC-AGI-1 for 67 Cents (22-minute read)

A small Transformer trained from scratch in 1.5 hours on a 5090 GPU scored 44% on the ARC-AGI 1 benchmark at an extremely low cost of 67 cents. This model demonstrates performance similar to TRM and HRM while being open source, proving meta-learning capabilities through pure deep learning without relying on the massive pre-training or synthetic data that existing LLMs depend on.

Sample Efficiency and LLM Criticism

The author argues that the high scores current LLMs achieve on ARC are not improvements in general reasoning ability, but rather the result of post-training and benchmaxxing through large amounts of synthetic data. LLMs exhibit low sample efficiency even on tasks that are easy for humans, and the fact that all LLMs' performance reset upon the emergence of ARC-2 suggests they memorized specific task solutions rather than possessing 'general reasoning.' In contrast, this model demonstrates meta-learning capabilities, quickly learning new rules from limited data.

Technical Approach

The core of the performance improvement is an architecture combining 3D RoPE positional encoding with per-task embedding. It also applies modernized Transformer structures such as the NorMuon optimizer, RMSNorm, and SwiGLU FFN. The model adopts a meta-learning approach that uses test inputs themselves but not ground truth labels during training, which aligns with the spirit of Test-Time Training (TTT) as permitted by the ARC community.

Issues with Benchmark Evaluation Criteria

The author points out that it is unfair for existing leaderboards to compare models by excluding the massive offline pre-training costs of LLMs. They emphasize that lifetime compute (total cost from initialization to inference) for all models should be included, and the use of synthetic data should be prohibited as it undermines the benchmark's original intent of testing 'fluid intelligence.' Additionally, the author criticized the discrepancy between the claimed model sizes and the actual number of trained parameters for HRM and TRM.