Small Transformer Achieves 44% on ARC-AGI-1 for 67 Cents

By leveraging a small transformer, 44% performance was achieved on the ARC-AGI benchmark, with a training cost of only 67 cents. The model was trained from scratch on an RTX 5090 GPU in 1.5 hours, delivering performance similar to existing Large Language Models (LLMs) or TRM/HRM but at a much lower cost and faster speed.

Technical Approach and Improvements

• Architecture Optimization: Efficiency was improved by applying SwiGlu instead of GELU, RMSNorm instead of LayerNorm, and using Flash Attention and Flex Attention kernels.
• Training Method Change: Performance improved from 40% to 44% by switching to a Supervised learning approach that includes only output tokens, excluding input tokens from training.
• Data Processing: Expressiveness was enhanced using 3D RoPE embeddings and task-specific embeddings, while convergence issues were resolved by applying the Normuon optimizer instead of AdamW.
• Preventing Data Leakage: Non-overlapping tasks from ARC-2 were added, but 773 puzzles overlapping with ARC-1 were filtered out to prevent data leakage.

Implications

This research emphasizes that Sample Efficiency is currently the most critical issue in AI, demonstrating that high performance can be achieved even with limited data and low-cost environments. By showing that actual performance improved despite worsening Validation Loss (Val Loss), it highlights the limitations of existing optimization methods and suggests new directions.