Literature Review on Running GUI Agents on Smartphones: AndroidWorld

Existing Android agent benchmarks have been criticized for repeatedly using fixed tasks and screenshots, effectively becoming "memorization tests." To address this, AndroidWorld introduces parameterized task templates, generating millions of unique task variations with variables changing on each execution.

Benchmark Structure and Evaluation Method

AndroidWorld performs 116 tasks across 20 apps in a real Android emulator environment. It ensures full reproducibility by incorporating initialize(), is_successful(), and tear_down() functions that directly inspect OS state via ADB (Android Debug Bridge) without human evaluators.

Performance Results and Implications

The performance of the new agent M3A, based on Gemini 1.5 Pro, GPT-4 Turbo, Gemma 2 27B, and others, is as follows:

• AndroidWorld: M3A 30.6%, SeeAct 15.5%, Human 80.0%
• MobileMiniWoB++: M3A approx. 68%, Human 100%
• Latency: M3A takes an average of 3.9 minutes per task, three times slower than humans

Notably, it was observed that certain tasks showed a 0% success rate when using a fixed seed but were successfully resolved when the seed was changed, suggesting that existing static benchmarks may have measured parameter combinations based on luck.