Fruit Fly Brain Simulation Learns Super Mario Bros. 3 in Real Time

Based on the fruit fly brain dataset from the FlyWire project, a process has been revealed where 138,639 neurons and over 15 million connections are simulated in real time to learn the first level of 'Super Mario Bros. 3'.

Simulation Architecture and Learning Method

• Brain Structure Replication: Based on research data by Shiu et al. published in Nature, AlphaLIF dynamics (voltage, spikes, alpha synapses, delays, refractory period) were applied to faithfully replicate the actual brain's connectivity structure.
• Visual Input Processing: The entire screen is converted into a 4-frame grayscale image of size 15×13 and fed into 10,855 visual input neurons.
• Learning Algorithm: Combining PPO (Proximal Policy Optimization) and self-imitation, the agent learns only from trajectories it discovers. Specific neural circuits such as LC4/LPLC2 (proximity detection) and Giant Fiber (escape response) are utilized as priors but not as cheating mechanisms.

Real-Time Observability

Users can directly observe the real-time learning process, rather than just a replay. The NES gameplay screen, the brain's activation state, the visual inputs triggering decisions, and the actual fruit fly's front legs pressing the NES controller are all streamed in real time.