Is AI Civilization Emergence Just Recitation? CIVOS Experiment Results Released

A problem has been raised regarding the difficulty of distinguishing whether the 'emergence' of society and civilization observed in multi-agent simulations is a genuine exploration process or 'recitation' of human history learned by LLMs. To address this, a control experiment framework CIVOS that manipulates only the availability of prior knowledge has been released.

Experimental Design and Results

CIVOS fixed the task structure and verified performance by dividing the state of prior knowledge used by agents into four conditions.

• Condition A (Full): Full availability of prior knowledge
• Condition B (Removed): Prior knowledge blocked
• Condition C (Corrupted): Prior knowledge contaminated
• Condition D (Baseline): Random simulation without LLMs

Experimental results showed that the discovery process significantly declined in Condition B, where prior knowledge was removed. This suggests that the observed civilization development relies heavily on the model's learned knowledge (recitation). Additionally, it was frequently observed that Condition C, which used corrupted knowledge, performed worse than the removal condition, confirming that incorrect prior knowledge can hinder exploration.

Statistical Rigor and Lessons

The research team disclosed cases where insufficient statistical power in initial experiments led to misinterpreting significant results as 'negative,' emphasizing the importance of multiple comparison corrections and a sufficient number of seeds (40 or more). After applying sign-flip permutation tests and Bonferroni correction, the performance degradation effect due to knowledge removal was found to be statistically robust.

This study warns that when evaluating the 'intelligence' of multi-agent systems, it is impossible to distinguish between actual problem-solving ability and simple recitation without controlling for the model's pre-training bias.