PNAS Warns of the Risks of 'Evolvable AI (EAI)'

According to a paper published in PNAS (Proceedings of the National Academy of Sciences) in 2025, AI risk does not necessarily have to be tied to the emergence of superintelligence (AGI). The key point is the moment AI becomes 'Evolvable.'

Evolvable AI (EAI) refers to systems that, regardless of intelligence or consciousness, possess the following four characteristics:

• Self-replication and mutation: Generating copies or variants of itself
• Trait transmission: Passing useful traits on to the next generation
• Change over time: Modifying itself to fit its environment
• Selection pressure: A structure in which the strongest version survives

The paper explains this by mapping the biological evolutionary process onto the digital environment:

• DNA $ightarrow$ prompts, model weights, fine-tuning, adapters
• Natural selection $ightarrow$ the internet, computing resources, APIs, capital, data accessibility

In conclusion, the paper warns that when AI agents operate on cloud servers and leverage existing digital infrastructure, they can, regardless of the level of intelligence involved, give rise to uncontrollable risks through the evolutionary mechanisms of replication, mutation, and selection.