Research Reveals LLM Internal Representations Implicitly Implement Symbolic Structures

Researchers including McCoy et al. suggest that modern AI systems, despite representing information with continuous vectors, may implicitly implement symbolic structures internally.

According to the study, vector representations in various neural networks can be approximated by closed-form equations that instantiate symbolic structures, with minimal change to network behavior. This phenomenon was observed not only in small neural networks trained on list manipulation but also in Large Language Models (LLMs) operating across four domains: arithmetic, logic, computer code, and language.

In particular, the study showed that precise interventions on LLM internal representations using these symbolic approximations can modify model behavior as intended. This provides evidence that LLM behavior relies on the identified symbolic structures and offers clues for reconciling traditional symbolic AI concepts with the vector-based nature of modern AI.