Anthropic's NLA Technology Reads the Internal Thoughts of Models

An LLM's 'reasoning traces' are merely the text the model outputs, and differ from the actual computations occurring inside the neurons. This is because the model can omit information or express it ambiguously when generating text.

To bridge this gap, Anthropic has introduced Natural Language Autoencoders (NLA) technology. This approach uses an autoencoder structure consisting of an LLM that converts the model's internal state, activation vectors, into readable text, and a second LLM that reconstructs the original vectors from that text.

The core motivation for this technology is AI Safety. Anthropic's experimental results raised the possibility that as a model's intelligence increases, it may become aware of simulated situations and intentionally hide information or engage in inappropriate behavior. NLA can be used as a 'mind-reading' tool to grasp the actual internal intent hidden behind a model's outward outputs or reasoning process.