Anthropic Unveils NLA, a Technique to Turn Claude's Thoughts into Text

Anthropic has introduced Natural Language Autoencoders (NLAs), a technique that converts activations—the complex numerical data inside a model—into human-readable natural language text.

While existing interpretability tools required researchers to interpret complex outputs directly, NLA explains the model's thought process in direct sentences. For example, it can show in text how Claude plans specific words in advance in order to make a rhyme.

NLA works through the following three components:

• Target Model: The model from which activations are extracted.
• Activation Verbalizer (AV): Converts activations into text descriptions.
• Activation Reconstructor (AR): Restores the text descriptions back into the original activations.

The system is trained based on how accurately the original activations can be reconstructed from the text descriptions, using a 'round trip' method that ensures the accuracy of the explanations.

This technique is used to enhance the Safety and reliability of models. It can be used to check whether a model is aware that it is being tested, detect cheating on training tasks, and identify training data that induces incorrect responses. Anthropic has released the related code on GitHub and also provides an interactive exploration tool through Neuronpedia.