EngramEdit Enables Decoupled Knowledge Updates in LLMs via Conditional Memory

EngramEdit introduces a method for decoupling factual knowledge updates from general-purpose computation in Large Language Models (LLMs) by leveraging conditional memory architectures like DeepSeek Engram. The approach addresses the challenge where different expressions of a fact activate different n-gram embeddings, and where updating shared embeddings risks altering predictions for unrelated facts.

Methodology

The technique operates in two main stages:

• Target Representation: It computes target memory representations that ensure the model predicts the updated fact across multiple linguistic expressions.
• Joint Embedding Update: It jointly updates shared n-gram embeddings to match these targets. To preserve unrelated knowledge, the method penalizes updates to frequently reused embeddings more strongly than those used less often.

Performance and Impact

Experiments demonstrate that EngramEdit enables independent factual knowledge updates with near-perfect editing success. Key findings include:

• Multi-hop Reasoning: Revised knowledge remains usable across unseen expressions and in multi-hop reasoning tasks, achieving nearly 3x the accuracy of the strongest baseline under chain-of-thought (CoT) prompting.
• Capability Preservation: Unrelated knowledge and general model capabilities are largely preserved, even as multiple factual updates accumulate.
• Architectural Shift: The findings establish conditional memory as an editable knowledge interface, extending its utility beyond simple model scaling to support precise, decoupled knowledge maintenance.