Microsoft Publishes NextLat Research

Microsoft Research proposed the Next-Latent Prediction (NextLat) methodology to overcome the limitations of the existing Next-token prediction approach. This method trains a transformer to predict its own next latent state, going beyond simply predicting the next token.

The key benefits of NextLat are as follows:

• Representation Learning: It encourages the transformer to compress past history into belief states, strengthening reasoning and planning ability.
• Data Efficiency: Predicting in latent space provides a denser supervision signal than predicting one-hot tokens.
• Faster Inference: It supports self-speculative decoding via recursive multi-step lookahead, enabling up to 3.3x faster inference.