Fractale-350M-base model released

Fractale-350M-base is a 386M parameter base model, pretrained from scratch using 10B tokens (code and English web data). The core of this model is that instead of using a long context window, it performs a memory function by leveraging 8 vector banks as the model's fast weights.

The key features are as follows:

• Memory mechanism: One gist vector is generated for each 512-token chunk and managed in a FIFO manner; these vectors are expanded into a low-rank MLP via a hypernetwork and integrated as part of the forward pass.
• Performance metrics: Experimental results showed that the cross-entropy (CE) difference (GAP) between using and not using the memory bank was +9.4 nats on code data and +7.3 nats on web data, demonstrating the effectiveness of the memory.
• Notable point: Despite the context window being intentionally limited to 512 tokens, the model was able to predict the next token by referencing past information through the memory bank.