ERNIE-5.5-Flash-Base: 280B MoE with only 14B active parameters per token, supporting 1M context

ERNIE-5.5-Flash-Base, released by Baidu, is a sparse MoE language model with a total of 280B parameters. Only 14B parameters are actually activated per token, maximizing inference efficiency. It is a checkpoint pre-trained on 33.5T tokens, representing a pure base model that has not undergone mid-training or post-training.

1 / 3

This model combines hybrid MLA and HCA attention to significantly reduce KV cache costs. Out of 43 layers, 37 use HCA, which caches only one entry per 128 tokens, while the remaining 6 apply MLA, which allows full context access. This enables long-context processing of up to 1M tokens during fine-tuning and inference, exceeding the 128K training context.

Each MoE layer contains 512 routing experts and 1 shared expert, with a multi-router design to improve the accuracy of expert selection. Through the ECHO mechanism, the final layer of the backbone also serves as a draft model for speculative decoding. It is provided in FP8 precision and is compatible with major inference frameworks such as Transformers, vLLM, and SGLang.

It supports English and Chinese and is suitable as a starting point for domain adaptation, instruction tuning, or reinforcement learning research. With efficiency as the top design priority, every component from attention to output heads has been redesigned. It was benchmarked on knowledge, reasoning, math, code, multilingual, and long-context tasks using the same evaluation pipeline as comparable models like DeepSeek-V4-Flash-Base.