DeepSeek V4-Flash Internal Analysis

DeepSeek V4-Flash's core change, mHC, is a structure that replaces standard residual connections. Using an FP8 conversion of this model on 8x H100 SXM (8x80GB), we measured hidden state, expert routing, and SVD structure. The converted model was about 310GB, and was configured with tensor parallel.

The model configuration was 284B total parameters, 13B active, 43 layers, 256 experts, 6 experts per token, and hc_mult=4. The experiment was conducted with a total of 7 prompts (5 short, 2 long: Roman Empire wiki 331 tokens, attention transformer code 641 tokens), and due to the Tilelang fused kernel, sparse attention scores could not be directly observed.

The key observations are as follows.

• BOS token magnitude grew 1,800x, from 28 at layer 0 to 69,632 at layer 42.
• Non-BOS tokens increased by about 70x, which was relatively within a normal range, and the explosion was concentrated only on BOS.
• mHC's hyper-connection replication appeared to become effectively redundant after 3 layers.

Ultimately, V4-Flash showed an extreme attention sink and a deterministic dimensional structure, and mHC appeared to function less as a means of expanding expressiveness and more as a device for channeling magnitude into specific dimensions of BOS. The remaining tokens behaved like a typical transformer.