Measuring Maximum Activation Values in Open LLMs

We re-measured the global and layerwise maximum activation of 27 open LLM checkpoints using the same pipeline.

• Measurement scope: Used 5,000 multi-domain samples and family-specific tokenizers, with identical hooks placed on embedding, hidden state, attention, MLP/MoE, SwiGLU gate, and final norm.
• Targets: Compared 27 dense, MoE, vision-language, intermediate-training, and instruction-tuned variants across 8 open families.

Global maximum values showed nearly four orders of magnitude difference even at the same parameter scale. Qwen3.5 and some MoE checkpoints were in the 10^2~10^3 range, while Gemma3-27B-it spiked up to approximately 7 x 10^5.

• Scaling: No simple monotonic relationship held across cross-family or cross-generation comparisons.
• MoE: Peaks were 14.0~23.4x lower than dense models of the same scale.
• Location: In 22 out of 24 cases, the global maximum was observed in the residual stream.

In a lightweight INT-8 sanity check, the measured maximum values moved together with low-bit reconstruction error depending on the activation-scale choice. In conclusion, maximum activation values are not a simple byproduct of scale but a property tied to model family, architecture, and training stage, and need to be reported alongside low-bit deployment.

The code has been released on GitHub.