AFM3 20B, 20% MLP Activation

AFM3 20B proposes an Instruction Following Pruning architecture that reduces the MLP layers activated depending on the input prompt.

• It activates only about 20% of all MLP layers, reducing the weights that need to be read and the memory bandwidth required at inference.
• It uses a MoE structure, but instead of switching experts per token or per layer, it was trained from the start to use the same combination of experts per prompt.
• The post explains that since FFN/MLP expert weights account for about two-thirds of active parameters, exploiting sparsity can improve read bandwidth efficiency relative to active parameters.
• As a comparison case, it mentions reducing a 9B dense model to a 3B active model.

Since dynamic pruning based on input may cause recovery latency when generating long outputs, further verification of actual serving performance and quality is important.