Command A+ Model Gets MLX Support

A cohere2_moe implementation for mlx-lm has been developed to run Cohere's Command A+ (218B MoE) model on Apple Silicon. A Pull Request has been submitted to ml-explore/mlx-lm and is currently under review.

Key Architectural Features:

• Shared Expert: Combines a single shared expert with the routed output using (routed + shared)/2.
• Routing: Uses Sigmoid routing instead of Softmax, normalizing the top 8 experts.
• Sliding Window: Applies a 3:1 ratio sliding window, using interleaved RoPE only on the sliding layers.
• Parallel Structure: Attention and MLP blocks operate in parallel from the same LayerNorm.

Performance and Technical Notes:

• The bias issue occurring during W4A4 quantization was confirmed to be an NVFP4 quantization artifact, which can be handled with the sanitize() function.
• Testing in a large-memory environment recorded a generation speed of 22.9 tok/s and a prompt processing speed of 57.6 tok/s.