Meta Releases Muse Spark Open: 816B-Parameter Open-Weight Multimodal MoE Model

Meta has released Muse Spark Open, an open-weight frontier multimodal mixture-of-experts (MoE) model. It is described as the most powerful US open-weight model, serving as the MoE expansion of the smaller dense model Muse Glimmer.

Architecture and Specifications

The model has approximately 816B total parameters with 42B active parameters per token. Key architectural details include:

• Context Window: Up to 1M tokens (max_position_embeddings 1,048,576).
• Inputs: Text, image, audio, and video.
• Outputs: Text.
• MoE Structure: Each MoE block contains 256 experts, with the router selecting 8 experts per token. Experts account for ~97% of total parameters.
• Attention: Grouped-query attention (128 query heads, 16 key-value heads, head dim 64) with QK-norm and learned residual gates.
• Encoders: A 1.8B parameter Perception Encoder for vision and a convolutional pyramid Audio Encoder generating 12.5 tokens/sec.

Deployment and Performance

Muse Spark Open is available in BF16 (~1.5TB) and NVFP4 (4-bit, for NVIDIA Blackwell GPUs) checkpoints. It runs at the speed of a 42B model while retaining the capacity of an 816B model.

• Hardware Requirements: BF16 requires 4 nodes x 8 H200 GPUs (TP=32). NVFP4 can run on a single B200 node.
• Llama.cpp: Integrated with prebuilt binaries. On a Mac M3 Ultra (512GB), a 2-bit quantized version achieves 18 tokens/s using 304 GB memory.
• MLX: Supports oQ3e and oQ4e quantizations. On M3 Ultra, oQ3e uses ~344 GB memory, achieving 225 tok/s prefill and 21 tok/s generation for 1024-token prompts.
• SGLang & vLLM: Supported for text, image, audio, and video inputs. NVFP4 variants require Blackwell-series GPUs.

Reasoning and Tool Use

The model enforces reasoning with configurable effort levels: low, medium, high, xhigh (mapped to integers 16, 64, 256, 512). Reasoning cannot be fully disabled; low is recommended for simple tasks. It supports text and multimodal tool calling via a Harmony-based channel template.

Fine-Tuning

LoRA fine-tuning is supported on the full 816B checkpoint. Using 64 H100 GPUs (8 nodes x 8 GPUs), a 500-step training run took approximately 2 hours with a throughput of 8,000 tokens/s. This resulted in mean token accuracy improving from 72% to 84% and loss decreasing from 1.26 to 0.57.