IQuestLab Releases IQuest-Q1, a 320B Parameter MoE Model for Agentic Coding

IQuestLab has released IQuest-Q1, a Mixture-of-Experts (MoE) model designed specifically for agentic coding, reasoning, and multi-step tool use. The architecture comprises approximately 320B total parameters, with an estimated 15B parameters activated per token, balancing capacity with inference efficiency.

Model Specifications

The model utilizes a hybrid attention pattern and specific transformer configurations to support its agentic capabilities:

• Architecture: 88 Transformer layers with 3,072 hidden dimensions.
• Attention: 48 Query heads and 8 Key/Value heads, with a head dimension of 128. The pattern alternates between 3 layers of Sliding Window Attention (SWA) and 1 layer of Standard Attention.
• Sliding Window: 4,096 tokens for SWA layers.
• Experts: 256 total experts with 8 activated per token.
• Context Window: Supports up to 524,288 tokens.
• Multi-Token Prediction (MTP): Features 2 independent MTP layers during training and 1 recursive MTP layer during inference, with a sliding window size of 512 for inference.

The model is available on Hugging Face, offering a high-parameter count with sparse activation for efficient deployment in complex coding and reasoning tasks.