Neutrino-1 8B

This provides detailed architecture specifications for Neutrino-1 8B, a dense decoder-only transformer model using Alibaba Cloud's Qwen3-8B as its base model.

The key technical features are as follows:

• Model structure: The architecture consists of 36 decoder layers, a hidden width of 4,096, and a feed-forward width (12,288) with SwiGLU gating.
• Attention mechanism: It uses Grouped-query attention at a 4:1 ratio to improve KV cache efficiency, consuming 0.60 GB of cache for a 4k-token session.
• Data compression and storage: The 252 transformer linear layers, which account for approximately 67.2% of the total file, use a ternary-family lane coding format, while embedding tensors are kept in int8.
• Weight characteristics: Of the 6.95B coded weights, approximately 62.63% are zero, with weight density varying depending on layer depth.
• Context length: It supports a context length of up to 40,960 tokens.