Carbon-X: 14.8B DNA-Language Model Achieves 66.7% Benchmark Accuracy at 1/1,800th of Frontier Cost

Details

Carbon-X is a 14.8B-parameter decoder-only model trained jointly on DNA sequences and natural language, achieving an average of 66.7% across five DNA benchmarks with thinking disabled.
Performance exceeds the base Qwen3-14B model by 27 points and approaches GPT-6.1 Sol (67.5%) at approximately 1/1,800 of the inference cost per problem.
The model uses a hybrid tokenizer with a BPE vocabulary for text and 5,515 DNA tokens (4,096 6-mers and 1,364 tail tokens), allowing DNA and text to share a single token stream without a separate encoder.
Weights fit on a single 80 GB GPU in bfloat16, enabling local processing of sensitive sequences; estimated cost to analyze 2.4 million ClinVar variants is about $300, compared to $32,000–$4 million for frontier models.
Architecture features 40 layers, hidden size 5120, GQA (40 query heads, 8 KV heads), SwiGLU, RMSNorm with QK-norm, and a 40,960-token context window.
Training involved 302B tokens of mid-training on DNA, biology, English, and instruction data, followed by supervised fine-tuning and reinforcement learning with verifiable rewards.

Key point