Empero Releases Quark: 165M-Parameter Model Trained End-to-End for $4,200

·2026.11.16 08:56

Key point

The 128K-context model achieves a 52.8% average on nine benchmarks, outperforming other open models of similar size trained on comparable token budgets.

Details

  • Empero released Quark, a 165M-parameter language model (140M without embeddings) trained through the full stack—data, pretraining, context extension, and reinforcement learning—for approximately $4,200 in compute costs (about 1,300 H100-hours).
  • Quark-150M-128K-Base achieves a 52.8% average score across nine zero-shot benchmarks, the highest among open models of 120M to 170M parameters trained on up to 500B tokens, leading on six of the nine tasks.
  • The Quark-150M-128K-Zero variant was trained using GRPO reinforcement learning alone, without supervised fine-tuning or distillation, raising the solved rate on simple terminal tasks from 3% to 65%.
  • All four checkpoints (Raw, 32K Base, 128K Base, and 128K Zero) are available on Hugging Face under the Apache-2.0 license, with the 128K model supporting up to 128K tokens of context.

Architecture and Training

  • The model uses 16 layers where 12 are local (attending to the last 256 tokens) and every fourth layer is global, reducing the key-value cache at 128K tokens to about 0.5 GiB in bf16.
  • Pretraining involved 500B tokens on a single 8×H100 node, with the 128K context version trained on an additional 10B tokens of long-context data.
  • Reliability testing showed the node went down seven times during pretraining with no loss spikes, and automatic recovery resumed training within two to four minutes.

Reinforcement Learning Results

  • RL training used real sandboxes for code tasks (95% of rewards) and LLM judges for other tasks, running for under 1,300 steps and about 32 hours.
  • Enabling a private thinking monologue before answering increased accuracy on commonsense puzzles from 44% to 67%.
  • Limitations include poor performance on web search tasks, failure in one of 15 harness formats, and a drop in long-context scores during the RL stage.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.