AI BriefingDharma AI Cuts Inference Costs by 77% for Qwen3-4B-Based Grading Model Using VocabX Technique
·2026.10.30 20:30
Key point
Reduced inference costs by 77% for a Qwen3-4B-based grading model through cost-weighted vocabulary selection.
Details
- Dharma AI has released the VocabX technique, which selects vocabulary based on generation cost rather than token frequency
- For a Portuguese essay grading model based on Qwen3-4B, response length was reduced by 71.1% and inference costs were cut by 77.0%
- Quality was also improved, achieving a benchmark score of 0.810, higher than the 0.746 score of the previous state-of-the-art method, In-place BPE
- Leveraging the fact that the cost of one output token is 7.4 times that of an input token as a weight, the number of rows in the output layer was reduced from approximately 152,000 to approximately 65,000
- Across 15 combinations including Qwen3, Llama-3.1, and Gemma-3, the token reduction rate was 1.2 to 9.3%p higher than existing methods
- Retraining is mandatory when changing the tokenizer, and tokens involved in the decision must be excluded from macro tokens
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Cartesia Research H-Net Achieves 1.3x Compute Efficiency Over BPE TransformersOpenAI and Anthropic cut LLM prices by up to 80% amid open-weight competition and compute efficiency gains