Model Optimization Strategies for Faster Codegen

Coding agent tasks have the characteristic of reusing large portions of existing files, re-reading the same repository at every turn. Typical inference stacks ignore this redundancy and waste resources by decoding every token from scratch each time.

To address this, we propose three key optimization strategies.

1. Training a Target-Model-Specific Speculator Speculative Decoding works by having a small Draft Model predict tokens, which the target model then verifies. A Draft Model trained on general data shows a 1.93x speedup, but when trained directly on the target model's coding outputs, the speedup increases to 3.07x.

Particularly for models under 30B, Overtraining—training on far more data than the Chinchilla optimum—proves effective. This allows building a small yet powerful Speculator specialized for a specific domain (coding).

2. Automatic Kernel Optimization (Autoresearch) Rather than manually tuning for every environment, kernels are automatically searched to achieve high performance even on low-cost GPUs. This delivers 97-162 tok/s performance even on $7K-class cards.

3. Interconnect Optimization Bottlenecks are eliminated through an All-reduce implementation over PCIe and interconnect technology that allows Prefix Cache sharing via TCP even in environments without NVLink.