AI21 cuts GitLab Duo Code Review cost by 98% via harness optimization

Switching all four stages to DeepSeek V4 Flash and adding context compaction reduced cost per PR from $0.589 to $0.0135 (97.7% saving) while maintaining 47.2% F1, compared to the baseline 46.4% F1.
The baseline cost was driven by the review stage (65% of total), where Claude Sonnet 5 generated approximately 18,600 reasoning tokens per review; compaction alone saved only 19.4% because the model increased its output tokens to 23,800.
DeepSeek V4 Flash served on H100s via vLLM costs 14x less per input token and 36x less per output token than Sonnet 5, but its 256K token deployment limit caused 3 of 50 pull requests to fall back to Sonnet 5, accounting for 77% of the remaining cost after the model switch.
Context compaction (file windowing and unified diff rewriting) reduced the median review prompt from 59K to 32K tokens, allowing all reviews to fit within the 256K window and eliminating fallback calls.
Claude Haiku 4.5 maintained quality on the mechanical stages (exploration and fetching) but degraded performance on analysis and review, limiting its standalone cost savings to 34.6% when combined with compaction.
Raising reasoning effort to "max" increased recall from 38.9% to 52.8% but only moved F1 from 35.0% to 35.4% at 3.5x the latency, offering no net quality improvement.
The study used Martian’s Code Review Bench with 50 public pull requests from Sentry, Grafana, Cal.com, Discourse, and Keycloak, scored by Claude Opus 4.5 and GPT-5.2 as judges.
GitLab has since moved Duo Code Review to Sonnet 5.5, making the specific configuration dated, though the method of measuring cost per stage and changing one variable at a time remains applicable.

AI21 cuts GitLab Duo Code Review cost by 98% via harness optimization

Key point

Details

Related