Bespoke Labs Releases Decision Model Nimble

Bespoke Labs has released Bespoke-Nimble-9B, a 9B parameter decision model trained on contrastive data that flips the correct answer by changing just one fact, along with its training recipe. This model applies LoRA to Qwen3.5-9B to output probabilities by reading only the logits of allowed answer tokens without generating answers.

Performance and Benchmarks

On its own holdout test (324 samples), Nimble recorded an accuracy of 90.12%, significantly outperforming the base model (66.36%), but fell short of the competing model Jev 1.13.0 (93.21%). On 13 public benchmarks with human labels (3,880 records), it achieved a macro average of 74.8%, trailing Jev (76.0%) by 1.2 percentage points. By type, Nimble had the advantage in rubric grading (Score), but Jev was 4.4 percentage points higher in true/false (Noul) judgments.

Training Data and Method

Training used 2,676 (1,338 pairs) of contrastive data, with labels inspected by models rather than humans (synthetic labels). During serving, token codes such as A/B/C are assigned to options in the prompt, and probabilities are calculated via softmax by reading only the allowed codes from the next token logits, eliminating the need for JSON parsing. In an Apple Silicon MLX environment, sharing a common prefill improves speed proportionally to the number of fields (7.25x for 8-field long contexts).

Version and License Status

The original release and v2 are under the Apache 2.0 license, allowing commercial use, but v3 is CC BY-NC 4.0 (non-commercial), and v4 and v4-max are exclusive to paid APIs with unpublished weights. v2 was trained on 4,926 examples, but only 2,676 were released, and its accuracy on the 324-sample evaluation set was 82.7%, lower than the original release. On H100, Nimble's median latency is 106.0ms, faster than the Jev API (246.7ms).