Reproduction and Performance Comparison Analysis of the Jev Decision Model

Results from the community projects 'Jev Decision Index' and 'JevBench', which reproduced and compared the performance of TypeSafe AI's decision model Jev, have been released. Jev is an API-only model that returns only probabilities without generating sentences, and since its weights are not public, various reproduction attempts are being made within the open-source ecosystem.

Decision Index: Accuracy-Centric Comparison

Based on Jev Decision Index 0.2.1, Jev scored the highest with 57.91. The closest open models were Surogate Rune 26B-A4B v3 (57.44) and Decider chat (57.33), indicating that no open model has yet surpassed Jev's accuracy. However, while Jev showed strength in the knowledge/reasoning domain, open models recorded higher scores in the remaining four domains.

JevBench: Considering Cost and Speed

In JevBench v1.5.7, which includes cost and speed, the rankings changed. Cygnet (73.7) and Winnow-12B Q8 (73.2) recorded higher official scores than Jev (72.1). Cygnet is an untrained model based on Gemma 4 12B, showing competitive performance at a lower cost ($0.028/1k) than Jev. Winnow-12B is a fine-tuned model tested in an RTX 5070 Ti environment.

Image JevBench and Korean Language Support Status

In Image JevBench, which supports image input, Jev was excluded because it does not support image input. Among open models, Imajev-4B (76.39) ranked second overall. Additionally, all leaderboards were evaluated based on English data, and only 2 out of the 70 models listed in the Decision Index support Korean, necessitating re-evaluation based on proprietary data for Korean business applications.