Technical Rebuttal of TypeSafe Jev Model's 'Calibrated Probabilities' Claim

The Jev model, released by TypeSafe, is marketed as a 'System One Model' capable of performing classification tasks without training data, with outputs that include calibrated probabilities. However, technical analysis has raised concerns that this calibration claim may not hold in general scenarios.

Limitations of Calibrated Probabilities

Model calibration depends not only on the model's inherent properties but also on the data distribution. While Jev may be calibrated for a specific training distribution, it may return the same uncalibrated probabilities if the user's actual operational data distribution differs. This issue arises because data characteristics vary by company, even for the same definition of 'spam'.

Empirical Errors and Alternatives

In actual tests, Jev demonstrated clear errors, such as predicting the probability of heads in a fair coin toss as 0.92. Additionally, consistency issues were found where the meaning of probabilities changed when using different Primitives for the same problem. Therefore, Jev's outputs should be treated as ranking scores rather than reliable probabilities.

Recommendations

For systems where actual probability values are critical (e.g., threshold setting, cost prediction), do not trust Jev's outputs as-is. Instead, perform self-recalibration using methods like Platt scaling with hundreds of labeled data points.