LangSmith Adds Jev Evaluation Model… Agent Evaluation Up to 450x Cheaper and 200x Faster Than LLMs

LangSmith has added TypeSafe AI's Jev model as an evaluation tool. Jev is a System One model that makes structured decisions without text generation, balancing the flexibility of LLM-as-a-judge with the speed of code-based evaluation.

Performance and Cost Efficiency

According to TypeSafe AI's benchmarks, Jev is up to 450x cheaper and 200x faster than LLMs for classification tasks. In comparisons with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6, Jev demonstrated superiority in accuracy, consistency, speed, and cost.

• Speed: Jev averages 0.44s/call vs. LLM judges at 2.16-2.83s
• Cost: For full judgments, Jev costs $0.34 ($0.00035/call) vs. Claude Sonnet 4.6 at $28.17
• Consistency: Variance is 92-913x lower compared to LLM judges

How It Works and Usage

Jev supports three question types: noul (yes/no probability), choice (multiple choice), and score (scale). It evaluates all questions in parallel within a single request, so verifying multiple criteria (PII, intent, etc.) incurs almost no additional cost or time.

• Setup: Register the TypeSafe API key in LangSmith Settings, then select TypeSafe as the Provider and jev-latest as the Model in the Evaluators tab
• Note: Currently, TypeSafe does not offer zero data retention, so evaluation data may be stored by the provider

Jev is best suited for making large volumes of narrow, typed decisions rather than open-ended reasoning, serving as a complement to existing LLM judges rather than a complete replacement.