Benchmark: Jev-Style Decision Models Do Not Consistently Outperform LLM Judges or Traditional Classifiers

Benchmark Overview

A comprehensive evaluation compared Jev-style decision models (Jev, Laya, DiffusionGemma) against LLM-as-a-judge approaches (Qwen3.6-35B, Nemotron-3.5, Shieldstral) and traditional pre-trained classifiers. The study utilized EvalHub's NeMo Guardrails benchmark library for prompt injection and toxicity tasks.

Key Findings

• Prompt Injection: Qwen3.6-35B achieved the highest accuracy at 89.31%, narrowly surpassing the traditional classifier deberta-v3-base-prompt-injection-v2 (89.01%). However, deberta-v3-base-prompt-injection-v2 offered the lowest median latency at 80.4ms, compared to 312.5ms for Qwen3.6-35B.
• Content Safety: Jev achieved the highest accuracy at 86.20%, followed closely by DiffusionGemma (85.53%) and Qwen3.6-35B (85.47%). The traditional classifier granite-guardian-hap-125m had the lowest median latency at 33.2ms but lower accuracy (80.27%).
• Prompt Engineering Impact: Performance varied significantly with prompt tuning. For example, Laya's content safety accuracy improved from 57.87% to 75.20% when using tuned risk definitions. Notably, Jev's accuracy dropped to 82.53% when evaluated using Laya's tuned prompts, highlighting that prompt strategies are not transferable across different decision model architectures.

Conclusion

The results indicate that while Jev-style models are competitive, they do not consistently outperform established methods. Traditional classifiers remain highly effective for specific tasks with abundant training data, offering superior speed and reliability.