Privatemode Benchmarks GLM-5.3-Flash as Vision-Capable Decision Model with Parity to Jev but Higher Cost

Privatemode has released a technique and benchmark suite converting GLM-5.3-Flash into a decision model comparable to specialized systems like Jev and Laya. By leveraging /completions with echo on vLLM endpoints, the method extracts tokenization data to perform token oracle masking and renormalization, enabling precise probability distributions for classification tasks.

Benchmark Results

Across 29 public labeled datasets (including intent routing, sentiment, and legal classification), GLM-5.3-Flash demonstrated statistical parity with Jev:

• Accuracy: Median gap of 0.7%p (Jev slightly ahead), with no statistical significance (p = 0.64).
• Comparison: GLM-5.3-Flash outperformed the smaller Laya (421M params) by a median gap of 13–15%p.
• Option Count Impact: The number of options affects accuracy more than the specific model choice. For example, in TREC (6→42 options), accuracy dropped for all models, but GLM-5.3-Flash remained competitive.

Cost and Latency

• Cost: For 1 million decisions, Jev costs approximately EUR 16, while GLM-5.3-Flash costs EUR 62. However, GLM-5.3-Flash uses fewer tokens for tasks with fewer than about 21 options.
• Latency: Location-dependent. In Germany, GLM-5.3-Flash is faster (180 ms vs. Jev's 264 ms). In the US, Jev is faster (164 ms vs. GLM-5.3-Flash's 299 ms).

Capabilities and Limitations

• Multimodal: Unlike Jev and Laya, GLM-5.3-Flash supports vision inputs (e.g., invoices, screenshots), achieving 70.2% accuracy on scanned business documents (RVL-CDIP).
• Long Option Lists: The endpoint returns max 128 options per request. Lists longer than 128 require two requests and merging, increasing latency to 719 ms.
• Reasoning Mode: Enabling reasoning improves accuracy (e.g., 89.9% vs 85.5% for two options) but increases cost to about EUR 350 per 1M decisions.
• Security: The implementation uses end-to-end encryption and TEE-based processing.