GLM-5.3-Flash Matches Specialized Decision Models in Zero-Shot Classification via Logit Extraction

Researchers from Privatemode demonstrated that GLM-5.3-Flash can function as a System One decision model, matching the performance of the specialized Jev model for intent classification without fine-tuning. By extracting logit probabilities from a single forward pass instead of generating text, the general-purpose LLM achieves comparable accuracy while supporting multimodal inputs.

Benchmark Performance and Methodology

The study evaluated GLM-5.3-Flash against Jev and Laya (421M parameters) across 29 public datasets. Results indicate statistical parity between GLM-5.3-Flash and Jev:

• Accuracy: Median gap of 0.7 percentage points (Jev favored, p=0.64, not statistically significant). Each model outperformed the other on 10 of 28 text datasets.
• Laya Comparison: Laya performed significantly worse, with a median gap of 13–15 percentage points (p<0.001).
• Option Count Sensitivity: The number of options impacts accuracy more than model choice. For example, in the TREC dataset (6→42 options), Jev dropped from 92.1% to 85.6%, while GLM dropped from 91.2% to 79.6%.

Technical Implementation

The approach utilizes vLLM and Privatemode’s infrastructure to bypass generation:

• Logit Extraction: Uses logprob_token_ids to retrieve probabilities for specific option tokens, avoiding the noise of top_logprobs.
• Constraints: allowed_token_ids restricts output vocabulary. Token IDs are dynamically fetched via /completions echo requests.
• Scalability Limits: GLM-5.3-Flash supports up to 191 options expressible as single tokens. For datasets like CLINC150 (151 options), two requests are merged to achieve 87.5% accuracy, surpassing Jev’s 78.4%, though latency increases to 719ms (vs. Jev’s 249ms).

Cost, Latency, and Multimodal Capabilities

• Cost: Jev is more cost-effective at EUR 16 per million decisions compared to GLM’s EUR 62. Enabling reasoning in GLM raises costs to EUR 350 per million.
• Latency: Varies by region. In Germany, GLM (180ms) is faster than Jev (264ms); in the US, Jev (164ms) is faster than GLM (299ms).
• Multimodal: GLM-5.3-Flash supports image input, achieving 70.2% accuracy on RVL-CDIP (scanned documents), a task Jev and Laya cannot perform.
• Token Overhead: GLM uses fewer tokens for <21 options but more for larger sets compared to Jev’s fixed overhead structure.