Cactus Adds Self-Confidence Probe to Gemma 4

Cactus added a 68k-parameter probe layer to a small on-device model (Gemma 4 E2B) and post-trained it so the model returns a confidence score between 0 and 1 for each response.

By routing to a cloud model like Gemini Flash-Lite only when confidence is low, the system achieves Gemini Flash-Lite-level performance on most benchmarks while sending only 15–55% of all queries to the cloud.

The probe architecture consists of LayerNorm → low-rank projection → attention pooling → MLP head, and predicts p(wrong) from the hidden state of an intermediate layer during decoding. It is returned as structured data rather than through response text parsing.

Performance comparison (AUROC):

• Probe: 0.814 average across 12 benchmarks
• Token entropy heuristic: 0.549

Notably, even though the probe was trained without any audio data, it achieved 0.79–0.88 AUROC on 4 audio benchmarks, while entropy only reached 0.32–0.52 (random level). This suggests the probe is reading a modality-independent accuracy signal from the hidden state.

Weights are released on HuggingFace (MIT), with execution code provided for Transformers, MLX, Llama.cpp, and Cactus. Support for Ollama, vLLM, and SGLang is also planned.