AI Briefingopenjev: 5x Faster by Extracting Classification Probabilities Without Text Generation
TheoLeeCJ/openjev
About the project
Eliminates the inefficiency of LLMs generating and parsing sentences when agents decide on routing or retries. By directly reading model logits to instantly return probabilities for each option, the answer generation process is skipped.
Running the Qwen3.5-4B model on an RTX 3090 takes approximately 1 second for 21 criteria judgments. This is more than 5x faster than the existing JSON array generation method, and by preloading state information for parallel processing, it can make over 20 decisions per second.
Reproduces an interface similar to TypeSafe's closed-source service Jev using open models. Provides a WebGPU demo that runs directly in the browser, allowing classification performance to be verified without complex installation.
TheoLeeCJ/openjev
Can we run something like Jev on a 3090 at home?
Python
This introduction was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report errors, attribution issues, or removal requests via Contact.