SMU and Meta AI Researchers Report Performance and Latency Advantages of Jev Recommendation Reranking Model Over LLMs

Hanjia Lyu from Singapore Management University and Yinglong Xia from Meta AI published an empirical study comparing the quality and latency of TypeSafe AI's decision-oriented model Jev with an LLM (Qwen2.5 7B Instruct) and recommendation-specialized models (SASRec, DCNv2) in the reranking stage of recommendation systems. The paper does not mention any relationship with TypeSafe AI or API support.

Performance and Latency Comparison

Jev showed advantages over pointwise LLMs in Hit Rate@10 and MRR as the number of candidates increased, with latency approximately 9.7 to 17 times shorter than pointwise Qwen. In 11 out of 12 experimental conditions, Jev lay on the non-dominated frontier in terms of quality and latency.

• Video Games & Books: Jev achieved the highest performance across all candidate counts (20 to 200). In the Books condition with 200 candidates, Jev's Hit Rate@10 was 0.251, approximately 1.6 times higher than Qwen Pointwise (0.161), and its MRR was 0.140, approximately 1.9 times that of Qwen (0.074).
• Movies and TV: In the condition with 200 candidates, the original 1-stage retrieval order of SASRec showed a Hit Rate@10 of 0.170 and MRR of 0.093, higher than Jev (0.150 and 0.080, respectively), suggesting that maintaining the original order may be better than reranking under specific conditions.
• Latency: Jev (hosted API) recorded 627 to 967ms with 200 candidates, whereas Qwen Pointwise running on a local GPU surged to 6,060 to 14,868ms proportional to the number of candidates. SASRec and DCNv2 were the fastest, with latencies under 1ms.

Research Limitations and Implications

Since Jev's architecture and serving internals are not public, it is difficult to clearly distinguish whether performance differences stem from the model itself or the serving environment. Additionally, the LLM comparison is limited to the single model Qwen2.5 7B Instruct, and it is unclear whether it underwent recommendation-data fine-tuning or serving optimization. Statistical significance tests and repeated execution results were not reported, so caution is needed when asserting rankings in conditions with small differences. In conclusion, Jev, which adopts a structured decision-making approach, may represent an efficient operating point compared to pointwise LLMs, but the optimal strategy may vary depending on the domain and number of candidates.