Open-Weight LLMs Catch Up in Accuracy

In regulatory life sciences work where accuracy is critical, an evaluation found that Open-Weight LLMs have caught up with closed-source frontier models. The two types of models were compared using the ClinReg benchmark, which evaluates three tasks: post-market safety review of literature, structured data extraction from clinical documents and regulatory submission drafting, and TLF generation based on clinical trial data.

Existing benchmarking in the life sciences field, such as CASP, LAB-Bench, and BixBench, has focused on drug discovery and preclinical stages. However, most of pharmaceutical R&D cost and duration is spent on clinical trials and regulatory work, and in this area, missing papers or incorrectly extracted values can have cascading effects on regulatory submissions.

According to the ClinReg evaluation results, in all three tasks, the accuracy of Open-Weight models fell within the performance variation range of closed-source models. At the same time, Open-Weight models were shown to run at roughly 3x lower cost than top-tier closed-source models, demonstrating the potential to achieve both accuracy and cost efficiency together in high-stakes regulatory work.