AI BriefingApple Research: LLMs and Humans Both Struggle with Machine Translation Quality Annotation
Key point
The study evaluated LLMs against human annotators across 70 language pairs using MQM and ESA schemes, finding both unreliable in most settings.
Details
Apple researchers evaluated the reliability of Large Language Models (LLMs) as alternatives to human judgment for Machine Translation (MT) quality evaluation. The study compared agreement between LLMs and human annotators using two prominent schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA).
Results from a long-context test set of 70 language pairs and publicly available WMT23 and WMT25 data show that while LLMs sometimes exceed human-human agreement, both vary substantially across annotation schemes, language pairs, and domains. Consequently, both remain unreliable for most settings.
The research identifies distinct challenges for each group:
- Humans struggle particularly with fine-grained MQM annotations and low-resource language pairs.
- LLMs face difficulties with minor errors, wrong language variants, and error span annotation.
The authors suggest that human-LLM collaborative annotation pipelines could address these reliability issues and improve overall performance.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.