[ACL 2024] LLM Reliability Evaluation Methodologies and Efficiency Research Trends

Of the ACL 2024 main conference papers, over 319 were LLM-related studies, with research addressing LLM capabilities, limitations, and reliability forming the mainstream. In particular, cautioning against viewing LLMs as an all-purpose tool, various methodologies were proposed to address the limits of reasoning ability and reliability issues.

In the LLM Evaluation & Faithfulness area, benchmarks and evaluation methodologies requiring complex reasoning beyond simple tasks were actively discussed. Key research examples are as follows.

• AppWorld: A benchmark that goes beyond conventional simple API calls, evaluating LLMs as Agents in a high-quality simulation environment built with 60,000 lines of code. Even GPT-4o showed a low success rate on complex tasks, demonstrating the need for evaluating interactive coding agents.
• CEF (Correlational Explanatory Faithfulness): A new metric that measures the agreement between the explanation (rationale) generated by an LLM and its actual final judgment, assessing how trustworthy the model's explanations are.

In addition, the Data Contamination problem arising from web data being included in training, and PEFT research aimed at improving model efficiency, have become key trends in research seeking practical utilization and reliability assurance for LLMs.