AraGen released for Arabic LLM evaluation

HuggingFace has announced AraGen, a new generative task benchmark and leaderboard for measuring the performance of Arabic LLMs from multiple angles.

To overcome the limitations of existing evaluation methods, AraGen introduces the following three key elements:

• 3C3H Metric: Using the LLM-as-a-judge approach, it comprehensively evaluates model responses across 6 dimensions: Correctness, Completeness, Conciseness, Helpfulness, Honesty, and Harmlessness. This allows for a balanced measurement of a model's knowledge (factuality) and alignment with user expectations (helpfulness).
• Dynamic Evaluations: To prevent data contamination, it operates blind test cycles on a 3-month basis. The test dataset and evaluation code are only released at the end of each cycle, after which they are replaced with a new, unreleased benchmark.
• Arabic-Specific Dataset: It provides a dataset that includes single-turn and multi-turn scenarios, enabling precise testing of Arabic models' capabilities across various domains and tasks.

As the first application of a language-agnostic, scalable framework to Arabic, AraGen aims to set a new standard for model evaluation.