Emirati Dialect Evaluation Benchmark Alyah Released

Existing Arabic LLM benchmarks have mainly focused on Modern Standard Arabic (MSA), leaving evaluation of the regional dialects used in actual everyday conversation insufficient. To address this, a new benchmark called Alyah has been introduced to evaluate the linguistic, cultural, and practical aspects of the Emirati dialect.

Alyah is a dataset composed of 1,173 samples collected directly by Emirati locals. Beyond simple vocabulary knowledge, it includes culturally-loaded expressions, idioms, oral poetry, and questions related to traditional heritage to measure a model's depth of understanding.

Key features are as follows:

• Data composition: Consists of 4-choice multiple-choice questions, with the correct answer index randomly placed to prevent positional bias in models.
• Evaluation categories: Covers a variety of areas including greetings and everyday expressions, religious and social sensitivity, figurative meaning, etiquette and values, poetry and creative expression, historical and heritage knowledge, and language and dialect.
• Purpose: Evaluates how accurately a model grasps Emirati-specific cultural nuances and context, beyond simple literal translation.