LG AI Research Wins NAACL 2024 Social Impact Award for Study on LLMs' Cultural Common Sense Understanding

A joint study by LG AI Research and the University of Michigan won the NAACL 2024 Social Impact Award. This study systematically analyzed how accurately large language models (LLM) understand diverse cultural contexts and common sense, as well as the cultural biases that arise in the process.

The research team designed experiments targeting 5 countries and 5 languages: China, India, Iran, Kenya, and the United States. They tested a wide range of models, from open-source models like Vicuna, Llama2 to closed-source models like GPT-3.5-turbo, GPT-4.

The key findings are as follows:

• Limits of cultural common sense understanding: The models showed high performance in environments rich in English and Chinese data, but performance dropped significantly when dealing with cultural common sense related to Iran or Kenya and their respective languages (Persian, Swahili).
• Relationship between language and performance: Asking questions in the language of a specific culture did not necessarily lead to improved performance. Rather, most models showed the highest performance when questions were asked in English rather than the country's own language.
• Cultural influence on general common sense: For universal common sense such as 'a healthy diet lowers blood sugar,' GPT models showed a pattern of uniformly improved performance when questioned in the respective country's language.

This study emphasizes the urgent need to develop culturally aware language models for the stability and fairness of AI systems, offering important insights for promoting inclusivity in AI technology.