AI BriefingCentaur Didn't Understand the Question
Key point
Doubts have been raised that Centaur's claim of mimicking cognition may be overfitting.
Details
Centaur, introduced in Nature in 2025, is an LLM-based model trained on psychological experiment data that drew attention for its ability to mimic human behavior across 160 cognitive tasks.
However, a follow-up study by Zhejiang University researchers published in National Science Open pointed out that this achievement could be overfitting. This means there is a strong possibility that the model did not actually understand the tasks, but instead memorized patterns in the training data and reproduced the correct answers.
To verify this, the researchers changed the instructions for multiple-choice questions to Please choose option A. If the model had truly understood the instruction, it should have chosen A, but Centaur kept selecting the answers from the existing data instead.
- Accuracy rate alone is not enough to determine cognitive ability.
- LLM-based psychological and cognitive models must also be evaluated for instruction comprehension and generalization.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Check Point Finds Jev Decision Model Vulnerable to Prompt Injection for ~$0.50Paper Instruments Releases Paper Office Python Suite to Boost AI Agent Document Editing Accuracy
Toss Builds 'Toss Benchmark' to Bridge Gap Between Public Leaderboards and Real-World Performance
LLM Reinforcement Learning Improves Performance by Reducing Computation on Easy Problems and Focusing on Hard Ones