Preprint Finds LLMs Show 'Irrelevant-Expertise Penalty' in Credibility Ratings

·2026.11.03 13:25

Key point

A preprint testing 20 arguments across four models finds that attributing arguments to experts in unrelated fields reduces perceived credibility compared to having no source.

Details

New preprint tests whether changing the attributed source of an argument affects LLM evaluation, using 20 arguments across 10 topics and four source conditions (relevant expert, irrelevant expert, non-expert, no source).
Credibility was the most source-sensitive dimension, with an 'irrelevant-expertise penalty' observed where attributing arguments to experts in unrelated domains reduced credibility relative to no source.
Agreement and logic dimensions showed less sensitivity to source attribution, and interaction effects based on argument strength were not statistically robust.
The study evaluated GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek Flash, noting that the preprint is not yet peer reviewed.

llm-evaluation bias gpt-5.6-sol preprint claude-opus-5.5 gemini-4-argon source-attribution deepseek-flash

Read originalShare on Threads

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.