Anthropic Introduces Conceptual Reasoning Index

Managing AI risks requires understanding how well models reason in domains where empirical feedback is scarce or philosophical and futurist arguments are necessary. To this end, Anthropic and Redwood Research developed the Conceptual Reasoning Index (CRI).

CRI is a metric that integrates the following three benchmarks to measure a model's abstract thinking capabilities:

• LMCA (Language Model Conceptual Argumentation): Evaluates argumentation skills across various topics such as decision theory, philosophy, and AI risks. It measures how well models judge and generate arguments using expert-evaluated argumentation data.
• ACCoRD (Assessment of Consistency in Conceptual Reasoning Domains): Measures whether the beliefs and preferences presented by models maintain logical consistency.
• DTBench: The third benchmark constituting CRI.

These benchmarks focus on measuring how effectively models can rely on Argumentation to reason in environments where there are no correct answers or empirical evidence is limited.