Paper Proposes Pair-Matched HodgeRank and Post-Control Risk to Complement g-mean² Monitorability Metrics

Details

The paper proposes two complementary metrics to address limitations in g-mean², which can be distorted by differences in valid positive behavior subsets across models and fails to reflect deployment risk when monitors act as rejection gates.
Pair-matched HodgeRank compares models only on shared valid instances to remove task-mixture bias, using inverse-variance-weighted least squares to estimate model scores while preserving g-mean² invariances to class imbalance and intervention shortcuts.
Post-Control Risk metrics simulate monitor-gated retries to measure the probability of deploying bad outputs (B(K)) and any outputs (A(K)), revealing that retries can increase the chance of passing ambiguous or harder-to-classify misbehaviors.
Synthetic experiments with 7 models show pair-matched HodgeRank accurately recovers known monitorability orderings as capability dispersion increases, whereas dataset-level g-mean² and Bradley-Terry methods degrade due to instance mix changes and discarded pairwise difference magnitudes.
Historical system card results for 7 agent models (including GPT-5.6 Sol) indicate that post-control gating reduces primary risk on average, but low B(K) can result from rare deployment, necessitating the reporting of conditional bad-output rates B(K)/A(K).

Key point