Four ML Puzzles, One Problem

It argues that the representations learned by ERM inevitably retain geometric blind spots. The explanation is that the encoder cannot fully ignore directions that correlate with labels in training data but become noise at test time, so the internal Jacobian sensitivity never reaches zero.

This single theorem explains four failure modes together.

• adversarial examples: sensitivity to spurious high-frequency features causes predictions to shift with small perturbations
• texture bias: since texture explains labels better than shape, texture keeps being used
• corruption fragility: corruptions like blur and noise hit the fragile directions that never disappear
• robustness-accuracy tradeoff: reducing the blind spot means losing predictive power that was used for existing accuracy

The paper also proposes the Trajectory Deviation Index (TDI). PGD adversarial training had a Jacobian Frobenius of 2.91, yet its clean-input geometry was the worst, with a TDI of 1.336. PMH, in contrast, showed a TDI of 0.904.

The authors report that this phenomenon was observed across 7 vision tasks, BERT/SST-2, and ImageNet ViT-B/16 backbones (CLIP, DINO, SAM). As language model size increased from 66M → 340M, the blind-spot ratio decreased from 0.860 → 0.765 → 0.742, and task-specific ERM fine-tuning worsened it by 54%. They claim PMH improved this by 11x with just one additional training term.