The Geometric Blind Spot Revealed by ERM and PGD

An encoder trained with standard ERM must retain nonzero sensitivity to directions correlated with the label, and even to directions that become nuisance at test time. The authors define this as a geometric blind spot.

• They claim this theorem holds across proper scoring rules, various architectures, and data scales.
• The phenomenon unifies non-robust features, texture bias, corruption fragility, and the robustness-accuracy tradeoff into a single framework.
• PGD adversarial training can lower the Jacobian Frobenius norm, but it can actually worsen the input geometry. As an example, they present TDI 1.336 vs. ERM 1.093.
• The new metric TDI (Trajectory Deviation Index) measures geometric anisotropy that CKA, intrinsic dimension, and Jacobian Frobenius norm alone fail to capture.
• Experiments observed the blind spot across 7 tasks, BERT/SST-2, and ImageNet ViT-B/16 backbones from the CLIP/DINO/SAM family, and reported that it worsens with foundation-model scale and greater task-specific fine-tuning.
• The proposed PMH lowered TDI with just one additional training term and no architectural changes, and also produced better results under non-Gaussian corruption such as blur/brightness/contrast.