GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks

Existing error slice discovery methods for vision models have mainly relied on clustering in representation space or predefined attributes. However, these approaches show clear limitations for instance-level tasks such as Object Detection and Segmentation, where contextual relationships and spatial patterns matter.

To address this, the proposed GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery) introduces a 'Generate and Verify' framework. This approach operates through the following stages:

• Hypothesis Generation: Combines an LLM's prior knowledge with visual grounding to construct relational error hypotheses.
• Slice Discovery: Identifies hypothesis slices at the instance level via a VLM (Vision Language Model).
• Statistical Verification: Verifies hypotheses through statistical trend analysis of instance-level errors.

Additionally, the researchers introduced a new benchmark, GESD (Grounded Error Slice Dataset), which includes expert-defined spatial slices derived from detection and segmentation failures.

Experimental results demonstrated GH-ESD's effectiveness, improving Precision@10 on the detection task of the GESD benchmark from 0.63 to 0.73, a gain of 0.10. This technique provides actionable interpretability for model improvement.