Metric-Dependent Annotation Saturation for Label Distribution Learning

Disagreement among annotators that arises during the data labeling process carries an important signal in itself. Research has confirmed a metric-dependent saturation phenomenon, in which the number of annotators required varies significantly depending on which evaluation metric the model uses.

Results from fine-tuning an NLI model using the ChaosNLI dataset are as follows.

• Entropy correlation: When measuring a model's ability to identify items with disagreement, about 20-50 annotators are needed for convergence.
• Distributional agreement (KL divergence): When measuring agreement of the distribution itself, just about 10 annotators achieve 87-95% of the performance gain, reaching saturation quickly.

Additionally, simple Label Smoothing fails to fully reproduce the item-specific signal contained in Soft Labels. In experiments using the DeBERTa and RoBERTa architectures as well, Soft Labels showed higher correlation than Label Smoothing. Therefore, for efficient AI training, the annotation budget should be set differentially depending on the target evaluation metric.