ProText: A Benchmark Dataset for Measuring (Mis)Gendering in Long-Form Text

ProText is a dataset for measuring gendering and misgendering in long-form English text across diverse styles. Based on Theme nouns such as names, occupations, titles, and kinship terms, it is designed to precisely examine bias by combining Theme category and Pronoun category together.

This dataset focuses on examining how state-of-the-art LLMs infer and impose gender during text transformation processes such as summarization and rewriting. It goes beyond existing pronoun resolution benchmarks, addressing broader gender representation issues without being confined to the gender binary.

The authors showed that a mini case study using just two prompts and two models was enough to capture meaningful differences.

• Bias was more pronounced when there were no explicit gender cues in the input.
• Models were confirmed to have a tendency to set defaults relying on heteronormative assumptions.
• As a result, they presented a measurement foundation that can jointly analyze gender bias, stereotypes, misgendering, and gendering issues.