Open ASR Leaderboard Adds Private Data

High-quality English ASR datasets provided by Appen Inc. and DataoceanAI have been added to the private track of the Open ASR Leaderboard.

• It includes both scripted (read) and conversational (spontaneous) speech, covering a wide range of accents including US, Australian, Canadian, UK, and Indian.
• The goal is to reduce benchmaxxing and test set contamination risks to create more reliable performance measurements.
• The default Average WER continues to be calculated using only public data, and private data is included in the average only when the toggle is turned on.
• The new aggregate metrics are Avg Scripted, Avg Conversational, Avg US, and Avg non-US, and split-level scores are not disclosed separately.
• The Rank Δ column shows rank changes depending on whether private data is included.
• Model submissions are accepted via GitHub PR, and after public set verification, Hugging Face calculates the private set metrics.
• Public set results can be self-reported by writing them in a YAML file on the model card, and they are also shown on the unverified leaderboard on the dataset page.

Since its launch in September 2023, the leaderboard has been visited over 710,000 times, and a Whisper-based normalizer is used for standardization. The UI code and evaluation scripts remain open source, and there are plans to add evaluations reflecting rougher real-world and noisy conditions in the future.