Supporting Researchers and Developers Building Multilingual AI with a New Open Dataset

GitHub announced the GitHub Multilingual Repositories Dataset, which helps researchers and developers easily find public repositories containing non-English content. Rather than containing the full content of repositories, this dataset is a metadata dataset designed to identify where multilingual collaboration occurs.

The dataset includes 80+ million classification rows across 40+ million repositories. For each public repository, it provides the following information:

• Language classification information for the README, the Issue with the most comments, and the Pull Request with the most comments (including a 150-character sample from the beginning of each text)
• Language identification results and confidence scores from three classifiers — fastText, gcld3, and lingua-py (only data with confidence above 0.5 is included)
• Repository metadata (creation time, disk usage, Stars, Forks, primary programming language, license, number of issues/PRs, etc.)

Users can leverage the results from all three classifiers to adjust Precision and Recall according to their research needs. For example, if high precision is required, one can extract only the data where all three classifiers agree.

This dataset can be used to discover development documentation or collaboration cases written in a specific language, research the activity of non-English-speaking developer communities, and build evaluation sets for AI coding tools that work well across various languages.