Project Lighthouse Part 3: Introducing project-lighthouse-anonymize

Airbnb has open-sourced the core anonymization code of Project Lighthouse, which was developed in 2020 in collaboration with civil rights and privacy organizations. This project is used to measure potential discrimination in user experiences by leveraging perceived race data that is not linked to user accounts.

The released project-lighthouse-anonymize Python library supports scalable anonymization of large datasets. To support this, two new technical papers were published on arXiv.

The first paper, Core Mondrian, presents a k-anonymity algorithm with a scalable architecture and parallel processing. This algorithm uses a hybrid model that immediately recursively processes small partitions while combining queue-based parallel processing for large partitions.

The second paper introduces a framework for measuring the quality of anonymized data. Key metrics include Pearson correlation coefficient, Revised Information Loss Metric (RILM), and Normalized Mutual Information (NMIv1), enabling analysts to reliably assess whether statistically valid conclusions can be drawn from the anonymized data.

The library is available for download on PyPI and GitHub, and it implements both k-anonymity and p-sensitive k-anonymity technical privacy models as proposed in the 2020 paper.