LG AI Research: A Benchmark Suite for Evaluating Neural Mutual Information Estimation on Unstructured Datasets

Presented at NeurIPS 2024, this research from LG AI Research's Data Intelligence (DI) Lab addresses the reliability issue of Mutual Information (MI) estimation, a problem that is easily overlooked in deep learning research.

MI is a key metric that measures the relationship between two random variables, and it is used in various fields such as Representation Learning, Self-supervised Learning, and Reinforcement Learning. However, while real-world data is complex unstructured data, existing MI estimation techniques have mainly been validated only on Gaussian Multivariates benchmarks, making it difficult to guarantee their performance on real data (images, text, etc.).

This research makes the following three key contributions:

• Proposing a methodology for generating data such that the accurate True MI value can be known even for arbitrary datasets
• Building a benchmark suite that can evaluate the accuracy of MI estimators across various settings, including Gaussian, image, and sentence embeddings
• Comparing MI estimator performance and summarizing experimental results based on 7 scenarios

To generate accurate MI values, the research team used a positive pairing technique leveraging Same-class sampling. By constructing a dataset that shares only class information in this way, they designed the theoretical MI value to match the entropy of the class variable, laying the foundation for rigorously verifying the performance of estimators.