LG AI Research 513

The main keywords of NeurIPS 2024 were LLM, Reinforcement Learning, and Multimodal, and in particular, the Datasets & Benchmark track featured a number of interesting studies. LG AI Research's Data Intelligence (DI) Lab presented a benchmark study in this track that enables more rigorous verification of deep learning model performance.

This research addresses Mutual Information (MI) estimation performance, a key metric for measuring the relationship between two random variables. Existing MI estimation techniques have mainly been evaluated only in Gaussian Multivariates settings, making it difficult to guarantee their reliability on Unstructured Data such as images or text.

To address this, LG AI Research made the following contributions.

• Proposed a data generation method that allows the ground-truth value of MI to be known precisely even for arbitrary datasets
• Built a benchmark covering various domains, including Gaussian, image, and sentence embeddings
• Compared MI Estimator performance based on 7 scenarios

Experimental results confirmed that existing Gaussian-based benchmarks have critical limitations in evaluating MI estimation accuracy in unstructured data settings. The study also revealed that the optimal MI Estimator can vary depending on the data domain and scenario, providing guidance for researchers.