Study on the Mismatch Between Reconstruction Error and Downstream Performance

A study has been published showing that even when the Reconstruction error of compressed data is low, the performance of the actual Downstream task using that data can degrade sharply. Through a Pre-registered study spanning 12 domains, the researchers confirmed that a code optimized for a specific task can show worst-case performance on another task.

The key findings of the study are as follows:

• Performance reversal phenomenon: Under the same Bit budget, even when performance is equivalent in terms of reconstruction error, the task winner can flip depending on the Sensitivity required by the consumer.
• Limits of loss-based compression: In a 4-bit KV-cache quantization case, the reconstruction based on Cosine similarity was very high at 0.995, but a mismatch occurred where the actual Perplexity worsened by about 3 orders of magnitude (1,000x).
• New approach: The study opens a new horizon in Task-oriented communication by presenting a method to recover the Read subspace through black-box consumer queries, along with a consumer-relative R(D) theory (Rate-Distortion theory).

This research suggests the importance of Loss-aware compression to maximize the performance of specific AI models, going beyond simple data restoration.