DiScoFormer: A Transformer That Estimates Density and Score Simultaneously

Existing kernel density estimation (KDE) suffers from reduced accuracy in high dimensions, while neural network-based score matching models have the drawback of requiring retraining whenever the distribution changes.

DiScoFormer leverages the Transformer architecture to simultaneously estimate a data's Density and Score with a single model. This model has the following characteristics.

• Shared Backbone Structure: By leveraging the mathematical relationship between density and score (score is the gradient of log density), it adopts a structure with a single backbone and two output heads.
• Use of Consistency Loss: Through a label-free consistency loss using the gradient difference between the density head and the score head, immediate adaptation is possible even for out-of-distribution data that was not trained on.
• Generalization of KDE: By leveraging the fact that the Attention mechanism is mathematically a generalized form of the Gaussian kernel, it combines the advantages of KDE with the neural network's capacity to handle high dimensions.

The research team trained the model using Gaussian Mixture Models (GMM), which provided accurate target values across various distributions and secured versatility.