[NeurIPS 2024 Series] B-splines-based Input Normalization for Tabular Data

Tabular Data has the characteristic that the distribution is very heterogeneous for each column (Feature). Existing methods such as Min-Max, Standardization, and Quantile Transformation are fixed approaches that cannot be optimized during the training process of Neural Networks, and finding a suitable normalization method individually for every variable consumes a great deal of resources.

To overcome these limitations, LG AI Research's DI Lab proposed Learnable Numerical Input Normalization technology using B-spline. This method uses B-spline, a parameterized curve, to dynamically adjust the normalization curve according to the learning difficulty and importance of the data.

The key principles are as follows.

• Dynamic Curve Optimization: The parameters of the B-spline are optimized simultaneously with the training of the Neural Networks to generate a curve suited to the data characteristics.
• Importance-based Weighting: Samples with low training loss are judged to be of low importance, so the slope of the curve in that region is lowered, mapping them to a narrower region of the normalized input space.
• Stability Assurance: The Knot Vector is maintained in a non-decreasing form to preserve the order of the raw data and ensure the stability of normalization.

This technology can be combined with state-of-the-art Transformer-based architectures such as FT-Transformer and T2G-Former, as well as Periodic Embedding, to achieve more efficient Tabular data learning. Experimental results on actual OpenML datasets demonstrated superior performance across 26 datasets.