Transformer for Tabular Data? Not a Transforming Robot

Tabular data is the data type most frequently encountered in practice, but modeling it is tricky because column meanings and distributions vary widely and numerical and categorical types are mixed together. This is also why traditional models like XGBoost and LightGBM have maintained their strength for so long.

The change appearing here is TabPFN (Prior-data Fitted Network). It is pretrained in advance on millions of synthetic tabular datasets, and when a new table comes in, it quickly infers based on those patterns. In other words, instead of training a new model for every problem, the field of table prediction is moving toward leveraging a pretrained Foundation model.

The core of TabPFN is that the boundary between training and prediction becomes blurred. The training data and the data to be predicted are put together into a single context, and all rows and columns reference each other to interpret relationships. Thanks to this structure, predictions can be made in a relatively short time without a separate, lengthy training process.

In practice, this is especially meaningful in the following situations.

• When data is scarce: It becomes a candidate for a stable baseline when samples are insufficient, such as for new stores, new categories, or specific events.
• When fast decision-making is needed: Since tuning and retraining don't take long, it can speed up initial validation.
• When a comparison model is needed: It can be used as a reference model to check whether an existing operational model has actually improved.

However, TabPFN does not completely replace existing models. When data is large and there are sufficient resources for feature engineering and tuning, XGBoost and LightGBM can still deliver better performance. The message is clear. The field of tabular data is also gradually shifting from "models trained anew each time" toward "bringing in a pretrained model to use."