TabFM: A Zero-shot Foundation Model for Tabular Data

Existing tree-based algorithms like XGBoost or Random Forest have the drawback of requiring complex hyperparameter optimization and feature engineering every time they are applied to a new dataset.

TabFM is a new foundation model that introduces In-Context Learning (ICL) to tabular data in order to skip this process. Without any separate weight updates, the model receives the entire dataset as a single prompt and directly interprets the relationships between rows and columns to make predictions.

This model operates through the following three core mechanisms:

• Alternating row and column attention: Applies cross-attention across rows and columns to learn complex feature interactions.
• Row compression: Compresses the rich information of each row into a single dense vector.
• In-context learning: Applies a transformer to the sequence of compressed vectors, increasing computational efficiency and enabling fast predictions even on large-scale datasets.

TabFM is currently available via Hugging Face and GitHub.