Machine Learning Applied to Credit Scoring

The 8Percent CSS (Credit Scoring System) team experimented with machine learning models to more precisely evaluate borrowers' credit. Credit scoring was treated as a classification problem predicting delinquency rather than actual default, and this piece introduces a model-centric approach.

The data consisted of personal information, loans, delinquency, and card information received from CB (Credit Bureau), along with the company's own indicators. CB data had the characteristic of having a Special Value for each variable; for example, ABC001 (card limit utilization rate within 60 days) used different special values depending on conditions such as the number of credit card institutions and the total credit limit sum, which helped infer card policies while also preventing division-by-zero issues.

Model building proceeded in three stages.

• Variable selection: mrmr was used to increase correlation between independent and dependent variables while reducing redundancy among variables.
• Data preprocessing: Categorical conversion, missing value handling, and date conversion were performed, and Min Max Scaling was also applied for neural networks.
• Special Value / Categorical handling: Special Values were separated and encoded in a manner such as (v, 0, 0), and categorical variables used One Hot Encoding.

Logistic Regression was used as the traditional baseline, but more complex machine learning models were also reviewed for performance. Among tree-based models, Decision Tree, Random Forest, Gradient Boosting, and XGBoost were tried. Gradient Boosting is described as a method that sequentially learns residuals, while XGBoost is explained as an implementation that leverages second-order approximation of the loss function and regularization.

Among neural network-based models, MLP, ResNet, and Transformer were trained. The Transformer was adapted for the classification problem by placing an MLP instead of Cross Attention, and referencing TabTransformer and FT-Transformer, it was designed to embed Special Values separately, then reduce the dimensionality of the Transformer output and produce the final prediction via MLP.

As a result, among the neural network-based models, the applied Transformer model performed best, but overall it fell short of XGBoost. The authors note that in tabular classification, tree-based models often outperform deep learning, and conclude that they will continue to explore various architectures and the potential use of LLMs for better risk control going forward.