Development of Harmful Content Detection Model Based on Open Chat Titles and Descriptions

The LINE AI Service Lab team developed a model that automatically identifies harmful content by analyzing the names and descriptions of Open Chat rooms. This project addresses the issue where previous models failed to meet the granular judgment criteria of certain countries, necessitating manual review.

During the data cleaning phase, cases where different penalties were assigned to the same name and description were handled. If the most severe penalty was applied two or more times, it was adopted as the final label; if applied only once, it was considered noise and excluded. When penalty reasons differed, a TF-IDF-based approach was applied to select the rarer reason across the entire dataset.

For model selection, Granite Guardian 3.1 2B was chosen as the final model. Released by IBM Research, this model classifies harmfulness by comparing the generation probabilities of the 'Yes' or 'No' tokens in response to the question 'Is the input harmful?'. The decoder model was selected over traditional encoder models because recent decoder models demonstrate strong performance even in classification tasks. Additionally, the selection was driven by its compliance with the Apache License, allowing for commercial use, and its lightweight 2B scale, which is suitable for real-time monitoring.