What We Learned Using 2 Trillion Tokens for Category Classification

Danggeun's Taxonomy team built a Taxonomy Management System to automatically classify post categories and attributes using an LLM. The core execution infrastructure is Dataflow(Beam), the Source of Truth for inference results is kept in BigQuery, and when needed, results are loaded into the in-house feature platform via a Kafka sink.

Both taxonomy definitions and pipeline configurations are managed entirely in YAML, allowing the same structure to be reused across different domains such as secondhand trading and meetups. LLMs were also actively used for category creation and expansion, establishing a process that spans new category creation referencing existing well-known taxonomies, sample-based expansion, merging similar categories, promoting new candidates, and regression evaluation. With this approach, they were able to define not only the existing 3-depth, roughly 1,400 categories but also a taxonomy of 6-depth or more, at 10,000 scale.

Inference was operated by switching between strategies such as Single Shot, Hierarchical, and Two-stage depending on the situation. Attribute inference was chained after category inference, and each stage was modularized as a DoFn-based component to maintain the Article → Category → Attribute flow.

Evaluation was done using the LLM as a Judge approach. Labels were collected by running samples through multiple models, ground truth was created via majority voting, and categories were measured by accuracy while attributes were measured by precision/recall. To verify that this system was actually valid, correlation with a small-scale human-labeled goldset was also checked.

On the operational side, cost optimization was as important as accuracy. Including images together with posts improved classification accuracy, and repeated experiments with prompt engineering and model combinations helped find the sweet spot between accuracy and cost. In conclusion, taxonomy classification was not a one-time task but an operational problem that requires continuous measurement and adjustment in line with large-scale traffic and evolving domains.