Using LLMs as Feature Engineering Tools Instead of Classifiers Improves Irony Detection Performance

LLM-as-classifier suffers from poor calibration of output labels, making precision/recall tuning difficult, and its built-in priors may mismatch the target distribution. To address this, we propose an approach that uses LLMs as feature engineering tools rather than final classifiers.

Methodology: Combining LLM Output with Logistic Regression We use the LLM output LLM(x) as an input feature for logistic regression. In the form p(y=1|x) = σ(α + β·LLM(x)), estimating α and β through training data converges to an empirical estimate. While β→∞ yields results identical to a standard LLM classifier, training provides the following benefits:

• Calibration/Threshold: Based on empirical proportions, it is calibrated in expectation, and threshold tuning becomes possible through actual probability outputs.
• Data Integration: Due to the nature of logistic regression, additional covariates can be combined, the model adapts to the baseline distribution via fit on training data, and target distributions can be addressed through reweighting.
• Interpretability: While the opacity of the LLM verdict itself remains, its contribution to the final decision becomes clear.

Case Study: Irony Detection (SemEval 2018 Task 3) We conducted irony detection experiments on 4,618 tweets using the gemini-3.1-flash-lite model. Using only a one-shot prompt yielded high performance with TPR 0.965 and F1 0.747, but calibration was poor with a Brier score of 0.259. After applying logistic regression to improve calibration, the Brier score improved to 0.175.

Performance Improvement and Comparative Analysis Beyond simple verdicts, we measured performance by adding 19 detailed dimensions extracted by the LLM (humor, realism, emotional mismatch, etc.) and rule-based features (reply status, URL inclusion, hashtag count, etc.).

• Verdict only: Brier 0.175 / F1 0.747
• + LLM features: Brier 0.131 / F1 0.768
• + rule-based features: Brier 0.127 / F1 0.779

On the SemEval 2018 Task 3A test set, the LLM hard label (F1 0.747) surpassed the competition-winning model (THU_NGN, F1 0.705). LLM features + Logistic Regression (F1 0.779) showed competitive performance overlapping with the confidence interval of the subsequent research SOTA (NTUA-SLP, F1 0.786).

Conclusion and Implications Treating LLMs as feature extractors rather than simple classifiers allows us to secure both the convenience of stock ML algorithms (calibration, data integration) and the performance of LLMs. The authors propose future directions including agentic classifiers where LLMs assess the rigor of their own investigation process instead of using fixed features, and statistical inference through reliable test sets.