Kurly Automates Review Curation with LLM and Chain of Thought… Achieves Accuracy Without ML Models

Kurly's Commerce Product Team conducted an LLM-based review curation experiment to address the low coverage and timeliness issues that were limitations of the existing best review summary. The existing approach required substantial resources for building and tuning machine learning models, but this was replaced by applying Prompt Engineering and Chain of Thought (CoT) techniques.

LLM-Based Review Curation Strategy

For review curation, four conditions were defined: recency, relevance, whether it is a negative review, and level of informativeness. Simple filtering Prompts were effective for clear-cut cases, but reliability issues were raised due to a lack of grounds for judgment on ambiguous reviews. To address this, the Prompt was designed so that the LLM outputs its grounds for judgment (Reasoning) along with its conclusion.

Effects of Applying Chain of Thought

By introducing the CoT technique, which breaks down complex verification steps, accuracy and transparency were secured. Verification steps were explicitly specified as "checking whether it refers to a different product," "checking for negative content," and "checking whether specific attribute information is included," guiding the model through a reasoning process at each step. As a result, a process emerged in which the LLM checked its own logical consistency and corrected errors, reducing the risk of hallucination.

Practical Implications

This experiment demonstrated that unstructured text analysis tasks can be performed efficiently by leveraging the LLM's reasoning capability without developing a separate ML model. In particular, it confirmed that explicitly specifying the Reasoning process is a key factor that both improves the accuracy of results and enhances the explainability of the system.