How to Quickly Get Started with LLM Evaluation Using OpenEvals

Deploying LLM-based applications to production environments requires a reliable Evaluation process. However, building an evaluation system from scratch is a very challenging task.

To address this, LangChain has released new packages called openevals and agentevals. These tools provide a common framework and pre-built evaluators so developers can easily get started with evaluation without going through a complicated process.

The key features of OpenEvals are as follows:

• LLM-as-a-judge: Uses an LLM to evaluate chatbot conversation quality, hallucination in summarization and question-answering, and writing consistency, enabling objective evaluation even without Ground Truth.
• Structured Data Evals: Verifies that structured data—such as information extracted from documents or JSON for tool calling—conforms to a specified format.
• Customization: Pre-built prompts can be easily modified, or few-shot examples can be added to adjust evaluation criteria to match human preferences.
• Transparency: Generates reasoning comments explaining why a given score was assigned to evaluation results, increasing the transparency of the evaluation process.

Going forward, LangChain plans to expand the library to include evaluation techniques specialized for RAG applications and Multi-agent architectures.