Introducing Align Evals: Streamlining the LLM Application Evaluation Process

One of the biggest challenges in developing LLM applications is that LLM-generated evaluation scores don't align with the quality perceived by actual humans. This misalignment provides misleading signals and wastes development time.

To address this, LangSmith has introduced Align Evals, a feature that calibrates evaluation models to human preferences. This feature provides a playground interface for iteratively refining evaluation prompts and a capability to compare LLM scores against human grading data.

Key features of Align Evals:

• Playground interface: Modify evaluation prompts while checking the 'Alignment Score' in real time
• Side-by-side comparison: Compare human grading results and LLM scores side by side, with an alignment feature to identify mismatched cases
• Baseline comparison: Compare the alignment score of the previous prompt version against the current version to check for improvement

Users can set evaluation criteria, create a 'Golden Set' by manually scoring a representative dataset, and then use it as a benchmark to optimize the LLM evaluation model. Going forward, analytics for tracking evaluation performance and automatic prompt optimization features are planned to be added.