Evaluating Amazon Bedrock Model Performance with promptfoo

When developing generative AI applications, it's difficult to be confident about how quality changes when you modify prompts or switch models. This is because the non-deterministic nature of LLMs makes it hard to judge quality using traditional unit testing methods.

promptfoo is a unit testing framework for evaluating LLM performance, allowing you to declaratively define prompts, models, and test cases in a YAML file and compare them all at once. In particular, using Amazon Bedrock is highly efficient because you can call and evaluate various models using only IAM authentication, without needing to issue separate API keys.

Evaluation methods are broadly divided into two types.

• Deterministic evaluation: Clear criteria such as JSON format validation, whether a specific string is included, and response latency are judged by code.
• Model-assisted evaluation: An LLM judge interprets criteria written in natural language and assigns a score. It can also evaluate subjective criteria like 'Is the answer friendly?', greatly reducing the effort of manual review.

This evaluation process can be integrated into a CI/CD pipeline to automatically halt the build when quality falls short.