smevals: An Evaluation Framework for Small and Large Models

smevals is a framework for evaluating specific capabilities of small and large models. High-level tasks such as text-to-SQL, image generation, and spec-compliance verification are defined as Evals, and each Eval consists of a collection of individual problems called Tasks.

The execution environment — including the model, system prompt, parameters, and tool settings — is managed as a Config. The results of running a Task with a Config are stored as an immutable record called a Run, and the same Task and Config can be run multiple times to check for non-deterministic results. Using smevals run -n 5 will run each Task up to 5 times.

If the execution program exits with a non-zero status due to a network error or similar issue, that Run is classified as a failed Run. Failed Runs remain on disk for debugging purposes but are excluded from grading and reports, and are not counted toward the repetition count.

A Grader is applied to a Run to produce a Grade. A Grader is a combination of Checks that run in sequence, and Checks range from simple verifications like text-containment checks to tasks such as rendering an SVG and having an LLM evaluate the resulting image.

• Checker: A named operation or reusable CLI program that performs each Check
• Required Check: If it fails, subsequent Check execution is halted
• Shared working directory: Files such as rendered images produced by an earlier Checker are used by subsequent Checks
• Grade: Records per-Check results, overall success/failure status, a numeric score, and notes for interpretation

Installation can be done via uv tool install smevals or pip install smevals, and it can also be run with uvx smevals --help.