Ollama Model Tester (GitHub repository)

Ollama Model Tester is a lightweight CLI tool that runs the same prompt against local Ollama models and saves all responses to disk, allowing you to compare results across different models or across repeated runs of the same model.

This tool uses only the Python standard library, giving it a dependency-free structure that requires no separate pip install step. It requires an environment running Python 3.7 or higher and a locally running instance of Ollama.

Users can utilize the tool in two ways:

• Interactive mode: sequentially enter model selection, prompt input, number of runs, temperature, and whether to stream
• Command-line flags: automate the process script-style using options such as --model, --runs, --temperature, --prompt-file

Results are saved in the ollama-runs/ directory, grouped by prompt hash. When different models are run against the same prompt, the results are collected in the same folder, making it very easy to compare response differences across models at a glance. Each file records not only the response content but also Ollama's metadata, such as token count and execution time.