Ollama Cloud Benchmark

For 3 prompts focused on medical reasoning, deepseek-v3.2:cloud, deepseek-v4-pro:cloud, deepseek-v4-flash:cloud, and glm-5.1:cloud were compared.

• Each combination was run 3 times each, for a total of 36 runs.
• Settings were temp=0.3, top_p=0.9, max_tokens=2000, calling /api/generate through the local Ollama gateway.
• Transient errors were handled with 1 automatic retry (5-second delay).

Performance varied widely by model.

• deepseek-v4-pro:cloud: averaged 124.8s, the slowest, but had the highest average token count at 3,149.
• deepseek-v4-flash:cloud: averaged 67.7s, with 0 hard fails and 0 silent retries, making it the most stable.
• deepseek-v3.2:cloud: averaged 55.1s, with 1 hard fail and 2 silent retries.
• glm-5.1:cloud: averaged 101.8s, with the highest throughput at 53.8 tok/s, but had 1 hard fail.

Overall, transient Ollama Cloud issues occurred in 6 out of 36 runs (17%).

• 3 were HTTP 500 errors, which recovered after 1 retry.
• 3 hit the 240-second timeout, and retries also failed.

Due to cloud endpoint constraints, token throughput was calculated based on Ollama's total_duration.