Promptimus: Automatically Optimizing Already-Good LLM Prompts

Promptimus is an automatic optimization method that further elevates already well-crafted LLM prompts without manual work. With just a few samples, it re-optimizes prompts refined on a source model to fit a target model, iteratively improving them based on user-defined performance metrics.

The inputs are a target LLM, an initial prompt template, a JSONL dataset of 20-50 samples, and a user-defined Python metric function. Samples don't need ground-truth answers, and the data is split into a dev set and a holdout test set to separate optimization from validation.

The process runs as a 4-stage loop.

• Evaluation: Measures baseline performance on the dev set, and a metric-analyzer generates checkpoint functions to break down failure points in detail.
• Feedback Generation: Analyzes bottleneck checkpoints together with success and failure cases to identify causes and directions for correction.
• Strategy and Revision Generation: standard mode produces full rewrites, while edit mode produces localized fixes based on find-and-replace.
• Candidate Evaluation: Runs the candidates, compares scores, and uses the best one as the starting point for the next iteration.

edit mode is especially well-suited to long prompts where structure matters, such as API schemas, regulations, or domain classification taxonomies. Even on prompts in the 50K-100K token range, it typically fixes only about 3-5 spots totaling 500-1,000 tokens, and its programmatic matching—handled in three stages of exact, whitespace-normalized fuzzy, and similarity matching—achieved a 97.3% success rate without any LLM calls.

In experiments, the same optimizer model and evaluation budget were used, with Claude Sonnet 4.6 fixed as the target model, and results were compared as averages over 5 random seeds. Out of 20 public benchmarks, it achieved the best performance on 16, tied on 1, and scored an average of 0.792, ahead of the previous best combination's 0.765. Notably, on multimodal tasks, edit mode boosted performance while preserving the existing prompt structure.