How Well Do Agents Utilize Testing and Verification Techniques?

An analysis of the impact of 26 verification technique instructions, including TDD, Formal Methods, and Fuzzing, when evaluating a Zstd Rust implementation using the GPT-5.6 Sol model, revealed that most skills failed to improve performance or instead increased costs.

Performance Analysis by Verification Technique

• Formal Methods: Instead of actual code verification, it produced abstract proofs or modeling errors (e.g., 8-bit overflow) using tools like Verus, Alloy, and Lean 4. ACL2 evaluations were biased due to Out of Memory (OOM) errors.
• TDD: The iterative approach caused Overfitting, where both implementation and tests failed together, making it unsuitable for Zstd verification.
• Fuzzing/PBT: When using QuickCheck or Hegel, it resulted in simple smoke tests or random inputs. Applying Hegel increased token costs by 900,000 to 1,800,000 without improving accuracy.
• Differential Testing: Out of 160 attempts, only 10 generated structured inputs, while the rest were meaningless repetitions of random bytes.

Cost and Efficiency Issues

• Using verification skills incurred additional costs of 16% and 18% in Medium and XHigh modes, respectively.
• Instructions like 'Make no mistakes' showed no statistically significant difference from the default settings, and unnecessary instructions led to performance degradation.
• Descriptive skills in long-text form were less efficient than directive instructions that alter the model's base behavior distribution; they were mostly ignored or led to unintended directions.

Implications

• LLMs tend to maintain their existing behavior distribution due to a lack of software testing training data.
• Specific domain skills, such as API control, may be more useful than generic testing workflow skills.
• Confidence in existing prompts drops sharply when switching model versions (GPT-5.5→5.6), necessitating model-specific optimization.