Validating AI Agents' Ability to Migrate Java Frameworks

Unlike existing code generation benchmarks, ScarfBench is a practical benchmark that goes beyond simple code transformation to evaluate whether an application actually builds, deploys, and maintains its behavior.

Key features are as follows:

• Target Frameworks: Performs migration between Spring, Jakarta EE, and Quarkus
• Evaluation Metrics: Measured not by whether code is generated, but by build success, correct deployment, and Behavioral Validation
• Data Scale: Includes 34 applications, 204 migration tasks, approximately 150,000 lines of code, and 1,331 expert-written tests

When tested on state-of-the-art AI agents, unlike their performance on existing software engineering benchmarks, the behavioral validation success rate for actual framework migration was under 10%, revealing a significant gap between code generation and maintaining actual application behavior.