AI migrated legacy COBOL programs to Java, but bugs came along for the ride

To verify functional mismatches that occur when converting legacy COBOL programs to Java, researchers proposed an agentic test-synthesis method called 'Locksmith Loop'.

After instrumenting and mocking both the original COBOL and the generated Java code, the two programs are executed on general-purpose hardware outside the mainframe. The AI agent explores input mocks to discover new branch paths and repeatedly generates mutations that maintain parity with existing behavior.

When branch exploration gets stuck, an analyzer identifies a 'Locked Paragraph'—a condition blocking deeper exploration—and generates additional test inputs based on it.

• The method was evaluated on 2 open-source programs and 1 internal production-like COBOL program.
• Target sizes ranged from 430 to 4,114 source lines.
• Near-complete coverage was reached for the 2 open-source programs.
• The internal program achieved 91.90% branch coverage.
• Across all approved test cases, the generated Java deterministically matched the COBOL reference implementation.

The findings present a method for using the original legacy program as a deterministic oracle when verifying the correctness of code generated by AI coding agents. However, this is an early-stage result limited to evaluation on just 3 programs.