AI BriefingStudy finds LLMs break 32-51 working PHP files during modernization
·2026.11.02 06:22
Key point
In a single-file test of 100 WordPress 4.0 files, the best LLM left only 59 passing behavioral checks compared to 83 in the unchanged originals.
Details
- Five LLMs were evaluated on modernizing 100 legacy PHP files to PHP 8.3 without project context or execution feedback.
- The best LLM resulted in only 59 files passing all 630 hidden behavioral checks, whereas 83 files passed when left unchanged.
- Each LLM broke 32-51 working files while repairing at most 10, significantly underperforming the rule-based tool Rector, which broke 7 and repaired 6.
- In files requiring no changes for PHP 8.3 compatibility, LLMs broke 36-60 percent of the code, compared to 8 percent for Rector.
- Most regressions stemmed from files failing to load due to type declarations rejected by PHP, errors that do not require project context to avoid.
- The study recommends that evaluations of LLM-based modernization measure against a no-change baseline and report modernization and behavioral preservation separately.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Researchers demonstrate data-free self-improvement in LLMs using on-policy self-training
NeurIPS 2026 paper proposes sampling coherent but cognitively unavailable research directionsTraining 1,000 Weights in Gemma 3 270M Unlocks Hidden Sorting and JSON Skills
Researchers Demonstrate Remote Code Execution Backdoors in LLM Agents via Pretraining Poisoning