Detecting GPT-4's Errors Using GPT-4

As models like GPT-4 advance, their errors are becoming more subtle, making it increasingly difficult for human trainers to identify them during the RLHF (Reinforcement Learning from Human Feedback) process. This acts as a fundamental limitation that hinders model Alignment.

To address this, CriticGPT was developed to catch errors in ChatGPT's code outputs. CriticGPT is a model trained to write critiques that point out inaccurate parts of answers.

Experimental results demonstrated the following achievements for CriticGPT:

• When human trainers were assisted by CriticGPT, they performed 60% better than when working without assistance.
• It produces fewer unnecessary nitpicks and less Hallucination compared to ChatGPT's self-critiques.
• For naturally occurring errors, trainers preferred CriticGPT's critiques at a rate of 63%.

CriticGPT can adjust the balance between precision and recall through test-time search, generating critiques optimized for RLHF. However, it is currently trained primarily on short code, and handling complex, long tasks remains a challenge for the future.