Reef Infra Releases RL Training Recipe Based on Conversation Logs

Reef Infra has released the OpenClaw-RL recipe, which reconstructs recorded inference traffic to utilize user feedback as reinforcement learning (RL) signals. This method processes learning data in batches by passing completed conversation turns and subsequent states to a separate PRM (Process Reward Model) judge.

Learning Mechanism

• Two update paths: Judged turns are split into PPO-style updates and on-policy distillation terms to update parameters.
• Feedback interpretation: Converts ambiguous user instructions, such as "maintain steps instead of bullet points," into explicit learning signals.

Deployment Considerations: Session Matching

• Session identifier required: Calls within the same conversation must include a stable x-reef-tag-session value.
• Risk of matching failure: Without a session tag, the client attempts to match using transcripts resent by the client, but matching may break due to conversation compaction or sliding windows. Unmatched turns are not collected as learning data, which can result in a situation where traffic is normal but no learning data exists.

Experimental Results and Limitations

• The repository includes a simulation experiment where a student refuses bold text or lists. In the recorded run, the agent achieved three consecutive first-answer passes in the 14th session.
• This is merely the result of one run out of the first 36 sessions and is not evidence that arbitrary user preferences converge quickly.
• This path requires a trainable model, GPU stack, and PRM, and separates feedback inference, session matching, and parameter updates, making debugging easier.