From RLVR to RLSVR (GitHub Repository)

RLSVR (Reinforcement Learning with Self-Verifiable Rewards) is a reinforcement learning method that, like self-supervised learning, transforms the task itself to create reward signals. It converts open-ended tasks into verifiable proxy environments, generating rewards through the environment's rules and interaction outcomes without a separate verifier.

Its implementation, SpyRL, is a multi-agent self-play environment inspired by 'Who Is the Spy?'. Agents perform the same task while holding asymmetric information, then vote to identify the spy; since the spy's identity is predetermined by the environment, the voting outcome can be used as a fully verifiable reward.

Key features are as follows.

• Extends self-play beyond domains with clear answers like math and code to summarization and creative writing.
• Maintains competition among agents through spy-identification rewards and vote-count-based zero-sum rewards.
• Normalizes the collective judgment of multiple players in a GRPO manner, reducing the impact of errors from a single verifier.
• Enables low-cost training by leveraging document-level data.

On Qwen3-8B, it achieved 75.4% and 77.3% A/B win rates in summarization and creative writing, respectively. For math reasoning, it improved the performance of Qwen3-4B and Qwen3-8B by 8.97% and 6.16%, respectively, across 7 benchmarks, while R-Zero and Absolute Zero showed only limited improvement on open-ended tasks.

The same repository's vision-zero branch also includes Vision-Zero, an image-pair-based VLM self-improvement method.