PopuLoRA: Co-evolving LLM Populations for Reasoning Self-Play

Verifiable-reward-based RLVR grows LLM reasoning on tasks whose correct answers can be automatically verified, but relying on a fixed task distribution can lock the curriculum into being too easy and too narrow.

Single-model self-play exposes this limitation. In a structure where one model creates tasks and solves them itself, over time the valid tasks converge to a difficulty level that its own solver can already handle, and the solve rate approaches 100%. As a result, AST depth, cyclomatic complexity, lines of code, and variable count all decrease together.

PopuLoRA separates generation from solving.

• teachers: generate verifiable tasks like code_o, code_i, code_f
• students: solve the matched teacher's tasks through multiple rollouts
• reward: teachers are rewarded on valid tasks the student fails to solve, students are rewarded for passing the verifier

Every role is implemented as LoRA adapters on a shared frozen base model, so population training can run on a single machine simply by increasing the number of adapters. Matching is done via TrueSkill-based prioritized fictitious self-play, and invalid or non-deterministic tasks are filtered out. Teachers are penalized for invalid tasks, and a zero-reward floor is applied to tasks that no one can solve. During the replacement phase, mutation and crossover evolve the LoRA weight space, and new children mostly recovered parent-level reward within 10-20 update steps.

Training unfolds as an arms race where the solve rate does not rise monotonically but fluctuates. When the teacher finds harder tasks, the student wavers; when the student catches up, the teacher raises the difficulty again. This structure lifted the entire population, and even the weakest member outperformed the baseline. The 4T+4S configuration ran 8 adapters with 1.31x wall-clock overhead.

As a result, PopuLoRA outperformed the compute-matched single-agent baseline on HumanEval+, MBPP+, and LiveCodeBench, and also showed improvements on AIME, AMC, MATH-500, GSM8K, and OlympiadBench. However, the math performance should be interpreted as a transfer signal arising from code-only training, and it was not isolated which factor is the direct cause.