Reinforcement Learning for Recursive Language Models (RLM)

Qwen3.5-4B was fine-tuned with RL so that a single shared policy learns both the parent and child roles of an RLM. Child rollouts directly inherit the advantage of the parent rollout, and the loss is computed by dividing by the number of children, allowing the model to learn recursive behavior without a separate reward.

The RLM operates inside a Python REPL. The model writes code to explore documents, and submits answers or calls sub-RLMs using FINAL, FINAL_VAR, rlm_query, and rlm_query_batched. Default tools include list_papers, search, extract_section, and get_paper_abstract, enabling the model to quickly locate supporting passages across multiple papers.

For data, papers selected from alphaXiv were grouped with up to 9 similar papers each, ground-truth evidence was generated using an OCR model, and a total of 1,000 synthetic queries were created. Groups contained up to 10 papers, and up to 3 questions were created per group. At test time, instead of OCR, noisy text from a PDF parser was used to mimic real-world operating conditions. For this task—requiring dynamic retrieval of variable-length source snippets—RLM was better suited than RAG, which retrieves a fixed top-k.

Several factors were key during training.

• Without a task strategy embedded in the prompt, even Claude Sonnet 4.6 took about 90 seconds to generate a rollout; with the strategy provided, this dropped to 30 seconds.
• Single-paper tasks required the retrieval strategy to be explicitly stated in the prompt, and without cold-start SFT, Qwen3.5-4B's pass@16 was effectively zero.
• SFT was performed only on a small scale, using rollouts generated by Qwen3.5-397B-A17B after filtering out REPL errors and cases with 0 F1.
• Because the structure re-appends the user prompt at every turn, a rollout could not be used as a single sample; each turn was decomposed into a separate sample and GRPO was applied.
• Rubric-based LLM judges proved more stable as a reward signal than verifiable signals like F1.

With this approach, the judge score for single-paper tasks rose from about 0.6 to 0.8. For multi-paper recursive tasks, instead of training only the root or using a frozen model, the child was also trained on-policy, so a single policy learned both the decomposer and sub-agent roles. As a result, a 4B model RL fine-tuned for the evidence selection task within the same RLM harness and REPL environment matched the performance of Claude Sonnet 4.6, while being far smaller in size and cost.