vLLM V0 to V1: Correctness Before Correction in RL

In GSPO RL moved to vLLM V1, rollout logprob and runtime behavior diverged from V0, causing clip rate, KL, entropy, and reward to fluctuate.

There were four key fixes.

• Turned on processed_logprobs to align logprob with the processed distribution used by the sampler.
• Explicitly set enable-prefix-caching=false and async-scheduling=false to remove V1 default differences.
• Used pause_generation(mode="keep", clear_cache=False) to bring inflight weight update behavior closer to V0.
• Applying fp32 lm_head aligned the final projection path, bringing reward close to the V0 reference curve.

processed_logprobs alone left mismatches, and lag and clip rate issues persisted even in batch invariance tests. The same kind of problem can appear in online RL such as PPO and GRPO. The conclusion is that rather than layering RL corrections on top, one should first verify that the rollout backend satisfies the trainer's assumptions.