Strengthening a Summarization Model with GRPO

Trained the Qwen2.5-0.5B-Instruct bf16 model with GRPO for a Reddit post summarization task, and yesterday's run recorded an average rollout length of about 64 tokens.

The reward design had two branches.

• length_penalty: -abs(response_length - MAX_LENGTH)
• quality_reward: maintaining the structure and quality of the summary based on ROUGE-L

At first there was a mistake of counting characters as tokens, and after fixing this, the plan is to check again whether the length reward alone games the system, or whether output quality collapses.

The setup ran on a cluster of 3 Mac minis.

• 1 node: handles GRPO training
• 2 nodes: generate rollouts with vLLM

Two variants were compared.

• length penalty only baseline
• length penalty + quality reward combination (BLEU, METEOR, ROUGE-L family)

Evaluation was conducted using the LLM-as-a-Judge method.

• Evaluation model: gpt-5
• Framework: DeepEval
• Evaluation axes: faithfulness, coverage, conciseness, clarity

In short, this is an experiment update that combines reward design, distributed rollouts, and automated evaluation to improve summarization performance on a small Qwen 0.5B model.