GRPO on Three Mac Minis

Qwen2.5-0.5B-Instruct was trained with GRPO for a Reddit post summarization task, and the experiments were run on three Mac minis.

Two training approaches were used.

• Using length penalty only
• Using quality reward + length penalty together

Evaluation was conducted using LLM-as-a-Judge based on DeepEval. The scoring criteria were the following 4 items.

• Faithfulness: whether it stays true to the original text
• Coverage: whether it captures the key content without omissions
• Conciseness: whether it avoids unnecessary length
• Clarity: whether it is easy to read and clear

The results showed 2.5/4 for the model with quality reward added, and 2.4/5 for the model using only length penalty.

Also, evaluation was done with 200 test samples from the smoltldr dataset, and in a one-sided t-test based on 5 rounds of evaluation results, the ROUGE-L-based quality reward was significant for the final composite score with a p-value of 0.0042.

The reward design is summarized as follows.

• length_penalty: -abs(response_length - MAX_LENGTH)
• quality_reward: ROUGE-L against the reference summary

In other words, this is an update showing that the combination of quality reward + length penalty, rather than simple length control alone, further improved the summarization quality of the small model.