Training a Coding Model to Paint Watercolors Using TRL and OpenEnv

An open-source project has been released that reproduces Surya Narreddi's viral watercolor generation project (using p5.brush) from an engineering perspective, utilizing TRL (Transformers Reinforcement Learning) and OpenEnv. This project applies human preference-based aesthetic rewards (RLHF) rather than verifiable rewards (such as math or code) to guide the model in learning specific artistic styles.

Reward Function and Training Environment

The reward function consists of four components and calculates scores by rendering the model's JavaScript code output in headless Chromium.

• Gate (0.05): Compilation success and anti-cheating measures
• Length (0.05): Code length guidance
• Pairwise Judge (0.60): Compares style with reference images using Qwen3-VL-30B-A3B-Instruct
• HPSv3 (0.30): Evaluates aesthetic scores via an open-source 7B preference model

The training dataset (Pool) consists of 178 human-curated images selected from sketches generated by four open-weight models, divided into two tiers, 'love' and 'okay', to provide initial training signals.

Technical Optimizations and Results

During the reproduction process, the LoRA configuration was modified to account for the MoE (Mixture of Experts) architecture characteristics of the Qwen3.5-35B-A3B model. Instead of the typical target module list for dense models, all-linear was applied to ensure the adapter reaches all linear layers. The learning rate was increased to 5e-5 and the scheduler was changed to constant_with_warmup to ensure training stability.

Experimental results showed that in runs with a higher weight for the Pairwise Judge, the reward score rose from 0.45 to 0.72, confirming that the human-curated pool successfully guided the policy. All code, datasets, and trained models are publicly available on the Hugging Face Hub.