NVIDIA Unveils Lightweight RL Framework Molt

Molt, developed by NVIDIA NeMo Labs, is a framework for agent-centric RL (reinforcement learning) post-training. To address the difficulties in research and rapid iteration caused by the complex code structure of existing Megatron-based frameworks, it minimizes the core RL code to around 9,000 lines, maximizing readability and ease of modification.

Key features are as follows:

• Scalability: Supports MoE (Mixture-of-Experts) models ranging from Qwen3.5-397B to GLM-5.2 753B in scale.
• Simple Stack: Implemented using only Ray, vLLM, and NVIDIA AutoModel/FSDP2 instead of Megatron, making the structure intuitive.
• Asynchronous Architecture: Maintains high throughput through an asynchronous loop composed of vLLM (Rollout), Agent (Python-based environments), Ray (Queue), and RL Trainer (FSDP2-based updates).
• Flexible Reward Design: Arbitrary Python code can be used as reward functions, including rule-based, heuristic, LLM-as-judge, and tool return values.

This framework maintains a token-level data path to ensure data alignment from rollout to training, and is designed to allow researchers to rapidly experiment with ideas even on large-scale models.