DAM-VLA: A Vision-Language-Action Framework for Robot Manipulation Based on Dynamic Action Models

Robot manipulation requires both generality that is robust to environmental changes and precision in specific tasks. Existing specialist policies such as ACT and Diffusion Policy were precise but weak in generality, and recent VLA models such as RT-2 and OpenVLA have extended language and visual understanding to action prediction.

DAM-VLA does not treat arm movement and gripper manipulation in the same way; instead, it branches the action type through the reasoning of a VLM and then invokes an action model suited to each situation. The paper identifies the key difference as: arm movement is a global task that occupies a large portion of the field of view, whereas gripper manipulation is a local task that focuses on a small region.

The architecture consists of three parts.

• The VLM encodes the observed image and language instruction to produce cognition latent and reasoning latent.
• The Action routing module generates weights w between the arm-movement model and the gripper-manipulation model based on task context.
• The Dynamic action model combines the VLM's high-level cognitive information with vision tokens to predict a temporal action sequence.

Additionally, dual-scale action weighting is used to coordinate the collaboration between the two action models at both the trajectory level and the action-chunk level. Rather than simply loosely connecting a VLM and an action model, this is a structure that uses the VLM's reasoning ability to finely determine which action should be handled by which model.

For performance evaluation, experiments were conducted in SIMPLER simulation and real-world pick-and-place tests. On the Google robot in SIMPLER, it achieved average success rates of 83% and 81% in both the VM and VA settings, and on the WidowX robot, it achieved the highest performance with an average of 71% under the VM setting.

The improvement margin was particularly large for highly task-specific tasks such as Open Drawer and Place Apple, and in the VA environment where diverse backgrounds, lighting, and distractors are added. In real-world environments as well, it showed a higher success rate than CogACT, and operated stably in both in-distribution and out-of-distribution settings.

As a result, DAM-VLA is presented as a framework that dynamically combines the semantic understanding of a VLM with specialized diffusion-based action models, jointly improving manipulation precision and generalization in dynamic environments.