AI BriefingMulti-Agent Reinforcement Learning for Real-World Deployment
Key point
LG AI Research proposed the AOCC architecture, which improves the efficiency of asynchronous multi-agent reinforcement learning (MARL).
Details
In real industrial settings, asynchronous situations frequently occur where each agent's action start times or durations differ from one another. Existing asynchronous MARL research has mainly focused on how to process data collected asynchronously.
Most MARL follows the CTDE (Centralized Training Decentralized Execution) structure. This approach trains a Decentralized Actor that decides actions and a Centralized Critic that evaluates value, separately. In particular, in MacDec-POMDP environments where each agent performs macro actions of different lengths, constructing histories is tricky because the observation timings among agents do not align.
Existing research used the Mac-JERTs method, which integrates the observations of all agents, but this risked distorting histories by duplicating data for agents that lack the most recent observations.
To address this, LG AI Research proposes the AOCC (Agent-Oriented Centralized Critic) architecture. This structure has the following features.
- Agent-oriented history encoder: Encodes each agent's individual history (Mac-CERTs).
- Aggregation module: Integrates the encoded information of each agent.
- Positional encoding: Adds positional information to each timestep to prevent confusion in observation timing across agents.
Through this design, histories are systematically organized without duplicate observations, enabling more effective learning of both the value function and the action policy.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.