Removing the Router Brought It Back to Life

Dynamically routing multi-timescale advantages (e.g., γ = 0.5, 0.9, 0.99, 0.999) into an Actor-Critic easily causes the policy to fall into irreversible policy collapse or get stuck in a strange local optimum.

The core of the problem is twofold.

• Surrogate Objective Hacking: When temporal attention is exposed to the policy gradient, optimization finds a shortcut that manipulates the attention weights instead of controlling the environment.
• Temporal Uncertainty Paradox: If you bypass this with a gradient-free approach, the router skews toward the short-term horizon where aleatoric uncertainty is low, resulting in a policy that is too short-sighted in delayed-reward environments.

The solution is Representation over Routing. Multi-timescale prediction is kept only on the Critic side to learn a strong auxiliary representation, while the Actor is updated using only the longest-term advantage.

With this decoupling, on LunarLander the agent doesn't get stuck hovering but learns fuel-efficient landing, and it claims to consistently exceed 200 points across multiple seeds. The implementation is organized as a pure PyTorch-based 4-stage MRE, making it possible to briefly reproduce the crash → hover → success progression.