Automating Game Balancing with Reinforcement Learning

Balancing, a key factor determining the fun of a puzzle game, has limits when performed manually by humans due to the numerous variables and random elements involved. In particular, for Cookie Run: Puzzle World, considering all the various cookies, maps, and enhancement systems requires about 11.52 million matches of testing, which would take 8,000 days if a person played them directly.

To solve this problem, Reinforcement Learning was introduced. While rule-based bots struggle to follow complex puzzle rules and constantly updated logic, reinforcement learning can learn optimal play on its own through trial and error.

For efficient learning, the following technical approaches were attempted:

• Cython-based Mock Environment: The existing Unity ML Agent had slow learning speed, taking 3 days per experiment, but a Cython Mock environment implementing the core logic reduced the learning time to half a day.
• PPO (Proximal Policy Optimization): The PPO model, which converges efficiently in high-dimensional Action Spaces, was adopted as the agent.
• ResNet-based Policy Network: ResNet, which excels at image recognition, was used to recognize the state of the jellies.

To address convergence issues that arise during training, performance is being improved by reflecting the game's physical characteristics—such as 'matchability' rather than the jelly's color itself—in the model.