Naruto Mobile: AI Sparring Partner Using Heterogeneous Deep Reinforcement Learning

Naruto Mobile is a popular mobile Fighting Game with over 100 million registered players. AI agents are deployed extensively to the game for a wide variety of applications such as level challenges and player training, which require them to fight like humans and imitate strong and weak players. Although deep reinforcement learning is an excellent approach to creating agents with diverse behaviors, it is difficult to apply to massive-scale games like Naruto Mobile which is built on a pool of more than 300 characters that have unique skills, speed, and attack range, as a traditional approach of self-play training at such scale may require a substantial computational cost and training time.In this paper, we present a new AI training approach called Heterogeneous Exploitation Self-Play (HESP) to improve AI agent generalization ability in Naruto Mobile and optimize its massive-scale self-play training so that the computational costs and train time are significantly reduced. The proposed algorithm has already been employed by the development team of Naruto Mobile to create AI agents, which, at the time of writing this paper, have been used in more than 300 million human-AI fighting matches. To the best of our knowledge, this is the first time that deep reinforcement learning has been employed by a commercial fighting game.