N-layered feudal network in an RTS game environment

Abstract

In an RTS, players act simultaneously in adversarial conditions. The agent must gather and remember information about the map and the opponent, while making decisions with long term consequences. FeUdal networks (FuN) tackle the domain using hierarchical reinforcement learning. A Manager sets goals for a Worker, which is intrinsically rewarded for accomplishing the goal. To explore the effectiveness of FuN, we propose an N-Layered FuN (NL-FuN), which generalises the Manager to fit an arbitrary number of tiers (3 tiers in this case). This requires adapting FuN to the StarCraft 2 domain. In addition, Atari-Net, FullyConv, and FullyConvLSTM are recreated and used as a baseline. Agents are implemented using PySC2 and TensorFlow. The scenarios used for training are: MoveToBeacon, DefeatSingleZealot, and BuildMarines. BuildMarines is the most complex of the three, having sparse rewards and requiring long-term planning. NL-FuN performs poorly in MoveToBeacon and similarly to FullyConv and FullyConvLSTM in DefeatSingleZealot but obtains a higher maximum reward than the baseline agents trained by DeepMind in BuildMarines in less time steps (540,000 vs 600,000,000).