Extending Reinforcement Learning FeUdal Networks

Abstract

In a publication by DeepMind, FeUdal Networks for Hierarchical Reinforcement Learning, a technique known as FeUdal Reinforcement Learning, involving running multiple Reinforcement Learning algorithms at different levels of spatial or temporal resolution, was extended by applying deep learning with the same principles. The two different layers of networks in the architecture operated at different time scales, allowing a higher level manager network to run at a more abstract "overarching" level. FeUdal networks achieve significantly higher scores than other deep reinforcement learning architectures on Atari games that are traditionally difficult, such as Montezuma's Revenge. In this poster, we explore two extensions to this architecture and determine how they affect performance. The first extension we look at is utilizing more than two layers of networks. The second is to experiment with how the modularity of this design can be exploited. We look at whether pretrained low-level networks can be plugged into a manager network to improve training time, and whether manager networks can be switched between environments to allow for transfer learning.