Reinforcement Learning of Locomotion based on Central Pattern Generators

Abstract: Locomotion learning for robotics is an interesting and challenging area in which the movement capabilities of animals have been deeply investigated and acquired knowledge has been transferred into modelling locomotion on robots. What modellers are required to understand is what structure can represent locomotor systems in different animals and how such animals develop various and dexterous locomotion capabilities. Notwithstanding the depth of research in the area, modelling locomotion requires a deep rethinking.In this thesis, based on the umbrella of embodied cognition, a neural-body-environment interaction is emphasised and regarded as the solution to locomotion learning/development. Central pattern generators (CPGs) are introduced in the first part (Chapter 2) to generally interpret the mechanism of locomotor systems in animals. With a deep investigation on the structure of CPGs and inspiration from human infant development, a layered CPG architecture with baseline motion generation and dynamics adaptation interfaces are proposed. In the second part, reinforcement learning (RL) is elucidated as a good method for dealing with locomotion learning from the perspectives of psychology, neuroscience and robotics (Chapter 4). Several continuous-space RL techniques (e.g. episodic natural actor critic, policy learning by weighting explorations with returns, continuous action space learning automaton are introduced for practical use (Chapter 3). With the knowledge of CPGs and RL, the architecture and concept of CPG-Actor-Critic is constructed. Finally, experimental work based on published papers is highlighted in a path of my PhD research (Chapter 5). This includes the implementation of CPGs and the learning on the NAO robot for crawling and walking. The implementation is also extended to test the generalizability to different morphologies (the ghostdog robot). The contribution of this thesis is discussed from two angles: the investigation of the CPG architecture and the implementation (Chapter 6).