People tracking by mobile robots using thermal and colour vision

Abstract: This thesis addresses the problem of people detection and tracking by mobile robots in indoor environments. A system that can detect and recognise people is an essential part of any mobile robot that is designed to operate in populated environments. Information about the presence and location of persons in the robot’s surroundings is necessary to enable interaction with the human operator, and also for ensuring the safety of people near the robot.The presented people tracking system uses a combination of thermal and colour information to robustly track persons. The use of a thermal camera simplifies the detection problem, which is especially difficult on a mobile platform. The system is based on a fast and efficient samplebased tracking method that enables tracking of people in real-time. The elliptic measurement model is fast to calculate and allows detection and tracking of persons under different views. An explicit model of the human silhouette effectively distinguishes persons from other objects in the scene. Moreover the process of detection and localisation is performed simultaneously so that measurements are incorporated directly into the tracking framework without thresholding of observations. With this approach persons can be detected independently from current light conditions and in situations where other popular detection methods based on skin colour would fail.A very challenging situation for a tracking system occurs when multiple persons are present on the scene. The tracking system has to estimate the number and position of all persons in the vicinity of the robot. Tracking of multiple persons in the presented system is realised by an efficient algorithm that mitigates the problems of combinatorial explosion common to other known algorithms. A sequential detector initialises an independent tracking filter for each new person appearing in the image. A single filter is automatically deleted when it stops tracking a person. While thermal vision is good for detecting people, it can be very difficult to maintain the correct association between different observations and persons, especially where they occlude one another, due to the unpredictable appearance and social behaviour of humans. To address these problems the presented tracking system uses additional information from the colour camera. An adaptive colour model is incorporated into the measurement model of the tracker to improve data association. For this purpose an efficient integral image based method is used to maintain the real-time performance of the tracker. To deal with occlusions the system uses an explicit method that first detects situations where people occlude each other. This is realised by a new approach based on a machine learning classifier for pairwise comparison of persons that uses both thermal and colour features provided by the tracker. This information is then incorporated into the tracker for occlusion handling and to resolve situations where persons reappear in a scene.Finally the thesis presents a comprehensive, quantitative evaluation of the whole system and its different components using a set of well defined performance measures. The behaviour of the system was investigated on different data sets including different real office environments and different appearances and behaviours of persons. Moreover the influence of all important system parameters on the performance of the system was checked and their values optimised based on these results.