Learning Representations for Machine Activity Recognition

Abstract: Machine activity recognition (MAR) is an essential and effective approach for equipment productivity monitoring. Developing MAR methods for forklift trucks, a vital piece of the industry, can benefit productivity efficiency, maintenance service, product design, and potential savings. With the growth of the Internet of Things, a large amount of sensory data has become accessible. Conventional MAR methods that have been developed primarily focus on data collected from external sensors, such as inertial measurement units (IMUs) and cameras. However, they are not effective for forklift applications: the IMU data does not reflect kinematic patterns due to a lack of large articulated parts, while the vision-based data collection requires many cameras to create sufficient coverage of an indoor environment, which, in result, risks the privacy and is less economical. Moreover, typical objectives in the existing MAR works are heavy equipment in construction sites where the working environment and tasks differ from the logistics sector. Therefore, it is necessary to develop intelligent and innovative approaches that are more suitable for forklift trucks.This thesis demonstrates developing and utilizing representation learning methods to solve forklift MAR problems, based on the assumption that forklift activities are formed by a series of basic movements that can be detected from the onboard communication, i.e., signals in a Controller Area Network (CAN). Most of the methods proposed in this thesis incorporate semi-supervised techniques to deal with the limited amount of labeled data and to capitalize on a large amount of unlabeled data in our experiments. Deep neural networks are implemented to overcome different challenges of recognizing forklift activities and learn various representations of the data: i) learning invariant features to reconstruct input CAN signals by applying autoencoders, ii) learning discriminative features to recognize forklift activities by fine-tuning pre-training networks, and iii) learning temporal coherence to capture activity transitions by implementing gated recurrent units. Apart from achieving promising classification performance for forklift MAR problems, the representations obtained also support visualization and interpretability of the data as they are three-dimensional. Our ongoing works are new experiments about learning domain-invariant features, where domain adaptation methods are implemented to recognize activities performed by forklift trucks from different sites.