Lip-motion biometrics for audio-visual identity recognition

Abstract: Biometric recognition systems have been established as powerful security tools to prevent unknown users from entering high risk systems and areas. They are increasingly being utilized in surveillance and access management (city centers, banks, etc.) by using individuals' physical or biological characteristics. The present study reports on the use of lip motion as a standalone biometric modality as well as a modality integrated with audio speech for identity and digit recognition.First, we estimate motion vectors from a sequence of lip-movement images. The motion is modelled as the distribution of apparent line velocities in the movement of brightness patterns in an image. Then, we construct compact lip-motion features from the regional statistics of the local velocities. These can be used alone or merged with audio features to recognize individuals or speech (digits).In this work, we utilized two classifiers for identification and verification of identity as well as with digit recognition. Although the study is focused on processing lip movements in a video sequence, significant speech processing is a prerequisite given that the contribution of video analysis to speech analysis is studied in conjunction with recognition of humans and what they say (digits). Such integration is necessary to understand multimodel biometric systems to the benefit of recognition performance and robustness against noise. Extensive experiments utilizing one of the largest available databases, XM2VTS, are presented.