User experience of spoken feedback in multimodal interaction

Abstract: The area of multimodal interaction is fast growing, and is showing promising results in making the interaction more efficient and Robust. These results are mainly based on better recognizers, and studies of how users interact with particular multimodal systems. However, little research has been done on users- subjective experience of using multimodal interfaces, which is an important aspect for acceptance of multimodal interfaces. The work presented in this thesis focuses on how users experience multimodal interaction, and what qualities are important for the interaction. Traditional user interfaces and speech and multimodal interfaces are oftendescribed as having different interaction character (handlingskaraktär). Traditional user interfaces are often seen as tools, while speech and multimodal interfaces are often described as dialogue partners. Researchers have ascribed different qualities as important for performance and satisfaction for these two interaction characters. These statements are examined by studying how users react to a multimodal timetable system. In this study spoken feedback was used to make the interaction more human-like. A Wizard-of-Oz method was used to simulate the recognition and generation engines in the timetable system for public transportation. The results from the study showed that users experience the system having an interaction character, and that spoken feedbackinfluences that experience. The more spoken feedback the system gives, the more users will experience the system as a dialogue partner. The evaluation of the qualities of interaction showed that user preferred no spoken feedback, or elaborated spoken feedback. Limited spoken feedback only distracted the users.