Towards an Emotionally Intelligent Interaction Strategy for Multimodal Embodied Conversational Agents acting as Companions

Abstract:   Existing Human Computer Interaction (HCI) strategies are seriously limited by current technologies. These are neither sensitive nor accurate enough to respond to users’ emotional states, the fundamental basis for effective communication in real time. This offered the challenge of investigating factors that would impact on the designing of effective and more emotionally intelligent interaction strategies for Companions. These were applied to a conceptual tool, the Affective Channel (AC), to endow Companions with emotional capabilities. This was implemented in the Wizard of Oz (WoZ) platform to evaluate Companions in real time. The WoZ is an experimental setup where existing immature technologies and a human operator combine to simulate Companion interaction with end users. In these aspects of my work is my original contribution to the HCI knowledge base.Experiments, focus groups and face to face interviews were carried out to ascertain users’ perception and expectations of virtual agents. ‘Descriptors’ thus identified formed the bases for the designing of user friendly Companions. Verbal and facial expressions data and other affective elements of effective human-companion interactionwere collected for use in the AC and the WoZ as stated above.Companion evaluations yielded the subsidiary contribution that Companions are perceived as empathetic, useful and trustworthy entities. Further, that they arouse positive emotions in children and also that they promote their learning improvement.These findings were the result of two experiments, one within subjects and one between subjects, conducted with thirty grade four pupils in a rural school in the poor Oaxaca region of Mexico.