Language and Vision. Using visual information in real-world language situations

Abstract: This dissertation is about the role of visual context information in communication. All accounts of language processing and communication agree that visual context is beneficial for understanding. Despite this, it is not clear through which mechanisms visual context works and whether these mechanisms also work in communication outside the lab setting. This dissertation aims to provide answers to these concerns and this is achieved through three different paths: 1. Testing known mechanisms under more realistic conditions. 2. Exploring new mechanisms for how visual context may inform communication 3. Resolve methodological issues that prevents experiments from being carried out under more realistic conditions. This dissertation reports the work of five different papers. Paper I provided concise answers to the problem of temporal imprecision in eye-trackers with low sampling frequencies. Paper II investigated the habit of looking at visual referents as they are mentioned and found that this habit works even in very demanding situations with complex speech and complex scenes. Paper III suggested a new measure for evaluating communicative success in unrestricted conversations. This paper also found that visual context information allow speakers to understand each other better. Paper IV found that visual context information, which use is often made mandatory in language experiments, is used even if it is completely optional to the communication task. Paper V found that images of high emotional content is perceived as more helpful in a communicative task. This paper also found that participants do not gravitate toward a similar use of emotional language, which is predicted by a current theory.