Sensory integration : success & failure

Abstract: The convergence of sensory signals plays an important role in perception. Two cases of convergence are examined in this thesis by means of modelling; the case when the converging signals are congruent, which leads to combination and an enhancement of perception, and the case when they are incongruent.The former modelling experiment considers the integration of phonemes and letters. Based on a series of reports on different aspects of letter-phoneme integration that has been presented from the Department of Cognitive Neuroscience at Maastricht University in the Netherlands we have developed a model for simulating some features of this course of events. Our model, the Artificial Cortical Network (ACN), is an artificial neural network whose essential parts are two types of modules, each employing its own learning law, none requiring manual intervention. The particular ACN architecture used in this thesis consists of three modules which are interconnected, where each module processes one of three different types of pre-processed stimuli: letters, phonemes and the bimodal combination of these. Modules of this architecture contain one or more neural lattices and inter-module information exchange is carried out using the coordinates of the neurons that respond the strongest to the inputs. The model is generally useful; it is, for example, equally suitable for modelling the integration of different features in a single modality, and the ideas behind it are new. The main features modelled using the architecture, are the combination of unimodal signals at the bimodal module and feedback from the bimodal to the auditory module. Simulation results of the architecture show the same characteristics as corresponding results from psychology and neuroscience. One important result is the qualitative enhancement of the response to a noise-perturbed sensory signal in one modality using a congruent one from a complementary modality. This is what happens at the opera when the libretto is shown above the scene: one better perceives what is sung. Another is the ability to use input to one modality as support for distinguishing the relevant signal from a collection of signals in another modality. This mechanism is most probably active when having a conversation at a cocktail, on the bus or in the subway. In these situations the (possibly unconscious) monitoring of the communicating peer's mouth movements may improve comprehension. The latter modelling experiment considers a case when integration "fails" due to irreconcilable incoming signals. The particular phenomenon under examination is that of binocular rivalry: the alternating periods of dominance and suppression occasioned by stimulation of corresponding retinal areas with dissimilar monocular stimuli. Typically an observer only "sees" one stimulus at a time and the rate of recurrence is about 1-2 seconds on average. The switching that occurs has some well established properties; such as a skewed unimodal distribution of dominance times and an increase of switching speed with stimulus contrast. In this thesis an artificial neural network model of a cortical area in which competition between populations of neurons that should not be co-active is presented. The focus of the model is for it to be able to simulate key properties of binocular rivalry; the skewed unimodal distribution of dominance times, and three other well-known properties of binocular rivalry. These properties are Levelt's second and fourth proposition and the "flipped" case of Levelt's second proposition. The essential driving forces in this architecture are fatigue and noise. Simulation results obtained from the model are in fair quantitative agreement with psychophysical experiments in all four aspects, using the same model parameters.