Validating User Engagement and Effectiveness of Training Simulations : A mixed-methods approach informed by embodied cognition and psychophysiological measures

Abstract: Simulation-based training has gained widespread attention recently as a response to drawbacks associated with traditional training approaches, such as high training costs (instructors, equipment, etc.), high risks (e.g. pilot training), and ethical issues (e.g. medical training), as well as a lack of availability of certain training environments (e.g. space exploration). Apart from their target training domains, many of aspects of simulations differ, such as their degree of physical realism (fidelity), scenarios (e.g. story), and pedagogical aspects (e.g. after-action reviews and collaborative learning). Among those aspects, designers have mostly focused on developing high-fidelity simulations with the expectation of increasing the effectiveness of training. However, some authors suggest that the above belief is a myth as researchers have failed to identify a linear relationship between the (physical) fidelity and training effectiveness of simulations.  Most researchers have therefore evaluated the correspondence between the behaviours of trainees in both real world and simulated contexts, however, the existing methods of simulation validation using behavioural measures have a number of drawbacks, such as the fact that they do not address certain complex phenomena of skills acquisition.Bridging the above knowledge gap, this research reports on empirical investigations using an improved methodology for validating training simulations. This research includes an investigation of the user experience of trainees, with respect to the acceptance of virtual scenarios provoking a similar psychophysiological response as in real world scenarios, and the training potential of simulations with respect to the positive transfer of training from a simulator to real world operational contexts. The most prominent features of the proposed methodology include the use of psychophysiological measures in addition to traditional behavioural measures and the use of natural (quasi-) experiments. Moreover, its conceptual framework was influenced by contemporary theories in cognitive science (e.g. constructivism and embodied cognition). The results of this research have several important theoretical and methodological implications, involving, for example, the dependency of the effectiveness of simulations on the perceived realism of trainees, which is more embodied than has been predicted by previous researchers, and the requirement of several different types/levels of adaptive training experience, depending on the type of trainee.