On the Industrial Applicability of Visual GUI Testing

Abstract: There is constant pressure on the software development industry to improve software quality and deliver new and innovative software faster and more efficiently. These demands affect all aspects of software development, from requirements engineering to testing. Today, much testing is performed with costly, tedious and error-prone manual practices, especially on higher levels of system abstraction such as in system and acceptance testing. Increased test automation has been proposed as a key solution to help alleviate these test-related problems and has found increasing use. Automated tests typically consist of scripts that can be executed to give frequent feedback on the system's quality, ensure that previously correct functionality has not been negatively affected by changes, and also relieve resources e.g. human testers. However, as a system evolves, changes to the requirements also require maintenance of the tests, which can be costly. Furthermore, most automated test techniques approach testing from a lower level of system abstraction, e.g. testing the low-level components and functions, but their use for system testing has been questioned. Test techniques have been proposed that automate the interaction with the graphical user interface (GUI) of a system. For example, record and replay tools can be used to automate system testing by emulating a user interacting with the system. Existing literature have pointed out several shortcomings of these techniques, for example they are sensitive to GUI layout and code changes, require access to source code, are dependent on a specific platform or operating system access etc. In this thesis we evaluate if Visual GUI Testing (VGT), a novel technique that combines test scripts with image recognition, can be used to automate high-level, software testing. The strength of VGT lies in the use of image recognition that makes the technique robust in instances where previous GUIbased test techniques had limitations. Yet, VGT is only sparsely applied in industry and the academic body of knowledge on the technique is limited; there is a lack of empirical evaluation and experience reports from industrial use. To fill this gap in knowledge we have conducted three empirical case studies applying different VGT tools in industrial software projects at two Swedish companies. We have also conducted an industrial survey with participants from multiple companies. These studies provide evidence that VGT can be successfully applied in industrial software development projects, but also detail challenges, problems and limitations that need to be overcome for more widespread industrial adoption. Future work should focus on evaluating the long-term maintenance costs of VGT test scripts, since our studies only present initial evidence that these costs are manageable in relation to the benefits of the technique.