Interactive Search-Based Testing of Embedded Software : Understanding the Benefits and Difficulties of Industrial Application

Abstract: The ubiquity of software has wide-ranging consequences for its development and testing. Increasingly often, software is developed and tested by engineers specialized in other areas. Embedded software, for example, is developed ad-hoc, for each product, by systems and domain engineers. Supporting testing activities in this context requires a highly flexible approach, powerful enough to create useful test cases, yet simple enough to not require specialized training in software testing. Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. It is, however, a set of complex techniques that cannot be used off-the-shelf as part of the software development process of a company. The objective of the work presented in this thesis is to investigate the applicability of Search-Based Software Testing in an industrial environment. A second objective was identifying additional knowledge gaps relating to integrating such techniques in existing software development processes. Investigating how meaningful interaction is to take place, what information is needed in order for both stakeholders to be able to achieve their objectives is a third goal. The findings are obtained by means of a series of case studies in a company developing both embedded software and the tools to enable embedded software development. A prototype Interactive Search-Based Software Testing (ISBST) system was developed that uses user interaction to drive the search-based component towards interesting test cases. The ISBST system was evaluated constantly, and improved based on the findings of each case study. The latest case study was an empirical evaluation of the system with the engineers, both software engineers and domain specialists, in the company. The empirical work includes both qualitative and quantitative data, with a focus on the exploratory study of the practical factors affecting the use of the ISBST system. A key early finding is that interactivity is essential when implementing search-based techniques in the industrial context described above. Repeated validations conducted with the company yielded additional information on the practicalities of interaction. The strength of SBST proved useful in investigating areas of the test space that were normally overlooked due to limitations in terms of resources. At the same time, developers were able to use their experience and intuition to guide the SBST system towards test cases that were more likely to be problematic. Overall, the results obtained indicate that the search-based techniques provide a useful complement to existing testing techniques. SBST, in its interactive form, can be a useful complement to existing testing techniques. An Interactive SBST (ISBST) system has been developed as a result of this research. Results show that this system is usable by the developers of embedded software, that often specialize on acquiring domain knowledge rather than software engineering expertise.