Structural System-Level Testing of Embedded Real-Time Systems

Abstract: People make mistakes. Software engineers are no exception to this fact. When software engineers make mistakes, these manifest in the form of buggy software - a major problem in today's industry. The existence of bugs is commonly detected using testing, the process of executing the software and checking if its behaviour complies with the specification. As limitations in time make testing of the entire software behaviour impracticable, testers need to make informed decisions on how to test the software to detect as many bugs as possible.In the realm of real-time systems (RTSs), software testing is made more difficult by non-deterministic factors such as interaction with the surrounding environment, (pseudo)parallelism, and timing requirements. Furthermore, RTS testing suffers from behaviour-altering perturbation from the instrumentation inserted in the system to keep track of test progress (i.e., probe effects).In our work, we analyse the main test criteria used for traditional software testing in order to see which of these scale to, and assist in, system-level testing of multi-tasking RTSs. We focus on one of these criteria, the all definition-use paths coverage criterion, as it highlights a central aspect of non-deterministic task interaction, and investigate what is specifically required for applying this criterion to testing of multi-tasking RTSs. Further, we examine the possibility of using execution replay for probe effect-free test monitoring. We evaluate this approach in real industrial settings by means of case studies.The contributions of this thesis are twofold: First, the use of structural test criteria in RTS system-level testing is facilitated by two different analysis methods. Second, the probe effect is handled by recording non-deterministic events during run-time, and by using this recording to create a monitorable deterministic replica of the first execution. By these contributions, this thesis shows how the non-determinism of multi-tasking RTSs can be handled during system-level testing.