Assessing Reusability in Automated Acceptance Tests

Abstract: Context: Automated acceptance tests have become a core practice of agile software development (e.g. Extreme Programming). These tests are closely tied to requirements specifications and these tests provide a mechanism for continuous validation of software requirements. Software reuse has evolved with the introduction of each new reusable artefact (e.g., reuse of code, reuse of frameworks, tools etc.). In this study, we have investigated the reusability of automated acceptance tests keeping in view their close association with textual requirements.Objective: As automated acceptance tests are closely related to software requirements, we have used existing research in software engineering to identify reusability related characteristics of software requirements and used these characteristics for automated acceptance tests.This study attempts to address the following aspects: (i) what important reuse characteristics should be considered when measuring reusability of automated acceptance tests? (ii) how reusability can be measured in automated acceptance tests?, and (iii) how cost avoided through reuse of automated acceptance tests can be calculated?Method: We have used a combination of research methods to answer different aspects of our study. We started by identifying reusability related characteristics of software requirements, with help of systematic literature review. Later, we tried to identify the reusability related characteristics of defect reports and the process is documented using an experience report. After identifying the characteristics from the previous two studies, we used these characteristics on two case-studies conducted on Behaviour driven development test cases (i.e., acceptance tests of textual nature). We proposed two approaches that can identify the reuse potential of automated acceptance tests and evaluated these approaches in the industry. Later, to calculate the cost avoided through reuse, we proposed and evaluated a method that is applicable to any reusable artifact.Results: The results from the systematic literature review shows that text-based requirements reuse approaches are most commonly used in the industry. Structuring these text-based requirements and identifying the reusable requirements by matching are the two commonly used methods for enabling requirements to reuse. The results from the second study, industrial experience report, indicates that defect reports can be formulated in template and defect triage meeting can be used to identify important test-cases related to defect reports. The results from these two studies, text-based requirements reuse approaches and template based defect reports, were included when identifying approaches to measure reuse potential of BDD test-cases. The two proposed approaches, Normalised Compression Distance (NCD) and Similarity Ratio, for measuring reuse potential were evaluated in the industry. The evaluation indicated that Similarity ratio approach performed better than the NCD approach, however, the results from both approaches were comparable with the results gathered with the help of expert analysis. The cost related aspects of reusable acceptance tests were addressed and evaluated using a method that calculates the cost-avoidance through reuse. The industrial evaluation of the method and guidelines show that the method is an artifact independent method. Conclusions: The evidence from this study shows that the automated acceptance tests are reusable, similar to text-based software requirements and their reuse potential can be calculated as well. The industrial evaluation of the three studies (i.e. approaches to measure reuse potential, calculation of cost avoidance and defect reports in triage meetings) shows that the overall results are applicable to the industry. However, further work is required to evaluate the reuse potential of automated acceptance tests in different contexts.