Facilitating Feature-Oriented Quality Assurance in Low-Maturity Variant-rich Systems

Abstract: Context: Many software systems exist in several variants customized for specific stakeholder requirements, such as different market segments or hardware constraints. This customization introduces a high level of complexity that renders traditional single-system quality assurance techniques inapplicable, since they need to consider variations and constraints between a system’s features—a.k.a feature-oriented or variability-aware analysis. While several analysis techniques have been conceived in the last two decades for this purpose, they mostly target a branch of variant-rich systems called software product lines, and are less applicable to systems that still rely on cloning strategies to engineer variants—a.k.a low-maturity variant-rich systems. Among other reasons, this is because such systems exhibit: immature architectures that do not systematically account for variability, redundancy that is not exploited to reduce analysis effort, and missing essential meta-information such as feature constraints and locations of features in source code. Objective: This research aims to facilitate quality assurance in low-maturity variant-rich systems. Through analysis of the state-of-practice, we propose techniques that can improve maturity and help developers of such systems mitigate some challenges posed by redundancy. Method: First, we conducted a survey and interviews with practitioners to identify industrial needs and practices for analyzing variant-rich systems, followed by a case study of some open source systems to understand where developers record information necessary for feature-oriented analysis. Then, we designed and systematically evaluated a technique and a tool that can improve the maturity of variant rich systems by supporting feature recording, and two techniques that can reduce analysis effort. Results: Our results stem from two main contributions: our analysis of the state-of-practice and techniques we propose for improving maturity to facilitate feature-oriented analysis in low-maturity variant-rich systems. For the former, we present results of a survey and interviews targeting 27 practitioners from 18 companies in 8 countries to understand industrial practices and needs for analyzing variant-rich systems. Then, we present our empirical understanding of features and their characteristics (facets) based on a case study of two industrial open-source systems. For the latter, we present design decisions and an evaluation of a tool and technique that help developers proactively and continuously record features. Then, we present empirical data on the potential of two techniques for reducing QA effort: one for predicting software defects at the level of features (evaluated on 13 open-source systems), and another for propagating test cases across forked projects, i.e., projects with similar but cloned features (evaluated on over 426,000 test cases from 2,153 projects). Conclusion: This thesis identifies the lack of adoption of feature-oriented quality assurance techniques in industry, and addresses the problem through tools and techniques for feature recording and quality assurance effort reduction.