Validation of a Standard- and Metric-Based Software Quality Model : Creating the Prerequisites for Experimentation

Abstract: Our long term research goal is to validate a standard- and metric-based software quality model. Today, ambiguous metric definitions lead to incomparable implementation variants in tools. Therefore, large scale empirical experiments, spanning different independent research groups, evaluating software quality models are not possible, since the generalization of the results of individual software measurements is impossible. We propose a public and extensible knowledge base -- a compendium -- for the unambiguous definition of software quality metrics and their connection to a standard-based software quality model. This constitutes the basis for the measurement and prediction of software quality in current and legacy systems. This compendium is published as specification for the metrics implementation. In order to provide well-defined metrics an unambiguous description framework for software metrics had to be developed. It includes formal definitions based on an extensible language-independent meta-model and language mapping definitions. We formalize an existing and well-defined meta-model for reverse engineering. Then, we define some software quality metrics based on this meta-model, as well as language mappings for Java and UML/XMI. A full set of metric definitions together with a quality model is provided in the compendium. Since we want our compendium to be a ``living document'', we need to maintain the provided knowledge base, the meta-models, language mappings, and metrics. Therefore, we design a maintenance process assuring the safe evolution of the common meta-model. This process is based on theorems controlling changes to the common meta-model, as well as abstraction mechanisms in form of views, hiding modifications to the common meta-model from existing metric definitions. We show feasibility by implementing our definition framework and the metrics specified in the compendium in a well known reverse engineering tool, the VizzAnalyzer. We evaluate our efforts in two industrial case studies supporting our claims and solution. We suggest that our description framework eliminates existing ambiguities in software metric definitions, simplifying the implementation of metrics in their original sense, and permitting the definition of standardized metrics. The compendium supports exchanging experiments and the evaluation of metrics. The maintenance process allows us to extend and modify definitions, e.g., add new programming languages, in a controlled and safe way. With the tools developed, it is possible to conduct the experiments we envision, and to evaluate the suitability of software quality metrics, in a next step.