Evaluating for a transition - Advancing evaluation of research and policy for a more sustainable energy system and society

Abstract: In response to global sustainability challenges, there are multiple public research andpolicy incentives being implemented to support a more sustainable energy system. These are aimed at advancing technology and innovation as well as guiding citizens tochange behaviour. But are those incentives accurately leading to a transition towards amore sustainable energy system and society?Evaluation of these incentives is key to addressing such a question, and such evaluations need to be able to provide knowledge that is oriented around transformative aspects and effects on system levels, and that is grounded in rigorous assessment approaches. The objective of this thesis is thus to advance knowledge on how the evaluation of research and policy incentives can be further developed to support a transition. The research presented will focus on Swedish evaluation practices of research and policy incentives aimed at energy efficiency in buildings. In order to provide foundations for how transformative evaluation can be designed and conducted, this thesis combines insights from transition research, evaluation theory, sociology of science, and policy analysis. In essence, transformative evaluation, as proposed in this thesis, will consist of two main tracks: it is transformative in that it departs from transition research; and it is rigorous and knowledge-based in that it supports assessments that build on and combine relevant evaluation-oriented insights. As such, transformative evaluation seeks to provide a broader approach that addresses the complexities of a systems-level perspective, and that contrasts and complements more traditional evaluation approaches that have a predominant orientation around the programme level.The review of the Swedish evaluation practices shows that evaluations are largely not currently used for, or focusing on, assessing transformative contributions. Key suggestions for advancing evaluation practices to support transformative insights include articulating the role of the evaluand in relation to societal goals and visions, heeding experimentation, and capitalizing on the learning that can be drawn from it. Suggestions for advancing evaluations towards rigorous and knowledge-based assessments include an increased application of triangulation, counterfactual analyses, and a wider variety of criteria than what is currently seen, including e.g. relevance, legitimacy, acceptability, adequacy, equity and responsiveness. A conclusion is that moving towards transformative evaluation can support both an alignment of research and policy instrument evaluations, as well as an understanding of cascading effects from individual research and policy incentives. In terms of adopting a transformative evaluation approach among Swedish state agencies, both benefits and challenges are noted by state representatives. Benefits include increased collaboration and enhanced evaluation use, while challenges relate to institutional and structural barriers, and issues surrounding agency and mandates to govern change processes.