Supporting the use of design knowledge : an assessment of commenting agents

Abstract: This thesiscontributes to an understanding of the usefulness of and effectsfrom using commenting agents for supporting the use of designknowledge in user interface design. In two empirical studies, wehave explored and investigated commenting agents from the aspectsof usefulness, appropriateness of different tool behaviour andforms of comments. Our results show a potential value of thecommenting approach, but also raises several questions concerningthe cost and actual effects. The use of formalized design is considered valuable, yetproblematic. Such knowledge is valuable in order to achieve reuse,quality assurance, and design training, but hard to use due to thelarge volumes, complex structures and weak reference to the designcontext. The use of knowledge-based tools, capable of generatingcomments on an evolving design, has been seen as a promisingapproach to providing user interface designers with formalizeddesign knowledge in the design situation. However, there is a lackof empirical explorations of the idea. In our research, we have conducted a three-part study of theusefulness of commenting tools. First, a Wizard-of-Oz study with 16subjects was performed to investigate designers' perceptions of theusefulness of a commenting tool, along with the appropriateness ofdifferent tool behaviors and forms of comment. We focus on toolmode (active/passive support) and mood (imperative/declarativecomments). Secondly, eight professional designers participated inan interview about support needs. Thirdly, a conceptual designprototype was tested by 7 designers, using cooperative evaluation.A broad set of qualitative and quantitative methods have been usedto collect and analyse data. Our results show that a commenting tool is seen as disturbing butuseful (since it affects the user's work situation). Using acommenting tool affects the designer's evaluation behaviour, i.e.,there is an indication of some form of knowledge transfer. Theshort-term result is an increased consciousness in terms of designreflection and guideline usage. In terms of preferred toolbehaviour, our results show that imperative presentation, i.e.pointing out ways of overcoming identified design problems, is theeasiest to understand. A high perceived mental workload relates toproblems detecting comments when using a commenting tool; thismeans that comments from an active agent risk being overlooked. In addition, a large part of this thesis can be described as areport of our experiences from using Wizard ofOz techniques tostudy user interface design support tools. We present ourexperience and advice for future research.