My current research is focused on establishing the usefulness of interaction design methodologies when evaluating Creativity Support Tools (CSTs). Specifically looking at user experience and the application of CST design thinking to generative music systems.
CSTs are distinct from productivity support tools, a need to further distinguish between these two types of tools has arisen as so many creative practitioners now rely heavily on digital tools in their creative practice. Until recently these tools have been evaluated under the presumption that their goals are primarily no different. While traditional Human-Computer Interaction methodologies are not irrelevant when evaluating CSTs they are not always appropriate nor are they holistic.
"if a user was to spend a long time performing a task with a productivity support tool, this may indicate that the tool was deficient. In contrast, if a user was to spend a long time working with a CST, this may actually indicate deep engagement, rather than a tool deficiency."
The problem with attempting to evaluate digital tools intended for use by creative practitioners is this: Whilst their goals are clear, a means by which to determine the users’ perceived success in achieving these goals is not. The evaluation of creative success is not empirically grounded in an objective methodology.
"terms such as ’creativity’ and ’imagination’ do not describe things that we can readily measure or objectively identify, they are concepts that frame other kinds of measurable and objectively identifiable things, as part of a loose theoretical framework."
The problem of evaluating creative success is compounded specifically in CSTs that incorporate computationally creative interactions. For example, Jnana is a generative musical accompaniment system. Built in max/MSP, it uses Max For live to integrate into Ableton Live. Jnana uses a Markov model to process a MIDI input in real time, allowing you to perform or compose with the system. Used as an accompaniment to a performance Jnana is at least a co-creator, and when used as a composition tool, Jnana becomes a kind of musical search tool with a simple yet strong ability to promote novelty. Making evaluative statements about the value of this novelty is difficult. Can we holistically quantify the process by which the sequence of interactions lead to a successful creative outcome? Though the goal here is not to create novelty, which potentially could be holistically quantified, but to enable creative search towards the discovery of creative value.
So in summary I think that the attribution of creative success is distinct from the operational utility of a system. This makes it hard for anyone but the user to evaluate creative success. Therefore one of the best methods of evaluation is one based on user-experience.