The Art of Evaluation

Tools of the Trade

On my third day in London, I got lucky and was offered a place on a sold-out workshop hosted by the Live Art Development Agency that just sounded too intriguing to be missed. Here’s the description that caught my attention:

Fed up with the standard evaluation surveys? Situations and the University of Central Lancashire have been developing an innovative new group based evaluation method to move beyond overt measures of impact and unlock the deeper story of an artwork’s effects on the imagination.

Titled Thinking Beyond Measure, the day-long event, part of the Public Art Now national programme of events, promised a mixture of practice and theory to explore the scope of results and potential applications of what the research team calls the Visual Matrix: an interpretation process based on a series of images that act as prompts to elicit associative thinking and make it easier for people to think and talk about their experience.

The case studies we discussed were Nowhereisland, an itinerant, durational and participatory project by artist Alex Hartley produced by Situations in 2012 as part of Artists Taking the Lead, the Cultural Olympiad series of major commissions –  and Verity, the Damien Hirst’s half-sliced pregnant bronze warrior loaned for 20 years to the Devon resort of Ilfracombe, one of Nowhereisland’s port of call.

The research team, led by Professor Lynn Froggett and Dr. Ali Roy from UCLAN in association with Situations, conducted an evaluation in Ilfracombe on both works in 2013, one year on. They used the Visual Matrix alongside other forms of evaluation to explore Nowhereisland and Verity’s respective role in reflecting local engagement and citizenship, as well as their legacy in terms of change and transformation.

I’ve summarised a few key points from the day but there would be much more to say – not least about the contrasts in findings between the two selected works and between different methods.

Process

First, some disclaimers: the workshops featured two sample 20-minute Visual Matrix and participant feedback sessions, one on each work, so a short version of the 2 first steps of the full process. Besides, workshop participants had, for the vast majority, no direct experience of the artworks – this was meant to be an exercise. By contrast, the case studies that we were somehow reproducing had gathered people with a varied range of exposure to both artworks, from passer-bys to contributors, all living in Ilfracombe and therefore able to reflect on a personal and community level on the effect of these two artworks on citizenship and their legacy of change.

1. The Visual Matrix

Ideal group size should be between 6 and 20, with at least 2 facilitators.

– Chairs are arranged in a snowflake formation – concentric circles that are slightly out of alignment to avoid direct eye contact.

No introductions are made: this is to avoid the bias of expertise and authority that can sometimes be overwhelming in a traditional focus group.

– Participants are explained that they will be shown a slideshow of 20 to 30 images, each lasting for 10 seconds, about which they can then express what they feel – what it reminds them of, in which state of mind they find themselves. They are expressedly ask to suspend judgement and refrain from interpreting or analysing what they see, and instead to feel what the images do to them.

– In the ensuing 1-hour session, which can occasionally be stirred – but not chaired – by the facilitators, participants should be reaching a state of “rêverie”, gliding from one idea to the other. They are not quite holding a group conversation, but rather letting their mind go back to the images and the experience itself  and absorb the new thoughts and images produced by the group.

– A form of documentation, such as note-taking, audio and/or video recording – whichever is most practical and less intrusive – is essential to this stage of the process.

2. Feedback with participants

After a short break, the participants, guided by the facilitators, start pulling together the themes that emerge from the Visual Matrix. This was an interesting process of convergence, pulling together the threads of information produced in the first part of the session, and we worked as a group to make sense of what had been expressed. One of the facilitator organised the ideas in a visual form.

3. Feedback with research group

Reconvening after another break, the research group – now on their own – starts to analyse the participants’ responses, working on their memories and notes of the Visual Matrix session as well as the synthesis co-produced by the participants. The nature and quality of the metaphors and the vitality – or lack of – with which they were produced are equally taken into consideration.

4. Wider discussion

The last phase of the process looks once again at the Matrix results and the layers of interpretation created by the participants and the research group, and can involve external advisors if appropriate: the aim here is to expand and generalise the results, for example to the realm of policy-making.

Benefits of the Visual Matrix

The Visual Matrix is inspired by Social Dreaming and aims at unlocking the deeper effects of an artwork on the imagination. Because it is based on imagery and metaphors, and not on expertise or status, such process makes it easier for anyone to participate. All thoughts are valid and they feed into one another to express a rich and nuanced response to an artwork or situation.

It is essentially a collective, participatory process, which seems appropriate to explore the collective resonance of complex works of public art (or other situations in the public realm).

It is also an open-ended creative process, and as such closer to the artistic process itself than sliding scales of enjoyment or debates about taxpayers’ ROI.

Considerations

The workshop allowed plenty of time for a group exchange about theoretical and practical considerations, and here are the ones that stuck out for me.

A. Training

The Visual Matrix method has been used for a while in different settings, but introducing it as part of a new evaluation framework for the arts would take some dissemination and training. It would be interesting to get to practice the interpretation steps and to be guided by an experienced mentor, to be able to reap the full benefits in a set amount of time.

B. Participants

Results are highly influenced by the group composition, and the dynamics between the participants will have a bearing not just on what they produce, but also on the ‘quality’ of the matrix – whether it is solid and keeps going in a steady state of “rêverie”, or breaks down into analysis and critical judgement.

Participants for the two case studies presented mostly responded to invitations from mailing lists and in the local media; they were self-selected and not screened against specific criteria.

My concerns here are as much about outreach – to attract a varied group of participants – and effective facilitation, to create the right setting and understand barriers and biases.

C. Image selection

The matrix is supported by visual materials – although other types of sensory prompts could be used, such as sound or movement – so it seems rather important to choose them well. It is probably also worth stating to the participants that the sequence of images is not meant to form a narrative sequence.

Practical Applications

This is an interesting method not just for evaluating the effects and legacy of public art, but also any collective experience: for example, applied to a volunteer programme, this method would allow to go much deeper than focus groups to uncover the intrinsic motivations of arts volunteers and the benefits of volunteering. It enables to measure success in terms of effect, not just figures.

Just like any other evaluation method, the trick is in the interpretation of the findings – and as it’s the part of the Matrix that we didn’t get to do by ourselves, I look forward to more workshops and guided applications to learn more about the process.

 

www.situations.org.uk
www.uclan.ac.uk
www.publicartnow.com