UMG/NCCPE event: Impact and evaluation – learning together

Report on the UMG/NCCPE event: Impact and evaluation – learning together
7^th Feb 2012, Great North Museum, Newcastle

Download this report as a pdf –  UMG NCCPE Event report, Feb 2012

An interest in issues around how university museums demonstrate their successes and their value brought around fifty people together in Newcastle for an engaging and thought provoking day at the Great North Museum this week.  Discussions were both wide-ranging and deep, so this is an attempt to summarise the key themes of the day, with apologies for any over-simplifications or omissions:

Paul Manners of the NCCPE provided an extremely useful keynote, outlining the differences between evaluation and impact, and how we might move use the former as evidence of the latter.  Evaluation is important for measuring what we did and how well we did it.  It might be undertaken for a number of reasons: for internal communication; as a means of learning from mistakes or successes; for reporting to funders; and as a tool for advocacy.  The results of evaluation might be presented differently, depending on the intended audience.  For example, an internal report might include a ‘warts and all’ analysis including the things that went less well; whereas a report to funders or external stakeholders might present the data in a more purely positive light.

Impact is different from evaluation in that it implies a requirement to account for the success of the activity as a whole.  Put another way, it’s the difference between ‘outputs’ (eg how many holiday activities did the museum run, and how many children came?); and ‘outcomes’ (ie what difference did we actually make to those children?).  The current concern with impact in the museum sector is perhaps an indication of the realisation that a ‘bums on seats’ (or ‘visitors through the door’) method of accounting for our activities is no longer sufficient.  We need to gain a more sophisticated understanding of the transformative potential of museum experiences.  Only then will we really be able to make the case for continued funding, from whichever source, in an increasingly difficult economic climate.

There are a number of different models that can be used to assess impact.  For example, the Generic Learning Outcomes framework, developed by the MLA several years ago, was an attempt to map the impact of museum visiting on visitors.  The Arts Council have defined their goals as excellence, reach, engagement, diversity and innovation, while other organisations have developed their own frameworks (eg the BBC, the New Economics Foundation, local authorities), all of which might have some relevance for museums.  So there is no shortage of models which museums could use to measure their impact, depending on where their priorities lie.

In the context of today’s discussion, much of the emphasis was on the Research Excellence Framework (REF), used to assess the activities of the Higher Education sector, and therefore of particular interest to those in university museums.  As well as the ‘excellence’ of an institution’s research outputs, the REF is concerned with the ‘impact’ of that research.  In other words, there is a requirement to demonstrate that research has been ‘effectively shared’ with audiences beyond academia, and this accounts for 20% of an institution’s overall score.

The day’s discussions therefore ranged between impact in general and ‘impact’ in REF-able terms.  As Paul Manners indicated, the concept of measuring impact was not universally welcomed by academics when it was introduced a few years ago.  It was seen as an attempt to measure the un-measurable and to impose a market culture on universities, and was fiercely resisted by many.  However, the general feeling within today’s gathering was that university museums ought to be in a good position to help their institutions demonstrate impact, since they provide an interface between academia and the wider public, and should therefore have a role in making new ideas available to audiences beyond Higher Education.  It was generally agreed that the REF might offer university museums the opportunity to reposition themselves at the centre of their host institutions’ activity.

At first glance, it looks quite straightforward.  Find an academic in your institution who has written an interesting (and REF-able) article or book: turn it into an exhibition.  Result: impact.  In reality, however, it’s a little trickier.  On the one hand, it’s not always easy to account for the passage of ideas from an article to an exhibition, since there are many other factors at play.  Secondly, a great academic paper does not necessarily translate well into an interesting and engaging exhibition which anyone will actually want to see.  Thirdly (and perhaps most frustratingly), lots of interesting work in museums is done by people (curators, artists in residence etc) who are not academic staff, hence their work is not ‘REF’-able’, and can’t be counted.

There is work to be done, then, on making university museums more easily able to demonstrate impact in terms of the REF.  We need to gather good case studies which show clear evidence of linkages between university research and public outcomes.  We need to value the very particular skills that museum staff often have in making complex ideas accessible to non-specialist audiences, and we need to stop thinking of exhibitions as the only vehicle for the dissemination of ideas (what about podcasts, websites, blogs, wikis etc? – all of which could be ‘curated’).  Above all we need to develop a vision for the role that university museums might play, in order to position ourselves strategically for the forthcoming REF and for the one after that.

At the same time, museums – including university museums – ought to be prepared to demonstrate their impact on and benefit to, society more generally.  In other words, while undoubtedly important, the REF is not the only game we are playing.  Eric Cross of Newcastle University outlined the way in which the redevelopment of the Great North Museum from a number of smaller institutions was spurred on by the University’s concern for greater impact in terms of the local and regional economy, public engagement, income generation and so on.  A number of other speakers also presented case studies of projects which offered really excellent outcomes for a variety of audiences.  We need to remember that universities are concerned with other agendas including enhancing the student experience, which might mean a concern for scores in the National Student Survey as well as the REF.

In summary, it seems to me that we all need to get better at evaluation, which means setting the right objectives in the first place, measuring the right things, and asking the right questions of ourselves.  We need to present the data, both as individual institutions, and as a sector, so as to demonstrate the remarkable achievements that university museums make.  (In fact we might need to juggle the same data to present evidence of different kinds of impact to different funding bodies or stakeholders).  Only with all of this in place will we be in a position to provide evidence of impact whether for the REF or against criteria set by other funders, and thus ensure that we are really worth what we cost, not merely worthwhile.

Zoë Hendon, Head of Museum & Collections
Museum of Domestic Design & Architecture
Middlesex University

Feb 2012