The Lirg/Sconul Impact Initiative has encouraged a greater focus by libraries on academic processes, but impact measures that can be used widely within the sector are still elusive, report Pauline Blagden and Philip Payne.
The nature of services in academic libraries supporting learning in further and higher education is changing at a fantastic rate, with librarians taking on new and expanding roles and moving into unfamiliar territory. It is essential, therefore, that librarians evaluate new initiatives. A number of impact assessment projects conducted by HE library services, in association with Lirg (CILIP’s Library & Information Research Group), Sconul (Society of College, National & University Libraries) and consultants David Streatfield and Sharon Markless have highlighted the challenges of library evaluation.
The Lirg/Sconul impact initiative
The Lirg/Sconul Impact Initiative was started in July 2003. The objective was to develop methodologies to enable higher education libraries to assess their impact. Ten institutions agreed to participate in the first phase and a further 12 in the second. The initiative has followed a facilitated action research model, in which participants have undertaken their own evaluation with support from experienced researchers.1 The initiative has been supported by the Sconul Working Group on Performance Improvement and by Lirg. Participants have received support from David Streatfield and Sharon Markless of Information Management Associates. This support has been primarily delivered through a series of events to which all participating institutions could send up to four delegates. It started with a two-day ‘implementation seminar’ which provided the opportunity to identify areas in which libraries wanted to assess their impact and to consider how they would undertake their particular projects. This was followed by a one-day mid-project review workshop, after six months, and a final review workshop at the end of each phase.
The projects
The 22 projects covered by the initiative focused on areas of development and change, particularly relating to the provision of electronic resources and supporting information literacy. The choices obviously reflect current concerns: the value of potentially expensive electronic resources must lie in their effective exploitation, and information literacy skills, while vital for independent and lifelong learning, can be time-consuming to teach. Libraries need to ensure that their resources are being effectively deployed and that they are making a difference, making such programmes clear candidates for a project of this nature. In all cases the initiative provided an opportunity to explore a topic of importance to the organisation, aligned with strategic aims and objectives. Each institution has followed a common approach:
- articulating service objectives
- developing success criteria
- creating impact measures
- identifying evidence needs
- selecting appropriate ways of collecting the evidence.
* * * *
Developing a community of practice on impact
- Consolidate the outcomes of the Lirg/Sconul Impact Initiative as training materials – tools, tips, case studies, etc, supported by a discussion list and website.
- Provide information about the context in which the tools were developed and used (e.g. type of library, assumptions, optimum conditions, etc)
- Badge the website as a clear outcome of the initiative but provide links to other relevant materials such as the eValued project.
- Encourage others to measure impact and deposit their materials on the website. All interested library professionals should be invited to contribute, irrespective of sector.
- Disseminate the outcomes of the initiative far more widely in the professional literature and at events/conferences.
- Run more courses on practitioner research, particularly on qualitative research methodologies.
Interested in participating in a community of practice on library impact? Please register your email address with Pauline Blagden (pauline.blagden@porthosp.nhs.uk)
or Philip Payne (p.payne@bbk.ac.uk).
* * *
What has been learnt?
Participating institutions had very different experiences of the initiative. Many found that their projects led to a deeper understanding of how the library supports academic processes and this has improved discussion with academics and stakeholders, as well as raising the profile of the library within its parent organisation. Others have found that being involved supported the development of library staff.
However, measuring impact is not easy and this came through strongly in the Lirg/Sconul Impact Initiative. Many institutions struggled with the use of the qualitative research methodologies which are particularly appropriate for measuring impact. Almost all the institutions found that measuring impact took longer than had been anticipated and that it was difficult to sustain that commitment alongside other pressing operational priorities.
* * *
Case study 1: University of the West of England
At UWE, we evaluated the impact of our library’s electronic information services (EIS). Spending on these resources has increased dramatically in recent years and we need to be able to justify this, and to understand how the rapid growth of e-resources affects our users.
We collected data using a variety of approaches: analysis of documentary evidence; questionnaire surveys; semi-structured interviews; and participant observation. The initiative also presented an opportunity to compare methodologies and share experiences with the other participants, in particular, with the project team from Bournemouth University who were also evaluating the impact of their EIS.
We gained some tangible evidence of the impact of EIS, and clearly demonstrated how this is likely to increase as a result of UWE’s developing e-learning culture. This represents valuable information for the strategic planning of the library service. The study also gave us insights into how we can maximise impact to achieve higher returns for our money.
At the same time we were able to evaluate research methodologies and to develop research tools and experience for future use.
Of particular interest were the ‘unintended consequences’ of the qualitative methods. The interaction with our users afforded by the interviews gave us a deeper understanding of the academic processes involved in teaching, learning and research, and an opportunity for us to raise the profile of the library.
Dianne Nelson, Faculty Librarian, Computing, Engineering and Mathematical Sciences, UWE.
* * *
The overall view is that the initiative was successful in promoting evidence-based practice, encouraging a greater focus by libraries on academic processes, and developing methods and skills for assessing impact. The initiative was less successful in the development of impact measures that can be used widely within the sector. At the final review meeting for Phase 2, held in December 2005, there was strong support for developing a ‘community of practice’, building on the initiative and aiming to support others who want to measure impact. A range of practical suggestions was offered about how this might be done (see panel, opposite).
* * *
Case study 2: University of Stirling
Stirling focused on measuring the impact of its subject resource web pages. These pages highlight useful electronic resources for students, covering all subjects taught at the university.
We used two different techniques: face-to-face interviews and a statistical analysis of our website. Students were recruited through our web pages and were paid £10 to participate in a 30-minute interview. After completing 23 interviews, we stopped, as we were not gathering any new data. The answers were then analysed to show common themes.
We discovered that these pages were among the most used on the library website and, after analysing information from the web statistics and interviews, seven recommendations were made by the project group. Most significant were: the pages should be redesigned, focusing on locating journal articles; publicity should be increased; and impact analysis should take place regularly across Information Services. The redesigned web pages can be found at www.library.stir.ac.uk/resource/subject/index.html
Although time-consuming, our participation in the initiative has provided staff with new skills, raised the profile of evaluating services within the service and allowed us to design web pages which reflect student needs.
Alasdair Stewart, Faculty Librarian
(Management), University of Stirling.
* * *
Lirg’s journal, Library & Information Research, has devoted a whole issue to Phase 1 of the initiative. It includes reports for each of the participating institutions and articles that provide an overview of the initiative.2 A forthcoming article by Poll and Payne provides a summary of the second phase and gives practical examples of how libraries can attempt to assess their impact from the initiative and from other library impact studies.3
References
1 S. Markless and D. Streatfield. ‘Gathering and applying evidence of the impact of UK university libraries on student learning and research: a facilitated action research approach.’ International Journal of Information Management 26, 2006, pp. 3-15.
2 Library & Information Research, (29) 91, Spring 2005 (www.cilip.org.uk/specialinterestgroups/bysubject
/research/publications/journal/archive/lir91/).
3 Roswitha Poll and Philip Payne. ‘Impact measures for libraries and information services.’ Library Hi Tech, 24 (4), 2006 [forthcoming].
Pauline Blagden is Service Development Librarian, Portsmouth Hospitals NHS Trust (pauline.blagden@porthosp.nhs.uk), and Philip Payne is Librarian, Birkbeck, University of London (p.payne@bbk.ac.uk).
Updated: 13 September 2006