Time will tell: how to measure the impact of the PN - Margaret Kendall & Jenny Craven

Substantial evidence of the success of the People’s Network initiative has been gathered at both local and national levels,1 in the three years since the deadline was met for the installation of computers in all public libraries. The major evaluation report from the Tavistock Institute2 found evidence of very heavy use, with many libraries needing to provide booking systems or time limits to control demand at peak periods. The proposed impact measures3 for 2005/06 will provide firm statistical evidence on the percentage take-up of available ICT time in every library authority in the country.

However, while the House of Commons select committee confirmed the value of the PN and deplored the re-introduction of charges for internet use by some authorities,4 recommendations for meeting the costs of maintaining and investing further in the initiative were deferred to the relevant government departments and local authorities.

In order to win the arguments for further funding, reliable quantitative and qualitative evidence, gathered over time, will be needed to show the PN has made a difference to people’s lives. In particular, evidence of how it has attracted users from groups at risk of social or digital exclusion will be needed to support the case locally and nationally. Longitudinal research is important because many of the effects of services can only be assessed over a period of time: they depend on either continued interaction or on a range of influences which affect a user.

A toolkit5 has been developed which provides guidance for public library managers on tried and tested methods of gathering and disseminating information on the impact of electronic services on individuals and groups.

Related research and guidelines
Since 1997, the Public Access Computing Project at the University of Washington has carried out a series of evaluation research projects on the effectiveness of the US Library Program,6 instituted by the Bill and Melinda Gates Foundation. They found that for many people on a low income or unemployed the public library was their only access point to computers and the internet.

They also discovered that there was much higher reliance on libraries for access to computers by African American and Native American communities, and that public access computing has had a significant impact on the visibility and reputation of libraries serving rural communities. An earlier research project by Cerlim, the Vital project,7 developed and tested a toolkit of questionnaire and interview schedules for users and non-users to enable similar ‘snapshots’ of impact to be assessed.

Such snapshot research projects may provide useful indicators, not only of increased use by people from groups previously under-represented among public library users, but also of a need for more proactive promotion of the services to those not yet included.

Maybury8 found that the internet user profile at Chester was dominated by the younger adult age groups and those involved in active employment, with the majority of her 78 questionnaire respondents (61 per cent) having heard about the internet facilities through the library itself.

Muddiman et al9 identified the need for ‘ICT and networking developments which actively focus on the needs of excluded people’ (p.58) in order to avoid resourcing strategies reproducing patterns of favouring existing library users. Their finding that only around a third of public library authorities in their sample carried out community profiling as a basis for service planning was startling and perhaps indicated the extent to which funding cuts in the late 1980s and 1990s had led to false economies.

The reassertion of the importance of community profiling in the Museums, Libraries & Archives Council’s proposed impact measures10 will redress this and help authorities identify needs for more proactive promotion to specific target groups. Combining local statistics identifying the extent of use of the PN computers with data from community profiling will make the results more meaningful.

Figure 1: Longitude Toolkit contents

1 Definitions
2 Planning longitudinal impact assessment

quantitative methods

qualitative methods

issues for consideration

3 Collecting your data

• statistical data
• questionnaires
• interviews
• focus groups
• diaries
• observation

4 Sampling methods

5 Analysing your data

qualitative data analysis
quantitative data analysis

6 Using and presenting your data

7 Longitude in context

• Library profiles to provide quantitative and descriptive data, e.g. usage statistics of the sample population. Although these measures will not provide evidence of impact themselves, they can be used to back up or illustrate other data collected. They also have the advantage that they are often readily available and may not require additional effort to collect. Following initial collection, it is not necessary to start from scratch every time the longitudinal evaluation process is conducted. For example, in Phase 2 of the project, both libraries were asked to provide a small amount of extra data to reflect any significant changes in services or in the general environment.
• Demographic data on participants in interviews and focus groups.
• Interview surveys to measure value and impact by asking participants to rate their own attitudes and perceptions using, for example, a Likert scale.
• Illustrations of usage to provide a richer picture of the services and their impact on the users.
• Focus groups to gather evidence of the impact of the library-based IT services to specific groups of users.
• Case studies extracted from both the quantitative and qualitative methods of both data collection exercises

From the above methods, the partner library authorities selected those most appropriate to their operational setting. For example, Birmingham had some existing groups of users who would be able to take part in focus groups, so decided to adopt this method; Cheshire had already successfully used the interview method in a previous project (Vital). Selection of appropriate methods is an important factor for libraries considering data collection of this type, and for this reason a variety of methods as well as those tested have been included in the toolkit.

Interviews were carried out by Cheshire with a pre-selected sample of subjects at the start of the project. Subjects were randomly selected from a sample of libraries and the interviews took place over a two-month period. A target figure of no less than 50 subjects was set by the project team. It was felt this would provide enough data to illustrate the impact within the timeframe of the project, taking into account that some people would not be available for re-interview in nine months’ time.

Focus groups
Focus groups were facilitated by Birmingham and included a combination of members of existing library groups such as a local history group and a coffee morning group, and individual library members who used the PN facilities. Although in most cases the groups involved in Phase 1 also took part in Phase 2, individual participants were not always the same. However, it was felt that they would represent a similar group of people and therefore retain the longitudinal aspect of the study.

Interview data collected showed opinions relating to the IT facilities in the library had not changed much over the nine-month period, although interestingly, whereas the individual value of IT access dropped, opinions relating to the importance of providing access had risen. Comments from participants confirmed that even where the value for them had diminished they felt in general it was important to provide IT access in the library.

Data gathered on the use of the IT facilities in the library showed that, in general, reasons for use had broadened over the nine-month period. For example, learning a new skill was mentioned by 19 per cent of participants in Phase 1 and by 39 per cent in Phase 2. Accessing government and local authority information was mentioned by 16 per cent in Phase 1 and by 30 per cent in Phase 2. Illustrations of use provided richer, qualitative data to back up the responses. For example, one participant in Phase 1 identified looking for general information as one of the reasons for using IT facilities in the library, going on to describe that this concerned jury duty. In Phase 2 the participant went on to describe continued use of the internet in the library to pursue an interest in the law: they are now thinking of looking for work in this area.

The focus group data demonstrated how an IT group which met once a week progressed over the nine months, their computer skills and confidence both improving. In Phase 1 participants described how they were apprehensive about using computers and sometimes found them difficult to understand. In Phase 2, they described how they had learned new skills, gained knowledge and increased in confidence (although many felt the need for further help or training).

Rich picture
The results confirmed that it is possible to demonstrate the impact library services have on the lives of the library users, and how qualitative data such as the illustrations and focus group discussions can be combined alongside more traditional measures such as library profiles, demographic data and usage statistics, to provide a rich picture of impact over a period of time.

These examples provide data over a very short period of time. It would be interesting to re-visit the participants (interviewees and focus group members) after a longer period of time to see whether there were any changes relating to their use of the library IT facilities and the impact (if any) this has had on them.

The Longitude Toolkit was developed using the tested methods described above, together with other recognised methods for data collection and analysis. The toolkit has a simple front page with links to relevant sections. These can be followed sequentially or topics can be selected randomly according to need. Figures 1, 2 and 3 show sample web pages from the toolkit. Figure 1, the toolkit contents, starts with some definitions of terms used (for example, qualitative, quantitative, longitudinal evaluation). It then moves through the process of planning for longitudinal impact assessment, data collection, sampling techniques, data analysis and presentation of data. Links are provided on the page to other work that can be used in context with the toolkit (for example, details about Framework for the Future, the Vital project, etc).

Figure 3. Longitude II guidance notes

Longitude Toolkit: collecting your data
Observation: Notes for guidance in observation
1 Devise a system were data can be recorded as unambiguously and faithfully as possible.
2 If you are using video or sound recording equipment, leave plenty of time for set up and testing.
3 Where possible, record observations on the spot, during the event. This may involve devising codes or abbreviations that can be transcribed later.
4 The record must be transcribed shortly afterwards to add detail and substance and to ensure it is understandable.
5 Remember that transcribing a record can often take as long, if not longer, than the original observation.

Example of observation protocol
User-Friendly Handbook for Mixed Method Evaluations:
www.ehr.nsf.gov/EHR/REC/pubs/
NSF97-153/C3APP_A.HTM.
Developed from Weiss, I. 1997. Local Systemic Change Observation Protocol.

Each section of the toolkit provides a brief introduction to the topic, together with further instructions, examples and references to further reading. Figures 2 and 3 concern toolkit techniques for collecting data. Figure 2 shows how the toolkit describes the advantages and disadvantages of conducting interviews. Links are provided to further guidance and examples of interview techniques. Figure 3 is guidance notes for observational techniques and a link to an example of an observational protocol.

The Libraries Impact Project has provided evidence of public libraries’ contributions to government priorities for education, children, health and older people.13 The methods for quantitative and qualitative data collection tested through pilot authorities are freely available and their use is recommended for further longitudinal research at a national level. In a similar way, the Longitude Toolkit provides support in measuring the impact of the People’s Network initiative over time at a local level and can be used alongside tools such as Inspiring Learning for All14 and the DCMS impact measures.


The government, in its response to the select committee report, sees the roles of public libraries ‘as enablers of access to digital skills and services including e-government, and facilitators of measures to tackle social exclusion, to build communities and develop citizenship’, not as something separate, but as part of their core function. It states: ‘It is crucial that library authorities work up the detail of how they engage with these agendas based on their expert knowledge of local needs.’15 J

References
1 P. Gannon-Leary, L. Banwell and S. Parker. ‘An evaluation of the development of the People’s Network in the North East.’ Library and Information Research, 27(87), 2003, pp. 5-16.
P. Brophy. The People’s Network: moving forward. MLA, 2004.
2 Books and Bytes: new service paradigms for the 21st century library. An evaluation of the People’s Network and ICT training for public library staff programme. Tavistock Institute, 2004 (www.mla.gov.uk/documents/pn_evaluation_full.pdf).
3 Improving Performance. MLA (www.mla.gov.uk/action/framework/framework_04a.asp). 4 Third Report of the Culture, Media and Sport Select Committee, Session 2004-05: Public libraries. February 2005
(www.publications/parliament.uk/pa/cm200405/
cmselect/cmcumeds/81/8109.html).
5 The toolkit was developed from a two-year research project undertaken at Cerlim in partnership with Birmingham City Libraries and Cheshire County Libraries, with funding provided by the MLA (Museums, Libraries and Archives Council).
6 US Library Program: Summary of research reports. Bill and Melinda Gates Foundation (www.gatesfoundation.org/downloads/libraries/
eval-docs/pdf/researchsummary.pdf).
7 J. Eve and P. Brophy. Vital (Value and Impact of IT Access in Libraries): methods workbook. 2001 (www.cerlim.ac.uk/projects/vital/index.php).
The Vital project was carried out in partnership with Cheshire, Birmingham and Cumbria library authorities.
8 J. Maybury. A Study to Investigate the Public Internet Service at Chester Library. Unpublished MA dissertation, Manchester Metropolitan University, 2003.
9 D. Muddiman et al. Open to All? The public library and social exclusion. Volume 1: overview and conclusions. Library and Information Commission Research Report 84. Resource, 2000.
10 See 3.
11 S. Sharp. ‘Fast forward to the future: e-enabling in Leeds libraries.’ The Electronic Library, 23(2) 2005, pp. 237-43.
12 R. Yeates. Longitude. Library and Information Commission Research Report 121. Resource, 2001 (www.viscount.org.uk/research/longitude/).
13 Libraries Impact Project. Pricewaterhouse Coopers, for Laser Foundation, 2005 (www.bl.uk/about/cooperation/pdf/laserfinal6.pdf).
14 Inspiring Learning for All. MLA (www.mla.gov.uk/action/learnacc/00insplearn.asp).
15 Government Response to the Third Report of the Culture, Media and Sport Select Committee, Session 2004-05: Public Libraries. Sections 18 and 19 (www.culture.gov.uk/).

Margaret Kendall (m.a.kendall@mmu.ac.uk) is Senior Learning and Teaching Fellow, and Jenny Craven (j.craven@mmu.ac.uk) is Research Associate, Cerlim, at the Department of Information and Communications, Manchester Metropolitan University.
The Longitude Toolkit and the Longitude II Project Final Report are available at: www.mla.gov.uk/action/pn/longitude.asp