Does your service make a difference? That’s not always easy to answer, says Christine Urquhart, who offers some practical guidance on impact assessment and some ‘health’ advice on negative impacts.
With the current emphasis on benchmarking, and finding evidence for the effectiveness of our services, it is often easy to assume that impacts are all positive, and that identifying the true impact of our services will be good news. That, alas, is not necessarily so. Negative impacts are more insidious than positive impacts, often appearing as the unwanted side-effect of some well-intentioned policy. But, unless we consider these risks, all the efforts to measure positive impacts may be worthless.
Many of the research projects we have done at the Department of Information Studies have been impact or evaluation studies, and the advice offered here is based on experience gained over 10 years of conducting impact studies.
Perhaps the first thing to consider is why we are thinking of doing an impact study in the first place. Are the results intended to justify our existence, or the addition of a new service? Are we hoping to extend services and show that some additional funding is necessary? Or is it simply part of the normal strategic planning process, to plan more effective provision? While the last is the one option we should all be choosing, impact assessment is often seen as an external imposition, something that funders or policymakers have asked us to do, which we would not be doing otherwise. That often sets us off with the wrong assumptions or aspirations that are difficult to satisfy. There is, possibly, a subtle difference between assessing the value, the worth of a service and whether it made a difference. While we usually want to show that our services are intrinsically worth funding, it is sometimes easier just to content ourselves (and the funders) with discovering the service has made a difference, or has had a measurable impact.
Distinguishing the value
Another problematic part of impact assessment is the difficulty of distinguishing between the value of the information provided and the value of the information provider. If the information provided made a difference to the customer, their opinion of the information service is likely to be favourable but, if the information provided was not useful, ultimately, the information service tends to be viewed the same way. Customers can be entranced by the results of very simple and quick searches, whereas other searches can be conducted very thoroughly but still come up with nothing that matches their perceived needs.
On the other hand, trying to value information leads us into all sorts of complications, because what one person considers data, just a load of figures or words, may seem like pearls of wisdom to another. In settings where there is a discrete set of users, with similar interests and motivations, it makes sense to try to assess the value of information provided, as it is possible to examine the outcomes of the information provision, and answer the all-important questions – what did the people do with the information provided and how did it make a difference? We can get a longer-term view of the importance of the information to the group and a better idea of the impact. Often, however, we have to content ourselves with an indication of the difference that the information made, or the impact of the service. It’s best to remember that value is often priceless, but impacts leave a measurable change in the shape of things.
Making a difference
There are three main ways that information services usually make a difference – educationally (learning and skills for learning), socially (community cohesion, personal well-being) and ways of working (changing practice or decisions made). Normally we’d expect all impacts to be positive, but the way we deliver services may get in the way of the impacts.
Often this is the ‘two sides of a coin’ problem. The positive viewpoint sees services being targeted at those in most need, the negative sees this as rationing. The positive viewpoint sees service provision as very fairly distributed, the negative sees service content diluted to the point where it seems hardly worth bothering at all. Positively, keeping to ‘core competencies’ ensures a safe and reliable service but, negatively, that may limit the experimentation and piloting of new services. And if new services are developed on some project funding, can these services be sustained? In any type of impact assessment it pays to dig behind the management jargon and consider whether adopting one course of action will have any hidden problems associated with it. The flip side of priority setting is rationing, after all.
Knowing our customers
If we want to make a difference, at least to some of our customers, then we need to know them well. That means being able to group them into categories of types of behaviour of interest to us, and our funders. Ideally we should know where we can make the most difference, and how. Realistically, we also need to be aware of the Pareto principle, in this case meaning that 80 per cent of the service usage will come from 20 per cent of the customers.
One of the observations I made when auditing library usage statistics many years ago was the limited perspective of some customers, who only ever used one of the services. The staff assumed that users had a global view of all the services offered, but I suspect they didn’t. And there were some libraries which devoted an enormous amount of staff time and energy to serving the needs of a few, where the ratio was nearer 90 per cent of the usage of a particular service coming from 10 per cent of the customer base. Of course, information obtained for one individual may trickle down and out to other potential users, but it could be wise to check that assumption.
So, we need to know what matters to our customers,1 what the perceived benefits of the information services are for them, and what the drawbacks are, and we need to know that for the various groups of customers that we have.
Users’ stories
For many public libraries the value and impact of their services are concerned with social aspects, and the Libraries Impact Project2 showed how library authorities could devise impact measures that were related to what policymakers wanted (e.g. the priorities set out in the government’s Every Child Matters).
The report stresses the importance of complementing the quantitative data from usage statistics with qualitative data – subjective views from users on the difference the service has made to their life. The stories of how a service has made a difference to an individual are often very powerful and compelling and offer lessons for further service development. However, in themselves they do not make a case for further funding unless there is the quantitative data to indicate that the service is reaching new users, or that existing users are making more use of services.
The qualitative data is very important in helping us to understand what makes a difference to our customers, what they think is important and what type of improvements they would like to see. Unfortunately, there are a few problems with the collection and use of such data. It can be viewed as normal audit of services, or needs analysis, but personal data is personal data nevertheless. Those managing the collection of qualitative data on service usage should be aware of the high ethical standards required in social science research in the UK nowadays.
Research Ethics Framework
In July 2005, the ESRC (Economic and Social Research Council) published its Research Ethics Framework,3 which sets out a mandatory framework for all ESRC-funded research. There are six key principles: the integrity and quality of the research; informed consent; confidentiality of data supplied and anonymity of respondents; voluntary participation of participants; avoidance of harm; and independence of research, being clear and open about any conflicts of interest. Much of the framework is common sense and, as it notes, ‘almost with exception, social science research in the UK has been carried out to high ethical standards’. What is different now is the need to be able to demonstrate you have done the right things, and that you have fully considered the rights and dignity of the individuals you have interviewed.
Before starting any qualitative data collection I would advise reading through the ESRC framework and other ethical guidelines (e.g. those published by the British Psychological Association) to ensure that the questions you are asking are fair, open and honest, and that you are complying with accepted guidelines for informed consent, confidentiality and anonymity, and with data protection regulations. There should not be too much to worry about, but it is a double check that your procedures are correct and thorough, and that there is an audit trail.
Consider your questions
More importantly, reflecting on the questions you are asking may lead you to ask yourself whether there are better ways of phrasing them, and whether being more honest about your aspirations in the data collection might actually be more helpful to you as well as the interviewees. Most ethics committees class some of the user groups of most interest to you – children, older people and disabled people – as ‘vulnerable’ groups, who deserve particular care from researchers. That alone means that ethical considerations are very important.
It is probably important, given the ethical considerations, to distinguish between the immediate impact of information provided, or services used, and the long-term impact. Usually, we can more easily, safely and ethically measure the immediate impact (e.g. knowing more, or understanding more clearly) than the longer-term impact which asks about decisions made, changes in behaviour, relationships with colleagues. Of course, the long-term impacts are often the key impacts, and we can’t neglect these, but sometimes impact assessment work has to be realistic. By all means devote some time to assessing the important long-term impacts but don’t neglect the immediate ones that provide some useful indicators, too, particularly when time is limited for data collection.
Confirm what you know
In many of the impact studies conducted in the health sector, we have used one set of questions for the immediate, cognitive impacts of the information provided. These questions ask whether the users know more now, whether the information confirmed what they suspected, and whether they had expected to find anything else. Confirmation of what is already known may not sound as exciting as finding out something new, but the value of reassurance should not be discounted, particularly when professionals want to check that their decisions are correct and in line with the best research evidence.
Impact assessment in the health sector has often sought the holy grail of showing that information supplied by the library service has had a direct effect on patient care. This does happen but, when it does, it has usually been the last piece in the complex jigsaw of changing practice. For example, in one project, a nurse recounted how finding out about a different method of pain control had changed practice on her ward, and the discussions had in fact made for better team working as well. The one piece of information had contributed to huge changes on that occasion but the situation was probably critically poised for change.
Sharing information
More often, the information provided may be the start of a long process of auditing care provided to patients with a certain condition, negotiation with other members of the healthcare team, and agreement on changes in practice and procedures. Professionals want to know what other groups’ experience with a treatment was, and what is best practice elsewhere. Increasingly, librarians’ efforts are being directed to supporting the professionals in critical appraisal of the research literature, training them to find the evidence, and then filtering to find the best evidence. That requires considerable investment in time to train, and it is difficult to trace a connecting line between that training and better decisions that may not be made until years later. However, it is possible that the training makes for better and faster searches by the professionals. Training may also help them to differentiate between the type of searches that should be delegated to the experts, the librarians, and the searches they can do themselves.
Savings in staff time
Networking of resources, the development of digital library services, has changed the ways people work, and one of the obvious ways of demonstrating benefits of networking is to show a saving in staff time. For an evaluation of the National Electronic Library for Health, we asked respondents how long, on average, it would have taken them to find and read the information if they had not got it via the NeLH.4 It is important to remember that, if they would not have bothered to find alternatives, the cost savings have to be considered as zero. Also, the effect of the learning curve in using electronic resources needs to be considered, and some respondents indicated that it took them longer than it might have done ordinarily. We reckoned that some of the respondents had just got the answers the wrong way round, but we had to take the answers at face value.
It was partly our fault for not phrasing the questions very well but, in fairness, it is not easy to write straightforward questions about estimated time savings. The estimation is inevitably subjective, but it is possible to use such estimates to develop scenarios, for different groups of staff, on the possible time savings (or time costs), that networking offers. That is what we did for the pilot evaluation of the NeLH. We considered scenarios for different extents of usage of the networked service, and estimated the savings in staff time costs that would ensue. Forecasting is an uncertain business, but a mix of highly conservative as well as highly optimistic forecasts gives a better idea of the likely costs and benefits.
North Wales Clinical Librarian Project
One of our recent projects was an evaluation of the North Wales Clinical Librarian Project.5 Like many of the clinical librarian programmes, this had received special funding, and we hoped we could demonstrate that there were cost savings. The reality, of course, was more complex. The clinical librarian was helping multidisciplinary teams in critical appraisal and one of the main activities was coaching junior doctors prior to their journal club presentations. We found that for small journal clubs, the costs were greater with clinical librarian input, but that savings were possible with the larger journal clubs. Whatever the size of the journal club, the qualitative evidence showed that the journal clubs were valued far more by the participants, as the clinical librarian input ensured that the presentation was worth attending.
We also examined the effect on the type of clinical decisions made, the impact of training sessions, and the impact of searches conducted for individual professionals. Just as we had found in the forecasts for the NeLH, we had to present various scenarios to get a better idea of likely impacts in the future, and what the lessons might be for future service development.
Assessing the impact is very useful, and often vital when trying to demonstrate that new services deserve continued funding. However, what we are also trying to do is to learn more about the way the services were received, and how we might improve the service. Careful examination of the quantitative evidence can help indicate some of the likely trends, but the qualitative evidence gives insights into what really counts for the users.
Learning from others
When starting out on an impact study, there are lessons that can be learnt from the experience of others. A systematic review6 of the value and impact studies in the health sector provides advice on the type of impacts that can be measured and what the results are likely to be. More general guidance,7 but guidance based on evidence, on conducting impact studies provides a checklist of what to do, and what to expect. There is usually no need to start devising your own questionnaires. It is far better to replicate others’ well tried and tested methods. There may in fact be data that is already being collected that could contribute to an impact study. A common criticism of libraries is that they collect usage statistics but rarely use them to help assess the impact of their services. In many cases the data is simply not rich enough, but there is a case to be made for expanding the routine data collection on service usage8 to make impact assessment part and parcel of normal service audit, rather than a special effort in time and money.
Counting the ways
So, how should we count the mysterious ways that our services affect users? First we need to be clear about what we need to know, and why. Second, and probably just as important, we need to know what policymakers and funders need to know. Third, we need to know what type of evidence will be convincing, and that is likely to be a mix of quantitative and qualitative evidence. And then there are the practicalities. Who are the service users we need to contact? Are our questions appropriate and ethical for this group? Can we obtain the data easily and reliably? Last but not least, are we sure that our professional assumptions will not bias our interpretation of the findings? Be sure that you are seeing genuine positive impacts and that the negative impacts are not ignored. The aim is to learn from an impact assessment exercise, and to enhance future service provision. *
References
1 C. Urquhart. ‘Performance measurement in library and information services: health advice from the Value and Evince studies.’ Library and Information Briefings (71), 1997.
2 Libraries Impact Project, conducted by PwC LLP. Laser Foundation, 2005.
3 Research Ethics Framework. Economic and Social Research Council (www.esrcsocietytoday.ac.uk).
4 C. Urquhart et al. NeLH Pilot Evaluation Project Report. Department of Information Studies, 2001 (http://users.aber.ac.uk/cju).
5 C. Urquhart, J. Durbin and J. Turner. North Wales Clinical Librarian Project. Final project report. Department of Information Studies, 2005(http://users.aber.ac.uk/cju).
6 A.L. Weightman and J. Williamson. ‘The value and impact of information provided through library services for patient care: a systematic review.’ Health Information and Libraries Journal, 22 (1), 2005, pp. 4-25.
7 C. Urquhart. ‘How do I measure the impact of my service?’ In A. Booth and A. Brice (eds), Evidence-based Practice for Information Professionals. Facet Publishing, 2004.
8 S. Ede. Snapshot Diagnostic of Data Flow in Relation to the Library Domain. Resource, 2002.
Christine Urquhart is Senior Lecturer at the Department of Information Studies, University of Wales Aberystwyth (cju@aber.ac.uk).
Updated: 25 July 2007