Measuring the work of an information service gives you valuable data on which to base improvements.
But this data should be collected for specific purposes, not just gathered for the sake of it. James Mullan looks at different reasons – and tools – for gathering metrics.
Library and information services have gathered information on usage of their services for many years. Traditionally these measurements have been tangible, for instance the number of items issued by a library management system. These measurements are useful but what value to the organisation do they actually demonstrate and how do they help you develop your service? Increasingly information managers are looking at using new metrics to meet the changing demands of their organisation.
Unfortunately there isn’t a simple definition of metrics. But, when you think about any sort of analysis, you think about numbers or statistics; the process of collecting these numbers and making them meaningful is metrics. Metrics can be a practical management tool, helping assess effectiveness, highlight user satisfaction and support applications for additional resources.
Qualitative v quantitative data
Much of the debate about metrics centres on whether you should be collecting quantitative or qualitative data. Quantitative data is data in numerical form, for example the total number of enquiries logged during a month. By comparison qualitative data is usually presented in words, perhaps a description of a service or a description of how well an enquiry was answered.
Compared to quantitative data, qualitative data is usually considered richer and more vital, deeper and more likely to present a true and fuller picture of a service or department.
Statistics
Statistics can show what value you provide and how you can develop your service – but only if you are able to manipulate the data to show trends and changes in the way your service is used and how this has an impact on the business you serve.
You probably already record transactional information (number of items issued, books ordered and enquiries answered) but to understand how this affects your service and how you can use these statistics to improve your service you need to combine this raw data with other information. For example, can you easily break down the number of enquiries answered by department, member of staff, time of day or the sources used to answer the enquiry? Being able to interrogate your data like this will, over time, show you changes in the number of enquiries logged, who is really using your service and what services are being used and so support any future requests for resources.
At CMS Cameron McKenna LLP we use a system called Heat to record all enquiries logged by the central research service (the exchange). In addition to the exchange, the IT Support Desk, the Business Analysis Team and the Human Resources (HR) Department use Heat.
The decision to use Heat as a tool to record enquiries and produce detailed statistics centred on several key benefits:
- Heat is linked to our HR system so that the user’s department, job title and contact details are automatically populated
- We could tailor Heat so that enquiries could be logged using a specific set of research types and sub-research types
- The IT Department was already using the Crystal Reports reporting tool to interrogate the enquiries logged on Heat
- We could create a set of codes for recording calls logged by clients.
Client calls are a growing area of work for the team, so being able to show how many calls we receive and details of which client contacts are using us is crucial. The statistics we produce on client calls are subsequently included in a monthly and quarterly client report. This demonstrates to the client, and to the internal Client Service Team, what ‘added value’ we have provided.
As part of the enquiry-logging process, the time the call was received and closed is recorded automatically; most importantly, the time spent on the call is always recorded. This has helped us identify which practice groups are our heaviest users. Counting the total number of enquiries may not be an accurate reflection of a practice group’s usage if all its enquiries take 45 minutes rather than five minutes. Calls are also recorded as being chargeable or non-chargeable. Chargeable calls are normally recorded against a file number and will eventually be charged to the client – being able to demonstrate that we are creating revenue for the firm is important.
Reports
The exchange then uses the Crystal Reports tool to produce a report, which details calls logged during the preceding month. Of note is that nowhere in this report is the total number of enquiries for the month displayed. This figure could be calculated, but a decision was taken not to include it, as it doesn’t provide much value on its own. Instead the enquiries are broken down by how they are received (email, in person, phone), when (morning, afternoon, evening) and from whom (partner, assistant, trainee). Very quickly it became clear that some of our preconceptions were wrong. Notably we believed most of our enquiries came from trainees when in fact the numbers logged by trainees and assistants are almost equal.
A number of other law firms have started using off-the-shelf or custom-built applications to log enquiries and produce reports on usage1 and I think we will see an increase in the use of these tools. In the US a long-running project called LibStats2 has created a tool that allows managers to produce detailed statistics on what enquiries they are answering and the time they take.
Monitoring electronic services
Accurate reports on the usage of your electronic products are particularly important when allocating budgets and when developing your services, because of their high cost. But until recently there were few tools available to collate information from a wide range of electronic services. Nowadays, vendors have started providing monitoring tools but the statistics produced can be unwieldy and meaningless unless a lot of analysis is carried out.
Independent companies have also started producing monitoring tools. One such example is Onelog. 3 Onelog is a web-based management tool that will log a user on to an internal system once and can connect them seamlessly to all other systems. Onelog also allows library managers to review usage of individual titles, enabling them to make informed choices about renewals of electronic products. 7Side 4 also produces a product called Research Monitor, which automatically captures client information and user information as they use the electronic service, ensuring that any cost associated with the service is applied.
Gathering usage statistics on electronic products can also be useful for measuring the organisation’s return on investment (ROI) from these products, as well as for analysing training needs by reviewing who is, and who isn’t, using the products.
Key performance indicators
Key performance indicators (KPIs) can help a service or organisation define and measure its progress towards goals. KPIs are quantifiable targets, agreed in advance, that reflect success factors. KPIs are not statistics but a combination of statistics and the setting of targets; combining these statistics and identifiable targets makes KPIs a very powerful measurement of your service.
A service may have a KPI of answering all enquiries within 24 hours, or answering x number of enquiries for a particular department. The key to using performance indicators successfully is to avoid measuring too much or measuring things that aren’t important to management.
You also need to ensure that you look at less obvious data. For example, setting a target of x days’ absence per person may not appear obvious to many, but it might to anyone scrutinising the reports. The British Library has an excellent example of a KPI report available on its website,5 which demonstrates how KPIs can be used in practice.
Overall, KPIs are excellent at demonstrating the activity of your service: these measurements can readily impress management but they will not show how ‘good’ your service is. To measure this you need to gather qualitative data.
One way of doing this is to look at the use of your service by a customer and measure the degree of satisfaction with the outcome, using the ‘Surveys’ tool.
It’s vital for any organisation or department to monitor its client base (both internal and external). Surveys can be a useful way of doing this. They can reveal trends in usage and other important information such as how happy your users are with your enquiry service and which members of staff answer their enquiries.
Tips for user surveys
Gone are the days when most surveys would mean creating a word document and distributing it either in paper form or via email. Nowadays, most surveys will use web-based software like Survey Monkey,6 Question Pro7 and Wufoo,8 which allow you to design, collect and analyse the data. They generally achieve a better response rate from your users, because they are more convenient.
Before you think about sending out a survey you should always test it with a few people – for clarity, and to ensure it works correctly. When sending your survey you should always explain why you are asking a user to complete it and indicate how long it is likely to take. Successful surveys will normally take only five or 10 minutes.
The questions should be clear and you should avoid jargon like ‘SDI’ or technological terms such as ‘RSS’. From experience, anonymity has always been a useful way of encouraging more ‘honest’ responses. However, simply designing the survey and distributing it is only half the task. ‘To demonstrate your value be sure to publish the results soon after, in summary, and outline what they mean. This is your opportunity to explain things, show off what you have done (especially if you are showing a comparative improvement on a previous survey) and get support for what you would like to do in response to issues.’9
Conclusion
Metrics and the tools available to managers have come a long way but they are still subject to abuse (or misuse). The ‘10 Best Ways to Lie with Metrics’10 provides examples of how you can manipulate metrics so they look and feel good rather than provide an accurate picture. The use of metrics is becoming more widespread. However, although there are many tools and methods to use to gather metrics, the collection of metrics in some areas remains difficult, notably for knowledge management projects and initiatives and the use and value of social media tools, like blogs and wikis.
When deciding on a set of metrics you should ensure that they will measure something that is important to your organisation and that they can be measured accurately, consistently and efficiently. You should always think carefully about what you are measuring and why. Are you measuring the right things – the ones which help you to improve your service?
References
1 M.Mancino. ‘Research revolution at Eversheds’. Legal Information Management 7(1) March 2007, pp. 8-14.
2 LibStats (www.wendt.wisc.edu/projects/systems/libstats/index.do).
3 One Log (www.itsltduk.co.uk/onelog.asp).
4 7Side Research Monitor (www.7side.co.uk/index.cfm?alias=rschmntr).
5 www.bl.uk/about/annual/2005to2006/pdf/kpis.pdf
6 Survey Monkey (www.surveymk.com).
7 Question Pro (www.questionpro.com).
8 WuFoo (http://wufoo.com/).
9 S.Mansfield. ‘Live long and prosper: demonstrating the value of your information service’. Legal Information Management 6(1) March 2006, pp. 22-27.
10 S.Maxwell, ‘10 best ways to lie with metrics’ from the Scott Maxwell Blog (http://scottmaxwell.wordpress.com/2004/02/23/10-best-ways-to-lie-with-metrics).
Further reading
- F.Ryan. ‘Proving the value of information services using metrics: using a relational database management system to defend and develop your service. Business Information Review 23(1), pp. 26-34.
- B.Hendricks et al. ‘Establishing the return on investment for information and knowledge services: a practical approach to show added value for information and knowledge centres, corporate libraries and documentation centres.’ Business Information Review 23(1), pp. 13-25
James Mullan is Information Officer, CMS Cameron McKenna LLP (james.mullan@cms-cmck.com).
Updated: 27 November 2007