For many NOF-Digitise projects, two areas presented real challenges – engagement with target audiences and assessing the impact of project outcomes. Julie Carpenter offers a consultant’s view on their importance in project planning and implementation.
When the NOF-Digitise programme1 was launched in 1999, ‘learning’ generally meant ‘formal education’, and ‘lifelong learning’ was a policy agenda that had not entered organisational practice. Things have changed considerably, and promoting learning is now core business for libraries. The cultural sector has benefited from guidance and tools for measuring informal learning outcomes from the Museums, Libraries and Archives Council (MLA) in Inspiring Learning for All.2 Its work on Generic Social Outcomes will also result in guidance for public libraries in identifying the extent of their contribution to community cohesion and civic values, in which learning is recognised as the key to success.
NOF-Digitise showed us that making a real impact on learning through online projects depends on understanding specific target audiences and closely targeting content and activities to meet their particular learning needs.
So what do we know now about how to develop and deliver a successful online learning project – and demonstrate your success?
More than ‘consultation’
Before your project becomes more than an idea you must know who the target beneficiaries are: indeed, the idea should have arisen from an expressed or perceived audience need (if it has not, question its validity immediately!). Libraries are increasingly involved in extending their reach within their community, so online learning projects are likely to address the interests of hard-to-reach groups, or be intended to extend access to online content among excluded communities. These are not the people that generally walk through the library doors, or normally turn to the library for help. For many public libraries, working in partnership with other public services or voluntary sector organisations is the best way to reach these potential target groups. But it might also seem like a way for the library staff to avoid direct contact with target communities!
You need a good understanding of your target audience’s social and economic circumstances, the various complexities of their lives, their aspirations, key competencies and preferred learning styles.3 Look for evidence elsewhere about these particular or similar communities and their life circumstances, for example in government or voluntary sector reports, or in research done in other sectors etc.
Finally, however, there is no substitute for direct engagement, through participatory research, with individual and community beneficiaries. This will help you define the project objectives, intended outcomes and desired impact. This means more than just ‘consultation’, and, though it may seem a difficult, time-consuming and often frustrating exercise, in my view it holds the key to successful projects with achievable and realistic goals.
‘Whilst consultation is a valuable tool to library services, it was felt that engagement with communities should ideally go beyond consultation. Engagement involves local people in identifying areas for development and getting actively involved in delivering these improvements or additional services.’4
The MLA, in its download library accompanying Inspiring Learning for All, provides useful ‘guidelines on involving users’,5 and advises:
- Start with a blank sheet of paper and allow users to set the agenda.
- Don’t rely on paper-based communication. Poor literacy and lack of English can provide a barrier to many users.
- Meet users at a time and place convenient to them, respecting their desire for confidentiality and informality. Libraries, museums and archives need to be willing to meet others on their terms and territory.
The next challenge is to keep your target audience continuously involved and participating in project direction, monitoring and review. You will need to explore ways of rewarding or remunerating participants for their time and contribution. You may find it’s not easy ensuring their commitment over time. However, if you can’t stimulate enthusiasm for your project among a few key people, is the project really on the right track? Perhaps you should look again at what you are trying to achieve and why!
Developing interactive features as project outputs is a good way of attracting audience participation and keeping your audiences involved. The evaluation of NOF-Digitise indicated that interactivity in online projects – the opportunity for users to register and take part in activities, personalise their websites, comment on, contribute to or change content – not only enriches the learning experience, particularly for the young and other groups facing barriers to learning, but also allows you to discover more about your target audience and the way they use and react to the project outcomes. Starting as individual ‘physical’ users, your audience can become a virtual learning community, exchanging views and sharing knowledge and enthusiasm among members and with you.
Impact assessment
Participation of your target audience is also essential to help you identify relevant impact goals, and set and monitor measurable impact indicators. What do I mean by ‘impact’? Outcomes and impact are very hard to untangle; Peter Brophy and Susi Woodhouse provided a useful definition for NOF-Digitise:6
‘Outcome measures relate to those things which happen as a result of the outputs. So the web pages accessed will, hopefully, lead to some at least being read.
‘It is when we start to measure impacts that the real answers begin to emerge. The question[s] that must be answered [include] “What good does this resource do?” or “What difference has it made?” … These are the most valuable measures of all – but they are also the most difficult.’
In other words, a learning outcome would be what you wanted your target audience to do or achieve as a result of your project; the impact of that learning outcome would be what happened to them as a result, or how the outcome changed them. Required outcomes and desired impact should be directly relevant to the needs, interests and circumstances of your target audience. Discussion and shared planning of project objectives, outcomes and anticipated impact with members of your target groups are, therefore, an essential part of preparing a final project plan.
Project planning with your target community should also help you to identify appropriate indicators to be used in monitoring and impact assessment. Indicators in this context are what you will need to look for – signs or signals – that show, by their presence or absence, whether the online learning project has had the desired impact. For instance, if your online learning project is about healthy eating, and a desired or potential impact was to change eating habits, one indicator of impact might be a reduction in the number of times per week community members eat meat; another might be a lively trade in vegetarian recipes within the virtual learning community.
Methods
Your choice of research techniques to engage with your audience in project planning and impact assessment will depend on who they are, what impacts you want to achieve and what resources you have. In most cases you are likely to use a mix of quantitative (e.g. questionnaire surveys, gathering statistical data) and qualitative (e.g. focus groups, interviews, observation) methods. The mix will allow you to gather evidence about indicators in different ways, to ‘triangulate’ or validate different assessment findings.
To continue with my example above, to find out whether behaviour (for instance, eating habits) within your target community has significantly changed you might start, before the project begins, by gathering information from community members in a questionnaire survey about how many times they eat meat in a week (an impact indicator): this will be your baseline data. Then you might follow up with the same group and the same question after your project outputs have been available and in use for some time, by repeating the questionnaire survey. Their answers will indicate not only whether the desired impact happened but to what extent (proportion of target community) it happened.
On the other hand, if you want to find out whether the project outcomes have had any impact on attitudes or feelings (for instance, a different attitude towards an individual’s health, or greater awareness of food issues) a discussion forum, such as a focus group, will be more effective, as feelings and attitudes are not ‘quantifiable’ or easily comparable across different individuals.
Both these examples point out the importance of developing an initial ‘baseline’ of knowledge and understanding about your target audience relevant to the areas in which you hope your project will have an impact. Impact assessment is about measuring change and progression, so you will need to know where you came from so you can compare it to where you end up!
Finally, experience in the NOF-Digitise programme indicated how difficult and often ineffective it is to engage with your target communities only online. Online surveys attract notoriously poor responses; web statistics can often be misleading as the underlying data and the data analysis techniques used may be flawed.7 If your project plan does include interactive features, and facilitates virtual communities of learning or interest, then you are likely to get good responses online from your participating audience members and rich evidence of impact from their online interaction with each other (for instance, through chat or discussion lists). However, I suggest you adopt a ‘blended’ approach to engaging with communities – nothing beats the personal touch of an interview or face-to-face conversation.
Resource implications
Participatory project planning and impact assessment requires considerable time and staff resources, which need to be realistically estimated, with adequate provision for direct costs, and built into your project budget. However, if you are seeking external funding there are few funders now that do not expect convincing evidence of audience needs and clear methodologies for monitoring and assessing impact as part of early project proposals. Some funders will provide modest grants for preparatory measures, which would certainly include effective engagement with target communities in planning. Also remember that the number of different individuals you use in focus groups, targeted surveys etc, or direct involvement in planning, need not be large. It is the quality of their responses and input, and how representative they are of the wider target audience, that matter most, not the volume of contributions.
Time to achieve good results may also be a challenge – projects are usually time-limited. While project outputs can be timetabled, and project outcomes may follow immediately (for instance, if a user completes part one of a healthy eating learning package there will be an immediate learning gain), the impact of that learning outcome may take a long time to emerge and may be subject to non-predictable influences (for instance, learning more about healthy eating may only change behaviour when a favourite fruit appears in the shops). People learn at different speeds and in differing circumstances – many, for instance, may not fully appreciate your project’s learning content until they have discussed it with others in the community. To be effective, therefore, impact assessment activities should take place at key points throughout the project cycle and some considerable time after the project has technically ‘finished’, i.e. all project outputs have been completed.
Conclusion
All this, you may say, is unrealistic for your organisation, hard-pressed for resources of all kinds, particularly staff, and wanting a ‘quick win’, perhaps, through repurposing some existing digital content for new audiences. If this is your reality, remember that many projects in NOF-Digitise felt with hindsight that they should have spent more time on the project planning and design stages before doing anything. Their relative lack of engagement with their target audiences made implementation much more challenging, and subsequent efforts to sustain their achievements were not helped by being unable to say definitively what impact their projects had upon whom. I rest my case.
References
1 www.biglotteryfund.org.uk/publications.htm
2 www.inspiringlearningforall.gov.uk
3 Much useful background information summarising different approaches to learning is now available on the web. MLA provides guidance on ‘learning styles activity’
(www.inspiringlearningforall.gov.uk/uploads/
Learning%20styles%20activity.doc).
Barbara Allan provides a concise summary of current approaches in e-learning which is relevant in other contexts in E-learning and Teaching in Library and Information Services (Facet Publishing, 2002 – new edition due autumn 2007).
4 Community Engagement in Public Libraries: a report on current practice and future developments. MLA and CSV Consulting, April 2006.
5 www.inspiringlearningforall.gov.uk/utilities/
download_library/index.aspx
6 Peter Brophy and Susi Woodhouse. Evaluation and Impact Assessment for NOF Digitise Projects (www.ukoln.ac.uk/nof/support/help/papers/impact-assessment/).
7 See Impact Analysis for Web Sites. Ukoln – AHDS Briefing 99
(www.ukoln.ac.uk/qa-focus/documents/briefings/
briefing-99/briefing-99-A5.doc).
Julie Carpenter is a Director of Education for Change Ltd (j.carpenter@efc.co.uk).
Updated: 13 September 2006