Researchers are engaging with the public more than ever before, but most have no idea how they are benefiting the people they interact with. Many researchers cite lack of time or funding as reasons for not evaluating the benefits of their work with publics. Those who do seek this information, often discover that impacts are complex, long-term and difficult to attribute to their work. Commonly used evaluation techniques (such as visitor surveys) rarely capture the breadth or depth of impacts arising from public engagement, or the long-term benefits.
Despite these challenges, researchers are increasingly expected to be able to articulate the value of their work with publics, to justify funding and demonstrate the impact of their research. This guide explains how to do this easily and effectively, based on research funded by Queen Mary University of London (Reed et al., 2018).
Why is it important to evaluate public engagement?
You can do better public engagement: Evaluating public engagement is essential if we want to improve our practice in this area. Evaluation enables us to better understand the interests and priorities of different publics, so we can better meet their needs and provide them with opportunities that they find meaningful, enriching and valuable. Knowing what works (and what doesn’t) can help us choose methods and activities that will engage publics more effectively, wasting less time and generating more impacts from the time we do invest. Evaluating the design of our public engagement can help us anticipate challenges and avoid using methods that are unlikely to work or that might lead to unintended negative consequences. When things don’t go according to plan, our evaluation can give us ideas about how to get things back on track or do things better next time.
You can communicate the benefits of public engagement more effectively: Evaluating public engagement can also provide important evidence of the benefits of research, informing funders and others with a stake in research impact. There is growing interest in evaluating the positive impacts of research for society, and this includes the benefits of engaging the public with research. The UK has led the way with the inclusion of impact in the Research Excellence Framework in 2014, which has subsequently been used to allocate pubic funding to Higher Education Institutes. Australia is now trialing the evaluation of research impact in its Engagement and Impact Assessment as part of Excellence for Research in Australia, and the European Commission are looking at how to evaluate impact more effectively in the successor to its Horizon 2020 funding programme.
What should I evaluate?
Evaluate the delivery and immediate outputs of engagement: Most evaluations of public engagement focus only on evaluating the delivery of an event or activity and its immediate outcomes. Indicators may be identified and tracked using a range of techniques that can quickly and easily tell you if you are getting the outcomes you expect.
However, if public engagement is as much about improving practice as it is about communicating benefits (see previous section), then it is just as important to evaluate the design of your public engagement. In particular, it is important to evaluate the extent to which the design follows known good practice principles, is adapted to your particular context and is underpinned by sound ethics. A well-designed public engagement process should typically:
Identify publics and stakeholders systematically, for example using the Fast Track Impact public and stakeholder analysis template
Understand and manage the expectations of these groups
Deliver tangible benefits that will be valued by each group in ways that are sensitive to their social and cultural context
Identify risks and assumptions and be prepared to adapt to changing circumstances
Engage experienced personnel who can manage events, facilitate workshops and organize engagement effectively
Evaluate the impacts of public engagement: Evaluations that focus only on the delivery of activities on the day and the immediate outputs of engagement often fail to articulate the broader, deeper and longer-term benefits of public engagement. However, evaluating the impact of public engagement can be challenging and time-consuming. To make this easier, the next section breaks this down into a series of simple tasks.
How do I evaluate the impact of public engagement?
To evaluate the impacts of public engagement, you need to know what specific impacts you’re looking for, design your public engagement to achieve these impacts, look for evidence that planned impacts are arising at the time and longitudinally, and be prepared to capture additional unanticipated impacts as they arise. Follow these steps:
Identify specific and measurable impacts that you would like to see as a result of your public engagement – consider what you might want to achieve in terms of changes in:
Understanding and awareness e.g. people understand an issue better than they did before
Attitudes e.g. a new appreciation for alternative views and more positive perceptions of people who hold different views
Behaviours e.g. more tolerance towards people who hold different views, more engagement with issues via other activities
Culture e.g. a change in the nature of public discourse around an issue
Capacity or preparedness e.g. access to knowledge or resources that were previously unavailable, which can be used at a time of need
Design your public engagement to achieve these impacts: build these goals into a logic model linking goals to specific activities that are tailored to the needs and interests of different publics, with activity and impact indicators you can use to track progress – see the Fast Track Impact Planning Template
Look for evidence that planned impacts are arising at the time and longitudinally: while you’re doing public engagement, design your data collection to look for specific planned impacts, and evaluate whether these impacts are arising longitudinally through planned follow-up with participants. For longitudinal work to be possible, you will need to incentivize participants to provide you with their contact details e.g. via a prize draw or joining a mailing list that provides additional free benefits or opportunities
Collect additional evidence of unexpected impacts as it arises, ensuring that evidence is safely stored somewhere for later reference
What sort of evidence do I need?
The sort of evidence you will need to convincingly demonstrate impact will differ from project to project. Play “devil’s advocate” and ask yourself what you would need to do to convince someone who does not believe that your public engagement has led any sort of impact.
In some cases, to be convincing you will need quantitative evidence e.g. an increase in visitor numbers and museum revenues after the installation of a new exhibit based on your research. In other cases, qualitative evidence will be more convincing and appropriate, for example illustrative quotes describing how engagement transformed people’s attitudes towards an issue or group of people.
Quantitative evidence might be most appropriate when:
You need to demonstrate cause and effect or a relationship between two things e.g. the number of students choosing to study a particular subject at University after watching TV programmes based on your research, or attributing a sales spike in a product to public engagement with your research on its health benefits
You need to describe exactly how much something has changed (or is likely to change) as a result of your public engagement e.g. awareness of an issue, or the prevalence of new behaviours
Qualitative evidence might be most appropriate when:
You need to explain why public engagement led to observed changes e.g. by challenging assumptions or attitudes that were underpinning previous behaviours
You need to represent a range of perspectives, showing how different groups have benefited in different ways from public engagement e.g. differences in the benefits that mothers, fathers or children derive from a public engagement event that targets families
What methods can I use to collect evidence of different types of impact?
Different types of impact often need to be evidenced in different ways. Decide what impact you want to see, and then based on this decide whether you need qualitative, quantitative or mixed methods. For example to evidence:
Understanding and awareness impacts: you will need to evidence how you have achieved learning outcomes among target publics, for example:
Quantitative surveys evaluating levels of knowledge and understanding around an issue before and after engagement e.g. entry/exit quizzes, website or in-app questions based on interactions with website or app content
Multiple-choice questions before, during and/or after public lectures using audience participation technologies
Qualitative content analysis of comments left in visitor books, feedback forms, stuck on post-it notes to walls or objects, written on a graffiti wall or written on post-cards that are posted back to participants after the event
Quotes from interviews or testimonials describing changes in understanding or awareness
Attitudinal impacts: you will need to evidence how attitudes have changed in response to your public engagement activities, for example:
Before and after Likert scale survey questions to quantify the extent to which attitudes have changed
Semi-structured interviews analysed using content analysis or Grounded Theory Analysis
Group or individual interviews based around pictures drawn by participants as part of the interview process to elicit implicit knowledge
Q methodology to quantify and categorise people who identify with contrasting attitudinal statements derived from interviews
Content analysis of social media comments linked to public engagement events
Quotes from interviews/testimonials describing attitude change
Behavioural impacts: you will need to provide evidence showing that behavior change has taken place, ideally at scale, for example:
Longitudinal questionnaires asking people to self-report changed behaviours and assess the extent to which engagement with research contributed to behavior change
Observation of behaviours during follow-on events and activities to observe and record evidence of changes in behavior
Analysis of secondary data showing evidence of behavior change, for example purchasing behaviours
Semi-structured interviews analysed qualitatively to describe behavior changes and the reasons for those changes in depth
Photo survey techniques where participants take photographs of their community or something linked to the research, you talk about their photos during interview to elicit implicit knowledge and photos are displayed in a community exhibition to gather further data about the changes that have taken place since your public engagement work
Focus groups in which findings from other techniques can be checked and critically discussed
Quotes from interviews or testimonials describing changes in behaviour
Cultural impacts: you will need to provide evidence that public engagement has led to changes in the way people think and talk about issues at a societal scale (or at least the scale of specific social units), for example:
Media mentions of your research and/or public engagement that are influential and well received e.g. in publications with large circulations, which are picked up by other influential media outlets or commentators, and widely discussed in the public domain
Media and social media analysis of public discourse around the issues from your research and/or public engagement, showing trends and attributing (at least in part) these changes to your work e.g. via testimonials from influential figures
Changes in cultural policy linked to your research and/or public engagement, evidenced via mentions and citations in policy documents and/or testimonials from policy-makers
Capacity or preparedness: you will need to provide evidence that public engagement has led to changes in specific capacities of individuals, groups or communities, and that this increased preparedness has enabled people to be more resilient or cope better with challenges when they arise, for example:
Social network analysis before and after your work, showing larger, more connected networks
Secondary data showing changes in capacity such as access to resources that were not previously available
Quotes from interviews or testimonials describing changes in behavior
Find out more
Find out more about the tools described in this section in the Public Engagement Evaluation Toolkit we helped create for Queen Mary University of London, available at www.fasttrackimpact.com/resources in early 2018.
Reed M.S., Duncan S., Manners P., Pound D., Armitage L., Frewer L., Thorley C. and Frost B. (2017) ‘A common standard for the evaluation of public engagement with research’. Research for All, 2 (1): 143–162. DOI 10.18546/RFA.01.1.13.
For more information and links, see other Fast Track Impact resources, listen to our podcast or check out our free magazine and The Research Impact Handbook.
If you are interested in impact training, find out about our courses here.