Many of us think that media engagement and impact go hand in hand, but this is not necessarily the case. Engaging with social and mass media can be one of the most powerful pathways to impact, but because the effects are so hard to measure, it can be challenging to evidence impacts. I meet social media and television stars, researchers who have had documentaries made about their research, and experts who appear on news channels and front pages around the world multiple times per year. But even at these impressive levels of media exposure, I am often the bearer of bad news: “Great, you’ve got global reach and you’re famous”, I tell them. “But until you can tell me that someone somewhere actually understood something that you said, and ideally as a result of that that, they benefited in some way, then your impressive reach has very limited significance or meaning”.
This is a real challenge. How do you know that anyone actually was listening, let alone understood what you said in that media interview? If they understood something, then how do you know that it was important to them in some way? For example, do they understand something new, are they now aware of an important issue, has it changed their attitude, or have they done something differently as a result?
Many researchers see it as their responsibility to respond to every media request their press office sends their way. They do the media interviews and give little thought to what happens next for those who listened. Of course, your press offices can give you lots of metrics around circulation numbers, viewer numbers and social media metrics, but this only gives you evidence of reach. If you have no evidence of significant impacts arising from your media engagement, then all your reach has little meaning. You might have reached every person on the planet, but if they didn’t understand a word you said, what was the point?
Plan for your impact and how you will evidence it at the same time
What What you need is some kind of plan to evidence whether your media engagement actually makes a difference or not. Just doing more and more media work, unless you have a plan to evaluate it, may not be a great use of your time. Clearly there are reputational benefits as you get more widely known, and being visible may mean that you get opportunities for impact that you would otherwise not get. Your university will certainly want you to say “yes” to the media opportunities they give you, as it increases their visibility too, and has benefits for them in terms of their reputation, and press offices are driven by quite different metrics from researchers.
The number of people hearing about research from your institution is a key metric for press offices. But in addition to knowing how many people were reached, researchers need to know whether it actually made a difference to those people. To do that, you need to have an impact plan and as part of that you need to think about how you might evaluate the significance of the impacts you get through media (or any other form of) engagement.
Logic models are a fast and powerful way of planning for impact in ways you can evidence. By thinking ahead with a tool like this, it is possible to identify things you would expect to see on the pathway to impact (milestones) and consider indicators of impact or other evidence that might show when expected benefits have been achieved.
Read my guide to impact planning and download the Fast Track Impact planning template here
1. Identify impact goals for specific beneficiaries
The first step in this tool is to identify specific impact goals and the group who will benefit from the achievement of those impacts. An impact goal should always be a benefit. Based on that definition, reaching people, “dissemination” or social media metrics are not impact goals, because there is no way of knowing whether anyone who has been reached in these ways actually benefited as a result. For all you know, they may have been unable to understand what reached them, misunderstood it or been bored or offended by it. You need to engage with the media with a clear impact goal in mind, for example, you may want to:
Raise awareness about an important but little-known issue
Enable people to understand something or change their thinking about
something important
Change opinions and attitudes that are limiting people or causing harm
Mobilise action or change behaviours in individuals or households
Influence decisions by people with power to affect broader positive change
2. Design media activities to reach those goals for those beneficiaries
Now you know what you are trying to achieve, it is possible to harness the media in very different ways that are specifically designed to deliver those outcomes. For example:
You may turn down an opportunity for an interview with CNN International and spend the time writing a press release for Farmers Weekly if you need to raise the awareness of farmers about a threat, technology or action they can take based on your research
You might plan a workshop or open day for farmers and integrate that into your article to advertise the event and get greater engagement with the group
Alternatively, you might link the article to a new online toolkit or other resource that enables wider engagement with the group across the country
Then, you might use the coverage in Farmers Weekly as a route into getting a speaking slot at the Royal Welsh Show
While you’re there, you might arrange to do an interview with the Farmer’s Guardian at the show to get wider coverage across that community
While you’re doing this, you might work with your press office to get the coverage on social media, and target influential accounts who engage with large numbers of farmers and hashtags like #agrichat to get wider readership and funnel social media readers to your workshop/open day, online resource and show event
At each of the events and via your website, you would incentivise people to leave their contact details so you can follow up with them long-term to provide more benefits and ultimately be able to follow up with a short questionnaire to evidence your impact, all GDPR compliant
This highly targeted approach to media engagement stands in contrast to the usual approach which tends to maximise reach without being particularly sensitive to the audience. Raising awareness amongst millions of members of the public about something farmers need to do may have little benefit if few farmers actually hear or understand the message.
Having said that, there is still a role for media opportunities that get you significant coverage, as the more prominent the coverage the more likely you are to include your target groups. You are also like to reach additional groups that you may not previously have been aware were interested in your work. This can be an important way of extending the reach of benefits from your research. However, you have to be ready to walk through the doors that open to you after a major media opportunity, so it is important to make sure that you are easily discoverable via a Google search and have time in your schedule to follow up with the opportunities as they arise.
3. Identify indicators you can use to evidence impact
Evaluation is built into my impact planning template. As you consider what impacts you want to see (point 1 above), and the kind of media engagement you’ll need to get those impacts (point 2 above), you need to consider how you’ll know if the media engagement is working, and whether or not it is generating the benefits you’re aiming for. Press offices are good at collecting data on the success (or otherwise) of your media engagement, collecting all the coverage and telling you the circulation and viewing figures, and social media metrics. Increasingly there are a range of other altmetrics that you can get around your media engagement that will tell you more about the number and types of people who are talking about the work, and where they come from. These kinds of metrics can be a useful starting point for evaluating the significance of your impact. However, in isolation all this data really tells you is your reach.
This is a key weakness of traditional media monitoring, because without evidence of any benefit for the millions of people you reached, it is difficult to argue that there is in fact any impact if you use technical definitions of impact (e.g. from REF) or academic definitions of impact, which all emphasise the significance of benefits as well as their reach. As a result, there is widespread scepticism across the impact community of the value of media engagement as a pathway to impact, and there are increasing reports of academics being told by impact teams to re-prioritise their time away from engaging with their press office, towards more targeted approaches that can generate evidence of significant benefits. This is unfortunate, given the potential scale of benefits that can arise from well-managed media engagement that has been embedded in a well thought out pathway to impact. Based on the points above, you have now created a plan that uses media engagement in a much more targeted way. Now, you just need to make sure you can measure the significance of those benefits in addition to their reach.
The logic model approach makes this easy, because it asks you to identify indicators of success, which do not require any specialist expertise. If your impact goal is to change people’s awareness of the benefits of maggot therapy and shift attitudes to reduce levels of disgust and increase the likelihood that a patient might accept the treatment on the NHS, then you know exactly what you are looking for as an indicator of impact. In this case, you would expect a sample of people exposed to your media coverage to have different levels of awareness and changed attitudes before and after engaging with the coverage, and you would then ultimately expect to see an increase in uptake of the therapy in hospitals. Developing indicators like this does not require any social science expertise – just common sense. Ask yourself what success would look like, and then ask how you might be able to evaluate that.
In some cases there might be things you would be able to quantify, and in other cases you might expect people to say certain things qualitatively if your work had made a difference to them. What methods would you then need to collect that data?
The final section of this guide considers a non-exhaustive range of options for evaluating the significance of impacts arising from media engagement, ranging from accurate but expensive methods to cheaper methods that will take more of your time, but that can still provide robust evidence.
Methods for evidencing the impact of media engagement
Methods for evidencing impact from media engagement range from highly accurate but costly methods to methods you can use for free if you plan ahead and have enough time. All the methods I’m going to suggest have three things in common:
They demonstrate the significance of impact by identifying and describing the benefits that arise from media engagement
They use a sample of the population who engaged with the media. It isn’t possible to survey every person who listened, watched or read about the research, but it is possible to find out if a sample of those people benefited in any way. The more expensive methods work with statistically robust samples, but if you are evaluating impact on a tight budget it is possible to triangulate your findings using other methods, so that you can still say something with confidence with a biased sample
They demonstrate cause and effect between the research and the benefits, so any impacts claimed are clearly attributable to the research. There are two types of causation you might want to try and prove. Necessary causation shows that the research was necessary to generate the impacts that arose via the media. Sufficient causation shows that the research could in theory have generated the impact, and you then need further evidence to build an argument that despite other factors also being plausible (or even probable), the research did indeed play a significant role. A third, weaker form is contributory causation, where the research may have been one of many contributing factors, and it is not possible to disentangle these competing/confounding factors to show that the research played a significant role.
In this guide, I will suggest three ways of evaluating impacts arising from media engagement, though there are many alternatives available.
1. Social media analysis
The first approach is inexpensive and may not take too much time (depending on the sort of analysis you do), but has some important limitations. Social media analysis might sound like a daunting prospect if you have not done it before, but it can be surprisingly accessible:
Quantitative content analysis can be used to count the frequency with which particular words or phrases appear in media (and trends can be tracked over time, looking for peaks that might correspond to public engagement activities). Similarly, this technique can be used to characterise a body of text that is known to relate to the public engagement that is being evaluated (e.g. newspaper cuttings or tweets on an event hashtag), based on the frequency of words within that body of text.
Although more time consuming, qualitative analysis of text that has been aggregated using a hashtag or keyword search can offer more nuanced insights into the nature of debate stimulated by public engagement. Changes in the amount and nature of discourse may be tracked over time.
Evidence of reach may be gathered for particular messages (e.g. number of retweets for particular tweets, and where possible the reach and impressions for that tweet), to evaluate which messages gained most traction.
It may also be possible to study the diversity of people discussing (or liking or retweeting etc.) media stories linked to research, or discussing the research directly, where profile information is available e.g. based on gender and interests. Similar information may be sought from comments under mass media articles, but these are typically less frequent and less likely to be linked to profile information.
Alternatively, more nuanced findings can be gained from a qualitative analysis of social media comments, identifying key themes and using these to build rich descriptions of different responses to the research, illustrated by quotes
Finally, it is possible to reach out directly to social media users who made comments to ask additional questions, collecting further qualitative data on the platform or inviting them to take an online survey or to a telephone interview
It is important to note that social media analysis comes with a number of limitations and ethical challenges. Social media users are unlikely to be representative of the overall audience engaging with the media, for example different platforms have distinct demographic or geographical biases. It is also important to get ethics permission for this sort of evaluation, and different universities often have quite different norms and rules around social media research. While some argue that those posting social media comments understand that the material is available for anyone to view publicly, it is very difficult to prove that you have obtained informed consent from those who posted the comments to analyse them.
2. The funnel approach
The second approach is still inexpensive, but requires a more significant investment of time, whilst still suffering from the same biases as social media analysis. Using this approach, the media opportunity is used to funnel a proportion of those engaging with the media to a website, social media and/or surveys. Using the funnel as an analogy, the media opportunity is the wide top of the funnel where many people engage, and the media opportunity is used to direct those who are interested towards the bottom of the funnel, where they end up in a targeted place where you have the opportunity to engage directly with them.
There are three steps:
1. Identify or create a target location (the bottom of the funnel) to which you will direct those engaging with the media, for example a website, landing page (of a website) or a social media account.
2. Direct those engaging with the media to the target location. This is easier to do in live interviews than it is in pre-recorded interviews that will be edited or newspaper articles, where the journalist may edit out the reference to your funnel. In some cases, it is possible to work with production teams to link to free resources at the end of a programme, or to work with their social media team to get links to your funnel put out alongside messages about a broadcast. In the majority of cases where this is not possible, it may be possible to work with your press office to co-ordinate social media around media activity. With some forward planning, you may be able to line up a few social media influencers with large and relevant followings to amplify these messages at the relevant time, to get further engagement. In this way, it may be possible to capture the interest of a proportion of those engaging with the media, taking them to the funnel. It may be possible to get more engagement if there is a clear benefit of visiting your website, for example exclusive unseen footage, a tooklit or guide, fun quizzes or games, a free e- book or some other benefit you think those engaging with the media would appreciate.
3. Engage with them once they arrive at your website or social media account. The goal of your engagement is: 1) to deliver further benefits, deepening the impact; and 2) to get their permission to follow-up with them in future to find out more about how they have benefited (from the media engagement and/or their further engagement via your own materials). For example, you might include a quiz to test people’s knowledge based on what they learned from the media they engaged with. Although this does not provide rigorous before/after data, you can ask if they were aware before or if they had only become aware after they engaged with your work via the media. You can also double this up as a Twitter poll to get a larger sample. However, to get more rigorous data, you ideally want to persuade a proportion of those visiting your site to give you their email address (in line with GDPR rules), and those visiting your social media account to follow you. One way to do this is to offer them a resource, with the “payment” being that they provide you with their email address in the understanding that you will follow-up to ask them how they are using what they have downloaded. While you may reduce the number of people who will then download your resource, you will find out the identify of those who do download it and you will be able to ask them questions that can help you improve your resource and get evidence of impact (if they have benefited from its use). As long as they remain on your mailing list, you will also get the opportunity to deepen their interest and generate more impact by providing them with additional resources or opportunities, such as attending public lectures or other events linked to the research and their interests.
3. Before/after polling data
The third approach is more rigorous and relatively time-efficient, but it is expensive. You can commission polling companies to do a sample before and after the media featuring your research goes out. Polling data is typically used to assess changing in understanding and attitudes, but could also be used to assess whether viewers for intentions to act or perform specific actions suggested to them in the content they have engaged with.
There are specialist polling companies that can target the viewers, listeners and readers of specific media outlets or programmes in a highly targeted way. For example, Prof Yamni Nigam from Swansea University had her research on maggot therapy featured in four episodes of the TV soap, Casualty, which has over 4 million weekly viewers. She commissioned a specialist TV polling company to find out what proportion of casualty viewers were aware of maggot therapy and its benefits for treating wounds that were resistant to anti-biotics, and the extent to which they viewed maggot therapy as acceptable or disgusting. After the episodes aired, she had evidence of an increase in awareness and understanding of maggot therapy and a reduction in what she called the “yuk factor”. This was important for her, because she had already convinced clinicians to offer maggot therapy on the NHS but uptake by patients was low due to the disgust they felt towards the treatment.
Alternatively, you might commission a generalist polling company to obtain a representative sample of the UK population or a particular target audience (e.g. demographic), to see what proportion engaged with the media you put out. For those who engaged with the media in question, you can ask how they responded to it, and for those who did not, you can get the polling company to provide them with the key messages and then probe for their response. For example, Newcastle University designed a before/after poll targeting a representative sample of the UK and German populations after media coverage of their research on the health benefits of organic milk coincided with a spike in organic milk sales across Europe. The polls were designed to be conducted a week after planned media work around research on the benefits of a range of organic foods. Respondents were asked whether they had bought organic food in the previous week, whether they had seen media reports about the health benefits of organic food, and if so, whether they thought these reports had influenced their purchasing decisions. Those who had not seen the media reports were given the key findings in a short summary based on the University press release, and were asked if they would be more likely to buy organic food on the basis of what they had heard. At the same time, sales of organic food were being tracked to see if there was a sales spike. This time, unlike the spike in organic milk sales, it would be possible to infer media coverage of the research as a major factor contributing to the increase in sales.
While polling data might seem like an expensive approach, it may be worth the investment if there is an important enough impact claim that could be proven with the data. However, in addition to the cash, you need to have the foresight to plan your polls well in advance of the media coverage, and this is not always possible.
Should I prioritise media engagement if I want impact?
Depending on how topical your research is, you may get a lot of media requests. Should you accept all requests, in the hope that the coverage will open doors and generate impact? Your answer to this question will depend on the nature of the impacts you hope to achieve. If you have no specific impact goals and would like to identify new impact opportunities, it may be worth accepting as many requests as you can to get your work more widely known, and see what opportunities arise as a result.
However, if you have specific impact goals (see the section on impact planning above), then you may want to be choosier. For example, if the key benefits you want to achieve are for a specific public or stakeholder group, you may want to decline mainstream media interviews in favour of targeting specialist news media outlets that are more likely to engage your target groups. If there is just one organisation you need to target to achieve change, such as the police force or a company that currently leads the market in the area you are innovating in, then a targeted strategy of face-to-face engagement with key individuals may be a better use of your time.
Consider the role of the media at the start of your research by planning for impact ahead of time, so you can plan ahead with your press office, and design a strategy to evaluate the impact of the media coverage you get. When you plan for impact, you can make what you do with the media really count.
Find out more
Listen to an interview with Yamni Nigam, Professor of Biomedical Sciences, and Clare Lehane, Impact Support Officer, at Swansea University. Yamni describes how she got her research on maggot therapy for wounds featured in four episodes of the popular UK soap Casualty, watched by 4.5 million people every week. They commissioned a polling company to do a before and after evaluation of the impact the episodes have on people’s perceptions of maggot therapy. Yamni’s story shows how curiosity-driven research can lead to impact, and how applying that same curiosity to impact can lead to powerful evidence of benefits to the people she has sought to help.
