REF Tips

Three ways to identify hidden impact case studies for REF2021

By Mark Reed

Many Universities will need to submit

significantly more impact case studies to REF2021 than they had to in 2014 after an announcement in November 2017 confirming that all independent researchers will need to be submitted to REF2021, accompanied by two impact case studies for the first 15 FTEs and then one more case study per additional 15 FTEs

Here are three ways to identify impact case studies that you might otherwise miss:

Identify historic impact case studies from staff that are no longer at your university. To be submitted to REF2021, they will need to be based on underpinning research published since 2000, so you don't have to look back that far. There are two ways you can do this. The easiest way is to look at university records of major grants awarded to your department since 1995, and then screen these for grants led by PIs who are no longer working at the university. Then research each project to see if any impacts have arisen that you might want to write up as an impact case study. The second approach is to ask long-standing members of staff who have been in post since 2000 if they can think of any ex-colleagues who are likely to have generated impacts linked to the research they did while they were at your university. I've done this via an online staff survey, but you might just want to take a few key people for coffee to quiz them.

Identify indirect impacts arising from fundamental non-applied research. It was possible in REF2014 to submit impact case studies based on work done by colleagues in other universities (anywhere in the world) who took your fundamental research and applied it to generate impact, for example, computer scientists who use algorithms developed by pure mathematicians to create software applications that generate revenue for companies. To find out if you have some of these, you'll need to do a citation analysis. This is made easier by citation analysis tools like SciVal. Essentially, you are doing a citation analysis for key non-applied papers and screening for citations in applied disciplines or journals. By scanning these papers, it should become apparent if there is an impact worth investigating further.

Strategically identify future impacts based on a research activity or body of work. HEFCE are consulting on Stern's recommendation to broaden underpinning research to include a body of work. This means that you may be able to identify case studies that are situated across the work of multiple researchers, potentially beyond your group or department. It is unlikely that any individual researcher will volunteer/suggest future impact work based on work across your group. For example, you may have a body of work that has the potential to contribute to post-Brexit policy in your area, and although no-one may be pursuing this yet, you may be able to identify a pathway to impact that could yield results prior to 2020 (given the timescale for Brexit) based on a body of existing work that is now relevant in this policy context. In some cases you may want to suggest additional new research to further strengthen the impact.

These sorts of impacts are not likely to be volunteered by researchers, so those of us in strategic roles need to consider these opportunities, in case we are missing important case studies.

Evidencing international policy impact

By Mark Reed

Linking policy impacts to specific pieces of evidence from research can be a major

problem for researchers working with policymakers. This attribution challenge is amplified significantly if you are working with overseas governments. It is greater still if you think multiple governments around the world may have taken up your work. How do you find out if your work is being used in government policy, let alone evidence it?

The first step for most researchers is an Internet search:

First, narrow your search down to countries you think are more likely to have used or benefited from evidence from your research (for example, based on the prevalence of the issues you address in your work).
If this is a long list of countries, rank according to those you think are likely to be able to benefit most (e.g. prevalence of a disease your work is targeting or share of international Greenhouse Gas emissions from a sector you are targeting).
intervention, or a specific number based on your analysis), use these as search terms.

If a targeted search for evidence of impact in these countries does not work via a search engine, then this does not necessarily mean there was no impact. The work may have informed policy but without citing your work or any identifiable number or phrase that might obviously link the policy to your work. At this point you have two options:

Systematically look through recent legislation published on the government’s website to find policies that address issues linked to your research that were passed into law after your findings were published, and search for any evidence of potential influence. These clues will need to be corroborated through interviews; and...
Conduct telephone interviews with members of the civil service and wider policy community to identify policy impacts. Getting names of relevant people can be tricky given rapid staff turnover in most organisations, and response rates to emails requesting information are often

poor, so the most efficient approach may be to pick up the telephone. Go from person to person till you find the most relevant teams and people. In some countries, it can be easier to identify and speak to others from the wider policy community of people who regularly interact with policymakers, for example charities and think-tanks.

If this still fails to uncover any evidence of international policy impact, then you can avoid wasting your time by using the contacts you make to start promoting your evidence. If you start your investigation with the goal of helping further inform policy, then you may get better responses to your initial enquiries, compared to just asking for evidence that your work has been used. Consider whether there are ways you can add value easily, for example, by turning your work into a policy brief, manual or handbook that can make it easy for members of the policy community to act on your evidence and learn from others who have implemented it already elsewhere. If this is difficult to do directly, then research the broader context in which policies are being developed in the country to identify organisations that may be trying to influence policy in areas relevant to your work, and consider whether you could empower them to achieve impact via your research. These organisations can be powerful allies who can affect change on your behalf, but tread carefully. Some organisations you work with may cherry-pick or distort findings to meet their goals. Depending on the context you are working in, aligning yourself with organisations that are trying to influence government policy can risk your personal safety as well as your reputation.

Finally, even if you have identified impacts already, it is always worth asking whether you can add to these impacts, for example by working with agencies that are charged with enforcing policy to ensure that the policy is implemented effectively on the ground in a manner that is consistent with the evidence you have generated. Your research may have been discussed in parliament, a committee, or have been cited in a policy document, but rather than leaving it there, you may be able to advise the government on how they might be able to actually turn this into legislation that can effect real change (especially if you have already learned how other governments have done this successfully).

Tracking research impact

Tracking research impact with hearts and minds

By Mark Reed

Almost every university in the UK now has a system to track the impact of its research. Some have designed their own systems; others are bolting on “impact modules” to their research management systems. If your university doesn’t have one yet, then it is probably in the pipeline.

However, all these universities share a problem. It doesn’t matter how good their systems are, if researchers are not motivated to enter their impacts.

As a researcher myself, I understand the challenge. I told the Research Councils about my impacts last week via ResearchFish. The week before, NERC contacted me to write an article about my impact for their magazine, Planet Earth. Every couple of months I have an informal call with another of my funders and tell them about the impacts of the project they are funding. Every quarter, I have to write a report on my research impact for the HEFCE programme that funds my Chair position. I’m drafting four impact case studies for the next Research Excellence Framework. On top of this, my university also expects me to log my impacts in its research management system (it is on my to do list).

I suspect that I am a victim of my own success, as few of my colleagues have to report impact as much as I do. However, most researchers I know agree that the administrative burden of reporting impact is rapidly ballooning.

This is a problem for two reasons. First, evidencing impact is important for universities’ reputations and bottom lines. Based on an analysis of REF2014 impact finance I published last month in the Fast Track Impact magazine with Simon Kerridge from the University of Kent, an average 4* impact case study was worth £324,000. Second, few researchers keep records of their impacts as they occur, and so tend to rely on memory when they report impacts (usually in a rush, just before

the deadline). This means reporting is often incomplete and lacking detail, leading to missed opportunities to deepen or properly evidence impact.

For me, this is a problem of hearts and minds. Most researchers’ hearts aren’t in impact reporting. They would prefer to be generating impact rather than entering it into an online system that they’ve forgotten how to access or navigate. Even if your heart is in it (as mine is), the repeated requests to input different impacts to different people in different ways are likely to become increasingly frustrating.

What we need is a culture of tracking impacts as we go, and the only way we can create this culture is by going for both hearts and minds. For me, the easiest way to get to the heart of impact tracking is to link it to your impact goals. I use my Fast Track Impact Tracking Template (available free here) to get researchers to visualise their impact goals (working back if necessary from the people who are interested in their work and why they are likely to be interested). It can be motivational to visualise the impacts your research might have, imagining yourself years from now, looking at what has changed because of your work.

However, I tell my colleagues that there is no point in visualising their future impact if they have no way of telling whether or not they are moving towards or away from it. By looking around themselves, in their mind’s eye, they can start identifying the specific things that have changed, that tell them they have reached the impact they set out to achieve. Now, rather than just waiting for evidence to appear, they are looking for specific things, and measuring them on a regular basis to check if their impacts are on track. If they are off track, they are empowered to change their pathway to impact in ways that are likely to get them back on track. Impact tracking now has a purpose. It isn’t just filling in forms to keep some nameless bean-counter happy. It is actually increasing the likelihood that they achieve the impacts I want to see. From this place of inspiration and empowerment, I’m motivated to track my impacts. We’ve got to the heart of the matter.

The mind is more of a challenge. What we need is a way of tracking impacts that is as effortless and painless as possible. Different people will look for different things. For me, I want to be able to track my impacts on the go, inputting things to my smartphone, whether or not I’ve got an internet connection, without having to learn a new web interface or having to remember a new username and password. Ideally, I want to be able to record things as I stumble across them, online or in my inbox, without leaving my internet browser or email programme. Each researcher needs to create their own system that will meet their needs, so they can effortlessly keep track of impacts on a day-to-day basis. Not everything they record will necessarily be worth reporting, but when asked to report impacts, they will have a wealth of material to sift through, and be much more likely to provide detailed and comprehensive information. This is not about replacing institutional repositories. It is about finding ways to collate material easily as you go, so it is easier to input to your institutional repository (or whatever form you are asked to report impact) when that time comes round.

So what are the options? A lot of my colleagues just use their email, putting things related to impact in a folder, filing leads, and emailing notes to themselves to store for safe-keeping. One person I met prints out anything related to his impact and puts it in a ring binder. Others I know are exploring OneDrive. There is no single right answer. We encourage researchers to come up with their own answer so that impact reporting becomes quicker and it is easier to provide high-quality information.

I have developed a system for researchers to keep track of their impacts on a day-to-day basis, prior to submitting them to funders and institutional repositories, using the productivity app, Evernote. Using this system, I collect evidence on the go in three simple steps:

Sign up for a paid Evernote account (£30 a year)—only one member needs to do this, the rest of your team can use the free version of the app or website, or just email impacts into your Evernote account
Start a new notebook, share the notebook with your team if they want to record impacts directly into the notebook in their own Evernote app (and anyone else who would like to have access to your impacts e.g. an administrator who is helping you input evidence to an institutional repository)
Give your team your unique Evernote email address to send in notes, photos, recordings, documents, clipped web pages and other evidence of impact to be collated in your shared notebook

Now I have relevant material quickly to hand when I need to report it to funders or my university. My team members don’t need to remember a new log-in or learn any new skills; if they can send an email, they can keep track of their impacts (no excuses!). They don’t have to download the app, visit a website or even be online unless they want to. For my most recent project, I’ve made a link on the project website that brings up an email addressed to my Evernote address automatically with a subject line that will deliver their email directly to the notebook for that project (I’m using Evernote to track impact for multiple projects). As Evernote is a

US-based cloud computing service, I make sure I don’t store confidential data about research participants or anything I have contractually agreed to keep confidential.

If you want researchers to enter high-quality information about their impacts into your shiny new institutional repository, consider first how you can create a culture of impact tracking. Simply having the repository is not the answer. We need to show people simple, quick and easy ways of tracking their impacts on a day-to-day basis that fit with their busy, mobile lives (the “mind” part). Beyond that, we need to show how evidencing impact on a regular basis can actually lead to bigger and better impacts. If we understand why researchers want to achieve impact (for many, this is more about instrumental motives or curiosity than it is about any intrinsic motivation to achieve impact), then this becomes an appeal to the “heart”. Impact tracking needs to come from the heart and the mind if it is to happen regularly and effectively.

If you want to find out more about my impact tracking system for Evernote, visit www.fasttrackimpact.com/evernote

Help!

Help! My impact case study is stuck at my previous institution

In keeping with REF2014, it is expected that

impact case studies will not be portable in REF2021. This means that when you move institution, you will not be able to claim any credit for your impact with your new employer—that privilege will remain with your previous employer. If that’s a problem for you, then there are some things you can do:

You can try and change affiliations on papers under review, but panels are likely to see through this. Instead, you need to start new research you can write up at your new institution that can become underpinning research for new impacts arising from your case study

Gold-plate it by getting some new funding you can cite (even if modest) alongside the papers to show that this really is new research funded, conducted and written up at your new institution
Make sure there are new impacts arising from your work since you moved
Ask people providing you with testimonials to mention timings and if you have made a difference since you moved, get them to say so explicitly
Look for any other evidence that is clearly dated after you moved institution
Talking about the previous work as a pathway to these impacts will be fine, and your previous institution can also benefit from the new impacts you’ve garnered. Collaborate with your previous employer to increase your access to resources, advice and additional opinions on your case study, especially if you are able to submit to different UOAs to avoid direct comparison.

Writing papers to corroborate your impact

Writing papers to corroborate your impact for REF2021

Finding evidence for some impacts is more challenging than for others, and may

require a substantial research effort in its own right. Designing a robust evaluation is a fairly technical task, and you will probably want to draw on social science expertise to design an evaluation that unambiguously demonstrates cause and effect and attributes impacts to your research.

To make this happen, you have a number of options open to you. You could commission an independent consultant to design and carry out the evaluation for you (we work with Laura Meagher—find out more at www.fasttrackimpact.com/impact-advice). However, despite the fact that the evaluation is independent, it is unlikely to look independent on a list of “sources to corroborate the impact” in an impact case study if it is published on your university website. The alternative, therefore, is to work with a stakeholder or public representative organisation linked to your impact case study to co-design, carry out and publish the evaluation (potentially with assistance from social scientists in your institution). If they do not have the resources or expertise to carry out the evaluation themselves, then you might want to pay for the independent consultant to do the work for them. They will have to acknowledge your funding, but the evaluation is independent and it is published on their website, so it looks independent as well.

Not everyone has access to the funding to make this possible, however. A further option therefore, which requires time and expertise instead of funding, is to conduct the evaluation yourself with social scientists in your institution and write it up as a peer-reviewed paper. In some disciplines, it may be possible to build an impact evaluation into the next paper you write which builds on your underpinning research. This may even add value to the paper, giving it greater academic impact. For some, it may be possible to write the evaluation as a stand-alone paper for a journal in their discipline. For other disciplines, this is not an option, and they are reliant on working with social scientists to co-design, carry out and publish the evaluation in journals from their own field. The challenge here is to pitch this as an opportunity, as there are few social scientists who are likely to want to spend significant amounts of time providing evaluation services to academics from other disciplines. However, with some exploration and discussion, it may be possible to get a social scientist excited about the research opportunities presented by an evaluation.

Although not independent, if you can design your evaluation robustly and get it published in a reputable peer-reviewed journal, it is unlikely that your evidence will be dismissed. In theory you might have “cooked” the results to make yourself

look good, but a strong research design in a good journal is likely to re-assure most panel members that you’re not trying to pull the wool over their eyes. Here are three types of impact evaluations leading to papers that I’ve helped design.

Design from scratch

I am helping one of my colleagues design an evaluation study that will provide a source to corroborate his impact. He approached me with evidence that a paper he had written had led to a massive spike in the consumption of a product across Europe. Sadly, the evidence was only in the form of a correlation, and did not prove cause and effect. The spike in consumption was significant and sustained, and coincided perfectly with the publication of his findings, but how could we demonstrate without doubt that his paper was the cause of this spike rather than a coincidence?

He explained to me that he was planning to publish another paper with even more striking and far-reaching findings early next year, and we identified an opportunity to do things differently. This time he would go back to his funders to commission a large-scale survey of consumers in the UK and Germany immediately before and after the publication of his work. We would ask them about their consumption of the products he was studying, and whether any changes we detected after publication were in fact as a result of reading about his work. I pitched the opportunity to a social science colleague of mine, Lynn Frewer, who jumped at the opportunity to work with us to test some hypotheses she was developing about consumer behaviour. Together we designed a methodology that should be sufficiently robust and novel to get our findings published in a good journal.

In a second example, I evaluated the impact of my own research with interdisciplinarian Ros Bryce, funded through an EU project concerned with science-policy dialogue. We conducted interviews with members of the policy community, asking them to identify whether they had come into contact with any of over 70 research findings, one of which was based on the work of my colleagues and myself. We then used qualitative interviews and Social Network Analysis to trace those findings as they were communicated from person to person until they either reached policy or not. The social scientist I was working with got a new job and we didn’t finish writing the paper, but in the intervening period, I met a PhD student, Ruth Machen, who was studying similar questions with the same group of policymakers. The resulting paper combines insights from the qualitative and quantitative analysis that Ros and I conducted, with Ruth’s in-depth work (she is now a Research Associate in Newcastle). It makes a significant methodological contribution to the new empirical understanding of the specific science-policy community we studied.

See: Reed MS, Bryce R, Machen R (2018) Pathways to policy impact: a new approach for planning and

evidencing research impact. Evidence & Policy

Policy report to research article

The second type of impact evaluation I’ve written up as a research article started life as a policy report. This example is more of an attempt to evidence rather than evaluate policy impact. I was faced with a problem where I had invested significant research effort to support a policy document (the United Nations Convention to Combat Desertification’s Global Land Outlook), but I did not have any control over the way our research would be cited in their report. I needed to create a publicly available evidence trail that would prove that a significant part of the report arose from my research.

To do this, we wrote the first draft of our contribution to the report as a research paper, and then converted this into simpler language for inclusion in the report with a request that the report cited the research paper which we then submitted to the journal Land Degradation and Development. The journal article includes a footnote on the front page explaining, “a condensed version of this paper is part of the UNCCD’s Global Land Outlook, published 17 June 2018”. Now, whether the report cites our work or not, we have a published link between the two, with clear overlaps in the text and ideas contained within the two documents.

See: Thomas RJ, Reed MS et al. (in press) Modalities for Scaling up Sustainable Land Management and Restoration of Degraded Land. Land Degradation & Development.

Bolt on

Finally, I published a paper earlier this year in Global Environmental Change which contained a substantive section detailing impact evaluation work I conducted with PhD student Kathleen Allen. We did the work in collaboration with the IUCN UK Peatland Programme, which funded her travel expenses to accompany me to run focus groups across the UK to evaluate the impact of research I had submitted to REF2014. The impacts were mixed, but made for interesting reading, enriching a paper that would otherwise have been more descriptive in nature. This enabled us to publish the work in a higher impact journal that might otherwise have been possible. I have not yet decided if I will use this paper as one of the “references to the research” or a “source to corroborate the impact”. It could be used for either purpose.

See: Reed MS et al. (2017) A Place-Based Approach to Payments for Ecosystem Services. Global Environmental Change 43: 92-106.

How to grade impact case studies

How to grade impact case studies for the REF

Giving feedback to colleagues about their

work can be a delicate affair at the best of times, but giving feedback about impact has additional challenges. First, there is typically a small pool of staff (usually in professional services) with the expertise necessary to accurately grade and provide feedback on impact case studies. Most universities are trying to grade impact case studies internally to understand their current REF position and inform their priorities for the next 2 years’ work. However, most universities have more impact case studies than they have skilled staff to evaluate, and the costs of employing consultants to grade such a volume of work is prohibitive.

For this reason, many universities are now considering how they might be able to train academics to provide preliminary grades for potential case studies. Very few attempt to predict REF grades at this stage in the cycle. The most common classification system grades case studies on the basis of their maturity (mature or immature) or uses a traffic light system (green for submitting, orange for case studies that need more work and red for those that have been deselected).

Grading case studies to the REF criteria

Although trying to grade case studies against the REF criteria is not necessary at this stage, paying greater attention to the grade definitions and levels may provide more constructive feedback to case study authors and increase the likelihood of good grades. Of particular relevance are two questions:

What potential does this case study have to evidence 3* or 4* significance and reach in this Unit of Assessment, assuming everything works?
How likely is it that (everything will work and) this case study will reach its potential and that this will be clearly evidenced and attributable to the research? How significant are the risks and uncertainties, and are there credible plans to overcome these?

The first question requires an understanding of the context in which the case study will be evaluated. There is evidence that significance and reach were evaluated very differently in different Units of Assessment (UoA) in REF2014.

For example, there were a number of examples of 4* reach in Panel D at the scale of an individual town or city, compared to the predominance of global reach in Panel A. In Panel B, an economic impact of over £1M may demonstrate 4* significance in UoA7 (earth systems and environmental science) but may be viewed less favourably in UOAs 11–15 where 3 and 4* economic impacts were more likely to be in the tens or hundreds of millions.

Therefore, the first task of anyone wishing to evaluate impact case studies objectively in relation to the REF criteria is to analyse REF2014 scores to identify high- versus low-scoring case studies for their UoA. Where there are enough case studies at identifiable scores, it can be useful to divide these into different types of case study, for example by subject area for diverse UoAs (such as materials versus electronic engineering case studies for UoA13), public engagement based cases versus case studies that engaged with stakeholders, or cases that focus on economic versus policy or societal benefits. Use this analysis to establish qualitative benchmarks for what you will consider to be 3* or 4* versus 1* or 2*.

Next, using these benchmarks, estimate whether or not the case study has the potential to reach 3* or 4* in its current form, and if not, whether or not you think it might in theory be able to reach this level if changes were made. Finally, identify in concrete terms the sources of risk and/or uncertainty that may compromise the ability of the case study to reach 3* or 4*. Based on the limitations, barriers, uncertainties and risks you have identified, try and suggest ways of overcoming these issues and identify questions that you can discuss with the case study author that will enable them to think of ways around these issues.

Together with the case study author, you will then be able to identify a series of concrete actions they can take to enhance the significance and reach of their impact. Alternatively, your analysis and discussion with the case study author may have identified that the case study has limited potential for ever reaching 3* or 4*, or that the risks/uncertainties are insurmountable and therefore likely to limit the grade of the case study.

With limited time and resources, which impact case studies do I prioritise?

The UK has seen a proliferation of positions supporting impact, ranging from professional services staff taking roles as impact officers to academic staff taking roles as directors of impact and impact champions. Funding pots dedicated to supporting impact are popping up in many universities. However, even in the best-resourced universities, there is significantly more demand for support than this professional services staff can supply. That means difficult decisions will have to be made about which case study authors should receive the most support.

What to do with lower-ranked case studies?

For high- and mid-ranked case studies at this point in the REF cycle, you will probably want to further strengthen and evidence impacts in collaboration with case study authors. But what about lower-graded case studies that are unlikely to score highly for the next REF? Do we give negative feedback and encourage colleagues to stop working on these impacts? The consequences of this course of action are likely to be negative for both the researcher, who may feel devalued and demotivated, and the publics and stakeholders they were working with, who may feel betrayed by the researcher and their university.

The easy option is to avoid giving negative feedback and encourage continued work at a lower level of priority, in the hope that sufficient progress is made in time for the next REF cycle. However, there are two other options:

First, you may look at the number of high-potential case studies that are highly likely to score 3* or 4* and decide that this case study is not likely to be needed, and encourage the case study author to invest their time and energy into the production of 3* and 4* research outputs.
Second, you may want to celebrate or reward impacts that are unlikely to score well in REF via other means, for example featuring the work on the university website, running a press release, entering it to an internal competition, or providing workload allocation.

Prioritising case studies for funding and other support

Universities across the UK are pouring an increasing amount of staff time and financial resources into supporting impact case studies for REF, but these resources are finite, so how do you prioritise your efforts? One approach is to place case studies in a matrix based on their potential to reach high scores, and the likelihood that this potential is realised.

The matrices on top below show how you can move case studies up your “long-list” by reducing uncertainty around their potential and likelihood (e.g. by interviewing case study authors and the publics and stakeholders they are working with), and then identifying actions that can increase the potential for impact (e.g. identifying new application contexts where there is greater potential for impact) and likelihood of impact (e.g. identifying and overcoming barriers that might compromise or lesson impacts), to move case studies onto the short-list.

With limited time and resources..

The matrix below shows where you might focus support, based on this analysis, first prioritising support for case study authors on your short-list to ensure they reach their full potential and are able to fully evidence their impacts. At this stage in the REF cycle, you probably want to prioritise efforts towards uncertain case studies so you have a better idea of their relative potential and likelihood of reaching that potential. Then, based on what you find out, you are identifying actions that could increase the potential for these case studies to reach your short-list. For some there will be limited actions you can suggest, and they may end up moving to a future REF cycle or being taken out of the REF system altogether to be celebrated in other ways (see previous section). As a result, these case study authors are only offered reactive support when they request it. In contrast, proactive support is targeted towards all case studies that are able to continue moving towards your short-list. Schedule regular meetings and milestones, and offer funding and other support to aid these case study authors on their pathway to more significant and far-reaching impacts.

Most funding for impact is competitive, creating significant paperwork for those administering the schemes for fairly limited amounts of money. Consider whether you might funnel funding through impact officers and others who are able to identify strategic priorities for funding based on the matrix approach below. Instead of funding the “usual suspects” who have the confidence to apply for competitive funds, you now have the opportunity to strategically approach those case study authors you know have opportunities to take specific actions that would improve their impact.

Stay in touch

Follow Fast Track Impact on Twitter

Keep an eye on our Facebook page:

www.facebook.com/

fasttrackimpact

Contact pa@fasttrackimpact.com