REF Tips
3 reasons why you should care about Research Excellence Framework (REF) if you're not working in the UK
This section of the magazine is dedicated to the UK's Research Excellence Framework (REF), which is how the UK government assesses the quality and impact of research done by Higher Education Institutions and distributes “quality rated” (QR) funding to those institutions. For UK researchers under institutions, REF has significant funding and reputational consequences. But why should non-UK researchers be interested in it? Here are three reasons why you should care about REF if you're not working in the UK:
-
REF has operationalised a widely accepted and measurable definition of impact based on its significance and reach. This provides a really useful, structured way of thinking about the impact of your research: first, ask yourself how you can make your impact more significant and meaningful for the people you want to benefit from your work; and then ask yourself how you can expand, replicate and scale up your impact
-
As a result, UK researchers are now leading the world in terms of the impact of their work. You can argue about whether they're leading in reality or just in terms of their ability to articulate their impact. The point however, is that the impact of UK research is now more visible than ever before. If you want the impact of your research to be more visible, then there are many lessons you can learn from REF about how to evidence and right up your impacts in convincing ways.
-
Research funders around the world are considering how to measure and communicate the impact of the research they fund. Get ahead of the game by learning from the successes on mistakes being made in the UK
​
Impact is made up of two things: significance and reach. Put simply, if my research saves someone’s life, I’ve made a significant impact; if my research saves millions of people’s lives, the impact is no more significant, but now it has reach as well. Although this is a fairly simplistic example, it illustrates the point well. If you’re not able to demonstrate something significant from your work, you might as well not even start thinking about REF. Tweaking something minor isn’t going to set people’s imaginations on fire, even if you do it on a global scale. If you start by trying to do something significant however, you can always consider how to achieve reach if it works on a small scale. One of the benefits of focusing on generating significant impacts to start with, is that impact becomes less intimidating. I don’t have to change the world any more; I can just change my home town or community.
Perhaps surprisingly (given the REF rules that stated only actual not potential impacts would count), there are a number of examples of top 4* scoring impact case studies in REF2014 that were restricted to a single town or city (e.g. Mapping Mediaeval Chester or the DECIPHer-ASSIST programme to reduce the number of young people smoking in Cardiff), organization (e.g. Jobcentre Plus or Hampton Court Palace) or part of the UK (e.g. the use of flags in Northern Ireland or the creation of new language law in Wales). What the majority of these cases have in common though, is some argument for transferability beyond the case study context, for example to other areas experiencing conflicts similar to Northern Ireland or to towns like Chester elsewhere in the UK.
​
Based on these case studies, it is possible to argue for three ways to enhance the reach of a case study:
-
Theoretical reach: Once you have established a significant impact with limited reach, consider the extent to which it could be transferred or extended to similar contexts elsewhere, estimating the potential benefits based on the assumption that benefits would be similar to the original context. The reach in this case is theoretical, and the strength of the theory will depend on how robust the assumptions are. If the bar is going to be raised in REF2021, it may not be a wise strategy to focus on this, based on the small number of successful examples of theoretical reach we saw in REF2014, and the large number of unsuccessful attempts to do this (not to mention the fact that the rules said all impacts had to be in the past).
-
Piloted reach: It is more rigorous and persuasive to actually test some of the assumptions of theoretical reach, to see if significant impacts can indeed be achieved in comparable contexts elsewhere. This will typically take the form of some type of pilot where work has been done in similar cities, organisations or communities elsewhere, and it has been possible to document significant impacts in these contexts. Although the impact has not reached national or international reach yet, it is much more plausible to argue that this is possible on the basis of pilot data. In some cases it may be possible to take this a step further if it has been possible to embed new practices or ideas in the work of an organization that has national or international reach, and is prepared to state its goal of achieving reach based on the research in credible terms.
-
Actual reach: finally, there are a few short-cuts to achieving actual reach faster than you might expect. The first short-cut is to find an organization that is seeking similar impacts, who you can help with your research and other capabilities, who can in turn work on your behalf to promote impacts based on your work e.g. from the business, Third Sector or policy world. Alternatively, find other groups similar to those you have worked with initially, targeting progressively larger groups e.g. from working with a small minority group that is only prevalent in your local area to working with a range of larger and more widespread minority groups. Work with your initial groups to identify larger groups via their networks and get their help to pitch the benefits in ways that are likely to appeal to those new groups.
How to enhance the reach of your impact
​
The Research Excellence Framework in 2014 laid out clear criteria for what makes a strong non-academic impact in terms of significance and reach. What was less clear was the extent to which these criteria would be interpreted differently in different subject areas. In particular, interpretations of reach differed significantly across subject areas, partly as a result of the epistemologies underpinning the disciplines involves, and partly as a result of relative scoring between case studies that all faced similar constraints or advantages, with all panels wanting to showcase a smaller number of the best impacts in their areas by giving them the highest scores. For example, it was perfectly acceptable in anthropology for a case study to demonstrate both significance and reach in a single settlement in a single country, whereas in medicine you might as well not bother submitting a case study if there wasn’t some sort of international dimension.
For this reason, it is essential that teams of researchers have a good idea of what is likely to be considered a strong impact in their subject area. The best way of doing this is by comparing and contrasting high and low-scoring case studies from REF2014. Although HEFCE tried to avoid making scores public, it is possible to identify 120 case studies that were given top 4* scores across 19 Units of Assessment (based on institutions that scored 4* for all their case studies in a given Unit of Assessment). All of these case studies are available via the Fast Track Impact website at: www.fasttrackimpact/resources. The problem is that it is not possible to identify 4* case studies for many Units of Assessment. The same applies for the lower scoring case studies: we know some case studies that scored 1* but these are not available via any published list online, and for many Units of Assessment it is not possible to identify case studies that we know definitely scored 1*.
-
Identify top scoring institutions for your Unit of Assessment (UOA): download the REF2014 results, filter for your UOA (columns E or F), then filter so it only shows you the impacts (column J), and then filter for 4* (column L), showing only the institutions from your UOA that had the highest percentage of 4* impacts. Now for those institutions, look across the table (columns L-P) to find institutions that only had impacts scored at 3* or 4*
-
Download a selection of impact case studies from the top scoring institutions via http://impact.ref.ac.uk. You may want to select impacts randomly, or you may want to go through more selectively, identifying impacts that are closer to the areas your group specialize in
-
Repeat for low scoring institutions so you can compare and contrast high and low scoring impacts
-
Discuss examples: print copies of the high and low scoring impact case studies, labeled clearly, and in your next UOA meeting, let everyone choose a high and a low-scoring example. Give them 10-15 minutes to quickly read a case study each (focusing on summary and details of the impacts so you’re not there all day) and then ask the group (or small groups if there are many of you) to discuss the key factors that they think distinguish between high and low scoring impacts. Get your group(s) to distill the key principles that they think are most useful and disseminate these more widely, so that anyone who wasn’t present can benefit.
How to design a workshop to discuss what makes a top-scoring REF impact case study in your subject area
In contrast, a 4* research output was typically worth between £5,000-25,000 (see below for our workings). In general case studies are worth far more than research outputs for all but the smallest submissions. Generally speaking, a case study was worth around 5 outputs at higher FTEs, with more variation at lower FTEs.
How much was an impact case study worth in REF2014
​
By Prof Mark Reed and Dr Simon Kerridge
Â
When the UK Government integrated impact into the Research Excellence Framework in 2014, it signaled how seriously it took the societal and economic impact of the research it funds.
Â
Higher Education Institutes received on average £308,000 (£44,000 per year between 2015/16-2021/22) for the most significant and far-reaching impacts. Given that many of the people who were responsible for leading these impact case studies earn salaries less than this, for most people, that’s taking things pretty seriously.
Â
We looked at Units of Assessment in REF2014 where a University had all its case studies graded as either 3* or 4* and found that:
Â
A 4* impact case study was worth £44,048 on average (range: £12,971-70,946) in 2016/17 (Table 1)
A 3* impact case study was worth £11,813 on average (range: £3,415-29,186) in 2016/17 (Table 2)
Â
The formula for calculating annual recurring payments for each of these case studies between now and the next REF may vary, but we can expect similar levels of funding per case study per year between now and 2021.
Â
It is of course possible also possible to calculate the value of 3* and 4* case studies from any submission where QR funding is allocated (using the QR sterling value, the number of case studies and the quality profile). Using the method described below, we can determine the worth of a 4* case study for any given submission – giving us the full range for English HEIs of £6,005-£90,490. This is per year for 7 years until the REF2021-informed funding kicks in. Note that these values include, where applicable, the London weighting (although the highest is actually from outside London). You can see this spread in the scatter graph below (showing funded English UOAs with <100 FTE staff):​​
Â
Table 1: Quality Rated (QR) funding allocated by HEFCE in 2016/17 per 4* impact case study, based on the case studies from Units of Assessment where 100% of the impact sub profile was graded at 4*
Table 2: Quality Rated (QR) funding allocated by HEFCE in 2016/17 per 4* impact case study, based on the case studies from Units of Assessment where 100% of the impact sub profile was graded at 4*
What does this all mean for UK researchers? Whatever our motives for generating impact from research, our employers are partly motivated by the financial rewards now linked to impact, and indeed the associated league table positions based on “impact excellence”. The extent to which this translates in any meaningful way into incentives for researchers depends on the way each institution chooses to use that funding. Most universities top-slice their QR funding before it gets to faculties or schools; whether this then reaches or benefits the researchers responsible for generating the impacts is another matter. In some cases, decisions about spending this money are being taken centrally without any input from faculties or schools, let alone the researchers involved in generating the impacts. In Kent, the policy is to allocate the QR funding to the schools
that “earned” it, based on their staff FTE submitted to the various Units of Assessment (UOAs). However there is a “central charge” levied on school allocations in order to determine their budget, so a top slice - but based on activity rather than allocation.
Some of us are pleased that at last, impact is being valued highly enough to be rewarded in this way. However there are also concerns about the power of these financial incentives to create game-playing tactics that will bring the academy into disrepute. We share both these feelings. Arguably, it is only because of the financial and reputational rewards associated with the REF that impact is now so widely (although far from universally) integrated into workload models and promotion criteria across the sector. These incentives are clearly motivating many researchers to engage with impact who had never fully considered the effect of their research before. However, it is these very incentives that are leading some researchers to chase impact for purely career-based motives, which has the potential to result in negative unintended consequences. As the rewards become greater, we must become ever more vigilant to these behaviors, and do all we can to build research cultures that value impact intrinsically, whether or not the benefits can be submitted to REF or are likely to score highly.
Mark Reed holds the HEFCE N8 Chair of Socio-Technical Research at Newcastle University. He is a recognized international expert in impact research with >100 publications that have been cited over 8000 times. He has won awards for the impact of his work as a research Professor and Research Manager for an international charity.
Simon Kerridge is Director of Research Services at the University of Kent, vice chair of CASRAI and immediate past chair of ARMA. He was a panel secretary for the criteria setting phase of the REF and led the Kent REF2014 submission. He was also part of The Metric Tide team that advocated against metrics for impact assessment.
3 things HEFCE's consultation has told us about the role of impact in REF2021
​
The role and shape of impact in the next REF got a lot clearer with the publication of HEFCE’s consultation on the second Research Excellence Framework in December 2016.
There were no big surprises, with the focus on how not whether to implement Stern’s recent recommendations. However there are a few significant points we can glean from the consultation:
-
Institution-level case studies could play a major role in the next REF, accounting for 10-20% or up to 25% of impact scores in two different proposals being consulted upon. However, this proposal has the potential to achieve the opposite of Stern’s intention to better capture interdisciplinary and collaborative impacts if it is perceived as a “showcase panel” to which institutions only submit their most iconic case studies
-
Larger units may only be allowed to submit one case study for every 12-20 staff they submit. Less research intensive Universities (that were more selective in the staff they submitted to REF2014) could have to find twice the number of case studies they needed in 2014 if they want to make a submission in 2021. For example, a unit with 80 academic staff that only submitted their 10 best researchers could have done so with two impact case studies in REF2014 but may need to find four case studies to be able to make a submission to REF2021. This may incentivize the submission of low grade and in some cases “unclassifiable” case studies that are not based on credible research in order to enable submissions to be made. Small units may only have to submit one case study, revealing their scores
-
Limits may be placed on the proportion of case studies that can be resubmitted from REF2014, and only cases where additional impacts have occurred may be eligible. The news that case studies from REF2014 will be particularly welcome for those whose case studies received low grades because they were still in progress. A significant proportion of case studies fall into this category. One significant group is policy impacts that had often passed into law but without evidence of the law yet being implemented or achieving results on the ground.
We'd love to hear from you if you have any ideas about issues you'd like us to cover got there or if you have questions you would like to have answered by Prof Reed in the next issue.
In particular, if you have an amazing productivity tip you'd like to share with other researchers, get in touch and we will try and feature as many of your ideas as possible.
​
Contact: pa@fasttrackimpact.com
Find out more about what we have learned from the HEFCE consultation about the role of impact in REF2021, including how institutional case studies may be backfiring on Stern on the Fast Track Impact Blog