Mark Reed
- Mar 9, 2020
- 11 min read

Everything you need to know about the final REF2021 guidance on impact in less than a minute

Updated: Apr 15, 2020

In this guide I will summarise all the key things you need to know about the submission of im

pact case studies in REF2021, and provide a few strategic pointers based on my experience advising teams across the sector who are preparing their case studies. For the full guidance on the submission of impact case studies to REF2021 see pages 68-76 of the Guidance on Submissions and do a keyword search for “impact” to find any specific guidance for your Main Panel or Unit of Assessment in the Panel Criteria and Working Methods. Here, I will focus on what is new in the final guidance.

If you’re short on time, here are the six most important and interesting things that have changed or been confirmed in the new guidance:

Panel A has changed its tune on its preference for new case studies and quantitative evidence, saying they will assess continuation case studies on merit alongside new ones, and accepting qualitative evidence without pre-judgement
A researcher’s outputs and impact can be submitted to different UoAs (but the submitting UoA will have to explain how the guest case study “relates” to their approach to generating impact in the unit)
New definition of a continuation case study has 1) no significant new research underpinning the impacts, AND 2) the impacts and beneficiaries are similar to those in 2014
Confirmed: impacts underpinned by research from different Universities or different units within the same University can submit identical descriptions of impact as long as they explains how the submitting unit “made a distinct and material contribution to the impact”
Case studies based on research by large teams within an institution will have to list all names, roles and dates within the 5 page limit, reducing space for gradable material
There is limited new guidance on public engagement, but it is clear that engagement is a pathway and not an impact, and the case study must still show engagement was "at least in part, based on the submitted unit’s research and drew materially and distinctly upon it”.

Everything else you need to know about impact in the new REF2021 guidance (if you've got more than a minute)

Evidence of impact

Main Panel A has climbed down from its preference for quantitative evidence of impacts. The guidance now clearly states that “the panels anticipate that impact case studies will refer to a wide range of types of evidence, including qualitative, quantitative and tangible or material evidence, as appropriate. Individual case studies may draw on a variety of forms of evidence and indicators. The panels do not wish to pre-judge forms of evidence… and no type of evidence is inherently preferred over another.”
In keeping with REF2014, public engagement case studies “must show that the engagement activity was, at least in part, based on the submitted unit’s research and drew materially and distinctly upon it”. The definition of underpinning research in REF2021 is "a body of work produced over a number of years or may be the output(s) of a particular project. It may be produced by one or more individuals" which must "include references to one or more key research outputs". This means that groups who created a large body of work could describe this wider body in the underpinning research section with examples of key outputs to illustrate in the list of outputs (the guidance says, "each case study must include references to one or more research outputs that best illustrate the research underpinning the impact"). Although the new definition of underpinning research is less tightly linked to outputs (for contrast, in REF2014, it was "research embodied in one or more outputs conducted by one or more individuals, teams or groups, within one or more submitted units), there is no indication that this body of work could include work by researchers beyond the submitting unit, which means that public engagement case studies that draw widely from across their discipline may still struggle to demonstrate that their impacts drew "materially and distinctly" on the submitted unit's research
Advisory roles that generate impact can only be submitted if the role or advice given “was at least in part based on the submitted unit’s research and drew materially and distinctly upon it”
Certain quantitative evidence of impact should be standardised based on guidance published by Research England last year (this is to make data more discoverable, comparable and synthesisable for post-hoc analysis). The guidance is fairly straightforward to implement, either by case study authors or by institutional teams prior to submission. This is a recommendation rather than a stipulation and there is no indication that scores would be compromised for case studies not following this guidance
We have similar options for non-publication or redaction of confidential case studies as REF2014, with the option for the submitting unit or sub-panel chair to identify individual panel members who should (based on national security vetting clearance) or should not (based on confidentiality, sensitivity or conflicts of interest) be given access to case study material. Redacted versions will need to be submitted by January 2021 and requests to restrict access to panel members will need to be made by the end of this year
You can submit corroborating evidence in any language. People who provide testimonials should be referred to by organisation/role (anonymously) with names entered separately in a different part of the submission system
Although testimonials must be statements of fact (evidence-based), it is acknowledge that statements of opinion are valid in some circumstances. We will have to say if the person giving the testimonial was part of the process of impact delivery (and presumably may in some cases have a conflict of interests) or is a “reporter” on the process
You can submit up to 10 testimonials and you can submit contact details for up to 5 of these to be contacted to corroborate evidence in your case study (if you want to include more than 10 testimonials, see my latest REF2021 intelligence

Although eligible, the panel criteria and working methods make it clear that pedagogical impacts based only on the submitting institution are unlikely to score well (paragraph 302, panel criteria and working methods)

Submission process:

A researcher’s outputs and impact can be submitted to different UoAs, either to the same UoA as they submit their outputs or to another UoA as long as the research underpinning the case study fits within the scope of the other UoA. However, the UoA submitting the case study will have to “describe how the selected case studies relate to their approach to achieving impact” in their environment statement
If a submission includes fewer than the required number of case studies, a grade of unclassified will be awarded to each required case study that is not submitted
For impacts underpinned by research from different Universities or different units within the same University, it is possible to submit identical descriptions of impact as long as the underpinning research section explains how the submitting unit “made a distinct and material contribution to the impact”
Names, roles and periods of service for researchers have to be given in the template, reducing the space available for graded material for case studies based on research by large institutional teams. The rest of the meta-data is additional to the five page limit. This includes details of research funding, despite the fact that this is listed under the “references to the research” section in the case study template. To save room for material that is more likely to contribute towards high scores, it would be unwise to submit funding data twice, but bear in mind that funding that only appears in the additional meta-data will not routinely be given to panels, so if funding was from prestigious sources, this would need to be mentioned in the narrative justifying the quality of the underpinning research to ensure panellists are aware of this

Continuation case studies:

A continuation case study is now defined as a case study that has 1) no significant new research underpinning the impacts, and 2) the impacts and beneficiaries are similar to those in 2014

- Crucially, to be considered a continuation case study, it must meet both criteria (no significant new research and similar impacts and beneficiaries)

- This means that case studies based on the same body of research as REF2014 are not considered continuations if they have generated new types of impacts for new beneficiaries. Similarly, case studies that generate similar impacts for similar beneficiaries to those claimed in REF2014 are only considered continuations if they are based on the same original underpinning research

- Those in Panel A who may wish to claim that their case is not a continuation from REF2014 (based on preferences against this expressed by Main Panel A in the consultation) will not simply be able to substitute or add new outputs to the underpinning research section. Instead they will need to argue that these new outputs represent “new research [has] taken place since the previous case study that has made a distinct and material contribution to the impact”, and show that there are significant new impacts and beneficiaries based on this new research in the REF2021 case study

Main Panel A have climbed down from their desire in the consultation to “encourage the submission of new case studies” with continuation case studies to be “considered”. In the final guidance, Main Panel A will “assess each case study on merit and wishes to receive information on how any continued case study relates to that submitted in REF 2014”. This is now consistent with the main guidance across all panels, encouraging “submitting units to submit their strongest case studies irrespective of whether they are new examples or represent continuing impact from those submitted in REF 2014”. However, the bias remains clear and it would be worth considering how case studies in this panel can legitimately avoid the continuation category (see options above)
The other panels do not want to receive information on how continuation case studies relate to those submitted in REF2014. The wording is strong here, saying that if this information is provided then it will not be taken into account. If your case study does not make sense without contextual information that overlaps with your REF2014 case study, it will be important not to explicitly link this to REF2014 and identify it clearly as a pathway rather than claimed impact

Underpinning research:

Underpinning research may be a body of research from an individual or group over many years (since 2000) but they must all be written or co-authored by researchers from the submitting institution. Research by PhD students will have to be co-authored by eligible staff to be included in underpinning research
In keeping with REF2014, underpinning research has to have been conducted at the submitting institution, but there is no longer a stipulation that the work must be published while the staff are at the institution. This means that in theory a member of staff could conduct research that underpins impact at your institution, and as long as you can prove the research was conducted (and presumably completed) at your institution, you should be able to claim the impact, even if they don't publish it till they are at their new institution and don't include your affiliation in the published work
We should “provide evidence of the quality of [underpinning] research”. Examples of such evidence are only provided by Panels C and D, but many will apply to A and B, and include: evidence of rigorous peer-review process for outputs; peer-reviewed funding; reviews of outputs from authoritative sources; prizes or awards made to individual research outputs; evidence that an output is an important reference point for further research beyond the original institution. We are told not to include citation data (where relevant the panels will be given this separately) or journal impact factors for outputs so I would be reluctant to use these indicators here. In some disciplines biases based on publication venue are strong enough that it is still worth prioritising prestigious outputs where there is a choice between two outputs that describe the same finding, but this should not be explicitly drawn attention to. I am collecting additional ideas for indicators in my latest REF intelligence blog if you want more inspiration.
It is not enough to just have one key output at 2* or above, as “a panel will grade a case study as unclassified if it judges that the underpinning research as a whole was not of at least two-star quality”. Although the panel criteria and working methods state clearly that not every output has to meet this threshold, it may be safer to focus on a single (or small number) or safe outputs rather than filling up output slots with research of questionable quality in case this undermines the case that the “research as a whole was of at least 2* quality”. However if only one output is deemed to be of 2* quality, this may be acceptable if it "is a key output underpinning the impact", as "this will normally be sufficient to demonstrate that the underpinning research as a whole meets the quality threshold" according to FAQs released on 29th August 2019
The case study template asks for a description of the research findings (under “underpinning research”) separately from a justification of research quality (under “references to the research”)
In your list of “references to the research”, you should note if the output has also been submitted as a REF output. Outputs without a DOI that are not in the public domain can be used as underpinning research as long as they can be made available to the panel

Impact strategy:

There is little guidance on how to describe impact strategy in the “environment” statement. At the institutional level it may include “integrity, open research, considerations of equality and diversity, and structures to support interdisciplinary research, where applicable in the assessment period and for the next five-year period”. At the UoA level, the guidance is even more vague, suggesting it covers “how the unit has sought to enable and/or facilitate the achievement of impact arising from their research and how they are shaping and adapting their plans to ensure that they continue to support the vitality and sustainability of the unit’s impact in the future”. Later, it suggests we may describe “how the unit recognises and rewards staff for carrying out research and for achieving impact”
Descriptions of impact aren’t limited to case studies. In the environment statement we can describe “wider contributions to the economy and society, including evidence of the wider activities and impact of research carried out in the unit that is not captured in the impact case studies”
Under staffing strategy and staff development in the environment statement, we can describe policies for impact leave as well as traditional sabbaticals (it would be interesting to know how many Universities have such policies)

This blog is my interpretation of the guidance, and I have been known to be wrong, so please contact me if you disagree with anything I've written, or think I've missed something important, and I will keep updating this blog as I learn more from you all.

You may also be interested...

REF2021 related:

Evidencing impact:

Find out how to write a winning impact summary and pathway to impact and explore our best practice library of pathways to impact.

Everything you need to know about the final REF2021 guidance on impact in less than a minute

Everything else you need to know about impact in the new REF2021 guidance (if you've got more than a minute)

Related Posts