My take on what the initial decisions on REF mean for impact

Summary: these are mainly tweaks that address issues for which most institutions already

found work-arounds. You can read the full announcement here.

There are 4 main proposed changes to impact:

• A reduction in the minimum number of case studies needed for a submission;

• The reintroduction of an impact narrative at the unit level;

• Removal of the 2* threshold for underpinning research; and

• New criteria around the rigour of engagement.

I'll take each in turn:

1. The reduction in the number of case studies needed to make a submission may encourage new groups to submit to REF, but in my experience this wasn’t a major barrier to entry as long as people didn’t mind that at least one of their cases might get a low or unclassifiable score. Nevertheless, this will open up the possibility of submitting to REF to a number of new groups who didn't have enough impact case studies to make a submission before. Modelling by Simon Kerridge has suggested that UoAs with 20-50 FTE are likely to have to submit a similar number of case studies to REF2021, but those with less than 20 FTE will have to submit significantly less case studies, and those with more than 50 FTE will also be able to submit proportionally less (with significant economies of scale for the largest units).

2. The re-introduction of an accompanying impact narrative may capture a broader range of engagement and impacts unlikely to score highly as case studies (and not captured in KEF for English and other participating institutions), which are nevertheless of value. Although this was done to an extent in REF2021 environment statements, this will undoubtably provide more granular information about impact activity across the sector. This could incentivise a broader range of impact activities, including public engagement work which was deprioritised in many applied disciplines that had easier routes to high scoring case studies. This narrative will also capture information about engagement (more on that in point 4 below). The narrative will be integrated as part of the new disciplinary statements required for REF2029.

3. The removal of the 2* threshold is useful, and may broaden the range of case studies that can be submitted, especially from institutions and disciplines who employ academics from professional practice who often have significant impacts based on limited research. It should be easier now for these academics to write up the work underpinning their case studies in a way that is sufficiently robust to demonstrate a contribution to knowledge without having to demonstrate the academic significance and originality of that contribution. However, guidance is needed on what constitutes "research" and it is expected that there will be some guidance on the rigour of research, to ensure impacts are based on high-integrity work. Given that this change in guidance was motivated in part by the inappropriate use of indicators (e.g. journal impact factors) of research quality, it seems logical that if we have to demonstrate the rigour of the underpinning research this would be in narrative form.

4. A new criterion is to be added on the rigour of engagement. After discussing this with the REF2029 team, they explained that this was proposed in response to consultation feedback from the NCCPE and others that the ethics of impact need to be scrutinised more carefully, in particular whether or not best practice has been followed on engagement. This is a big issue, and guidance on the ethics of impact is much needed. However, it will be interesting to see how panels operationalise this, given that not all impacts require engagement. For example there are many technology and commercialisation pathways to impact, and impacts that arise directly from research (e.g. policies based directly on recommendations from peer-reviewed sources) or indirectly via a ripple effect from other impacts (e.g. global impacts from governments who replicate a Scottish Government policy which was based on UK research, where the UK researchers had no contact with any of the overseas governments). The REF2029 team have explained that they would not expect submissions to be penalised if the impacts they are achieving require little or no engagement, where this can be explained and justified. They also explained that although the wording of the announcement focussed on engagement, the intention is to consider the ethics of impact generation more broadly, for example how negative unintended consequences or grimpact are identified and managed. My hope is that consideration of the ethics of impact is broadened beyond further, for example evidence in impact narratives that impact is integrated into research ethics procedures and EDI training, or considering the introduction of positionality statements in impact case studies. Engagement will be assessed in the impact narrative in a questionnaire format, which will ask questions about the rigour with which engagement was conducted. It is expected that this will focus at the level of the UoA rather than requiring evidence of the quality of engagement underpinning specific impacts (or case studies), for example training and structures put in place to ensure high integrity engagement.

What will this mean for impact culture? If rigour is broadened out to consider wider ethical issues around the generation of impact, then this has the potential to drive significant change. Added to this, done well, the reintroduction of impact narratives could capture and therefore incentivise a much wider range of impact activities than are typically captured in case studies, which tend to focus on the more instrumental, easy-to-measure and mature impacts. As Tadhg Caffrey has pointed out in his insightful reflections on the announcement, REF2021 incentivised a narrow range of hero stories from individuals that grossly simplified the messy reality of impact. He points out that the proposed reforms have the potential to incentivise more diverse stories from a more diverse range of people, better recognising the role of teams, including professional services colleagues in more complex narratives, that better capture reality. Although there are many important details to be worked out, these changes are both progressive and welcome.

To explore these issues in greater depth, I facilitated a discussion with Katrina Firth from the REF2029 team, which you can watch here:

Logo copied from REF2029 Twitter page, here