By Dr Jimmy Twin, ISCRR Research Communications and Translation Manager.
No matter the discipline, research is carried out to achieve some form of positive impact. This impact varies according to why the research is needed – from discovering new knowledge, to achieving a societal change such as saving lives. Thankfully, there is a growing focus on the importance of understanding and measuring research impact, including the design and implementation of formalised impact assessments. This article will share some of the learnings ISCRR has had thus far in our work measuring research impact.
Since ISCRR was established in 2009, we have been involved in several research impact activities including return on investment analyses. In 2018, WorkSafe Victoria and ISCRR started to formalise a process to measure research impact across the research program. A research impact framework was developed, based on the work carried out by the Institute for Work and Health. The framework developed focussed on recording impacts according to three levels:
Underpinning these three impacts are measures relating to building an evidence base and research capacity.
This framework was conceptualised as a continuous improvement cycle, to adopt learnings in order to optimise research impact measurement methodologies for future assessment time points, and for future projects. Research impact data is intended to be collected throughout the project lifecycle and beyond, through conversations with those who used the research, and a formal research impact survey six months post-project completion.
During 2019 and 2020 this framework was piloted on 17 projects, 14 which were completed during the 2018/19 financial year, and 3 from 2019/20. For an initial pilot exercise, the results were incredibly positive.
For each project, we identified what impacts each research project intended to achieve. In total, 71 of these impact goals were identified. These varied from the dissemination of research reports to key stakeholders to improved injured worker outcomes. As impacts can take years to be achieved, we were pleasantly surprised to see 65% of these impacts achieved. Nearly all (96%) of the intended research dissemination and diffusion impacts were achieved, 73% of the impacts intended to inform decision making, and perhaps most surprising, there was evidence indicating 36% of the impacts contributing towards societal change had been achieved.
The impact goals that tended to be integral to the majority of research projects were those that aimed to inform decision making. Of the 17 projects analysed, 14 helped WorkSafe make evidence-informed decisions (82%), with the remaining three expected to inform decision making at a later date. The aim of this pilot was to begin the process of implementing a research impact assessment framework, and we already understood it would be too early to record impacts for many projects.
During our pilot, what drove high quality data collection and detailed research impact reporting? Simply put, good stakeholder engagement. We saw higher quality data with projects where there was strong stakeholder engagement from the project design stage and where champions were identified. And we feel that these champions also drove initiatives that improved impact outcomes.
It was likely that data supporting impact goals were missed, particularly when champions left the business and made potential impact learnings difficult to obtain.
With any new implementation process, both enablers and barriers were identified. It was clear from the outset that a proactive approach to data collection was required. Retrospective analysis lacks the planning for research impact data collection, and those champions may have left the organisation making connecting the dots difficult.
For some projects, certain impact goals defined at the beginning of the project were not feasible, indicating impact goals several steps ahead of the research project. It is incredibly tempting for researchers, with the best intentions, to make broad statements about the potential impact of their research, such as reducing workplace injuries. And although these future impact goals are worthy to note, this process highlighted the need to ensure research impact goals for each project are both realistic and measurable.
Another important consideration was the need to validate findings. In some instances, we received information that research findings led to key decision-making impacts or potentially contributed towards societal change. However, follow-up validation of findings was needed which requires additional resourcing.
So where to now? WorkSafe was strongly encouraged by our pilot work and we are working with them to assess the next phase of their impact assessment program. We have adopted a case study approach to validate research impact findings, and we are currently working on several exciting research impact case studies to showcase some of these great research success stories.
ISCRR’s current translation strategy is based on this research impact framework, and we continue to improve and refine our processes to best meet business needs and to identify potential opportunities to both measure and guide research impact success.
Dr Jimmy Twin presented these findings at the XXII World Congress on Safety and Health at Work. Slides from this presentation can be found here (developed with assistance from Scientell).
This article was written by Dr Jimmy Twin, ISCRR Research Communications and Translation Manager. If you’ve found this article useful, please let us know via social media: Twitter or LinkedIn