Professor Marcus Munafo is a biological psychologist at the University of Bristol. He gave a talk on research reproducibility at the Academy’s ‘Succeeding in Team Science’ career development event for early career researchers. For those who were unable to attend the oversubscribed Team Science event, he followed up his talk with a webinar and now this blog post, which explore the issue in more detail.
For many years, but particularly in the last ten years, scientists have discussed the extent to which published findings in the biomedical research literature are robust and reproducible.
Certainly there are indications that we can do better. The pharmaceutical industry argues that findings from the academic literature are no longer robust enough to inform their drug discovery programmes. Empirical estimates routinely find that less than 50% of published findings can be replicated.
Crisis or opportunity?
Some describe the current situation as a “crisis”, but in my view that framing is not particularly helpful (or has at least served its purpose). Many of the issues described are not new: low statistical power, problems associated with basing claims on p-value thresholds, and so on have all been discussed extensively in the past. After all, science is still continuing to advance and make new discoveries. But perhaps science could be more efficient, and self-correct more rapidly. In other words, we should see the current situation as an opportunity – to reflect on how we work, and whether we can improve on the current status quo.
The challenge is that academia is too conservative. Although we might develop cutting-edge technologies in our work, many of the ways we work are antiquated. Our dominant model of statistical inference – null hypothesis significance testing – is 100 years old; many research groups still revolve around individuals, reminiscent of the self-funded scientists of the 19th century; and our primary means of disseminating knowledge – the scholarly journal – is 400 years old. In an era of team science and online repositories for study materials, data and outputs, is it time for science to modernise?
What are people doing?
Various key elements of the scientific ecosystem are beginning to change:
- Funders are increasingly placing an emphasis on research reproducibility, requiring grant applicants to state how they will ensure their findings are robust, and awardees to make their data available on completion of their research.
- Publishers are introducing reporting checklists to increase the quality and transparency of published research.
However, system-level change will require institutions and individual researchers to take action as well. Whilst there are plenty of standalone examples of this, these are currently somewhat fragmented.
To support scientists and institutions to make these changes, we recently set up the UK Reproducibility Network – a peer-led initiative to develop and share best practice across universities within the UK, and work with funders, publishers and other stakeholders to ensure that this activity is coordinated. So far we have individuals at 35 universities, who are setting up local networks across their departments and faculties, and support from 15 stakeholders. Whist biomedical research forms the majority of our constituency, we are keen to bring together individuals and stakeholders from across the research spectrum.
What can I do?
If you are interested in joining the UK Reproducibility Network contact Marcus Munafò (marcus.munafo@bristol.ac.uk) or follow the UKRN on Twitter (@UKRepro).
If a local network already exists at your institution UKRN can put you in touch; if not, UKRN can help you to set one up.
Initiatives at every level
Many of the UK Reproducibility Network’s initiatives are at the grass roots – setting up Open Research Working Groups to promote open practices; early career researcher led “ReproducibiliTea” journal clubs to discuss and critique meta-research studies.
Others are intended to change cultures within universities from the top down; for example, we have developed a role description for a senior academic lead focused on research improvement, reporting to the senior management team. This is being considered by a number of universities.
Yet others are for funders and publishers – the Editors4BetterResearch initiative (which promotes, among other things, open research practices), and the Registered Reports Funding model (which combines the grant and journal peer review process and offer both funding and in principle acceptance of the resulting publication from the outset).
There is now a mature literature describing the epidemiology of research reproducibility. We are now entering the next phase – developing and testing interventions that are intended to improve the effectiveness and efficiency of scientific research. Time will tell which initiatives will work but, critically, this will not be a one-off effort. Rather, the development of meta-research as a discipline in its own right will allow us to continually assess our ways of working, and ensure that the incentive structures within science, and our underlying culture, are aligned with what is best for science and, by extension, society.
Professor Marcus Munafo, November 2018
_________________________________________________
Our Career Development events are for pre- and post-doctoral biomedical and health researchers, to give practical skills and the chance to network with colleagues. They also offer clinical academic trainees help in navigating the clinical academic training pathway and developing a research career. We hold three or four per year; to find out about upcoming events, please see the events page.
If you would like to receive the Academy’s bi-monthly Career Development newsletter, with updates on our events, funding and other opportunities for early-career researchers, please sign up here.
The Academy of Medical Sciences, jointly with the Biotechnology and Biological Sciences Research Council, the Medical Research Council and Wellcome Trust, held a symposium in 2015 to explore the challenges and opportunities for improving the reproducibility and reliability of biomedical research in the UK. One year following the event, we issued an update on actions that have been taken since the meeting, alongside our three symposium partners. The importance of research reproducibility was recognised in the Academy’s recent report on ‘Enhancing the use of scientific evidence to judge the potential benefits and harms of medicines’ with a recommendation that reproducibility efforts should be recorded in the environment statement in the next Research Excellence Framework process.