Reproducibility and reliability of biomedical research

The Academy of Medical Sciences, jointly with the BBSRC, MRC and Wellcome Trust, held a symposium on 1-2 April 2015 to explore the challenges and opportunities for improving the reproducibility and reliability of biomedical research in the UK.  

Questions about the reproducibility of scientific research have been raised in a number of different arenas over the last few years, including the general and scientific media, sparked in part by an increase in the number of retracted papers and a failure by industry to replicate findings in ‘landmark’ papers. The consequences are potentially significant for many areas of biological and broader scientific research. The meeting considered the implications for the future of biomedical research in particular, but we hope that the outcomes will be of interest across the disciplines.

A report of the symposium was published in October 2015, alongside a statement from the four sponsor organisations. Both documents are available to download on the right.

The Academy was also involved in the Nuffield Council on Bioethics’ work on research culture and ethics, which led to the publication of a report on the ‘Culture of Scientific Research in the UK’ in December 2014. Our Director of Policy, Rachel Quinn, was a member of the working group and our President, Professor Sir John Tooke FMedSci, co-signed the report foreword. This was important background to our work on the reproducibility and reliability of biomedical research and informed our symposium.

In October 2016, the InterAcademy Partnership for Health published a statement, A call for action to improve the reproducibility of biomedical research. The statement was endorsed by 46 national Academies, including the Academy of Medical Sciences. The document is available to download on the right. 

The organisation of the symposium was overseen by a steering committee, who met for the first time in early November 2014. The steering committee members were:

• Professor Dorothy Bishop FRS FBA FMedSci (Chair), Professor of Developmental Neuropsychology, University of Oxford
• Professor Doreen Cantrell CBE FRS FRSE FMedSci, Professor of Cellular Immunology, Wellcome Trust Principal Research Fellow, and Vice Principal and Head of College, University of Dundee
• Professor Peter Johnson FMedSci, Professor of Medical Oncology, University of Southampton
• Professor Shitij Kapur FMedSci, Professor of Schizophrenia, Imaging and Therapeutics, King’s College London
• Professor Malcolm Macleod, Professor of Neurology and Translational Neuroscience, University of Edinburgh
• Professor Caroline Savage FMedSci, VP and Head, Experimental Medicine Unit, GlaxoSmithKline
• Professor Jim Smith FRS FMedSci, Director of Research, Francis Crick Institute, and Deputy CEO and Chief of Strategy, MRC
• Professor Simon Tavaré FRS FMedSci, Director, Cancer Research UK Cambridge Institute, University of Cambridge
• Professor Melanie Welham, Executive Director of Science, BBSRC
• Dr John Williams, Head of Science Strategy, Performance and Impact, Wellcome Trust (member until June 2015)
• Dr Nicola Perrin, Head of Policy, Wellcome Trust (member since July 2015)

We are committed to improving the reproducibility and reliability of biomedical research in our own activities, as well as working with others to do so. In October 2016, 12 months after publishing our symposium report, we issued an update on our activities, alongside our three symposium partners: the Biotechnology and Biological Sciences Research Council, the Medical Research Council and Wellcome.

The symposium highlighted that there should be a collective ethos of responsibility in addressing irreproducibility by stakeholders across the biomedical research community, including funders, publishers, universities, and academics. We looked across our activities, at things we were already doing and things we could change, to ensure that we were acting wherever possible to  improve research reproducibility – and our work here continue. The October 2016 update summarises such activity from all four partners and our own efforts include:  

Openness and Transparency

The Academy expects all of its award holders to maximise the availability of research data. We now require all lead applicants to provide an ORCID iD when completing an application form, and require researchers to make their publications available through Europe PMC.

For further information, please read the Academy’s position statement on open access.

The Academy is a signatory of the Concordat on Openness on Animal Research. We are committed to being open about when, how and why animals are used in the projects we fund, as also outlined in our statement on the use of animals in research.

Research design

The Academy endorses the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines from the National Centre for the Replacement, Refinement, and Reduction of Animals in Research (NC3Rs), to improve the design, analysis and reporting of animal research. Please read the Academy’s statement on the use of animals in research.

We have updated our guidelines to encourage all grant applicants to consult the NC3Rs Experimental Design Assistant, a free online tool to guide researchers through the design of their experiments.

Funding decisions

The Academy supports the San Francisco Declaration on Research Assessment, which recognises the need to improve the ways in which the outputs of scientific research are evaluated. Our panel members and peer reviewers are requested not to use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, or to assess an individual scientist's contributions.

We collaborate with NC3Rs, who help us to peer review applications involving the use of non-human primates, cats, dogs, pigs or equines.

Education and training

The Academy is reviewing the training needs of its sectional committee members and its grant panel members, to inform future development of training opportunities. In September 2018, our Careers Team partnered with Professor Marcus Munafo to host a webinar on reproducibility for early career researchers, which can be listened to below.

Raising awareness

We have led conversations on this issue with Academies in other countries, for example, we initiated an international statement on reproducibility, which was endorsed by 45 of the world’s medical Academies. We also promoted this issue when we published the symposium report, for example, Professor Dorothy Bishop spoke to BBC Inside Science.