Trial By Error: More on Cochrane’s New Risk of Bias Tool

By David Tuller, DrPH

As Virology Blog has reported, the lead author of the revised version of Cochrane’s Risk of Bias tool, published last week in BMJ, is a long-time Bristol University colleague of Professor Esther Crawley. In that capacity, he is a co-author of two high-profile studies that violated key principles of scientific investigation—the Lightning Process study, published by Archives of Disease in Childhood two years ago, and the 2011 school absence study published in BMJ Open.

I have reported at length on the problems with these studies here and here, and in many subsequent posts. In the Lightning Process study, the authors recruited more than half the sample before trial registration, swapped primary and secondary outcome measures after gathering data from these early participants, and failed to disclose these salient details in the published paper. The outcome swap allowed Professor Crawley, Professor Sterne and their colleagues to report positive rather than null findings for their official but revised primary outcome.

In July, after more than a year of investigation, Archives of Disease in Childhood, a BMJ journal, published a “correction” that acknowledged the methodological missteps. Yet the journal still allowed Professor Crawley, Professor Sterne and their colleagues to republish the biased findings from the initial paper.

Last week, Virology Blog sent an open letter protesting this untenable decision to Dr Fiona Godlee, editorial director of BMJ. The letter was signed by 55 scientists, academics and clinicians from Harvard, Stanford, University College London, London School of Hygiene and Tropical Medicine, Berkeley, Columbia, Queen Mary University of London, etc., and expressed dismay at the journal’s “scientifically and ethically indefensible” actions. Some of the signers have subsequently made their own calls for retraction of the Lightning Process study.

*****

Given this background, it is perhaps not surprising that Professor Sterne’s revised Risk of Bias tool makes it easier for unblinded studies to be identified as having a “low risk” of bias, as the smart folks on the Science For ME forum have noted. Professor Sterne and his co-authors themselves make the point, in their discussion section.

Here’s what they write:

“We expect the refinements we have made to the RoB tool to lead to a greater proportion of trial results being assessed as at low risk of bias, because our algorithms map some circumstances to a low risk of bias when users of the previous tool would typically have assessed them to be at unclear (or even high) risk of bias. This potential difference in judgments in RoB 2 compared with the original tool is particularly the case for unblinded trials, where risk of bias in the effect of assignment to intervention due to deviations from intended interventions might be low despite many users of the original RoB tool assigning a high risk of bias in the corresponding domain. We believe that judgments of low risk of bias should be readily achievable for a randomised trial, a study design that is scientifically strong, well understood, and often well implemented in practice.”

This paragraph packs a lot of problems into a thicket of sentences that are not easily untangled. In discussing the risk of bias in studies with unblinded interventions, the authors only mention bias that might occur when there are “deviations from intended interventions.” While it is true that such deviations can cause bias, they are not the only source of bias in these studies with unblinded interventions. In particular, when studies combine unblinded interventions with self-reported outcomes, the self-reported outcomes are likely to be impacted by expectations, hopes and other aspects of participants’ relationships to the assigned interventions and/or to the providers of the interventions, even in the absence of deviation from intended or assigned interventions. The possibility or likelihood of such bias is why other fields of medicine do not generally develop guidelines or policy from studies with this specific design–unblinded interventions relying on self-reported outcomes.

Moreover, the statement in the revised Risk of Bias tool’s discussion section appears to reflect a conviction that randomization alone is likely to ensure a “low risk” of bias. One argument presented is that randomized trials have often been “well implemented.” It is hard to know exactly how to respond to this non sequitur. The fact that some or many randomized trials have been “well implemented” seems irrelevant to whether a specific randomized trial—whether of ME/CFS or any other disease or condition—has also been “well implemented” and should be considered at “low risk” of bias. As for being “well understood”…Yes, it is well understood that randomized trials combining unblinded interventions with self-reported outcomes will generate results that suffer from unknown levels of bias.

Other aspects of this revised Risk of Bias tool are also troubling. The appendix provides detailed guidelines on how to interpret its various planks or decision points. One section covers whether study data were analyzed “in accordance with a pre-specified analysis plan that was finalized before unblinded outcome data were available for analysis.” Per the guidelines, “Changes to analysis plans that were made before unblinded outcome data were available, or that were clearly unrelated to the results (e.g. due to a broken machine making data collection impossible) do not raise concerns about bias in selection of the reported result.”

This expansion or elaboration of the meaning of “pre-specified” is significant. The revised Risk of Bias tool considers “changes to analysis plans” to be “pre-specified” if they occurred at any time before the unblinding of outcome data. This revision might perhaps be justified or at least acceptable for trials that rely on objective outcomes. Yet it ignores or overlooks what happens in trials with unblinded interventions that rely on self-reported outcomes, like the Lightning Process and PACE trials. In these cases, investigators are likely to be aware of outcome trends long before any outcome data is actually unblinded. No matter what Professor Sterne and his co-authors assert, changes to analysis plans made after the start of such trials should raise real concerns about bias arising from selective outcome reporting.

It bears noting that much research from the biopsychosocial field, including research into ME/CFS and so-called “medically unexplained symptoms,” or MUS, is of this type. So this revision of the Risk of Bias tool could have a huge impact on Cochrane reviews in these domains. The revision will certainly make it much easier for reviewers to scoop up bad science with the good while officially rating it all as being at “low risk” of bias.

The argument adopted by Professor Sterne and his co-authors—that changes made up until the point when outcome data have been unlocked can essentially be deemed “pre-specified”–has been routinely used by the PACE authors to excuse their own rampant outcome-switching. The argument was further deployed by Lillebeth Larun, the lead author of the contested Cochrane exercise review for ME/CFS, in defending her inflated assessment of the PACE trial. The revised tool’s fallacious reasoning is also convenient for Professor Sterne himself, since it could presumably be used to overlook the outcome-swapping in his own Lightning Process trial.

Is anyone paying attention here? BMJ? Cochrane? Hello???

Comments on this entry are closed.

  • Couch Turnip 4 September 2019, 5:17 pm

    I’m glad there are some who are paying attention. Good luck with getting any response from BMJ or Cochrane.

    Sterne was not an author of the feasibility trial paper but we know from its ‘Acknowledgements’ section that he gave advice on the methodology of the trial. Does that mean that he gave advice on the methodology of the full LP trial, since it all turned into the same amorphous blob?

  • Couch Turnip 5 September 2019, 6:46 am

    One paediatrician signed Tuller’s letter to Godlee about the BMJ’s and ADC’s decision on the Lightning Process Trial paper. I take my hat off to him. But where are all the other paediatricians who are so keen to stress the importance of safeguarding children? Why won’t they take the time to examine this case when a group in their own midst apparently stands accused of methodological and ethical transgressions? UK paediatricians surely know that doctors can be capable of all sorts of wrongdoings, we only have to think of high profile cases in the UK press. I’m not suggesting that the LP researchers have broken any UK law, but doctors should be alert to issues raised about their own colleagues and not protect them by laughing in the face of criticism and/or by refusing to examine it.

    Sir Terence Stephenson is now the Chair of the HRA. According to this UCL child health webpage (https://www.ucl.ac.uk/child-health/news/2019/aug/professor-sir-terence-stephenson-appointed-health-research-authority-chair) he holds and has held some very senior and eminent positions in the UK medical establishment and in relation to child health, including having been the President of the Royal College of Paediatrics and Child Health. Perhaps David should ask him whether he thinks the breaches in the LP trial paper are acceptable and what he thinks of this study in relation to safeguarding children.

  • jimells 5 September 2019, 11:27 am

    This is a “Get Out Of Retraction Free” card for PACE, the exercise review, and the 99% of psychology research that is junk. My crystal ball predicts that Regius Professor Sir Simon will soon be joined by Professor Sir Jonathan Sterne, for Service to the Empire. I hope he enjoys the ride on SS PACE as it continues its painfully slow journey to Davy Jones Locker.

    They’ve convinced themselves that magic wands can make icebergs disappear, but in the end, reality always bites very, very hard.

  • chirurgie esthetique Tunisie 23 September 2019, 5:44 am

    Cet article est exceptionnel, bonne continuation 🙂

  • blepharoplastie tunisie 3 October 2019, 8:53 am

    bonne continuation

  • chirurgie esthetique tunisie 3 October 2019, 8:56 am

    c manifique