Trial By Error: The Cochrane Controversy

By David Tuller, DrPH

Cochrane–formerly called the Cochrane Collaboration–is respected worldwide for its systematic reviews of medical treatments. These reviews are often cited as the definitive source of information about treatment efficacy and safety. In taking on the thankless task of assessing the data on commonly used interventions, Cochrane performs an invaluable public health service and has advanced the cause of evidence-based decision-making in medicine.

But like any organization, Cochrane can get things wrong—as it has in the case of chronic fatigue syndrome. (Cochrane generally uses the term CFS, so I will also when referring to these systematic reviews.) Cochrane’s review of cognitive behavior therapy for CFS was published in 2008, pre-PACE. The most recent review of exercise therapies for CFS, which mainly included studies of graded exercise, was published in 2014. These systematic reviews and previous versions, all of which reported benefits from the treatments, were conducted by Cochrane’s Common Mental Disorders group.

Last month, The Times and The BMJ covered the growing international concerns about the PACE trial. Both publications ran articles about Virology Blog’s most recent open letter to The Lancet, which cited PACE’s “unacceptable” flaws and called for a fully independent reanalysis of the trial data. The letter was signed by 114 experts, ten members of Parliament, and 70 patient and advocacy organizations. To counter this sort of public criticism and support their unwarranted claims, the CBT/GET ideological brigades and their enablers regularly cite Cochrane’s systematic reviews.

Most recently, Professor Fiona Watt, executive chairwoman of the UK Medical Research Council, released a statement in response to The Times’ coverage of the Lancet open letter. The MRC, the main funder of PACE, has previously defended the conduct of the study. In her letter, Professor Watt reaffirmed this support without providing any response to the specific concerns raised about PACE—such as the paradox that 13 % of the participants were already “recovered” on the key outcome measure of physical function at baseline, before any treatment at all.

Professor Watt’s defense of PACE rested heavily on the fact that other researchers have similarly reported benefits from CBT and GET. She noted pointedly: “This evidence is summarised in three Cochrane reviews. Cochrane reviews are systematic reviews of primary research in human healthcare and health policy, and are internationally recognised as the gold standard in evidence-based healthcare.” [It is not clear which is the third Cochrane review being referenced here.]

It takes nothing away from Cochrane’s reputation to note that systematic reviews are only useful if the studies they include provide valid and reliable data. If the studies themselves are fundamentally flawed, and if the purported experts conducting and writing the reviews refuse to acknowledge or cannot understand these shortcomings, then any synthesis or summation will generate similarly problematic conclusions. This is what appears to have happened with the systematic reviews for CFS treatments conducted by the Common Mental Disorders group.

**********

Let’s dispense with one issue right away: This illness should not be housed in the Common Mental Disorders group. Whatever the historical reasons for this arrangement, it undoubtedly must lead observers to assume that Cochrane as an organization endorses the framing of CFS as a psychiatric illness. Patients object to the situation not because they are prejudiced against psychiatry and people with mental disorders–as the PACE authors and others have claimed—but because their illness is not a mental disorder and because the Common Mental Disorders group has already demonstrated its inability to assess the research accurately.

(Cochrane editor-in-chief David Tovey and psychologist James Coyne have previously engaged in a public debate over issues related to conflicts-of-interest of Cochrane reviewers in this domain. I am not addressing those issues in this post.)

The Common Mental Disorders group’s most recent version of the exercise systematic review drew skeptical scrutiny from very smart advocates soon after its 2014 publication. Patient-researchers Tom Kindlon and the late Robert Courtney, in particular, submitted cogent and comprehensive comments that exposed the systematic review’s serious flaws and refuted its unfounded claims. When Cochrane republished the review last year, it included the exchanges between the correspondents and Lillebeth Larun, the lead author.

Larun, a researcher and associate professor in the department of assessment interventions at the Norwegian Institute of Public Health, provided inadequate defenses to the concerns raised by Kindlon and Courtney. I won’t dissect the arguments here. But Larun’s response to one problem is worth highlighting for its audacity in re-purposing English to justify poor methodology.

In PACE, the investigators switched their methods of assessing their primary outcomes from those detailed in their protocol. These outcome switches—which produced numbers that favored a more positive interpretation of the results—took place after data collection. The PACE investigators have nonetheless referred to these revised assessment measures as “pre-specified” because, as they have explained, the changes were made before they examined their data.

The issue is significant because it impacts how Cochrane reviewers should assess PACE’s risk of bias. In Cochrane’s own guidelines for assessing a study’s risk of bias across multiple domains, the requirements for being considered at “low risk” of bias when it comes to reporting results include the following: “The study protocol is available and all of the study’s pre-specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre-specified way.”

That sentence seems clear. “Pre-specified” in this case means “specified in the protocol before the beginning of the trial.” It is indisputable that PACE was not reported in this “pre-specified” way. Yet Larun, parroting the PACE authors, has chosen to re-define the word so that it can encompass what actually happened in the trial. Here is what she wrote about the outcome-switching: “These changes were drawn up before the analysis commenced and before examining any outcome data. In other words they were pre-specified, so it is hard to understand how the changes contributed to any potential bias.” She assessed PACE as having a “low risk” of bias.

Larun’s position is unsustainable. Clinical trial investigators write protocols so that everyone understands what the goal-posts are. No matter what Larun and the PACE authors might argue, “pre-specified” does not mean “specified post-data-collection-but-pre-data-viewing”—and that’s per Cochrane’s own risk-of-bias guidelines, which Larun apparently decided she could ignore. Moreover, PACE was an open-label trial relying on subjective outcomes. In such cases, investigators are likely to know the outcome trends long before they look at any actual data. In this context, to define the reported PACE outcome measures as “pre-specified” is ridiculous.

**********

With more than 600 participants, PACE was the largest treatment trial for the illness. Even so, removing it from the exercise systematic review would not change the overall conclusions. But both the exercise and CBT systematic reviews suffer from other deficiencies that render their findings suspect and essentially meaningless. Putting aside PACE’s outcome-switching and other unique flaws, the trial exemplifies two major problems that plague much or most of the CBT/GET research for this illness. The first is that many studies use overly broad case definitions. The second is that the studies are open-label trials that rely on subjective outcomes.

The first problem means that study samples are likely to include a heterogeneous collection of people suffering from chronic fatigue for any number of reasons, including depression and anxiety disorders, but not necessarily the illness supposedly being investigated. It is possible that some of these other participants could benefit from CBT and GET, complicating any efforts to interpret the findings.

The second problem—combining open-label status with subjective outcomes–means that positive self-reports from participants in treatment arms could easily be due to bias. Since participants know their treatment allocation as well as whether the treatment is supposed to help them, their responses are likely to be influenced by hopes and expectations. It is not clear why systematic reviews should include such trials at all, just as it is not clear why anyone would spend much money conducting them in the first place.

Do systematic reviews of pharmaceuticals generally include such trials and assess them as providing robust evidence with a low risk of bias? If not, why is that appropriate in the case of this illness and these studies? In any event, if Cochrane feels it must include these inherently unreliable trials in systematic reviews, then its guidelines should automatically designate them as having a high overall risk of bias, even if they boast other laudatory traits.

Systematic reviews that includes studies with these thorny problems will feature some of the same defects themselves. Unfortunately, such reviews will provide little or no information about people suffering from the illness in question as defined through more precise definitions. In this case, that means not only the CDC’s 1994 Fukuda definition (for CFS) but also two superior ones drawn up by international committees of experts–the 2003 Canadian Consensus Criteria (for ME/CFS) and the 2011 International Consensus Criteria (for ME). (There is also the US Institute of Medicine’s 2015 clinical case definition for systemic exertion intolerance disease, or SEID, but that’s another issue.)

And such systematic reviews will provide no information about whether objective measures support the positive subjective reports. Larun has acknowledged that including objective outcomes in the exercise systematic review would be helpful. However, to justify having excluded them from the current exercise review, she noted that they were excluded from the systematic review protocol. Of course, that reasoning just raises the question of why objective findings were excluded from the protocol.

When it comes to this illness, objective findings have generally not supported the published subjective results. Including objective data in the systematic review would therefore have required a downward reassessment of the reported benefits of the interventions. Perhaps that is one reason it was decided to leave these data out of both the protocol and the review.

**********

The Common Mental Disorders group has written a second exercise systematic review, using individual participant data from the various trials rather than just the published results. This IPD review was reportedly supposed to have been published last year, but it remains unpublished. It was known that Cochrane—to its credit—sent it out for peer review to people beyond the usual orbit. These further peer reviews were said to have been scathing. This would not be a surprise to anyone outside the biopsychosocial bubble-think.

Cochrane is aware that concerns about PACE and this entire field of research have extended beyond the patient and advocacy communities. It knows the US National Institutes of Health and the Institute of Medicine (now the National Academy of Medicine) released major reports three years ago that declared the illness to be organic and not psychological in nature. It knows that the US Centers for Disease Control has rejected CBT and GET as treatments for what it now calls ME/CFS; that the US Agency for Healthcare Research and Quality has downgraded its recommendations for CBT and GET after stratifying the analysis by case definition; and that the UK National Institute for Health and Care Excellence is pursuing a “full update” of its guidance.

In other words, international support for the CBT/GET paradigm is crumbling. Yet members of the Common Mental Disorders group still champion these treatments, basing their arguments on deficient research. This presents a challenge for Cochrane. The challenge involves not just what to do with the unpublished IPD review but how to handle the published reviews as well. These reviews, and in particular the exercise review, continue to exert a harmful impact on patient treatment options, as I noted in a recent post about the Mayo Clinic. That will continue as long as CBT/GET promoters can hide behind Cochrane’s skirts.

In the near future, Cochrane needs to make some tough decisions, announce its plans, and then clean up the mess created by the Common Mental Disorders group. Should the current systematic reviews be withdrawn? (Absolutely, from my perspective, with the reasons clearly outlined.) Or should they be slapped with warning labels while experts unaffiliated with the Common Mental Disorders group reconsider the entire enterprise and develop a new strategy for assessing studies and analyzing the data?

Would any subsequent systematic reviews be required to differentiate results based on case definition? Would these reviews highlight objective outcomes? If open-label trials relying on subjective outcomes are to be included, would they be appropriately assessed as having a high overall risk of bias?

Professor Watt’s recent defense of PACE suggests that deference to authority still outweighs scientific reasoning in powerful sectors of the UK medical-industrial complex. Cochrane will likely face pushback for seeking to address the flaws of these systematic reviews, so taking corrective action won’t be easy. But to protect patients’ health, it must be done.

Comments on this entry are closed.

  • Peter Trewhitt 3 September 2018, 2:19 pm

    Thank you David.

    Getting the faults in the Cochrane reviews relating CBT and GET recognised is essential if these harmful treatments a ditched.

  • Sandra 3 September 2018, 2:54 pm

    What a beautiful piece of clear, rational writing. David, you have so cogently laid out the problems with the Cochrane reviews. Thank you! Cochrane must respond appropriately for, as you said, patients’ lives and well-being are on the line. Step up to the plate, Cochrane, and do your part to clean up this mess.

  • Anton Mayer 3 September 2018, 3:32 pm

    Cochrane is not to be taken seriously as long as it pretends that unblinded trials with subjective outcomes are reliable. Especially for conditions that the trial authors believe to be perpetuated by cognitions.

    The acceptance of this kind of clinical trials is holding back progress and is detrimental to patients. Alternative trial designs and outcome measures exist.

  • Rachel Riggs 3 September 2018, 3:39 pm

    Jeremy K. Cutsforth-Gregory was my GET/CBT-prescribing Neurologist at Mayo. At the time we thought he was the loveliest, and it was so cool that Mayo had an openly same-sex married physician on staff. We thought it was indicative of a forward-thinking institution. Well, I barely dodged that bullet! Thankfully, I found an article written by Cort on Health Rising about the PACE debacle just days before putting 20k on a credit card and heading back to Mayo for their month-long program. Perhaps you could contact Dr. Cutsforth-Gregory directly and ask him about that….

  • Ellen Goudsmit 3 September 2018, 4:03 pm

    Larun’s own Norwegian review was more critical. But then she didn’t have White as advisor. Reviewers, and I’ve had contact with them re my own research, all from cbt fanclub.

  • Sally James 3 September 2018, 4:24 pm

    Thanks David. As others have said this is a great explanation.

  • Simone 3 September 2018, 6:00 pm

    Thank you for this article, David. There are so many changes happening in the field, and the psychosocial model is weakening, but the Cochrane reviews are really holding us back. Until they are addressed, the psychosocial crowd will always have them as an authoritative voice to support their position. Equally, once they are addressed, the psychosocial crowd will be left without any major support. I think it will be a watershed moment.

  • David Foley 3 September 2018, 8:49 pm

    Thanks to David Tuller, and especially to Robert Courtney, for drawing attention to some of the problems with the recent review of exercise therapy from Larun and colleagues. While the inaccuracies and contradictions Courtney identified are serious enough, Larun’s refusal to properly address these issues in her replies to Courtney’s comments is even more worrying. When mistakes are identified they should be promptly corrected. It seems that Larun is unwilling to acknowledge the problems with her work, and the underlying research that she was reviewing, but unable to offer any meaningful defence. Those overseeing this review at Cochrane should have recognised then that they needed to step in – instead they have left the review’s identified flaws in place while others use the review’s unsafe findings to shape their understanding of this contentious area of research. Cochrane need to explain what went wrong here, and how they intend to ensure that when they are informed with problems within their reviews appropriate action is taken.

  • Paul Fox 4 September 2018, 5:38 am

    This is another excellent piece from David. What we would do without him, I can hardly imagine.

    When David writes “Professor Watt’s recent defense of PACE suggests that deference to authority still outweighs scientific reasoning in powerful sectors of the UK medical-industrial complex.”, I think that he hits the nail on the head. To speak against those in power is still very dangerous for one’s career, at least here in the UK, so it is understandable that those concerned for their livelihoods behave in the way that Prof Watt apparently has where PACE is concerned. However, understandability in a standpoint does not necessarily confer acceptability.

    Unquestionably, the responsibility of the position of Chair of the Medical research Council is to the public. For the holder of that office to close ranks with others in positions of power when to do so is so clearly against the public interest, and when what they are promoting is manifestly wrong, is a grave and egregious action. That it was, nevertheless so predictable, and that it is unlikely to dent Prof Watt’s standing, says so very much about the British way, from the humblest of local hobby clubs to the very top of society.

  • Mary 4 September 2018, 7:23 am

    A good article but why is the fact that the OBJECTIVE data published by the PACE authors shows little improvement left out? Mark VanNess clearly demonstrates that the severely ill patients pre trial were still severely ill post trial based on the objective data. All the vague ….subjective wishy washy arguements can’t explain away the objective facts. Yet this article ignores them????

  • Graham McPhee 4 September 2018, 9:00 am

    “Professor Watt’s recent defense of PACE suggests that deference to authority still outweighs scientific reasoning in powerful sectors of the UK medical-industrial complex.” Well said, David.

    That lies at the heart of it all! It is still very, very difficult to get established authoritative figures to look at the situation and comment. All we ever get is the argument that “the big boys said we could do it.”

  • Lene Christiansen 9 September 2018, 5:32 pm

    Thank you, David Tuller, for writing yet another article about the flaws in The Pace Trials. Every scientist should be able to see the flaws so why didn’t they..?

  • Nancy Blake 15 September 2018, 9:12 pm

    Margaret Williams documented Professor Wessely’s interference with the earlier Cochrane review. I have noted that the current Guideline Review process plans to rely on Cochrane Reviews, as part of an overall attempt to keep the Guideline firmly within the BPS model and the psychiatric remit. Wessely is alive and well. Google Nancy Blake ME/CFS and look for my Positive Health article ‘What Can We Expect From The Current Review of Guideline CG53, ME/CFS ‘

  • William A. Perrin 25 September 2018, 2:44 am

    Thanks David for sharing this article. Cochrane’s controversy has been explained quite well here. If Cochrane were to come forward responsibly and address it may help to regain their trust.