Trial By Error: What’s Up With Cochrane’s Exercise Review?

By David Tuller, DrPH

On June 17th, Cochrane announced that it had received a revision of a much-contested review of exercise therapy for treatment of chronic fatigue syndrome (as the organization has long called the illness or cluster of illnesses also referred to as myalgic encephalomyelitis, CFS/ME, and ME/CFS). In a posted statement, Cochrane noted that “the process has taken longer than hoped; the amended review is being finalised and it will be published during the next 2 months.”

Well, it seems like the process continues to take “longer than hoped,” since it is now more than three months since that notice was posted. Publication of the revision is more than a month overdue. Perhaps Cochrane is planning to publish it imminently. Or perhaps the process of “finalising” the amended review has been more complicated than anticipated, raising further questions for Cochrane about whether it should endorse and publish another iteration of what has been a misbegotten mess from the start.

We don’t know what changes were made for the version mentioned in the June 17th statement. However, we do know that Dr David Tovey, Cochrane’s now-retired editor-in-chief, rejected a revision submitted in mid-May because it overstated the available evidence.

Critics–mainly very smart patients–have long made this same point about the 2014 review, which was an update of one published a decade earlier. Most of the eight studies included investigated graded exercise therapy. The review, which was conducted by Cochrane’s Common Mental Disorders Group, is so methodologically flawed that the best move would be for Cochrane to scrap the whole thing, transfer ME/CFS to a more appropriate section of the organization, and start from scratch.

In any event, Dr Tovey expressed his views in a set of e-mails released under Norway’s liberal freedom of information laws. (The review’s lead author, Lillebeth Larun, is a researcher at the Norwegian Institute of Public Health). On May 13th, Atle Fretheim, a director of research and innovation at Larun’s institute, sent Dr Tovey the long-awaited revision of the exercise review. On May 24th, Dr Tovey informed Dr Fretheim that Cochrane did not plan to publish it.

“With great regret and disappointment, I am afraid that I am not able to publish this version,” wrote Dr Tovey. “As we have discussed, the scrutiny that this review and this decision will be subject to means that we cannot publish a new version unless it is clear that the review is as watertight as humanly possible…In particular, I remain concerned that the presentation in general, but in particular of the short term results on fatigue, remains too optimistic.”

In discussing exercise therapy elsewhere in the e-mail exchange, Dr Tovey referred to it as “an intervention that really probably has very little to contribute”—a view at serious odds with the review’s claims of benefits. (Dr Tovey is now retired, and Dr Karla Soares-Weiser has assumed the position of Cochrane’s editor-in-chief.)

The wrangling over the review has embroiled Cochrane’s editorial leadership in the protracted dispute between advocates of integrity in science and the GET/CBT ideological brigades, who continue to have significant support among pooh-bahs within the UK academic and medical establishments. Experts in study design—that is, experts outside the precincts of the psychiatry departments at Oxford, Kings College London and some other select British institutions–recognize that unblinded trials relying on self-reported outcomes, like those included in the Cochrane review, generate unknown amounts of bias and cannot provide reliable evidence.

I should disclose that I have been told second-hand by a reliable source that Larun considers my work on the Cochrane review tantamount to “harassment.” If she does in fact believe this, it would demonstrate that she cannot distinguish between “harassment” and tough but fair criticism. I have never contacted Larun. When I was in Norway last November, I met with Dr Fretheim and another official from Larun’s agency, the Norwegian Institute of Public Health. I explained to them in sharp terms why I thought the Cochrane exercise review was deficient. Expressing my professional opinion about an influential paper to health officials cannot reasonably be defined as “harassment,” no matter how much the lead author dislikes my views.

*****

To recap briefly what’s happened in this saga till now: After Cochrane published the exercise review by Larun and three co-authors, two patients, Tom Kindlon and the late Robert Courtney, wrote extensive critiques in the correspondence. Larun provided deficient and intellectually bankrupt responses. Courtney then sent a formal complaint to Cochrane. Last year, after investigating Courtney’s concerns, Cochrane acknowledged their validity and asked Larun and her co-authors to address them. Unfortunately, Courtney was already dead. I have written most extensively about the Cochrane situation here. (In July of last year, I met with Dr Tovey to press the argument that the review was deficient in multiple ways.)

One of several key issues was the review’s claim that PACE was at low risk of selective outcome reporting bias—even though the investigators changed their primary outcomes well into the conduct of the trial. In the correspondence following publication of the review, Larun insisted that the primary outcomes in PACE were “pre-designated” because they “were drawn up before the analysis commenced and before examining any outcome data.” In advancing this claim, Larun was parroting the bogus argument often promoted by the PACE authors themselves to excuse their dramatic outcome-switching. Apparently it needs to be repeated that, in open label trials with self-reported outcomes, trends in results should be apparent to investigators long before they see the actual data.

By sometime last year, Dr Tovey had apparently decided that the review should be withdrawn, pending the revisions. Last October, Reuters UK science “reporter” Kate Kelland, a bestie of the so-called Science Media Centre, broke the news of the attempted withdrawal with a profoundly biased article that portrayed Cochrane, and Dr Tovey personally, as caving in to rabid patient activists. The Reuters article quoted from an e-mail exchange between Cochrane editors and the review team about the proposed withdrawal. Although Reuters did not disclose the source of the e-mails, they were presumably shared with Kelland by those with an interest in pushing back against Cochrane’s decision to withdraw the review.

Predictably, given Kelland’s ties to the SMC and apparent sympathy for the plight of the unhappy researchers, the article failed to discuss the methodological issues at stake or the scientific reasons behind the decision to withdraw the review. In response, Virology Blog organized and posted a statement in support of Cochrane’s decision; the statement outlined the review’s serious flaws and was signed by more than 40 scientists, clinicians and academics from Harvard, Columbia, London School of Hygiene and Tropical Medicine, Berkeley and many other leading institutions.

Unfortunately, the distorted Reuters article stigmatized patients and lent credence to the manufactured argument that Cochrane was surrendering to an irrational and anti-scientific rabble. In the face of this public smackdown, Cochrane switched gears. A week after the Reuters article appeared, the organization posted a notice that a revision was in the works–but it did not withdraw the current review. In late November, Cochrane announced that it had rejected this revision, leaving the situation in some limbo.

The next step was a March announcement of further changes to the review that were “aimed at improving the quality of reporting…and ensuring that the conclusions are fully defensible and valid to inform health care decision making.” Some version of that revision was the one rejected by Dr Tovey in May. Presumably a re-revised version is what was being referenced in the June 17th statement.

Incidentally, the phrasing of the March announcement suggests that the conclusions of the current version are not “fully defensible and valid to inform health care decision making.” Given that, it is fair to ask Cochrane how the organization justifies having backed down on its earlier decision to withdraw the review pending acceptable revisions.

*****

After Dr Tovey rejected the revision on May 24th, Dr Fretheim responded by proposing further possible tweaks to settle their differences. The subsequent exchange featured spirited and somewhat testy debates over the obstacles to publication—in particular a conflict over the quality and/or certainty of the evidence supporting exercise as a beneficial treatment for fatigue.

According to the 2014 review, the evidence that exercise therapy reduces fatigue is “moderate-quality”—even though all the studies included in the review were open label studies with self-reported outcomes. Since self-reported outcomes in open-label studies are by definition subject to unknown amounts of bias, the “evidence” cannot reasonably be described as of “moderate” quality.

Dr Tovey’s e-mails make clear that he considered the quality of evidence in support of the claims for exercise therapy and fatigue reduction to be “low,” not “moderate.” As he wrote, “the conclusion that this is moderate certainty evidence seems indefensible to me, and as we know, I am not alone in this…It seems absolutely clear to me that this is low certainty evidence.” In response, Dr Fretheim suggested a compromise, in which the evidence would be characterized as “low-moderate.” It is unclear if or how this issue has been resolved in the revision now in the pipeline for publication.

Besides that dispute, some other changes appeared to have been agreed upon. According to an attachment sent to Dr Tovey by Dr Fretheim, the revision would include a statement that “patients diagnosed using other criteria may experience different effects.”

If the revision is indeed published, it will be critical to highlight this point prominently. The trials included in the review recruited patients based on either the Oxford criteria or the Fukuda criteria. Both of these case definitions are fatigue-based. Unlike more recent case definitions, such as the 2011 International Consensus Criteria for ME, they do not require the presence of post-exertional malaise and are more likely to generate heterogeneous study samples. As has been frequently noted, exercise therapies could very well be helpful for many people whose fatigue arises from undiagnosed depression or anxiety disorders, or other conditions.

Although not discussed in the e-mails released, Cochrane should also have insisted that any revised review include the various objective outcomes from the studies. Given that objective outcomes in ME/CFS trials have repeatedly failed to match the benefits found in self-reported results, including them would have required a downgrade in the ratings for exercise therapy. The 2014 review was based on an earlier protocol that did not include these objective measures. But it was a mistake for Cochrane to approve the update without ensuring that objective measures were added to the analysis. Any revision without them will be biased toward the positive. Cochrane should understand that it is unacceptable to publish a review—or republish a review–while ignoring key findings that would have forced a reassessment of the reported results.

On May 29th, Dr Fretheim proposed submitting the matter to a mutually agreed upon “arbitrator.” Perhaps that has happened, perhaps not. No e-mails from after May have yet been released. At this point, Cochrane is in a fix. Its editors must know, or at least suspect, that publishing this new version could be a big, big mistake. If Cochrane endorses a revision that turns out to be more propaganda for the purported benefits of exercise therapy, its reputation is likely to suffer far greater damage than if it pulled the plug on this disaster now and began the process all over again.

Comments on this entry are closed.

  • Peter Trewhitt 24 September 2019, 4:08 am

    Cochrane had promise to announce their decision about moving the responsibility for ME from the Common Mental Disorders Group some time ago, but all seems to have gone quiet in relation to that too.

  • Alicia Butcher Ehrhardt 24 September 2019, 4:09 am

    Thanks to you and people who have protested since the beginning, this is hitting the fan. I don’t envy them the position they created for themselves, and continue to shore up – but it’s their own fault.

  • LF 24 September 2019, 5:20 am

    Do You know if the desicionmakers read meta-analyzis regarding recoveryrate presented as f.ex here:
    https://www.mdpi.com/2075-4418/9/4/124

  • Lady Shambles 24 September 2019, 5:29 am

    A cynical person might ponder that the delay represents an opportunity to ‘re-frame’ in a manner acceptable to the BPS cabal, and the same cynical person might wonder whether Dr Karla Soares-Weiser is more ‘agreeable’ to such, or not! Certainly Dr Tovey was clipping the wings of the proponents of the BPS version of ‘science’ and only time will tell whether the new Editor in Chief is on the side of logic and reasoning or happy to be a puppet for the purveyors of junk science.

    One doesn’t need to be cynical to be aware that Cochrane’s reputation is already in the gutter…and is so despite this present drama . If they continue along this pathway they’re ‘toast’. After all their whole raison d’etre is unflinchingly truthful assessment and presentation of science. Given that solitary requirement is now missing then they no longer have a workable model do they?

    One also doesn’t need to be cynical to observe that the best criteria for ME (not synonym) MUST be employed by all clinicians and researchers in this field in order to prevent uninterpretable results. I’m very thankful to David for acknowledging the strength of the ICC. I’m just frustrated that it is not being adopted by all those who need to adopt it as their default set of Criteria…and by that I include large patient organisations who are frankly playing with the lives of those with the discrete disease ‘ME’ by not adopting it as their model of the disease and the one that should be politically demanded…. in the US and the UK and elsewhere..

  • Anton Mayer 24 September 2019, 7:13 am

    Proponents of therapies like CBT/GET and the lightning process ignore the issue of bias affecting self-reported outcomes. Authorities like Cochrane and NICE and the medical establishment are all too willing to go along with this.

    The logical outcome is the absence of a genuinely effective treatment because attention, resources and effort is continuously diverted towards treatments that only appear to work because nobody is subjecting them to a proper test.

  • Anton Mayer 24 September 2019, 7:17 am

    I want to get better or at least reach a level of function where I can do something meaningful with my life. The two main obstacles to finding an effective treatment is the low level of funding and the appallingly low scientific standards. Cochrane and the scientific journals are a big part of the problem.

  • CT 24 September 2019, 8:29 am

    And so we wait while patients continue to suffer. There seems to be such a lack of urgency with these things, almost to the point of contempt. I’m not suggesting that they should rush and get it wrong, but what on earth takes them so long?

  • Susanna Degaardt 24 September 2019, 8:31 am

    Thank you so much Dr David Tuller for shedding clear lights on this continueing scientific scandal!

    Here in Sweden, uninformed health care authorities of some counties unfortunately still lean upon this flawed Cochrane report and its ”treatments”, since it is still not withdrawn. Even as 99% of people with ME here are diagnosed by the strict ICC- and CCC-criterias.

    The misconceptions this report still fuels about a very serious organic disease, cause so much harm and extra societal costs. They still make ill people become permantly worse, they leave us, our local doctors and our burdened families in a vacuum of ignorance, with hardly no biomedical clinics or biomedical research.

    Thank you!

  • jimells 24 September 2019, 9:32 am

    If the Wessely School and associates were knowingly selling bad drugs instead of bad treatments they would be facing criminal complaints. In the US they could potentially be charged under the RICO Act (Racketeer Influenced Corrupt Organization). In other words, they are a criminal conspiracy protected by very important people.

    I wonder how much longer that protection can last…

  • Richard Vallee 24 September 2019, 11:41 am

    “the scrutiny that this review and this decision will be subject to means that we cannot publish a new version unless it is clear that the review is as watertight as humanly possible”

    This does not inspire confidence in Cochrane’s process, that if people aren’t looking over their shoulders and doing the job Cochrane authors and editors are unable to do, the process can and does produce harmful recommendations based on blatant pseudoscience and low-quality research of maximum bias and conflicts of interest.

    It’s been almost a year since the review has been undefensible. That it is still published after all the evidence of being a massive mistake compounds Cochrane’s initial failure in properly vetting their own work product. Meanwhile people are still being subject to harmful, and useless, medical advice, some of which may end up permanently disabled (until a cure is found anyway) or in such distress as a result of severe disability that they may commit suicide, as Cochrane’s and others’ recommendations directly lead to stigmatization from medical professionals and ostracization from community, friends and family.

    It is clear from the internal communications, even the initial peer review itself, that this review should never have been published and does not even come close to meeting standards of quality medical care, Cochrane’s or anyone else’s.

    At this point, whatever Cochrane choose to do, they has already done plenty of harm and elected to continue doing so even as they know about said harm and the complete lack of evidence for the review as it is published, all the way back to the initial review, spanning 15 years.

    There is already a clear and urgent need for reform in the process. Unfortunately the new planned assessment tool (RoB2) goes exactly in the wrong direction and will increase the already significant problem of knowingly doing harm. Given this, I can’t even see a need or reason for Cochrane to continue to exist at all without a complete reorganization and massive changes to applying their own rules on their own work.

  • AB 25 September 2019, 8:27 am

    If decision makers continue to deflect valid criticism as “harassment” then they are not fit to be in positions of responsibility.

    Patient lives are put in danger and that should be their primary consideration.