Trial By Error: And Cochrane Makes Another Move…

By David Tuller, DrPH

Yet another interesting development has taken place at Cochrane—and this new development again suggests that things there might be moving in the right direction. On Friday, the organization withdrew a protocol for what is called an “individual patient data” (or IPD) review of exercise therapy for the illness it has referred to as chronic fatigue syndrome.

Given recent related news involving Cochrane, this development led to some understandable confusion. So just to clarify: This is NOT the exercise review that has been the subject of controversy over whether it would or would not be temporarily withdrawn. That basic exercise review focused on the published data from the papers included. Just ten days ago, Cochrane rejected a revision of this basic exercise review and announced the upcoming development of a plan for a “full update.”

An IPD review is a more in-depth analysis than a basic review because it uses the actual trial data from the included studies rather than relying on the published numbers. The goal of an IPD review would be to provide what could be portrayed as even more authoritative findings to bolster those arising from a basic review. In any event, no IPD exercise review has been published. What has just been withdrawn is the protocol for such an IPD exercise review, which Cochrane published in 2014.

The lead author of the IPD exercise review protocol was also the lead author of the currently contested basic exercise review. More controversially, the co-authors included the three main PACE investigators as well. These circumstances triggered a public debate between Cochrane and critics about appropriate standards for assessing conflicts of interest. Cochrane’s policies allow those who contribute trial data for use in IPD reviews, as the PACE investigators were doing in this situation, to be included as co-authors.

Based on the IPD exercise review protocol, the authors produced the IPD exercise review, which was expected to be published last year. Perhaps because of the growing international controversy over PACE and other GET/CBT studies, Cochrane wisely decided to consult with a wider group of readers than usual and sent out this IPD exercise review to experts not affiliated with the GET/CBT ideological brigades. At least some of these outside reviews were scathing. By last summer, the apparent delay in publication of the IPD exercise review had already led people outside Cochrane to assume that it was unlikely to be published—an assumption confirmed by the new statement.

Here’s the notice that appeared on the site of the IPD exercise review protocol on Friday:

“This protocol has been withdrawn and it is no longer being progressed to a Cochrane review.
This protocol was published in 2014 and a new protocol is required for any future individual participant data (IPD) meta‐analysis on this topic.
The editorial group responsible for this previously published document have withdrawn it from publication.”

Significantly, the statement indicates that the withdrawal was made by the “editorial group” that prepared the document. That presumably means the Common Mental Disorders group, which has long owned this illness at Cochrane. If that’s the case, it means that those in charge at the Common Mental Disorders group now accept that Cochrane will not be publishing the actual IPD exercise review. That in turn would leave no reason for the IPD exercise review protocol to remain in the literature.

Certainly the IPD exercise review protocol withdrawal is very different from this fall’s messy almost-withdrawal involving the basic exercise review. To recap: In October, Reuters broke the story that Cochrane planned to temporarily withdraw the review, pending a revision to be submitted by the authors. Cochrane had based its decision on the basic exercise review’s unacceptable methodological and scientific failings, which a patient had authoritatively documented in a formal complaint. Yet Reuters largely portrayed Cochrane as having caved under belligerent pressure from an aggressive patient lobby.

After the Reuters article appeared, Cochrane backed away from its apparent decision to temporarily withdraw the review. While leaving the review in place, it announced that the authors were nonetheless pursuing a revision in response to the concerns raised. This was the revision rejected by Cochrane late last month. Given that editors considered the revisions necessary in the first place because of the inadequacies of the published basic exercise review, the fact that it remains in the Cochrane library raises continuing questions about its status as a credible and legitimate piece of scientific research.

In any event, figuring out exactly what is transpiring within Cochrane’s inner sanctum is challenging—especially since it appears difficult for the organization to withdraw a review without the consent of the authors. My hope is that these recent steps are all part of a slow but deliberate plan to finally transfer the illness from the Common Mental Disorders group to a more appropriate department. The withdrawal of the IPD exercise review protocol will help facilitate such a move, since this also eliminates any expectation that Cochrane will publish the actual IPD exercise review from the Common Mental Disorders group.

Along with these recent positive actions, I hope that everyone involved—including those who have worked hard producing the multiple iterations of these reviews—can amicably agree that anything else previously generated by the Common Mental Disorders group related to this illness should also be withdrawn. That would allow whatever new group at Cochrane takes on the challenge to start its investigations with a clean slate.