By David Tuller, DrPH

This week I raised concerns about a second systematic review that cited the dung heap known as the Lightning Process study, published by Archives of Disease in Childhood two years ago. This new review did not mention the paper’s egregious deficiencies. After the appearance in April of an earlier systematic review that highlighted the study, I sent letters of concern both to Dr Fiona Godlee, editorial director of BMJ, and Dr Terry Segal of University College London, that review’s senior author.

To date, neither has taken any public steps to address the issues. Dr Godlee and her editorial team have apparently fallen asleep at the wheel when it comes to this domain of inquiry, for reasons that remain opaque. Dr Segal, so far, has given no sign that she is concerned about having cited a study in which the investigators violated core ethical and methodological principles by recruiting more than half the participants prior to trial registration, swapping outcomes based on the early results, and then deciding not to disclose these details in the published paper.

The first systematic review was of pediatric “CFS/ME,” as the authors called the illness. The new systematic review is of pediatric treatments for so-called “medically unexplained symptoms.” (I write “so-called” because the MUS category has often been hyped as a diagnostic entity by investigators who tend to believe that those identified as suffering from this condition are in urgent need of psychological treatment.)

For the record, I am pleased overall with this second review. It notes very pointedly that most of the trials in question—including those touting cognitive behavior therapy as an effective MUS treatment—cannot produce reliable evidence. Rather, they have been designed in ways that generate a “high degree of bias.” The review authors didn’t necessarily address all the concerns I might have raised about this body of research; nonetheless, they appear to have approached their task with a level of honesty and integrity that has been largely absent in this domain of scientific inquiry.

Their assessment could prove useful going forward in efforts to counter the metastasizing National Health Service program called Improving Access to Psychological Therapies. IAPT is built on bogus claims that these sorts of studies are in fact robust examples of quality research. The same fake argument, of course, has been used for years by the CBT/GET ideological brigades with regards to PACE and the promotion of potentially harmful treatments as effective means of achieving “improvement” and even “recovery” from ME, CFS, ME/CFS, CFS/ME, or whatever one chooses to call the illness.

In response to this new review, I wrote to two of the co-authors, both from University College London. I made sure to express appreciation for their clear-eyed view of the poor quality of the research as well as alerting them to the disturbing details about the Lightning Process study. I also wrote again to Dr Godlee, of course, to remind her of her editorial obligations to safeguard the legitimacy and accuracy of the medical literature.

In the case of the first systematic review, Dr Segal, the senior author, at first ignored both my public letter to her and a private one in which I invited her to discuss the matter with me confidentially. I only heard back from her after I cc’d her on a subsequent letter to the editor of Current Opinion in Pediatrics, the journal that published the problematic paper. That e-mail apparently caught Dr Segal’s attention, at least enough for her to decide to let me know she and her colleagues would “consider” my concerns.

Since then, Dr Segal has continued to be MIA (“missing in action”). I have no idea whether she and her co-authors have actually considered my concerns or completely ignored and dismissed them. In any event, her continuing silence on this matter is unacceptable. As I have repeatedly pointed out, it takes only a matter of minutes to check the relevant trial documentation and see how the Lightning Process study investigators, led by Professor Esther Crawley of Bristol University, engaged in questionable strategies in the conduct and reporting of their trial.

In contrast, after I sent my letter about the second systematic review to two of the co-authors on Monday, one of them–Professor Roz Shafran–responded with a short note thanking me for touching base.

**********

Here’s what I wrote back:

Thanks for the quick response. To reiterate, I was very pleased to read your overall assessment of the studies. Given the bias built into the various study designs used, it would seem imprudent or unwise to base public health or medical policy on such a compromised body of research.

As far as the inclusion of the Lightning Process study, can you let me know how you plan to address the issue? This seriously deficient piece of work should not be cited as if it were a legitimate piece of research. I believe the citation and the references to the study in the review should be removed.

As a second-best alternative, it should at least be made clear to readers that serious violations have been documented as well as acknowledged by Archives in the editor’s note, although the journal has so far failed to resolve the matter. The reasons for this failure remain opaque, but Archives and BMJ have managed to create the distinct impression that they are more concerned with avoiding or minimizing reputational damage than protecting children from the possibly harmful effects of bad research.

Thanks again. As I mentioned, it is a shame that Archives and BMJ have put you in this fraught position. It is really inexcusable to post an editor’s note in a place where it is not designed to be seen by readers of the paper in question. I assume you must not have seen the editor’s note. Is that assumption correct?

I hope I can look forward to seeing a few changes in the review based on this information.

Best–David

**********

I heard back once more the same day from Professor Shafran, who promised to discuss the issue with her co-authors. Again, I responded:

Thanks. I appreciate that. Those pushing these treatments continue to insist that the various studies, like the PACE trial, are robust, when this is obviously not the case. I have had a very hard time understanding the behavior of BMJ, The Lancet, and all of those involved in producing and publishing these deficient studies.

The Lightning Process study in particular seems to me to a likely case of research misconduct. I do not see how it is possible for researchers to forget to mention in their paper that they recruited more than half their sample before trial registration and then swapped their outcome measures on the basis of the early results. Failing to disclose salient and relevant information can certainly be construed as a form of research misconduct, according to any normal definition.

The Committee on Publication Ethics discussed a case that sounds very similar to the LP study. The dates mentioned are slightly off but they could have been changed to try to anonymize the case. Besides that, the documented facts seem to converge with the details outline on the COPE Forum site:

https://publicationethics.org/files/u661/COPE%20Forum%20Agenda%20and%20materials%2013_May_19_FINAL2.pdf

If this case is indeed about the Lightning Process study, it would appear that it is to be retracted soon, according to the COPE forum account. However, I have not been able to get confirmation that this COPE case is in fact about the LP study, so that is still speculation…

**********

I look forward to finding out how Professor Shafran and her colleagues ultimately respond to the concerns. Because their systematic review suggests that they are approaching these studies with some degree of skepticism, I am somewhat optimistic that they will choose to do the right thing. We shall see.