Trial By Error: Another Review Mentions LP Study and Prompts More Letters

By David Tuller, DrPH

In recent weeks, I have tried to bring attention to the troubling fact that a major systematic review of pediatric CFS/ME (as the authors called the illness) cited Bristol University’s Lightning Process study as evidence that the intervention was “effective.” Now another systematic review—this one of mental health outcomes of treatments for kids diagnosed with so-called “medically unexplained symptoms”–has also cited the discredited study, which was published in September, 2017, by Archives of Disease in Childhood.

This new review–“A systematic review of randomised controlled trials using psychological interventions for children and adolescents with medically unexplained symptoms: A focus on mental health outcomes”–was conducted at University College London and has been accepted for publication by the journal Clinical Child Psychology and Psychiatry. The university recently posted an “in-press” version before publication in the journal.

The review includes eighteen studies of a variety of conditions identified as belonging in the MUS category by those who believe in the reality of the MUS category. These include chronic fatigue syndrome, fibromyalgia, chronic pain, and headaches. Besides the Lightning Process, fourteen of the studies investigated some form of cognitive behavior therapy; three others examined biofeedback. Outcomes included anxiety, depression and other mental health indicators.

Overall, the reviewers professed to be unimpressed with the quality of the studies. Here’s what they concluded: “Psychological interventions may be effective in improving mental health outcomes within this population, however, evidence for the efficacy of these interventions is limited due to a high risk of bias within the majority of reviewed studies. Future research using rigorous methodological and non-CBT interventions is recommended.”

Hm. Good advice. Wonder if anyone will pay attention?

In any event, I wrote to two of the authors of the new review to alert them to the problems with the Lightning Process study. I followed up with another letter to Dr Fiona Godlee, editorial director of BMJ, urging her once again to take the necessary steps to shield children’s health from the impact of recommendations arising from this sub-par research.

Below I have first posted my letter to two authors of the systematic review. After that, my letter to Dr Godlee.

**********

Dear Professor Shafran and Dr Bennett—

I recently saw the “in-press” version of the new paper you have co-authored–“A systematic review of randomised controlled trials using psychological interventions for children and adolescents with medically unexplained symptoms: A focus on mental health.” Clinical Child Psychology and Psychiatry has accepted the review and University College London recently posted it online before publication.

I am pleased with your assessment that the available studies generally suffer from a “high risk of bias” and therefore are not especially useful in assessing whether the investigated treatments are actually effective, whatever the reported results. I fully agree.

However, I am writing not for that reason but because the review includes the Lightning Process study published by Archives of Disease in Childhood in September, 2017. The decision to include this study—”Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial”–is problematic. In this study, the investigators from the University of Bristol violated core methodological and ethical principles of scientific research, and it should not have been published in the first place.

By way of introduction, I am a senior fellow in public health and journalism at UC Berkeley’s Center for Global Public Health, which is part of the School of Public Health. I have been a journalist for decades and more recently received a doctoral degree in public health from Berkeley. For the last few years, I have examined–and critiqued–some of the research in the ME/CFS field, including the PACE trial.

I have chronicled my findings in more than 150 posts and articles on the site Virology Blog, hosted by Professor Vincent Racaniello, a microbiologist at Columbia University. I have also written about the issue for The New York Times and elsewhere. My examination of ME/CFS research has been mentioned or cited in Science, Nature, The Wall Street Journal, The Guardian, and Slate, among other publications.

As I documented on Virology Blog in December, 2017, the Lightning Process investigators recruited more than half the participants before trial registration, swapped outcome measures based on those early results, and then failed to disclose any of these details in the published paper. Since all major medical journals have a policy of not publishing studies in which participants were recruited before trial registration, Archives should not have accepted it for publication. In January, 2018, Professor Racaniello sent Archives an open letter alerting the journal to the problems; the letter was signed by more than 20 experts and posted on Virology Blog.

On June 19th, 2018—a year ago last week—Archives posted an “editor’s note” about the Lightning Process article. This note acknowledged the concerns, indicated that the investigators had provided “clarifications,” and suggested that the matter was under “editorial consideration.” For unexplained reasons, that period of “editorial consideration” has not yet ended.

Moreover, the editor’s note is not visible from the paper itself; I presume you did not notice it for that reason. It is unfortunate that the journal’s failure to properly discharge its editorial obligations has led you to cite a study that has no legitimate place in the medical literature. I urge you to review the matter and take the appropriate steps to address it as soon as possible.

(By way of transparency, I plan to post this letter on Virology Blog. I also plan to send a letter to Dr Fiona Godlee, BMJ editorial director, on which both of you will be cc-d.)

Best—David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

**********

Dear Fiona–

Several weeks ago I sent you a letter in which I urged you to prioritize children’s health by finally addressing the many serious concerns about the Lightning Process study. That study, from a Bristol University team led by Professor Esther Crawley, was published two years ago in Archives of Disease in Childhood, a BMJ journal, under the following title: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.”

The immediate impetus for my last letter was a review of pediatric CFS/ME published in April, which cited the Lightning Process trial as evidence that the intervention was an “effective” treatment for children with the illness. Since the reported results were generated after the investigators committed serious methodological and ethical violations of the scientific process, they cannot be taken as legitimate. So far, the authors of that review, which was published in Current Opinion in Pediatrics, have failed to take any action, despite having been alerted to the issues.

Now another review—this one of mental health outcomes among children diagnosed with so-called “medically unexplained symptoms”–has also cited the study without any mention of its disqualifying shortcomings. This new paper—called “A systematic review of randomised controlled trials using psychological interventions for children and adolescents with medically unexplained symptoms: A focus on mental health”—has been accepted by the journal Clinical Child Psychology and Psychiatry. University College London has posted an “in-press” version of the review, which includes seventeen other studies besides the Lightning Process trial. (In a positive development, the review authors noted that the studies they included generally had a “high risk of bias.”)

As I documented on Virology Blog in December, 2017, the Bristol University investigators recruited more than half the participants before trial registration, swapped outcome measures based on those early results, and then failed to disclose any of these details in the published paper. Since Archives should never have published the study in the first place, the paper has no legitimate place in the medical literature—except perhaps as a case study of how not to conduct a clinical trial.

The current situation is untenable. On June 19th, 2018—more than one year ago—Archives posted an “editor’s note” about the Lightning Process article. This note acknowledged the concerns, indicated that the investigators had provided “clarifications,” and suggested that the matter was under “editorial consideration.” This editor’s note was not visible from the paper and it is likely most readers have not noticed it, which perhaps explains why review authors have cited it. It is past time for Archives to conclude its lengthy process of “editorial consideration” and take the necessary corrective actions to safeguard the well-being of a highly vulnerable population—kids with a debilitating and stigmatizing illness.

Because of ongoing debates about proper treatments for people with ME/CFS as well as for those diagnosed with “medically unexplained symptoms,” I believe it is important to bring widespread attention to the deficiencies of the Lightning Process trial, especially now that it is being cited in systematic reviews. I am therefore cc-ing two of the co-authors of the new pediatric MUS review; the senior author of the previous review that cited the study; four physicians involved with the development of new guidelines for ME/CFS under the auspices of the National Institute for Health and Care Excellence, as well as two patient representatives on the NICE committee; Tom Whipple, science correspondent for The Times; Sue Paterson, director of legal services at Bristol; Professor Alan Montgomery, senior author of the Lightning Process study; Professor Vincent Racaniello, host of Virology Blog and a Columbia University microbiologist; and Professor Chris Ponting, vice chair of the CFS/ME Research Collaborative.

Thanks much for your attention to this matter.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

Comments on this entry are closed.

  • Lady Shambles 24 June 2019, 7:00 am

    Given their sensible observations about the quality of the available literature, and presuming that they hadn’t noticed the ‘note of the concern’ ( perhaps because it’s been so well hidden?), is there some hope Professor Shafran and Dr Bennett might wish to interrogate more dodgy ME or MUS related ‘research’ and read the whole archive of Tuller and Racaniello posts at this blog? (Links at right). It would be rather nice to think two authors, who seemingly have the luxury of open-mindedness, might pursue this further…. Maybe beneficence is about to have its day? It won’t be too soon.

  • Wendy Boutilier 24 June 2019, 7:26 am

    MUS is a popular diagnosis with the Psychiatric Collaborative in the UK & Europe. It’s difficult to treat because they pack everything they can think of under it.
    “There is no proven primary care treatment for patients with medically unexplained symptoms (MUS). We hypothesized that a long-term, multidimensional intervention by primary care providers would improve MUS patients’ mental health.”
    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1924714/

  • Couch Turnip 24 June 2019, 8:17 am

    This highlights again that the deliberation over the Lightning Process study has taken way too long. As I understand it, COPE guidelines suggest swift action in such cases so that other researchers don’t cite problematic papers. All I can think is that Archives of Disease in Childhood pays scant attention to those COPE guidelines, even though the BMJ was a founding member of COPE. If COPE can’t enforce its guidelines/principles, (even in relation to a paper in a BMJ journal), and if journals can’t effectively police themselves, then perhaps a new body is required to ensure that standards and ethics are upheld particularly in medical journalism.

  • Justin Reilly 24 June 2019, 1:47 pm

    Great work once again, David!

  • Alicia Butcher Ehrhardt 24 June 2019, 3:45 pm

    Makes my blood boil. I really can’t afford that!

    Follow the money: these things affect someone’s financing – and that someone has no interest in producing good science because of that.

    Not a witch hunt, obviously, but some serious questions about WHY the misinformation continues to be spread.

  • AB 25 June 2019, 7:05 am

    Thank you Dr Tuller for following up these systematic reviews which include bogus and questionable studies.

    We need both patients and doctors to have confidence that when a treatment is assessed or recommended that this is on the best possible evidence.

    Systemic reviews should not be including studies with major flaws. It would be best of course, that medical journals would not publish studies with these major flaws. they should be weeded out by the review process, which obviously is not happening.

    One error flows into the other. It means that systemic reviews have become unreliable and that should be a concern of every medical body.

    Patients should not have to be pointing this out. We should have confidence in systematic reviews and medical journals. What is happening behind the scenes that has resulted in this loss of integrity and therefore loss of confidence?

  • Erik Johnson 25 June 2019, 2:25 pm

    What kind of idiot would say “Medically Unexplained Syndrome”?

    A “syndrome” is the collated signs and symptoms that characterize illness entities for which the etiology is not known.

    No need to say something that is medically unexplained is medically unexplained.

  • Andrew Kewley 26 June 2019, 8:35 pm

    The O’Connell review is not that bad – it accurately reports that all of the studies had a high risk of bias (leading to low internal and external validity), low statistical power and other issues relating to reproducability, including ill-defined hypotheses, poor manualisation etc. It rates LP as having poor internal and external validity (it was given the lowest rating).

    Of course the review is let down by the unnecessary mention of Descartes and a lack of awareness that it is basically impossible to control for all of the biases in an unblinded trial. Controlling for participant time and attention is just scraping the surface as to the many uncontrolled biases.

  • Steve Hawkins 28 June 2019, 3:31 pm

    Dr. Tuller’s, deliberations are building the makings of a paper on how the medical publishing establishment attracts those seeking to make careers out of cheap and simple, largely paper-based ‘studies’ that few people will take the trouble to question or even read. The ‘studies’ then produce exponentially increasing supplies of ‘references’ on which to build further spurious but reputation-building ‘studies’ and ‘metastudies’, and so on, damaging nobody but carefully selected patient categories, chosen precisely for the likelihood that most medical professionals, and an all too spinnable media, will disregard–and even actively ridicule–their ‘beliefs’.

    A nice little earner for rather too many would-be ‘scientists’, these days, it seems. 🙁

    The mechanisms of propagation, and profit from, bad science should prove to be a rich vein for real scientific study to mine in themselves. Dr Tuller can be proud of his growing expertise and reputation in this field! (y)

    [Incidentally: I’ve been hearing media comments lately, where it is being said that government departments–and others subject to FOI requests–now routinely grade journalists by their ‘vexatiousness’. It seems that people who use the FOI Acts for their intended purposes, whatever the field, are being deliberately defamed with this new buzzword. More sinisterly, the fact that league tables of ‘vexatiousness’ are being compiled, points to a readiness to have the best journalists barred, black listed, or worse, as once democratic governments veer more and more strongly to the authoritarian and even totalitarian, as envied by the likes of Mr Trump.]