Trial By Error: My Letter to LP Study’s Senior Author

By David Tuller, DrPH

Alan Montgomery is a professor of medical statistics and clinical trials at the University of Nottingham’s School of Medicine. He is also the senior author of the Lightning Process study published in 2017 in Archives of Disease in Childhood, a BMJ journal. Professor Montgomery formerly worked at University of Bristol, along with Professor Esther Crawley and the other co-authors of the Lightning Process study.

As I documented more than a year ago, the Lightning Process study is a methodological and ethical mess that violates core scientific principles. Despite the potential public policy impact of the reported findings, Archives of Disease in Childhood has so far failed to fully resolve the issues. Given the journal’s inability or unwillingness to take the prompt action necessary to protect this vulnerable patient group from bad research, I decided to reach out to Professor Montgomery directly. Considering his credentials, it is perplexing that he agreed to be senior author of the paper, since it improperly identified the study as prospective and failed to mention major outcome-swapping half-way through.

I e-mailed the following letter to Professor Montgomery on Sunday evening (California time), with the subject heading “Retrospective Registration and Outcome-Swapping in Lightning Process Trial.” I cc’d Dr Nick Brown, editor-in-chief of Archives of Disease in Childhood, and Dr Fiona Godlee, editorial director of BMJ. In addition, I cc’d three members of Parliament who have expressed dismay about the poor quality of research in this field and are seeking a parliamentary debate on the issue: Carol Monaghan MP, Darren Jones MP and Nicky Morgan MP. Finally, I cc’d Adam Lowe, Sally Burch and Dr Charles Shepherd, three members of the committee that is developing the ME/CFS guidance for the National Institute for Health and Care Excellence.

**********

Dear Professor Montgomery:

I am a journalist and public health researcher at the University of California, Berkeley. For the last few years, I have been investigating problematic studies in the ME/CFS field. Much of my work has appeared on Virology Blog, a site hosted by a Columbia University microbiologist, Professor Vincent Racaniello. This open letter will be posted there as well.

In September, 2017, Archives of Disease in Childhood published a study called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.” You are the senior author of that paper; your eight co-authors are all affiliated with the University of Bristol.

The Lightning Process study was an open-label trial relying on self-reported outcomes—a design likely to generate results fraught with bias. Beyond that, as I have documented repeatedly during the past year on Virology Blog and in letters to BMJ editors, the trial featured serious methodological and ethical violations.

In this case, a pediatric Lightning Process feasibility trial was extended into the full study even as primary and secondary outcome measures were swapped based on the early results. These major changes were not mentioned in the Archives paper, although they allowed the findings to be presented in a more favorable light than would otherwise have been possible. The changes are only apparent from a review of other documents, such as the trial registration, the feasibility trial protocol and the full-trial protocol.

My specific concerns include the following:

*The study was registered as a prospective trial. But 56 out of 100 participants were recruited as part of the feasibility trial, starting almost two years before the registration date. For unexplained reasons, the Archives paper did not mention this fact. Like all major publishers of medical journals, BMJ has a longstanding policy of not considering any clinical trials in which participants were recruited before registration. In considering and then publishing the Lightning Process paper, Archives of Disease in Childhood violated its own stringent guidelines.

*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After the collection of data for the feasibility trial, these outcome measures were swapped for the full trial—school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. As with the trial registration issue, the Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had been recruited as participants in the feasibility trial.

*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant attention in the Archives paper. Thus, the outcome-swapping biased the findings and improved the apparent success of the intervention. Had the measures not been switched, the paper would have had to report null results for the primary outcome.

*In both the feasibility trial protocol and the full-trial protocol, the researchers promised to seek official school attendance records as a way of vetting the accuracy of self-reported school attendance. The Archives paper did not provide official school attendance records, even though it highlighted benefits at 12 months in self-reported school attendance. No explanation for the absence of these official school attendance records was provided, leaving two possibilities: Either they were not obtained, or they were obtained and failed to confirm the benefits in self-reported school attendance at 12 months.

*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results reported for school attendance at six months, the original primary outcome, were largely ignored. The undisclosed outcome-swapping thus ensured that the reported findings received more positive media coverage than they otherwise would have.

Transparency is essential in scientific research. Yet the Archives paper failed to include salient information about trial recruitment and registration, the decision to swap outcomes midway through, and the disposition of the official school attendance records—all factors that impact how the data should be interpreted. Last June, a published “editor’s note” suggested that the concerns raised about the study were valid and that the paper’s fate was under ongoing consideration. Seven months later, no further actions have been announced, even though a thorough assessment of the matter requires only a brief review of the relevant documentation.

It is worth reiterating that the Lightning Process is a pseudo-scientific intervention combining life-coaching, neuro-linguistic programming and osteopathy. Participants are taught that they can overcome illness by controlling and changing their thought patterns. Lightning Process practitioners have declared–without citing legitimate evidence–that they can successfully treat multiple sclerosis, eating disorders and other serious conditions. Government regulators have admonished some practitioners for making misleading claims.

Phil Parker, the creator of the Lightning Process, previously taught a course on how to heal people with “divination medicine cards and tarot.” The archived website for this course explains that “divination is useful in creating a strong connection with healing/spirit guides.” The course also featured lessons in “the use of auras for diagnosis of a client’s problems” and in how to “prepare a space appropriately so that any energy polluting the room will not interfere with the work you are doing.”

Here is Phil Parker’s biography from the same archived website:

“Phil Parker is already known to many as an inspirational teacher, therapist, healer and author. His personal healing journey began…whilst working with his patients as an osteopath. He discovered that their bodies would suddenly tell him important bits of information about them and their past, which to his surprise turned out to be factually correct! He further developed this ability to step into other people’s bodies over the years to assist them in their healing with amazing results. After working as a healer for 20 years, Phil Parker has developed a powerful and magical program to help you unlock your natural healing abilities. If you feel drawn to these courses then you are probably ready to join.”

Professor Montgomery, this situation has dragged on for too long. The facts here are not in dispute. I urge you, as senior author of the Lightning Process study, to take responsibility for the paper’s missteps. Did any of your co-authors recognize or point out that the paper presented a misleading and inaccurate account of the research? If not, why not?

Given that the trial was improperly registered as prospective, Archives of Disease in Childhood should not have published the paper in the first place. The appropriate action, therefore, is to retract it along with a full explanation of how these methodological and ethical anomalies occurred. You and your co-authors should also apologize to any families that have spent hundreds of pounds for the Lightning Process based on your reported positive findings.

I have cc’d Dr Nick Brown, editor-in-chief of Archives of Disease in Childhood, and Dr Fiona Godlee, editorial director of BMJ. In addition, I have cc’d three members of Parliament who have expressed dismay about the poor quality of research in this field and are seeking a parliamentary debate on the issue: Carol Monaghan MP, Darren Jones MP and Nicky Morgan MP. Finally, because the National Institute on Health and Care Excellence might consider the Lightning Process during its ongoing efforts to develop a guidance for ME/CFS, I have cc’d three members of the committee tasked with the assignment.

Thank you for your prompt attention to this matter.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

Comments on this entry are closed.

  • Tanya Selth 7 January 2019, 7:07 am

    As someone with severe ME/CFS I’d like to say thank you David Tuller for trying to make sure our science is good science!! Hopefully the day will come someday soon where so many biased ME/CFS studies are being published in journals.

    Yours Sincerely
    Tanya Selth

  • Susanna Degaardt 7 January 2019, 9:15 am

    Thank you for shedding light on this scientific scandal!

  • AB 7 January 2019, 11:51 am

    It’s a great letter. Thank you Dr Tuller.

    It remains a mystery why the University of Bristol and Professor Montgomery would get involved with a “magical healer” and then conduct a research study that was so flawed and badly prepared.

    Especially when there are so many accounts of both adult and child ME patients being harmed by similar programs which are run for profit outside the NHS.

    Psuedo-scientific theories should be subject to a high level of scrutiny otherwise we run the risk of bringing medicine into disrepute. I think what has happened here seriously questions the judgement of the Bristol approach to treating children with ME.

    Because the patients were children should have increased the care Prof Montgomery and colleagues should have taken to ensure that scientific methods were adhered to.

    I know from meeting adults who have been on similar courses of this “treatment” that they are encouraged to think of themselves as being well and to downplay their physical symptoms. We need accurate measurements of progress and deterioration to determine how much is real and how much is covered over to please the therapists, parents and others.

    With children who trust the therapist and their parents to make the right judgement for them we need to think hard about putting them in a situation of making their health worse and with no avenue of complaint left.

    I do hope Professor Montgomery reflects on this.

  • Maria from Sweden 7 January 2019, 2:11 pm

    Thank you for your work/Maria

  • Hannah 7 January 2019, 3:39 pm

    Other than sounding like an infomercial for a cult, logic would dictate that this self-healing ability that he (Phil Parker) teaches (understandably for a price) would work for all illnesses – so it has to make one wonder how fast the scientific community who is validating the Lightening Process would withdraw their support if this treatment was being espoused to the cancer community, for example. I would imagine withdrawal for support would be pretty swift.
    Logic would also dictate that if one could be taught to simply self-heal there would be absolutely no need for the practice of medicine and related professions.
    My question to the scientific community who is validating the research published on the effectiveness of the lightning process – Can you self-heal?

    Thank you David for your tireless and ceaseless efforts in doing what the majority of ME sufferers are too ill to do and being our voice.

  • Mary 7 January 2019, 10:58 pm

    Why are studies without any objective measures of physical functional capacity being funded and this glaring anomaly not being called out by the universities and by the journals publishing the “results”.
    Outcome swapping and the inclusion of pre trial results is what you teach school kids not to do. Fundamental mistakes and that high school kids would be called into account for.

  • Joanna Colin 8 January 2019, 12:05 am

    Thank you so much for not letting these fibbers get away with sneaky lies. The damage they do should be punishable.

  • Craig Morris 9 January 2019, 7:39 am

    Thank you so much for investigating in a way that is beyond so many people with ME and their families. It’s great to see you take the time to look beyond the headlines and hold bad science to account. People with ME deserve a lot better and highlighting how bad things are with ME might just help the UK government decide to fund some research to find out what’s wrong with us and what can be done about it.

  • Anne Highfield 12 January 2019, 7:10 pm

    Thank you, David.
    I want to hug you for doing this.
    Great job.
    Thanks