By David Tuller, DrPH
Last week I posted three comments sent to Dr Fiona Godlee, editorial director of BMJ, in support of retraction of the biased and discredited Lightning Process. All three–Professor Ola Saugstad of University of Oslo, Professor Vincent Racaniello of Columbia University, and Professor Elisa Oltra of University Catolica de Valencia San Vicente Martir–were among the 55 experts who signed Virology Blog’s recent open letter to Dr Godlee. That letter expressed concern and dismay that Archives in Disease in Childhood, the BMJ journal that published the study two years ago, had acknowledged the methodological violations documented on Virology Blog but nonetheless republished the original, seriously compromised findings. When I sent the letter to Dr Godlee, I cc’d many of the signers.
Since Professor Oltra sent her note, additional calls for retraction of the paper have come from: Dr Alan Gurwitt, a retired Yale psychiatrist; Dr Marlon Maus* [corrected shortly after posting from Maos–sorry, Marlon!], a colleague from Berkeley’s School of Public Health; and Professor Jonathan Edwards of University College London. In his letter to Dr Godlee, Professor Edwards mentioned Cochrane’s revised risk-of-bias tool, just published by BMJ, and noted a troubling detail–the revision’s lead author, Professor Jonathan Sterne of Bristol University, was also a co-author of the Lightning Process study.
Dear Dr. Godlee:
This message is to add to the others you have already received regarding the correction and republication of the Lightning Process study, originally published by Archives of Disease in Childhood in September 2017. That article was entitled “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical for paediatric chronic fatigue syndrome: randomized controlled trial.”
I fully agree with the points outlined in Dr Tuller’s letter, which many other clinicians and scientists also signed. This study must be retracted in order to protect the interests of children. The decision to republish the study with the original findings was grossly inappropriate.
I am a retired psychiatrist and one of several co-authors of “Myalgic Encephalomyelitis/Chronic Fatigue Syndrome Diagnosis and Management in Young People: A Primer,” published in Frontiers in Pediatrics (June 2017). In the process of reviewing the literature for that primer, I reviewed what was available on the Lightning Process. I came to the conclusion that this treatment was utter nonsense. Its proposed use for ME/CFS emanates from the misguided belief in the UK that the illness has psychological rather than pathophysiological underpinnings.
Alan Gurwitt, MD
Psychiatrist in Private Practice (retired)
Associate Clinical Professor (retired)
Yale Child Study Center
Yale School of Medicine
New Haven, Connecticut, USA
Dear Dr. Godlee,
As an ophthalmologist and Professor of public health I wish to add my voice to that of others regarding the unsound study “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical for paediatric chronic fatigue syndrome: randomized controlled trial.” After reviewing the background for the study and its conclusions I concur completely with the interpretation by Dr. Tuller and others. The medical profession is rife with examples of the victimization of patients that have been diagnosed with psychiatric diseases when in fact later studies and research have proven them physical diseases. I believe that this may be such case and we must utilize the precautionary principle and I suggest you consider retracting the study until further evidence is obtained.
Marlon Maus, MD, DrPH, FACS
Professor, School of Public Health, UC Berkeley
I note the recent publication in BMJ of an account of the finalised Risk of Bias 2 tool (RoB2) used in the context of the GRADE system used by Cochrane and NICE for assessing quality of clinical trial evidence. The document focuses on reducing the stringency of RoB assessment principally in unblinded trials.
My immediate reaction to this was that it is a retrograde step, since, increasingly, unblinded trials appear to be given more credence than they deserve. My immediate reaction was then followed by a more serious concern when it was brought to my notice that the corresponding author, Jonathan Sterne from Bristol, is also an author on the paper reporting on the SMILE trial of the Lightning Process that has been the topic of previous correspondence and is now again.
Amongst the proposals for increasing leniency when considering bias in trials, the RoB2 document re-assesses the impact of changing outcome measures after trial initiation but before data analysis and also makes reference to problems with bias due to beliefs held by the patient or treatment delivery team when outcome measures are subjective. It will have not have escaped your notice that amongst the many flaws in the SMILE trial are the issues of outcome switching midstream and the problem of severe risk of expectation bias relating to subjective outcome measures.
So I find myself writing to you again to support the request from David Tuller, and now fifty or more other colleagues, to take the problems with the SMILE trial seriously. I think I had previously suggested that a retraction or clear indication of the flaws of the trial was needed. Looking at the revised manuscript I cannot see that anyone would consider the problem adequately addressed since the inappropriate conclusion remains the same.
Things are beginning to look surreal. It is almost as if the SMILE trial was an exercise in demonstrating that scrutiny by GRADE and RoB systems can be circumvented with impunity.
The GRADE system starts from the premise that a randomised trial provides high quality evidence unless it suffers from one or more defects including bias. The weakness of this can be illustrated by proposing a trial in which patients are randomised to being taught that they will only get better if they think and say they feel better, whether they do or not, or to being told to say how they really feel, and then using how the patient says they feel as outcome. One might think such a trial would be absurd, and never proposed. Yet this appears to be more or less what the SMILE trial is. It is pretty much how patients describe their experience. On the other hand, as Dr Oltra points out, we are not told what really went on, and it looks as if according to RoB2 the risk of bias is not scored high if we do not know what went on!
RoB2 does mention the possibility that bias might arise from patient or therapist beliefs but gives as examples the rather extreme cases of a physiotherapist assessing the benefit of her own treatment or a patient having a belief in homeopathy. There appears to be no recognition of the fact that expectation bias due to beliefs about the value of any sort of treatment are ubiquitous in trials (and in any scientific experiment, including work in the lab) and are the basic reason why we blind ourselves to test and control. In ME/CFS we have seen dramatic responses due to expectation bias with conventional drugs with no therapeutic effect, including rituximab and anti-virals. The RoB2 analysis appears either very naïve or disingenuous.
As we are all aware, the SMILE trial had a highly unsatisfactory structure, being initiated as a ‘feasibility study’, with a large number of patients recruited, and then being registered as a ‘formal study’ after switching of outcomes in the knowledge of the progress of the first 50 or so patients. This might fall under Dr Sterne’s allowed situation of before data analysis but it is a classic cherry picking scenario.
‘Feasibility trials’ and ‘pragmatic trials’ appear to be increasingly popular. As far as I can see these techniques are designed to make trials look as if they have a valid structure for gathering reliable evidence when they almost certainly do not. I have a strong impression that ‘methodological experts’ associated with clinical trials units and other related departments may have a conflict of interest in terms of co-authorship on publications of poor quality trials.
I had for some time thought that the problems of poor quality trials in ME/CFS were at the periphery of academic medicine. Now I get the strong feeling that they may be typical of a general process of degrading the quality of clinical science in the UK. We appear to be moving towards acceptance of methodology that for decades we have known yields meaningless results. I get the impression that bodies like Cochrane and the BMJ are sleepwalking into a situation where they rubber stamp commercial ventures of no merit. In this wider context I can only emphasise the view of all those copied above, that the SMILE trial paper should be retracted and that a fully independent investigation should be completed as soon as possible.