By David Tuller, DrPH
Same-Day Update:
I forwarded our response to Dr Brown’s letter (see below) to Dr Fiona Godlee, BMJ’s editorial director. I also cc-d others on my e-mail to Dr Godlee. Here’s what I wrote:
Dear Fiona–
Dr Brown, the editor-in-chief of Archives of Disease in Childhood, sent a message to Professor Racaniello and me last Thursday about the results of the in-depth investigation of the Lightning Process study. Professor Racaniello sent Dr Brown our response this morning. I am forwarding you a copy of that response.
As you know, the investigators of the Lightning Process study violated BMJ policy on prospective trial registration, selected their primary outcome after recruiting and collecting data from more than half their participants, and failed to mention these details in the published paper. So we applaud the journal’s acknowledgement of the study’s documented flaws. However, the decision to republish the paper with its original findings intact is unacceptable and potentially harmful to children, as we explain in our letter.
I am forwarding our response as well to some of those involved with the issue who have been cc-d on earlier letters about this matter: Sue Paterson, director of legal services at Bristol University; Professor Alan Montgomery, the senior author of the Lightning Process study; four physicians involved with the effort to develop new ME/CFS guidelines under the auspices of the National Institute for Health and Care Excellence; two patient representatives on the relevant NICE committee; Dr Terry Segal, senior author of a recent systematic review that highlighted the Lightning Process study; Tom Whipple, a Times science reporter who has written about ME/CFS; Professor Chris Ponting, vice chair of the CFS/ME Research Collaborative; Carol Monaghan MP, who has expressed alarm at the poor quality of much ME/CFS research; and Professor Racaniello.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
**********
In December, 2017, I wrote a long post about the methodological and ethical violations committed by a team of Bristol University investigators in their conduct and reporting of a pediatric trial of the Lightning Process. The study was published three months earlier by Archives of Disease in Childhood, a BMJ journal, under the following title: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.”
The issues I raised in the post involved retrospective registration, outcome-swapping, and the failure to disclose salient details in the published paper. On January 30, 2018, Professor Racaniello sent a letter of concern to Dr Nick Brown, the journal’s editor-in-chief, signed by more than 20 experts (including me).
Dr Brown promised to look into the matter, and he kept his word. On June 19, 2018, the journal posted an editor’s note acknowledging the validity of the issues raised and explaining that the situation was under “editorial consideration” in light of “clarifications” provided by the investigators. Unfortunately, this editor’s note was not visible from the paper itself, so was likely not seen by most readers.
I repeatedly highlighted the obscure placement of this editor’s note in blog posts and private messages. Last October, I finally received a short note from Dr Fiona Godlee, BMJ’s editorial director. Dr Godlee suggested that the organization’s “systems” did not make it easy for the IT department to address the problem–although she assured me she would see what she could do. Apparently she wasn’t able to do anything, since the editor’s note never moved.
Last Thursday, Professor Racaniello and I received an e-mail from Dr Brown. He alerted us to the news that a “corrected” version of the Lightning Process study had just been posted, along with a lengthy correction notice and a new editor’s note explaining the changes. He thanked us for bringing the issues to the journal’s attention. I have posted that message below, followed by our response, which Professor Racaniello sent this morning.
I consider it a major victory to have Archives of Disease in Childhood acknowledge that everything I documented is true. The correction notice itself is unusual in its length and in the sheer number of points it needs to cover. But allowing the investigators to re-affirm their original reported findings despite the study’s disqualifying methodological and ethical violations represents a perplexing abrogation of BMJ’s editorial responsibilities.
**********
Dear Professor Racaniello and Dr Tuller
Thank you both once again for making us aware of your concerns about this paper
Having acknowledged your comments as valid, we embarked on the (highly unusual) process of subjecting the paper to a full re-review, both methodological and senior statistical
During this process, it became clear that a number of facets of the manuscript needed to be addressed and clarified. To do this to both the reviewers’ and our own satisfaction has required a number of iterations, multiple discussions and a substantial amount of time.
Our conclusion, in short, was that the issues did warrant, as you suggested, the publication of a corrected version to take the place of the original. As a courtesy, we wanted to let you know that this has (literally) just gone ‘live’. The new (corrected) version is linked to both a full correction note (in which every change to the original is listed) and an editor’s note detailing the decisions leading to and chronology around the process we have undertaken
Thank you again for your comments and for your patience during the (unavoidably) lengthy process
Nick Brown
**********
Dear Dr Brown,
Thank you for your letter. We are pleased that Archives of Disease in Childhood has published a correction and editor€™s note confirming what Virology Blog documented in December, 2017: The investigators of the Lightning Process study violated BMJ policy on prospective trial registration, swapped outcome measures after gathering data from early trial participants, and failed to disclose these salient details in the published paper.
However, we are surprised you believe addressing these concerns is only a matter of clarification€ and that you have therefore allowed the study€™s reported findings to stand. This decision appears to call into question previous statements from BMJ leaders about the critical role of prospective trial registration in protecting the integrity of the published literature. Examples include the following:
*In 2013, Dr Godlee testified before the House of Common€™s Science and Technology Committee about BMJ€™s efforts to implement a zero-tolerance policy for trials in which recruitment of participants preceded registration. As far as we are aware, in the last two years we have not published any trial that has not been prospectively registered,€ testified Dr Godlee–a claim she could not truthfully make today.
*In a 2017 article, Dr Elizabeth Loder, The BMJ€™s head of research, discussed how the medical and research communities might render these trials a thing of the past.€ One solution, proposed Dr Loder, “might be to treat them as the research equivalent of medical ‘Never Events€™€¦These are ‘particularly shocking medical errors (such as wrong-site surgery) that should never occur.€™€
*Last year, Dr Trish Groves, BMJ Open€™s former editor-in-chief, tweeted the following: Prospective registration of #clinicaltrials isn€™t red tape: it€™s done for sound ethical and scientific reasons.€
When it comes to the Lightning Process study, you seem to regard the registration issue as a red tape€ mix-up that can be resolved through a simple clarification while the findings themselves remain intact. In this case, BMJ has apparently abandoned the sound ethical and scientific reasons€ behind its principled position on trial registration and adopted a more expedient or flexible approach. In the editor€™s note, you declare that non-adherence to the prospective registration requirement does not warrant retraction–but you offer no rationale for this relaxed interpretation of long-standing BMJ policy.
Your current stance is not compatible with the tough, no-nonsense views articulated by Dr Godlee, Dr Loder and Dr Groves. As Dr Loder noted in her 2017 article, Trustworthy clinical trials = prospectively registered trials that faithfully report their prespecified outcomes.€ In a 2018 BMJ Open article, Dr Loder and colleagues described prospective registration as “an important safeguard” against selective reporting of results. The investigators of the Lightning Process study not only failed to register their trial prospectively but also failed to faithfully report” their prespecified outcomes.
These investigators swapped primary and secondary outcomes after collecting data from early participants, who were initially recruited for a feasibility study and then included in the full trial sample. For the feasibility study, school attendance at six months was designated as the primary outcome, and self-reported physical function at six months as a secondary outcome. At the end of the full trial, school attendance at six months yielded null results, but this measure had been demoted to a secondary outcome. The new primary outcome, self-reported physical function at six months, had positive results.
In a clinical trial, it is problematic to identify the primary outcome after collecting data from more than half the participants. Whatever the intentions of the investigators, their undisclosed outcome-swapping allowed them to present positive rather than null results for what they reported as their primary outcome. It is irrelevant whether or not they analyzed outcomes data from the feasibility study before deciding to swap measures for the full trial. In open-label trials relying on self-reported measures, like the Lightning Process study, trends in outcomes are generally apparent to investigators long before the data are formally scrutinized.
In the editor€™s note, you discuss the journal€™s extensive and lengthy review of the multiple issues raised about the study. With regards to the outcome-swapping, you explain that the journal€™s review involved seeking assurance from the authors that the change in primary outcome was not influenced by (positive) findings in the feasibility phase.€
This statement is striking. Individuals subject to potential bias are often not considered impartial and authoritative arbiters of whether this potential bias has influenced their decision-making. Perhaps you and Dr Godlee believe it is reasonable and appropriate to take such assurance from the authors€ at face value while ignoring the self-evident impact of their methodological choices on the reported results. We do not.
Beyond the truism that humans tend to be blind to their own biases, it is worth noting that these investigators withheld important information about trial registration and outcome-swapping in their public version of events. In a significant departure from reality, the published paper portrayed the trial as if it were prospectively registered, with pre-designated primary and secondary outcomes. This sort of misrepresentation does not inspire confidence in any subsequent assurances of impartiality.
Archives of Disease in Childhood would not have published this paper in the first place except for two major failures: 1) The investigators€™ failure to provide editors with an accurate account of how they conducted the trial; and 2) The journal€™s failure to detect and address unacceptable deficiencies, despite its obligation to subject the paper to rigorous processes of peer review and editorial oversight.
Given the undisputed facts, BMJ€™s decision to publish a “corrected” version of this study that re-affirms the original findings is scientifically and ethically indefensible. It is also potentially harmful to the health and well-being of children.
Because of public interest, this exchange will be posted on Virology Blog and forwarded to others engaged with the issue.
Best,
Vincent Racaniello, Ph.D. | Higgins Professor
Department of Microbiology & Immunology
Columbia University, New York
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments are closed.