By David Tuller, DrPH
I have sent the following e-mail to Edward Sykes, the head of mental health and neuroscience at the Science Media Centre. The e-mail concerns the Lightning Process study published last year in Archives of Disease in Childhood. The SMC promoted the findings, which received widespread media coverage.
I have cc€™d Fiona Godlee, the editor in chief at The BMJ and editorial director at BMJ Company, given that the editorial team at Archives of Disease in Childhood has failed to respond to the concerns in a timely manner. This failure creates the impression that BMJ harbors an indifferent or lax attitude toward violations of key scientific principles, including principles promulgated by the company itself, in pediatric research–or at least pediatric research involving this particular illness.
**********
Dear Edward,
Last September, the Science Media Centre promoted the following study published by a BMJ journal, Archives of Disease in Childhood: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.” The SMC held a briefing about the paper, issued a round-up of expert opinions and published an in-depth behind the headlines statistical analysis.
I wanted to make sure you and the SMC were aware that this study violated BMJ€™s own strict policies about not publishing trials in which participants were recruited before registration. I have documented the case fully on Virology Blog, and in January Archives of Disease in Medicine received a letter from 21 experts about the issues. The journal has so far failed to either refute the concerns raised or take corrective action. Below are two posts about the issue:
Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood
The Lightning Process paper noted that recruitment began in September, 2010. Yet the trial itself was registered in July, 2012. According to the documentation, 56 out of the 100 participants were recruited as part of a feasibility study that began almost two years before trial registration. Moreover, primary and secondary outcome measures were swapped mid-way through based on the data already provided by these feasibility trial participants–a surefire way to bias the reported results.
Not surprisingly, the investigators were able to report positive results for the revised primary outcome measure of self-reported physical function. The original primary outcome measure in the protocol, which became a secondary measure in the Archives paper, was school attendance at six months. This outcome measure achieved null results. The fact that 56 out of 100 children in the trial were recruited before trial registration was not disclosed in the Archives of Disease in Childhood paper. Neither was the fact that primary and secondary outcome measures were swapped based on data provided by these early participants.
Media coverage arising from SMC€™s briefing and other efforts understandably focused on the positive reported results for the revised primary outcome and ignored the null results from the original primary outcome. Unfortunately, the experts consulted by SMC failed to notice or mention that the outcome measures were swapped halfway through based on the feasibility study data. Do you agree that these methodological choices biased the results and that the decision to omit the information from the published record breaches principles of transparency in science? Why were these issues missed by those reviewing the study for SMC?
Given these anomalies, does the SMC intend to contact journalists who attended the briefing or received online materials about the Lightning Process study so as to update them on the matter? Do you feel you have a responsibility to follow up when studies you have promoted turn out to violate basic scientific principles, such as the need to disclose that outcome measures were swapped based on data provided by more than half the participants in a trial?
I have cc€™d Fiona Godlee on this e-mail. Perhaps she can explain why a journal under her purview has not yet acknowledged the obvious: The Lightning Process study violated BMJ policy on prospective trial registration and should not have been accepted for publication in the first place. If my concerns were a result of misinterpretation or misunderstanding, surely Archives of Disease in Childhood would not need more than four months to find evidence to refute them.
I am raising the issue with you now because the Archives of Disease in Childhood paper is at risk of being given credence in deliberations on public policy involving children, such as the current NICE effort to develop a new guidance for ME/CFS or the ongoing roll-out of the problematic Improved Access to Psychological Therapies program. This flawed study has no place being cited in support of any public health initiatives, clinical guidelines, or recommendations on medical care. But because of apparent editorial deficiencies at BMJ, the paper somehow passed peer review, was accepted for publication, and remains in the literature.
Given the SMC€™s role in disseminating the findings of the Lightning Process study, perhaps you yourselves or those who provided expert opinions for the round-up should invest a few minutes to review the relevant documents. Those documents include the paper itself, the trial registration, and the application for research ethics committee approval to both extend the feasibility trial into the full trial and simultaneously swap primary and secondary outcomes.
Then the SMC could resolve the matter once and for all by confirming one of two things: either all participants in the Lightning Process study were recruited after trial registration, or more than half were recruited before trial registration and outcome measures were swapped based on their data. If you determine the latter to be the case, you can also confirm that publication of such a trial violated not only BMJ€™s own policy but the longstanding policy purportedly adhered to by members of the Committee on Publication Ethics.
Thanks for your attention to this matter.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments are closed.