Trial By Error: A Letter to Health Officials About BMJ’s Lax Editorial Standards

By David Tuller, DrPH

I am not sure what is going on at BMJ and why editors there seem incapable of acknowledging their flawed decision-making when it comes to two papers that should never have been accepted for publication. One violated BMJ’s policy that all trials must be properly prospectively registered, with no participants recruited beforehand. The other violated official U.K. guidelines by exempting itself from ethical review on false grounds.

Both of these studies involved children and could be cited or raised during policy-making deliberations in the domains of public health and/or medical care. In not acting promptly, BMJ is certainly creating the impression that it places reputational concerns above its interest in ensuring children’s wellbeing. These misplaced priorities suggest that something is seriously amiss with BMJ’s ethical barometer. The failure of BMJ leadership to step in and take corrective action suggests that this is a company-wide problem rather than one afflicting just two journals.

In any event, I am alerting people potentially involved in developing or assessing public policy that these two papers breach scientific and/or ethical principles and should not be taken into account when considering any issues related to children’s health. My latest letter, posted below, is to the Rt Hon Jeremy Hunt MP, who is the U.K.’s Secretary of State for Health and Social Care, and Professor Dame Sally Davies, who is the Chief Medical Officer for England. As with previous letters, I am cc’ing Fiona Godlee, the editor in chief of The BMJ and editorial director of BMJ Company. (This letter is similar to the one I sent yesterday to Carol Monaghan, the Scottish National Party MP and staunch PACE critic.)

The letter’s subject line read: “BMJ’s casual attitude toward scientific/ethical violations in pediatric ME/CFS research”

**********

Dear Rt Hon Jeremy Hunt MP and Professor Dame Sally Davies: 

I am a senior fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. For the last three years, I have been investigating the research underlying treatments for the illness variously known as chronic fatigue syndrome, myalgic encephalomyelitis, CFS/ME, and ME/CFS. Through my investigative series (“Trial By Error”) on a well-regarded science site (Virology Blog), I have helped to highlight the many flaws of the PACE trial–a topic addressed this year in the House of Commons.

When it comes to poor research in the ME/CFS domain, the public debate to date has focused largely on PACE. Yet it would be unfair to criticize The Lancet alone for promoting inadequate research in this field. BMJ journals have done the same. Because of your key roles in the domains of public health and medical care, I felt it was appropriate to bring your attention to two such problematic studies published in BMJ journals–Archives of Disease in Childhood and BMJ Open.

Both studies involve children with the illness, and both violate core scientific and/or ethical principles. Yet because they are in the literature, they could be in a position to impact public health policies and the medical treatments available to kids, if they haven’t already. Although the editors of the two journals have been informed of the issues, they have so far failed to take the appropriate and necessary corrective steps. This failure is arguably placing children at risk of harm.

The Archives of Disease in Childhood paper, called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial,” was published last year. The BMJ Open paper, called “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics,” was published in 2011.

In the Archives of Disease in Childhood study of the Lightning Process, more than half the participants were recruited as part of a feasibility study starting almost two years before trial registration. Even as these feasibility study participants were folded into the full trial, primary and secondary outcome measures were swapped based on their results–an obvious recipe for biasing the findings. The paper in Archives of Disease in Childhood reported positive results but did not disclose these relevant details—a disturbing omission.

To protect against bias, BMJ and other leading journals all have a longstanding policy against publishing trials in which participants were recruited before registration. The Lightning Process study clearly breached this policy. And yet Archives of Disease in Medicine continues to provide practitioners of the Lightning Process with bragging rights that their pseudo-scientific program is effective in treating children with this illness. The news coverage around the reported results has likely driven more desperate parents to spend their hard-earned money on the Lightning Process.

In the BMJ Open study, the investigators sought to test the hypothesis that children with chronic fatigue syndrome could be identified through school absence records. But they exempted the study from ethical review on the false grounds that it was “service evaluation” and not “research.” The study relied on primary data collected directly from identified participants, not on anonymous secondary data, and included a hypothesis and generalizable conclusions. According to U.K. guidelines, a study with these features cannot be classified as “service evaluation” but qualifies as “research” and requires ethical review.

A peer reviewer for BMJ Open raised tough questions about the study’s claim that it was not “research” but “service evaluation” exempt from ethical review. Although the investigators failed to provide an adequate response for the lack of ethical review, BMJ Open overlooked the issue and accepted the paper anyway. Ironically, the journal published it under the heading of “research” but has since defended the decision to categorize it as “service evaluation.”

You can read more details about the documented problems with these studies, and letters of concern sent to the journal editors by more than a dozen scientists, clinicians and other experts, in the following Virology Blog posts:

Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood
Trial By Error: No Ethical Review of Crawley School Absence Study
Trial By Error: A Letter to BMJ Open

I have cc’d Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ Company, on this e-mail. Perhaps she can explain why journals under her purview have not yet acknowledged the obvious: The Archives study violated BMJ policy on prospective trial registration, and the BMJ Open study was not “service evaluation.” Neither should have been accepted for publication in the first place.

Does Dr Godlee believe these incidents suggest a need for greater scrutiny of BMJ’s editorial and peer-review processes, at least when it comes to this domain of inquiry? Does she have suggestions for what recourse should be available when journals fail in their responsibility to monitor themselves, with potentially negative public health and medical consequences?

I am raising these issues with you now because this BMJ research is at risk of being given credence in deliberations on public policy involving children, such as the current NICE effort to develop a new guidance for ME/CFS or the ongoing roll-out of the problematic Improved Access to Psychological Therapies program. Like PACE, these flawed studies have no place being cited in support of any public health initiatives, clinical guidelines, or recommendations on medical care. But because of apparent editorial deficiencies at the journals in question, these papers somehow passed peer review, were accepted for publication, and remain in the literature.

Thank you for your attention to this important matter. I would be happy to answer any questions you might have, whether about these two studies, the PACE trial, or other papers from this subpar body of research.

Best–David Tuller

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

{ 0 comments… add one }

Leave a Comment