Trial By Error: A Letter to BMJ Open

By David Tuller, DrPH

Three weeks ago, Professor Racaniello e-mailed a letter of concern to Archives of Disease in Childhood about its recent study of the Lightning Process as a treatment for ME/CFS in kids. The journal’s editor, Dr. Nick Brown, answered within an hour, assuring Professor Racaniello that he took the matter seriously and that the journal would review the issue. I understand that this process can take time, and I look forward to the response.

In the meantime, I followed up with Dr. Brown over the weekend, after an observant patient alerted me to an interesting statement on the BMJ website. As a BMJ journal, Archives of Disease in Childhood promises on its website to adhere to the organization’s policies and refers readers to the BMJ Author Hub for further information. This hub includes a page about clinical trial registration.

Here’s the key passage: “In accordance with the ICMJE [International Committee of Medical Journal Editors] Recommendations, BMJ will not consider reports of clinical trials unless they were registered prospectively before recruitment of any participants. This applies to trials which commenced after 1 July 2005.”

In my e-mail to Dr. Brown, I noted that the Lightning Process study violated this explicit policy about prospective trial registration. In the study, more than half the subjects were feasibility study participants who were recruited before the official trial registration date—in some cases almost two years before. Therefore, the trial was clearly not registered “before recruitment of any participants,” as required per BMJ’s stated policy.

The statement about prospective trial registration in the BMJ Author Hub is not ambiguous. Given the Lightning Process study’s methodological choices, the trial cannot reasonably be described as “prospective.” If Archives of Disease in Childhood had adhered to the BMJ’s stated trial registration policy, it would not have published the study in the first place. Under the circumstances, perhaps a retraction rather than just a correction is warranted.

Moving on, Professor Racaniello this morning e-mailed the following letter about a different study to Trish Groves, editor-in-chief at BMJ Open. He cc’d Richard Sands, an associate publisher, and Fiona Godlee, editor-in-chief of The BMJ.

**********

Dear Dr. Groves:

In 2011, BMJ Open published a paper called “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics” [1]. In the study, the investigators implemented what they called a “pilot clinical service” to test their theory that many children miss school because of chronic fatigue syndrome, and that school absence records could be a useful way to identify them. The study concluded that this approach was indeed effective in identifying new patients.

The investigators, all from the University of Bristol, collaborated with three schools in the region. The schools sent letters to the families of 146 students with records of significant absence, inviting them to meet with a pediatrician from the Bath CFS/ME clinical service and a school staff member about the issue. The pediatrician was the lead investigator of the study; in many or most cases, the staff member was the school attendance officer. More than 100 families responded to the letters and attended the meetings. Ultimately, 28 of the students were identified as having chronic fatigue syndrome.

Parents in the U.K. can be prosecuted if their children miss too much school. Receiving such school letters could have felt coercive to the families involved, and any subsequent meetings could have caused further anxiety and distress. This is precisely the kind of challenging situation in which ethical review and approval are essential. Yet the investigators exempted this study from ethical review on the grounds that they were conducting “service evaluation,” as reported on Virology Blog by David Tuller of the University of California, Berkeley [2].

The study described in the BMJ Open paper does not meet accepted definitions of service evaluation, including those from the U.K. Health Research Authority [3]. According to these official guidelines, studies with hypotheses and generalizable conclusions are considered “research” and are required to undergo ethical review.

Service evaluation studies, which do not require ethical review, are designed to assess current care but not to produce findings that can be extrapolated to other populations. They typically rely on anonymous secondary data that have been routinely collected from patients in the course of their regular treatment—not on primary data that investigators themselves have collected firsthand in meetings arranged specifically for the purposes of a study.

The case here is clear-cut. This study was not an evaluation of the clinical services being provided to current patients. It was designed and described as a road-test of an innovative methodology to identify undiagnosed or unknown CFS/ME patients among students with records of chronic absence.

We are specifically concerned about the following:

*To support the argument that the study was service evaluation and therefore exempt from ethical review, the investigators cited a 2007 research ethics committee opinion about a project that had nothing to do with the outreach and data-collection activities described in the BMJ Open paper. The 2007 opinion affirmed that the addition of a few questionnaires to those filled out by patients already attending the CFS/ME clinical service qualified as service evaluation. The claim that this opinion can reasonably be interpreted to cover the methodology piloted in the school absence study cannot withstand scrutiny.

*In this study, only 28 of the 146 students whose families were contacted were ultimately identified as having CFS/ME. In other words, more than 100 families of students not ultimately identified as having CFS/ME received letters inviting them to a meeting with a doctor and a school official about a medically and legally sensitive issue. It is unclear how contacting people apparently unaffected by CFS/ME as part of a pilot program to identify potential patients could be viewed as service evaluation of current care for the illness.

*The BMJ Open paper included an explicit hypothesis as well as conclusions that were presented as generalizable, albeit with some limitations. Hypotheses and generalizable conclusions go beyond the usual understanding of what constitutes service evaluation and are recognized as elements of what is considered research, as described by the Health Research Authority.

*BMJ Open itself published the paper under the heading of “research.” Nevertheless, the journal has defended the investigators’ decision to classify the study as service evaluation—even though a pre-publication reviewer raised serious questions about the lack of ethical review. BMJ Open has not explained how reaching out to the families of students unknown to the CFS/ME clinical service could be considered service evaluation of current patient care.

*To support its position, BMJ Open has referenced a statement issued last year by the University of Bristol in defense of its faculty’s work on chronic fatigue syndrome [4]. But the Bristol statement focused on the use of anonymous secondary data compiled from CFS/ME clinics across the country. The statement did not address the methodology for collecting primary data directly from students that was used in the school absence study.

In any event, the BMJ Open paper’s detailed description of the data collection for this “pilot clinical service” provides ample evidence that the study was not “service evaluation” but “research.” It is perplexing that the journal has failed to acknowledge this undeniable fact. This pediatric study should not have been conducted, much less published, without a full ethical review. Since it was conducted and published anyway, BMJ Open must now take prompt steps to clarify what happened and address this serious matter in a fully transparent manner.

Sincerely–

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, NY, USA

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, CA, USA

Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, UK

Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, VA, USA

Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, IL, USA

Steven Lubet
Williams Memorial Professor of Law
Northwestern University Pritzker School of Law
Chicago, IL, USA

David F. Marks, PhD
Editor
Journal of Health Psychology
& Health Psychology Open
London, UK

Marlon Maus, MD, DrPH, FACS
Director, DrPH Program
School of Public Health
University of California, Berkeley
Berkeley, CA, USA

Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, VA, USA

Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs
Solve ME/CFS Initiative
Los Angeles, CA, USA

Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, CA, USA

John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, CA, USA

Samuel Tucker, MD
Assistant Clinical Professor of Psychiatry (retired)
University of California, San Francisco
San Francisco, California, USA

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
University of California, Berkeley
Berkeley, CA, USA

William Weir, FRCP
Infectious Disease Consultant
London, UK

**********

[1] Crawley EM, Emond AM & Sterne JA. 2011. Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics. BMJ Open; 1: e000252. Accessed at: http://bmjopen.bmj.com/content/bmjopen/1/2/e000252.full.pdf

[2] Tuller D. 2017. Trial by error: no ethical review of Crawley school absence study. Virology Blog; Aug 28. Accessed at: http://www.virology.ws/2017/08/28/trial-by-error-no-ethical-review-of-crawley-school-absence-study/

[3] U.K. Health Research Authority. 2017. Decision tools. Accessed at: http://www.hra-decisiontools.org.uk/research/docs/DefiningResearchTable_Oct2017-1.pdf

[4] University of Bristol. 2017. University of Bristol statement about the CFS/ME National Outcome Database (NOD). Accessed at: http://www.bristol.ac.uk/media-library/sites/red/documents/research-governance/University%20of%20Bristol%20statements%20about%20CFS_ME%20research.pdf

Comments on this entry are closed.

  • Joan byrne 19 February 2018, 1:07 pm

    Excellent letter and thank you for writing this Prof Racaniello. It is hard to believe that the BMJ would publish such an unethical piece of work. Let us hope they see immediate sense and acknowledge their grave error in doing so. Time to right the wrong.

  • Rivka 20 February 2018, 9:16 am

    When do theses malice actions on the part of the medical community cross the line into criminal acts? Their targets are sick children and their desperate bewildered parents.