By David Tuller, DrPH

Last week, I noticed that Nick Brown, the editor-in-chief of Archives of Disease in Childhood, had appended an “editor’s note” to the Lightning Process study on June 19th. The note is stunningly inadequate.

Here it is:

This study was published online in Archives of Disease in Childhood after peer review in September 2017. The trial tested the effectiveness of a neurolinguistic programming intervention (used widely but never formally tested) in children and young people with chronic fatigue recruited between 2010 and 2013. Though the number of participants was small, analysis suggested a benefit in terms of physical function (measured by the standard SF 36 scale) at both 6 and 12 months after intervention.

Since publication, the study has been criticised for failing to meet ICMJE and BMJ policy on trial registration and for not fully adhering to CONSORT guidance on trial reporting. The journal has been criticised for not detecting these issues during editorial and peer review. We have acknowledged these comments and reviewed our processes in relation to this paper and relating to EQUATOR guidance in general. In addition, we have received clarifications from the authors which are under editorial consideration.

********

Today, I sent the following e-mail to Dr. Brown, cc-ing NICE and the Health Research Authority as well as BMJ editorial director Fiona Godlee, who has also so far remained silent on the egregious problems with this trial.

Dear Dr. Brown–

Last week I noticed the editor’s note that you apparently appended to the Lightning Process study on June 19th. In a subsequent letter to Dr. Godlee, I suggested that I found this editor’s note unsatisfactory. I thought I should explain to you directly why I feel that way.

I have cc’d stakeholders in the NICE guidance process on this message, since you had promised them you would respond “fully” to the concerns about the study. I don’t know if you updated any of them on this development; in any event the note is clearly not a full response. I have also cc’d officials at the Health Research Authority, given that the local research ethics committee ensured bias in this study’s reported findings with a wrong-headed decision–approving the investigators’ request to extend a feasibility trial into a full trial while simultaneously swapping primary and secondary outcomes.

Your editor’s note repeats the study’s positive reported findings but is vague on the criticisms raised. No one reading the note would grasp that the registration issues and non-adherence to CONSORT guidelines mentioned actually render those reported findings unreliable and invalid. (Obviously, per BMJ policy, the study should not have been published in the first place.)

Moreover, in not refuting the criticisms, the editor’s note appears to acknowledge that they are true without having to directly concede the point. If you had determined the criticisms arose from a misconception or misreading of the documentation, you would presumably have responded accordingly. There would be no reason to review your editorial and peer-review systems if the concerns were unwarranted.

Yet if you have decided the criticisms are warranted, it is perplexing that you have left the paper itself as is. You have also chosen not to share salient details with readers—about the flaws themselves, about their implications for the viability of the paper’s reported findings, and about identified deficiencies in the journal’s publication processes. Nor have you indicated when you will provide further updates. You note that you have received “clarifications” from the investigators and that these are “under editorial consideration.” But you have provided no insight into whether these “clarifications” shed light on the study’s problematic methodological choices and how much time this “editorial consideration” will take.

As I have repeatedly pointed out, ascertaining the accuracy of these serious concerns would take minutes. Yet after almost six months of investigation, you, Archives, Dr Godlee and BMJ are still unable to forthrightly admit what has been authoritatively documented: 1) The Lightning Process study violated BMJ policy by recruiting 56 out of 100 participants before trial registration; 2) The investigators further biased their findings by swapping outcome measures based on their initial results; 3) They failed to disclose all these significant changes in the Archives paper. None of this is acceptable scientific practice, as you presumably know.

The editor’s note further mentions that Archives has been criticized for its peer-review processes. If that sentence is referring to criticism from me, I believe I have been more critical of your failure to address the matter after it was formally brought it to your attention in January. I recognize that people, including editors, can make mistakes—as I myself do. That’s why correcting them is an indispensable editorial function; failing to do so is a betrayal of readers and the public trust. At this late stage, posting an opaque note that no one will see, much less understand, suggests that Archives is prioritizing reputational concerns over editorial integrity and transparency, at least in this domain.

Let’s remember that we are talking about a woo-woo treatment based on neuro-linguistics programming, positive affirmations and osteopathy. The founder, Phil Parker, previously trained healers in the use of Tarot and spiritual guides. Here’s his biography from a website promoting one of his courses:

“Phil Parker is already known to many as an inspirational teacher, therapist, healer and author. His personal healing journey began when, whilst working with his patients as an osteopath. [sic] He discovered that their bodies would suddenly tell him important bits of information about them and their past, which to his surprise turned out to be factually correct! He further developed this ability to step into other people’s bodies over the years to assist them in their healing with amazing results. After working as a healer for 20 years, Phil Parker has developed a powerful and magical program to help you unlock your natural healing abilities. If you feel drawn to these courses then you are probably ready to join.”

As I have previously said, Archives has now endowed its prestige on practitioners of Parker’s Lightning Process—based on a study that violated well-established ethical and scientific principles. These violations occurred in a trial involving children with a stigmatized disease. BMJ professes to care about children’s health and wellbeing. Given that no one is disputing the facts, your refusal to clean up the mess you created when you published the study is disappointing, disheartening, and difficult for me personally to understand.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

The TWiV team travels to Texas A&M University, home of the Center for Phage Technology, where they speak with Ry Young and Jason Gill about their work on viruses that infect bacteria.

Click arrow to play
Download TWiV 502 (54 MB .mp3, 90 min)
Subscribe (free): iTunesGoogle PodcastsRSSemail

Become a patron of TWiV!

Show notes at microbe.tv/twiv

By David Tuller, DrPH

Earlier today, I e-mailed the following letter to the members of the House of Commons Science and Technology Committee, which has been investigating issues of scientific and research integrity. I thought it would be a good idea to make sure they knew that this quality appeared to be lacking in some studies in the ME/CFS domain. (If the letter sounds familiar, that’s because it is pulled together from previous versions I’ve sent to others.)

I cc’d both Richard Horton, editor of The Lancet, and Fiona Godlee, editorial director of BMJ.

The subject line: “Concerns about lack of research integrity in Lancet, BMJ papers on ME/CFS”

**********

Dear Rt Hon Norman Lamb MP (and other members of the House of Commons Science and Technology Committee)–

My name is David Tuller. I am a senior fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. For the last three years, I have been investigating the research underlying treatments for the illness variously known as chronic fatigue syndrome, myalgic encephalomyelitis, CFS/ME, and ME/CFS. I am contacting you because of your investigation into scientific integrity—a trait that appears to have been in remarkably short supply in this domain of inquiry.

Through my series of reports (“Trial By Error”) on a well-regarded science site (Virology Blog), I have helped to highlight serious problems with the PACE trial–a topic addressed in a recent Westminster Hall debate on the illness. Just this week, Virology Blog sent—and posted online–an open letter to The Lancet and editor Richard Horton about the “unacceptable methodological lapses” in PACE. The letter was signed by more than 100 scientists, academics, clinicians and other experts, 10 MPs, and more than 60 patient/advocacy groups from around the world.

Here is a link to the open letter: http://www.virology.ws/2018/07/10/trial-by-error-yet-another-appeal-to-the-lancet-with-more-on-board/

Here is a link to my 15,000-word investigation of PACE, published in October, 2015: http://www.virology.ws/2015/10/21/trial-by-error-i/

Here is a link to an updated list of all Virology Blog’s posts about PACE and related ME/CFS issues: http://www.virology.ws/mecfs/

I have cc’d Dr Horton on this e-mail. Perhaps he can explain why his journal published a study in which 13 % of the participants were identified simultaneously as being “disabled” and “within normal range” for physical function on the same measure—and why the published paper does not mention that salient fact. Perhaps he can also explain why the investigators did not tell participants about their close ties to disability insurance companies—even though they promised in their trial protocol to disclose “any possible conflicts of interest” as part of the process of obtaining informed consent.

When it comes to poor research in the ME/CFS domain, however, it would be unfair to criticize The Lancet alone. BMJ journals have also published questionable studies. Given your ongoing inquiry into these matters, I felt it was appropriate to bring your attention to two such problematic studies—one in BMJ Open and one in Archives of Disease in Childhood.

The BMJ Open paper, called “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics,” was published in 2011. The Archives of Disease in Childhood paper, called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial,” was published last year.

Both studies violated core methodological and/or ethical principles. Yet because they remain in the literature, they are in a position to impact public health policies and the medical treatments available to kids. Although BMJ editors have been informed of the issues, they have so far failed to take the appropriate and necessary steps. This failure is arguably placing children at risk and raises questions not just about BMJ’s poor decision-making in the prepublication period but about lack of accountability for correcting the record when irrefutable errors are identified.

In the BMJ Open study, the investigators sought to test the hypothesis that children with chronic fatigue syndrome could be identified through school absence records. But they exempted the study from ethical review on the false grounds that it was “service evaluation” and not “research.” The study relied on primary data collected directly from identified participants, not on anonymous secondary data, and included a hypothesis and generalizable conclusions. According to UK guidelines, a study with these features is not “service evaluation” but “research,” and ethical review is required.

A peer reviewer for BMJ Open raised tough questions about the study’s claim that it was not “research” but “service evaluation.” Although the investigators failed to provide an adequate response for the lack of ethical review, BMJ Open overlooked the issue and accepted the paper anyway. Ironically, the journal published it under the heading of “research” but has since defended the decision to categorize it as “service evaluation.”

In the Archives of Disease in Childhood study of the Lightning Process, more than half the participants were recruited as part of a feasibility study starting almost two years before trial registration. Even as these feasibility study participants were folded into the full trial, primary and secondary outcome measures were swapped based on their results–an obvious recipe for biasing the findings. The paper in Archives of Disease in Childhood reported positive results but did not disclose these relevant details—a disturbing omission.

To protect against bias, BMJ and other leading journals all have a longstanding policy against publishing trials in which participants were recruited before registration. The Lightning Process study clearly breached this policy. And yet Archives of Disease in Medicine [sic] continues to provide practitioners of the Lightning Process with bragging rights that their pseudo-scientific program is effective in treating children with this illness. The news coverage of the reported results has likely driven more desperate parents to spend their hard-earned money on the Lightning Process. Last month, Archives appended an opaque “editor’s note” to the article; the note indicated that the journal was looking into concerns about some irregularities, but did not provide details or answers.

You can read more about the documented problems with these studies, and letters of concern sent to the journal editors by more than a dozen scientists, clinicians and other experts, in the following Virology Blog posts:

Trial By Error: No Ethical Review of Crawley School Absence Study
Trial By Error: A Letter to BMJ Open
Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood

I have cc’d Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ Company, on this e-mail. Perhaps she can explain why journals under her purview have not yet publicly acknowledged the obvious: The BMJ Open study was not “service evaluation,” and the Archives study violated BMJ policy on prospective trial registration. Neither should have been accepted for publication in the first place.

Does Dr Godlee believe these incidents suggest a need for greater scrutiny of BMJ’s editorial and peer-review processes, at least when it comes to this domain of inquiry? Does she have suggestions for what recourse should be available when journals fail in their responsibility to monitor themselves, with potentially negative public health and medical consequences?

I am raising these issues of scientific integrity with you now because this BMJ research is at risk of being given credence in deliberations on public policy involving children. That includes such initiatives as the current NICE effort to develop a new guidance for ME/CFS or the ongoing roll-out of the problematic cost-cutting program called Improving Access to Psychological Therapies. Like PACE, these two flawed studies have no place being cited in support of any public health initiatives, clinical guidelines, or recommendations on medical care. But because of apparent editorial deficiencies at the journals in question, the papers somehow passed peer review, were accepted for publication, and can be cited as authoritative sources.

Thank you for your attention to this important matter. I would be happy to answer any questions you might have, whether about these two papers, the PACE trial, or other studies from this subpar body of research.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

By David Tuller, DrPH

This morning I sent another letter to Fiona Godlee, editor-in-chief at The BMJ and editorial director at BMJ. Hopefully she will take action soon on the two pediatric papers whose publication has demonstrated that something is seriously amiss at the journals under her stewardship, at least when it comes to this domain of research. The ongoing refusal of the journal editors and Dr. Godlee herself to take responsibility for this mess and clean up their mistakes is rather shocking. The immediate prompt for my new letter was a recently published article by Phil Parker, founder of the Lightning Process, touting the Archives study as evidence that his training program has healing properties.

As it turns out, the editor of Archives of Disease in Childhood last month slapped a notice on the Lightning Process study, dated June 19th. It is of course a completely inadequate statement that leaves the paper as is, at least for now. It is included only in the “responses” section of the online page for the paper, as far as I could tell. That means no one reading the current version of the paper online would likely have any idea that it is under investigation for serious methodological violations, and that these violations raise concerns that its findings are invalid and unreliable.

**********

Dear Dr. Godlee—

In my letter to you last week, I raised questions about two pediatric studies published in BMJ journals. The first, a 2011 BMJ Open study of school absence, did not appear to have undergone a required ethical review; the investigators exempted it on the false grounds that it was “service evaluation” and not “research.” The second, a 2017 study in Archives of Disease in Childhood of a pseudo-scientific program called the Lightning Process, breached BMJ policy on prospective trial registration and biased its findings by swapping outcome measures half-way through. In both cases, the editors have failed to respond adequately and with an appropriate sense of urgency.

As it turns out, the Archives editor quietly appended an opaque and inadequate notice last month to the Lightning Process study acknowledging that the journal had received expressions of concern about the study’s methodological anomalies. The statement claims that the journal has reviewed its oversight processes and has received “clarifications” from the authors that are “under editorial consideration.” But the statement falls short in multiple respects.

It fails to mention that the journal was alerted to the issues in a letter from 21 experts more than four months earlier. It does not clearly state that the concerns raised are warranted. It does not explain why the inquiry has taken so long when the facts are very clear. It also does not include a deadline by which the journal will conclude its “editorial consideration” of the “clarifications” and take action on the documented issues with trial registration, outcome-swapping, and lack of disclosure of salient information. So for now this paper remains in the literature as is.

In my letter last week, I expressed particular concern that Lightning Process practitioners would be able to cite the Archives study as proof that their woo-woo intervention works. I wanted to let you know that, sure enough, Lightning Process founder Phil Parker has now done just that. In a new journal article called “Understanding the Lightning Process Approach to CFS/ME: a Review of the Disease Process and the Approach,“ he and colleagues present the Archives study as the highlight of a purported “developing evidence base for…efficacy.” (The Parker article appeared in a Romanian “experiential psychology” journal, so we can probably agree that its reach is likely to be limited. But even journals of questionable merit and provenance can provide enough of a veneer of academic credibility to help disseminate and popularize dubious information.)

The Lightning Process is a goulash of osteopathy, positive affirmations, and neuro-linguistic programming. Participants are told they can improve their health by changing their thought patterns. Parker, the founder, previously taught something called the “European College of Holistic Medicine Healing Course.” According to an archived website, this training included “the use of auras for diagnosis of a client’s problems” and “divination medicine cards and tarot,” since “divination is useful in creating a strong connection with healing/spirit guides.” Participants in the course, according to the website, could also expect to learn how to “prepare a space appropriately so that any energy polluting the room will not interfere with the work.”

As I have pointed out multiple times, the Lightning Process study featured multiple methodological lapses. More than half of the 100 participants were recruited before trial registration. Given BMJ policy, Archives should have rejected it on those grounds alone. Beyond that, the investigators swapped primary and secondary outcomes based on those initial results, a recipe for generating biased findings. Then they failed to disclose these details in the Archives paper—a troubling omission of salient information.

BMJ has taken almost six months to investigate a matter that could have been cleared up in a few minutes. Now it has posted a notice about the matter but still left all the questions unresolved. Had editors acted with the appropriate dispatch once alerted to the problems, Parker could not have leveraged BMJ’s prestige in promoting the Lightning Process in his recent journal article.

How much longer will it take for you to address this issue? How many more articles invoking Archives of Disease in Childhood will Phil Parker publish between now and then? How many desperate parents with sick kids will be seeking out the Lightning Process, based on the positive reports from the Archives paper?

In this case, as I have already noted, BMJ editors appear to be prioritizing reputational concerns over professional and moral obligations. The situation has enabled Lightning Process proponents to perpetuate questionable claims by citing BMJ research, so it is imperative that you pursue appropriate remedies sooner rather than later. In the interests of transparency, I am cc-ing representatives of the Health Research Authority, the CFS/ME Research Collaborative, and others already aware of my concerns that BMJ has provided bragging rights to Lightning Process practitioners.

Thank you for your prompt attention to this matter.

Best–David

By David Tuller, DrPH

Last month, Professor Racaniello sent Lancet editor Richard Horton an open letter about the PACE trial signed by 94 scientists, clinicians, academics and other experts. The letter, a follow-up from one sent in 2016, cited the study’s “unacceptable methodological lapses” and called for a fully independent investigation. Since Dr Horton has not responded, I re-sent him the letter earlier this evening (Monday, July 9, PST). In addition to more experts and ten members of the House of Commons, dozens of organizations from around the world are now also urging The Lancet to take prompt action.

The subject line of today’s e-mail:

Re-sending open letter about PACE, with additional signatories

[click to continue…]

By David Tuller, DrPH

Professor Michael Sharpe blocked me on Twitter many weeks ago but apparently can’t restrain himself from tweeting at me again. Maybe I’ve gotten under his skin.

Yesterday he tweeted what must have felt to him like a slam-dunk question: He wanted to know how many clinical trials I have conducted. The answer is none. But I don’t need to be a clinical trial expert to know that it is impossible to be “recovered” (or “within normal range,” per The Lancet) and “disabled” simultaneously on a primary measure, as happened in PACE. I don’t need to be a clinical trial expert to know that you can’t publish newsletters in which you promote therapies under investigation as already proven to be effective.

[click to continue…]

TWiV 501: Outbreak

Vincent visits the Smithsonian Institution and speaks with Sabrina Sholts, Jon Epstein, and Ed Niles about the exhibit Outbreak: Epidemics in a Connected World.

Click arrow to play
Download TWiV 501 (54 MB .mp3, 90 min)
Subscribe (free): iTunesGoogle PodcastsRSSemail

Become a patron of TWiV!

Show notes at microbe.tv/twiv

p53Of all the potential uses for genome editing via CRISPR/Cas9, perhaps the most exciting involves introducing therapeutic changes into human pluripotent stem cells (hPSCs) which can then be differentiated into many different cell types. A serious obstacle to such engineering is the finding that the tumor suppressor p53 inhibits CRISPR/Cas9 editing in hPSCs.

[click to continue…]

The entire TWiV team visits The University of Texas in Austin to record episode #500 with guests Jinny Suh, Jason McLellan, and Jon Huibregtse.

Click arrow to play
Download TWiV 500 (73 MB .mp3, 120 min)
Subscribe (free): iTunesGoogle PodcastsRSSemail

Become a patron of TWiV!

By David Tuller, DrPH

Update (7/6/18): I sent a follow-up e-mail to Dr Godlee yesterday to correct an inaccuracy in what I wrote about the 2011 report on PACE in The BMJ. I have included that follow-up e-mail at the end of this post.

**********

Earlier today, I e-mailed the following letter to Fiona Godlee, the editor-in-chief at The BMJ and editorial director of BMJ. I cc’d the Health Research Authority, the CFS/ME Research Collaborative, NICE, Carol Monaghan, etc.

The subject line: BMJ’s failure to address problems with pediatric studies of ME/CFS

**********

Dear Dr. Godlee—

I’m sorry if this sounds like an impertinent question, but I have to ask: Does BMJ care about the health of children? Frankly, the only conclusion I can draw from my recent efforts to address the methodological and ethical violations in two BMJ papers involving children with ME/CFS is that concerns about reputational damage have overridden other considerations, at least in this domain of inquiry. This is an unattractive conclusion, but as far as I can determine it is the only one that fits the facts.

[click to continue…]