Trial By Error: My Letter Responding to Norway Health Leader’s Efforts to Denigrate My “Activist” Work

By David Tuller, DrPH

Updated on March 2, 2022:

Yesterday I received a thoughtful note from Professor Miek Jong in response to my recent letter to her (see below) regarding the effort on the part of a leading Norwegian doctor to denigrate my work by describing me as an “activist.”

I have sent Professor Jong the following answer:

Dear Miek–

Thanks so much for your gracious response. It’s great that you’ve had experience at Berkeley and UCSF! 

I don’t mind my work being mentioned in this debate. What I find disturbing is that Dr Flottorp and her colleagues apparently deride anyone opposing their views as an “activist,” notwithstanding any professional and academic credentials or the merits of the issues raised. Moreover, they do so as if the word “activist” automatically disqualifies someone from having a legitimate perspective. 

When an intervention essentially tells patients that to acknowledge having an illness or symptoms or even negative thoughts represents a failure, it is self-evident that their responses to subjective questions about how they feel are likely to be infused with an unknown amount of bias. This should not be a controversial or confusing concept for experienced investigators to grasp, but apparently it is.

Good luck holding down the fort in this debate! 

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

In my previous post about a whine de coeur from northern European members of the GET/CBT idealogical brigades, I mentioned that the lead author, Signe Flottorp, had cited my work in a 2020 letter. In that blunt message, she characterized me as a “well-known ME-activist.” The letter is part of an exchange that was apparently obtained through a freedom of information request and released on social media.

Dr Flottorp is the research director at Norway’s National Institute for Public Health. The exchange was with Miek Jong, the head of NAFKAM, an agency that collects information about alternative medical approaches. In the exchange, Dr Flottorp complained to Dr Jong about a cautionary NAFKAM report about the Lightning Process (LP). Among her complaints was that NAFKAM did not publicize results from the pediatric LP trial conducted by pediatrician Esther Crawley, a methodologically and ethically challenged professor at Bristol University.

In a response to Dr Flottorp, Dr Jong linked to Virology Blog—specifically, to a post about a letter I organized on behalf of myself and other experts concerning the LP study in question:

“The study that you mention of Crawly et al in 2018 was not included in our factsheet about LP because the methodology of the study was heavily criticized after publication by a group of 21 scientists from highly ranked universities world-wide https://www.virology.ws/2018/01/30/trial-by- error-a-letter-to-archives-of-disease-in-childhood/, and still under re-evaluation by the journal when our latest factsheet on LP was published in 2018 https://adc.bmj.com/content/104/10/e3.”

In Dr Flottorp’s rebuttal, she wrote this:

“You state that NAFKAM did not use data from the Crawley study on LP in a factsheet in 2018 because of the criticism raised by a group of 21 scientists from highly ranked universities world- wide. First: this is another argument than the argument that NAFKAM builds on systematic reviews, not on primary studies. Second: the letter you cite was organised by David Tuller and published on his Virology Blog. David Tuller is a well-known ME-activist, supported (also economically) by ME patients (information from the webpage of ME-foreningen). The journal (Archives of Disease in Childhood) has not retracted the study, but republished a corrected version. Reading your reply made me aware that David Tuller also supports the ME activists’ fight against the planned Norwegian LP trial in a Virology blog May 2: Trial By Error: Norway’s Proposed Lightning Process Trial. In this blog he has linked to and google translated Nina Steinkopf’s blog Omstridt LP-studie er ikke godkjent enda.”

After running through more verbiage, Dr Flottorp ended her letter with this demand:

“Being quite disappointed by your reply, I have to repeat that I request NAFKAM to withdraw and apologize the warning on LP issued in 2011 and repeated 02.03.20.”

**********

So. Since Dr Flottorp invoked my name and work,  I figured the situation warranted an intervention on my part. I have sent the following letter to Dr Jong;

Dear Dr Jong:

It has come to my attention that Signe Flottorp, research director at the National Institute for Public Health, mentioned me and my work in a blunt letter sent to you on May 13, 2020. (I have cc’d Dr Flottorp.) This letter, which concerned NAFKAM’’s report on the Lightning Process, was apparently obtained through a freedom of information request. It was recently posted on Twitter. In the letter, Dr Flottorp made strong demands involving the NAFKAM report because she did not approve of the content.

As part of her argument, Dr Flottorp invoked my name—accurately–as the organizer of a letter about the 2017 report of the pediatric trial of the Lightning Process published by Archives of Disease in Childhood, a BMJ journal. The lead investigator of that trial, Esther Crawley of Bristol University, has also been involved with a proposed Lightning Process trial in Norway. A previous proposal for that trial was rejected last year by federal ethics authorities because of apparent conflicts of interest and a problematic study design. My opposition to the proposal, as outlined in the blog cited by Dr Flottorp in her letter, was based on the self-evident flaws with the proposal and related concerns about the LP.

In her letter to you, Dr Flottorp characterized me as an “ME activist.” (I don’t have ME, nor do I have any loved ones who do.) I don’t personally consider the word “activist” as dismissive–unless it is used in a dismissive manner, as it was by Dr Flottorp. Whatever my “activist” status, I am a member of the public health faculty at the University of California, Berkeley–one of the world’s leading research institutions. 

My investigation into poor research in the domain of ME/CFS and related conditions is my academic project. (My position is supported through crowdfunded donations made directly to UC Berkeley.) It is inappropriate for anyone discussing my work in a professional context to omit my academic qualifications. Moreover, Virology Blog, the science site where I post much of my work, is hosted by Vincent Racaniello, Higgins Professor of Microbiology at Columbia University. (He is also cc’d on this letter, as is Cesilie Aasen of Norway’s Directorate of Health, who was cc’d on Dr Flottorp’s letter.)

In short, I am an “activist” in my efforts to highlight questionable science. As such, I advocate strongly for my views, just as Dr Flottorp and her colleagues are “activists” in their strong advocacy for their own beliefs.

Turning to the pediatric Lightning Process trial itself, it is worth explaining exactly what Dr Crawley and her colleagues did and why I mounted objections. In conducting the study, the investigators recruited and collected data from more than half the participants before trial registration. At that point, they decided to swap primary and secondary outcomes.

BMJ—like all major medical publishers—has a long-standing policy of refusing to consider trials in which participants were recruited before registration. This policy is designed specifically to prevent bias through strategies like selective outcome reporting and exactly the sort of post-registration outcome-swapping that happened in the pediatric trial under discussion.

In this particular case, the outcome swap transformed how the investigators were able to report the results. The original primary outcome—school attendance at six months—had null results. The revised primary (and former secondary) outcome—self-reported physical function—had positive results. Thanks to the advantageous decision of Dr Crawley and her colleagues to swap an objective measure of function (school attendance) in favor of a subjective one (self-reported physical function) after data had been collected from more than half the sample, they were able to hype the study based on the revised primary outcome.

Archives of Disease in Childhood would presumably have flatly rejected the manuscript had the authors disclosed how they actually conducted the trial. So what happened? Well, Dr Crawley and her colleagues were not honest in their description of their work. They wrote up this trial–in which more than half the participants were recruited before registration–as if it were a fully prospective study. And they did not mention that they swapped outcomes after collecting data from more than half the participants.

It is hard to understand how this untrue account of the trial could have happened unintentionally. However it happened, the situation certainly appears to meet standard definitions of research misconduct, in my opinion. In any event, here’s the only thing any LP trial relying on subjective outcomes will show: Sick people who are in a program in which they are told over and over to affirm they are well and to deny they feel crummy will be more likely afterwards to report that they are doing well and don’t feel crummy than sick people who haven’t been through such a program.

As Dr Flottorp noted, I organized a letter about the pediatric LP study that was signed by a group of highly credentialed experts. Other experts chose to join me in signing the letter not because I am a persuasive “ME activist” but because anyone who reviews the facts can see the trial stinks–it violated core principles of scientific research.

After investigating the issue for more than a year, Archives of Disease in Childhood acknowledged the problems I had documented. However, in what I can only assume was a face-saving move, the journal decided not to retract the paper. Instead, it published a 3,000-word correction and a revised version—with the exact same findings. It also published a 1,000-word “editor’s note” that used tortured logic to defend the journal’s decision-making. I believe the failure to retract the paper represented a serious breach of BMJ’s responsibility to protect the integrity of the scientific literature.

After the revision was published, I organized another letter–this one to BMJ’s then-editorial director, Fiona Godlee. The new letter was signed by 70+ experts from Harvard, Columbia, University College London, Berkeley, and many other top universities around the world, plus dozens of patient and advocacy organizations. Here’s a key section: 

“The editor’s note explains that the journal examined the outcome-swapping issue by ‘seeking assurance from the authors that the change in primary outcome was not influenced by (positive) findings in the feasibility phase.’ This statement is perplexing. Individuals subject to potential bias are not generally considered impartial and authoritative arbiters of whether this potential bias has influenced their decision-making. Study design is supposed to seek to minimize bias precisely because humans tend to be blind to their own biases. Reputable science in high-impact journals should not have to rely on investigators’ ‘assurance’ that they have resisted the natural temptation to let their perceived interests guide their selection of primary outcomes.

Moreover, in relation to any assurances provided by the Bristol investigators, it is worth remembering that they withheld critical information about trial registration and outcome-swapping from their public version of events. Archives of Disease in Childhood would not have published the paper in the first place except for two major failures: 1) The investigators’ failure to provide an accurate account of the trial; and 2) The journal’s failure to detect disqualifying flaws, despite its obligation to subject the paper to rigorous peer review and editorial oversight.

Archives of Disease in Childhood has now re-published positive primary results generated by outcome-swapping in a study that did not meet a strict publication requirement specifically intended to prevent outcome-swapping and other kinds of selective outcome reporting. In doing so, the journal has rewarded the Bristol investigators for their lack of candor and their methodological missteps.”  

The full letter to Dr Godlee can be read here. I also wrote an article about the entire disturbing mess for the online health site STAT, which can be read here. 

I apologize for the length of this letter. But when I read Dr Flottorp’s missive apparently casting doubt on my motivations with its reference to me in terms that she considered derogatory (“ME activist”) and with no mention of my Berkeley affiliation, I felt a need to respond in full. And I wanted to reinforce the main point: The pediatric trial in question was a methodological and ethical disaster. If my work in exposing its egregious flaws and opposing an adult LP trial favored by Dr Flottorp makes me an “activist” in her view, so be it.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley