virology blog

About viruses and viral disease

The Lancet

Trial By Error, Continued: My Questions for Lancet Editor Richard Horton

by

By David Tuller, DrPH

In January, I posted a list of the questions I still wanted to ask the PACE authors, who have repeatedly refused my requests to interview them about their ethically and methodologically challenged study. Richard Horton, editor of The Lancet, has similarly declined to talk with me, ignoring my e-mails seeking comment for the initial investigation, posted on Virology Blog last October, as well as for several follow-up articles. Now Dr. Horton has doubled-down on his efforts to keep a lid on the controversy by rejecting a letter that he personally solicited—a major breach of professional courtesy to the 43 well-regarded researchers and clinicians who signed it.

As Dr. Racaniello explained this week at Virology Blog, he submitted the letter on behalf of the group in March, in response to an express invitation from Dr. Horton. The invitation came right after Virology Blog posted an open letter, based on my investigation, that outlined the trial’s major missteps. Dr. Racaniello presumed from the wording of Dr. Horton’s invitation that the letter would, in fact, be published, as did the other signatories. On Monday, having been dissed by The Lancet, Dr. Racaniello finally posted the letter on PubMed Commons. He also called the PACE trial “a sham.” (I’ve called it “a piece of crap.” I might also have referred to it somewhere as “doggie-poo,” but I’m not sure.)

In rejecting the letter that he himself solicited, Dr. Horton certainly appeared to be trying to squelch the growing public controversy over PACE and its recommendations that graded exercise therapy and cognitive behavior therapy are effective treatments for chronic fatigue syndrome (or myalgic encephalomyelitis, CFS, ME, CFS/ME, or ME/CFS, or some other name). But The Lancet’s effort to shield PACE is doomed, not only because the study is so bad but because the emerging science presents a completely different portrait of the illness. On Monday, a paper in Proceedings of the National Academy of Sciences reported distinct patterns of metabolites in the plasma of ME/CFS patients—an important finding that, if confirmed, could finally lead to diagnostic tests. The PNAS paper and other recent research support the conclusion of reports last year from both the Institute of Medicine and National Institutes of Health: ME/CFS is a devastating physiological disease.

Back in January, Columbia statistics professor Andrew Gelman blogged about the harm Dr. Horton was already inflicting on his journal by not addressing the serious questions that serious critics were raising about PACE. The longstanding claim of the PACE authors, The Lancet and the trial’s other defenders—that the opponents were a small cabal of irrational, dangerous, and anti-psychiatry patients—has been exposed as false. The PACE authors, The Lancet and their colleagues wielded this narrative for years to discredit those challenging the trial. To their dismay, this tactic is no longer working.

The Lancet’s decision to reject the Virology Blog letter will only compound the journal’s growing reputational damage over the issue. It also seems deeply short-sighted, in light of last month’s powerful court decision ordering Queen Mary University of London, the professional home of principal PACE investigator Peter White, to release the raw trial data. That would allow others to determine whether the PACE investigators altered their outcome assessments strategies to produce results more likely to get published in The Lancet and other journals. (The answer should not surprise anyone except those in extreme stages of denial.)

The decision involved a freedom-of-information request filed two years ago by Alem Matthees, an Australian patient. Since the published results did not include the results per the assessment methods outlined in the PACE trial protocol, Matthees wanted the data necessary to calculate those results for the two primary outcomes of fatigue and physical function, as well as for the original definition of “recovery.” Last October, the Information Commissioner’s Office, an independent agency, found that QMUL had no grounds for refusing to provide the data. QMUL appealed that ruling to the First-Tier Tribunal, which issued the recent decision.

The U.K. medical-academic-media establishment has wholly endorsed the PACE trial’s unreliable findings and accepted the authors’ unsubstantiated claims that they have been subjected to a concerted campaign of threats and harassment. In contrast, the tribunal demonstrated a refreshing unwillingness to play along. In robust language, the tribunal smacked down the specious arguments raised by the university in its attempt to shield the data from public disclosure.

The chance that any participant could or would be identified from the anonymized data was “remote,” the tribunal found. The scenarios envisioned ed by QMUL’s data security expert, who sketched out far-fetched strategies that “activist” patients might pursue to re-identify and then harass trial participants, were “grossly exaggerated” and “a considerable amount of supposition and speculation,” wrote the tribunal. In fact, noted the tribunal, the only incident of “harassment” proven by QMUL’s experienced legal team was that someone somewhere once heckled Trudie Chalder, a principal PACE investigator who also testified at the tribunal hearing. (I also have some thoughts on Dr. Chalder’s testimony, but will hold those for another time.)

In contrast to the QMUL portrait of PACE opponents, the tribunal cited Virology Blog’s open letter to The Lancet as evidence of a robust scientific debate, noting that “the identity of those questioning the research…was impressive.” The tribunal also noted that QMUL’s decision to share data with friendly researchers but not with others had created the impression that it was acting out of self-interest, not principle. “There is a strong public interest in releasing the data given the continued academic interest so long after the research was published and the seeming reluctance for Queen Mary University to engage with other academics they thought were seeking to challenge their findings,” declared the tribunal in the decision.

The PACE authors, QMUL, Dr. Horton, and The Lancet are stonewalling the obvious, at the expense of millions of sick patients. Although Dr. Horton will never grant me an interview, I want to highlight some of the questions I have about his actions, claims and thoughts, in case someone else gets the chance to talk with him. This list of questions is certainly not exhaustive, but it’s a decent start.

So, Dr. Horton–Here’s what I’d like to ask you:

1) Do you agree that the invitation you sent to Dr. Racaniello certainly implied, even if it didn’t explicitly promise, that The Lancet would publish the letter? Since the letter submitted by Dr. Racaniello, on behalf of himself and 42 other experts, reflected the points made in the Virology Blog open letter that triggered your invitation, what changed your mind about whether it added something to the debate? Since you personally solicited the letter from Dr. Racaniello and his colleagues, do you feel you should have sent him a personal apology, rather than leaving your correspondence editor, Audrey Ceschia, to answer for your behavior?

2) In your invitation to Dr. Racaniello, you noted that the PACE authors would have a chance to respond, alongside the published letter. That was a fair plan. When did that plan of offering them a response morph into the plan of offering them a role in discussions about whether to publish the critical letter in the first place? What impact did their views have on your decision? Did the PACE authors argue, as they have in the past, that they have already answered all these criticisms?

This repeated claim that they have answered all questions is simply untrue. They have never explained, for example, how it is possible to be disabled and “within normal range” on an indicator simultaneously, and why 13 % of their participants were already “within normal range” on one or both primary outcome sat baseline. When anyone asks legitimate questions, they evade, ignore or misstate the issues—including in the correspondence following The Lancet’s 2011 paper. (This pattern of non-response is clear from their non-responsive responses to the charges raised in my Virology Blog investigation, and my rebuttal of their non-responses.)

3) What’s your reaction to the First-Tier Tribunal’s decision ordering the release of the PACE trial data? Do you agree with the tribunal’s observation, referring to Virology Blog’s February open letter to you and The Lancet, that the roster of scientists and researchers now publicly questioning the methodology and findings of PACE is “impressive”?

4) Do think QMUL should spend more public money to appeal the decision?

If QMUL decides to appeal, do you think this will fuel the already-widespread assumption that PACE had null findings per the original protocol methods of assessment?

5) The PACE interventions, as described in The Lancet, are based on the premise that deconditioning rather than any pathological process perpetuates the illness, and that increased activity and a new psychological mind-set will fix the problem. The descriptions of the interventions categorically exclude the possibility of a continuing organic disease as the cause. Do you think this portrait of the illness squares with the view emerging from this week’s study in PNAS and other recent research, including last year’s reports from the Institute of Medicine and the National Institutes of Health?

6) The IOM report identified “exertion intolerance”—the prolonged relapses patients often suffer after minimal activity–as the core symptom of the illness. Yet a key aspect of the PACE rehabilitative interventions, GET and CBT, is urging patients to increase their activity and to interpret a resurgence of symptoms as a transient event, not a sign of deterioration. Given the IOM’s focus on “exertion intolerance” as the central phenomenon, isn’t the PACE approach contraindicated?

7) Does it bother you that you published a paper in which 13% of the sample had already, at baseline, met the outcome thresholds for one or both primary measures? These outcome thresholds, which represented worse health than the entry criteria, were variously defined as being “within normal range” (the Lancet paper), “back to normal” (Dr. Chalder’s statement at the press conference for the Lancet paper), and “a strict criterion for recovery” (the Lancet commentary by colleagues of the PACE authors). Can you point me to any other studies published in The Lancet, or anywhere, in which positive outcome scores represented worse health than entry criteria?

8) Does it bother you personally that the PACE authors did not inform you or your editorial staff that a significant minority of patients were already “within normal range” on at least one primary outcome at baseline? (I presume they didn’t mention it to you because, well, it’s hard to imagine you would have published the paper if you or anyone there had been told about or noticed the inexplicable overlap in the entry criteria and the post-hoc “normal range” thresholds.)

9) During a 2011 Australian radio interview not long after The Lancet published the first PACE results, you said the following about the trial’s critics: “One sees a fairly small, but highly organised, very vocal and very damaging group of individuals who have I would say actually hijacked this agenda and distorted the debate so that it actually harms the overwhelming majority of patients.” Given that the First-Tier Tribunal expressed a different perspective on the stature and credibility of those criticizing PACE, do you still agree with your 2011 characterization of the trial’s opponents?

10) During the same interview, you stated that the PACE trial had undergone “endless rounds of peer review.” Yet the trial was also “fast-tracked” to publication, as indicated on the version of the article in the ScienceDirect database. Can you explain the mechanics of “fast-tracking” a paper to publication while simultaneously subjecting it to “endless rounds of peer review”? How long was the fast-track process for the PACE paper, and how many actual rounds of review did the paper undergo during that endless period?

11) Can you explain why the Lancet’s endless peer review process did not catch the most criticized aspect of the paper—the very obvious fact that participants could be simultaneously disabled enough for entry yet already “within normal range”/”back to normal”/”recovered” on the primary outcomes? Can you explain why the reviewers did not request the authors to provide either the original results promised in the protocol or else sensitivity analyses to assess the impact of the mid-trial changes they introduced?

12) Do you think it was appropriate for the PACE investigators to publish a mid-trial newsletter that promoted the therapies under study and included glowing testimonials from earlier participants about their excellent outcomes? Can you point to other published studies that featured such mid-trial dissemination of personal testimonials and explicit descriptions of outcomes? The PACE authors have stated that the newsletter testimonials did not identify participants’ trial arms and therefore could not have created any bias. Do you agree with this novel and creative argument that influencing all remaining participants in a trial in a positive direction is not a form of bias?

13) Did The Lancet’s peer review process include an evaluation of the PACE trial’s consent forms, given the authors’ explicit promise in the protocol to abide by the Declaration of Helsinki? The Declaration of Helsinki requires investigators to disclose “any possible conflicts of interest” not just to journals but to prospective participants. Yet the PACE consent forms did not disclose the authors’ close financial and consulting ties with the insurance industry. Do you agree this omission violates their protocol promise, and that given this violation the PACE authors failed to obtain legitimate informed consent from their participants? Without legitimate informed consent, did the PACE authors have the right to publish their findings in The Lancet and other journals? What should happen to the PACE papers already published, since the authors do not appear to have legitimate informed consent from participants?

14) Who do you think should be held responsible for the $8,000,000 in U.K. government funds wasted on the PACE trial? Who should be held responsible for the harm it has caused? What responsibility, if any, does The Lancet bear for the debacle?

Once Again, Lancet Stumbles on PACE

by

Last February, Virology Blog posted an open letter to The Lancet and its editor, Dr. Richard Horton, describing the indefensible flaws of the PACE trial of treatments for ME/CFS, the disease otherwise known as chronic fatigue syndrome (link to letter). Forty-two well-regarded scientists, academics and clinicians put their names to the letter, which declared flatly that the flaws in PACE “have no place in published research.” The letter called for a completely independent re-analysis of the PACE trial data, since the authors have refused to publish the results they outlined in their original protocol. The letter was also sent directly to Dr. Horton.

The open letter was based on the extensive investigative report written by David Tuller, the academic coordinator of UC Berkeley’s joint program in journalism and public health, which Virology Blog posted last October (link to report). This report outlines such egregious failings as outcome thresholds that overlapped with entry criteria, mid-trial promotion of the therapies under investigation, failure to provide the original results as outlined in the protocol, failure to adhere to a specific promise in the protocol to inform participants about the investigators’ conflicts of interest, and other serious lapses.

Virology Blog first posted the open letter in November, with six signatories (link to letter). At that time, Dr. Horton’s office responded that he would reply after returning from “traveling.” Three months later, we still had not heard back from Dr. Horton–perhaps he was still “traveling”–so we decided to republish it with many more people signed on.

The day the second open letter was posted, Dr Horton e-mailed me and solicited a letter from the group. (He did not explain where he had been “traveling” for the previous three months.) Here’s what he wrote: “Many thanks for your email. In the interests of transparency, I would like to invite you to submit a letter for publication–please keep your letter to around 500 words. We will then invite the authors of the study to reply to your very serious allegations.”

Dr. Horton’s e-mail clearly indicated that the letter would be published, with the PACE authors’ response to the charges raised; there was no equivocation or possibility of misinterpretation. In good faith, we submitted a letter for publication the following month, with 43 signatories this time, through The Lancet’s online editorial system (see the end of this article for a list of those who signed the letter). After several months with no response, we learned only recently by checking the online editorial system that The Lancet had flatly rejected the letter, with no explanation. No one contacted me to explain the decision or why we were asked to spend time creating a letter that The Lancet clearly had no intention of publishing.

I wrote back to Dr. Horton, pointing out that his behavior was highly unprofessional and requesting an explanation for the rejection. I also asked him if he was in the habit of soliciting letters from busy scientists and researchers that his journal had no actual interest in publishing. I further asked if the journal planned to reconsider this rejection, in light of the recent First-Tier Tribunal decision, which demolished the PACE authors’ bogus reasons for refusing to provide data for independent analysis.

Dr. Horton did not himself apologize or even deign to respond. Instead, Audrey Ceschia, the Lancet’s correspondence editor, replied, explaining that the Lancet editorial staff decided, after discussing the matter with the PACE authors, that the letter did not add anything substantially new to the discussion. She assured us that if we submitted another letter focused on the First-Tier Tribunal decision, it would be “seriously” considered. I’m not sure why she or Dr. Horton think that any such assurance from The Lancet is credible at this point.

The reasons given for the rejection are clearly specious. The letter for publication reflected the matters addressed in the open letter that prompted Dr. Horton’s invitation in the first place, and closely adhered to his directive  to outline our “serious allegations”. If outlining these allegations was not considered publication-worthy by The Lancet, it is incomprehensible to us why Dr. Horton solicited the letter in the first place. Perhaps it was just an effort to hold off further criticism for a period of months while we awaited publication of the letter, unaware of the journal’s intention to reject it. It is certainly surprising that The Lancet appears to have given the PACE authors some power to determine what letters appear in the journal itself.

Dr. Tuller’s investigation, based on the groundbreaking analyses conducted by many savvy patients and advocates since The Lancet published the first PACE results in 2011, has effectively demolished the credibility of the findings. So has a follow-up analysis by Dr. Rebecca Goldin, a math professor at George Mason University and director of Stats.org, a think tank co-sponsored by the American Statistical Association. In short, the PACE study is a sham, with meaningless results. In this case, the emperor truly has no clothes. Dr. Horton and his editorial team at The Lancet are stark naked.

Yet the PACE study remains in the literature. Its recommendation of treatments that are potentially harmful to patients–specifically, graded exercise therapy and cognitive behavior therapy, both designed specifically to increase patients’ activity levels–remains highly influential.

Of particular concern, the PACE findings have laid the groundwork for the MAGENTA study, a so-called “PACE for kids” that will be testing graded exercise therapy in children and adolescents. A feasibility study, sponsored by Royal United Hospitals Bath NHS Foundation Trust, is currently recruiting participants. It is, of course, completely unacceptable that any study should justify itself based on the uninterpretable findings of the PACE trial. The MAGENTA trial should be halted until the PACE authors have done what the First-Tier Tribunal ordered them to do–release their raw data and allow others to analyze it according to the outcomes specified in the PACE trial protocol.

Today, because of the urgency of the issue, we are posting on PubMed Commons the letter that The Lancet rejected. That way readers can judge for themselves whether it adds anything to the current debate.

Please note that the opinions in this blog post are mine only, not those of any of the other signers of the Lancet letter, listed below

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, New York

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, California

Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, England, United Kingdom

Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, Illinois

Bruce Levin, PhD
Professor of Biostatistics
Columbia University
New York, New York

Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley
Berkeley, California

******

Dharam V. Ablashi, DVM, MS, Dip Bact
Scientific Director – HHV-6 Foundation
Former Senior Investigator
National Cancer Institute, NIH
Bethesda, Maryland

James N. Baraniuk, MD
Professor, Department of Medicine
Georgetown University
Washington, D.C.

Lisa F. Barcellos, PhD, MPH
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California
Berkeley, California

Lucinda Bateman MD PC
MECFS and Fibromyalgia clinician
Salt Lake City, Utah

Alison C. Bested MD FRCPC
Clinical Associate Professor of Hematology
University of British Columbia
Vancouver, British Columbia, Canada

John Chia, MD
Clinician/Researcher
EV Med Research
Lomita, California

Lily Chu, MD, MSHS
Independent Researcher
San Francisco, California

Derek Enlander, MD, MRCS, LRCP
Attending Physician
Mount Sinai Medical Center, New York
ME CFS Center, Mount Sinai School of Medicine
New York, New York

Mary Ann Fletcher, PhD
Schemel Professor of Neuroimmune Medicine
College of Osteopathic Medicine
Nova Southeastern University
Professor Emeritus, University of Miami School of Medicine
Fort Lauderdale, Florida

Kenneth Friedman, PhD
Associate Professor of Pharmacology and Physiology (retired)
New Jersey Medical School
University of Medicine and Dentistry of NJ
Newark, New Jersey

Robert F. Garry, PhD
Professor of Microbiology and Immunology
Tulane University School of Medicine
New Orleans, Louisiana

Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, Virginia

David L. Kaufman, MD,
Medical Director
Open Medicine Institute
Mountain View, California

Susan Levine, MD
Clinician, Private Practice
Visiting Fellow, Cornell University
New York, New York

Alan R. Light, PhD
Professor, Department of Anesthesiology
Department of Neurobiology and Anatomy
University of Utah
Salt Lake City, Utah

Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, Virginia

Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs
Solve ME/CFS Initiative
Los Angeles, California

James M. Oleske, MD, MPH
Francois-Xavier Bagnoud Professor of Pediatrics
Senator of RBHS Research Centers, Bureaus, and Institutes
Director, Division of Pediatrics Allergy, Immunology & Infectious Diseases
Department of Pediatrics
Rutgers – New Jersey Medical School
Newark, New Jersey

Richard N. Podell, M.D., MPH
Clinical Professor
Rutgers Robert Wood Johnson Medical School
New Brunswick, New Jersey

William Satariano, PhD
Professor of Epidemiology and Community Health
University of California, Berkeley
Berkeley, California

Paul T Seed MSc CStat CSci
Senior Lecturer in Medical Statistics
King’s College London, Division of Women’s Health
St Thomas’ Hospital
London, England, United Kingdom

Charles Shepherd, MB BS
Honorary Medical Adviser to the ME Association
London, England, United Kingdom

Christopher R. Snell, PhD
Scientific Director
WorkWell Foundation
Ripon, California

Nigel Speight, MA, MC, BChir, FRCP, FRCPCH, DCH
Pediatrician
Durham, England, United Kingdom

Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, California

Eleanor Stein, MD FRCP(C)
Assistant Clinical Professor
University of Calgary
Calgary, Alberta, Canada

John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, California

Ronald G. Tompkins, MD, ScD
Summer M Redstone Professor of Surgery
Harvard University
Boston, Massachusetts

Rosemary Underhill, MB BS.
Physician, Independent Researcher
Palm Coast, Florida

Dr Rosamund Vallings MNZM, MB BS
General Practitioner
Auckland, New Zealand

Michael VanElzakker, PhD
Research Fellow, Psychiatric Neuroscience Division
Harvard Medical School and Massachusetts General Hospital
Boston, Massachusetts

Mark Vink, MD
Family Physician
Soerabaja Research Center
Amsterdam, The Netherlands

Prof Dr FC Visser
Cardiologist
Stichting CardioZorg
Hoofddorp, The Netherlands

William Weir, FRCP
Infectious Disease Consultant
London, England, United Kingdom

John Whiting, MD
Specialist Physician
Private Practice
Brisbane, Australia

Marcie Zinn, PhD
Research Consultant in Experimental Neuropsychology, qEEG/LORETA, Medical/Psychological Statistics
NeuroCognitive Research Institute, Chicago
Center for Community Research
DePaul University
Chicago, Illinois

Mark Zinn, MM
Research consultant in experimental electrophysiology
Center for Community Research
DePaul University
Chicago, Illinois

A request for data from the PACE trial

by

Mr. Paul Smallcombe
Records & Information Compliance Manager
Queen Mary University of London
Mile End Road
London E1 4NS

Dear Mr Smallcombe:

The PACE study of treatments for ME/CFS has been the source of much controversy since the first results were published in The Lancet in 2011. Patients have repeatedly raised objections to the study’s methodology and results. (Full title: “Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome: a randomized trial.”)

Recently, journalist and public health expert David Tuller documented that the trial suffered from many serious flaws that raise concerns about the validity and accuracy of the reported results. We cited some of these flaws in an open letter to The Lancet that urged the journal to conduct a fully independent review of the trial. (Dr. Tuller did not sign the open letter, but he is joining us in requesting the trial data.)

These flaws include, but are not limited to: major mid-trial changes in the primary outcomes that were not accompanied by the necessary sensitivity analyses; thresholds for “recovery” on the primary outcomes that indicated worse health than the study’s own entry criteria; publication of positive testimonials about trial outcomes and promotion of the therapies being investigated in a newsletter for participants; rejection of the study’s objective outcomes as irrelevant after they failed to support the claims of recovery; and the failure to inform participants about investigators’ significant conflicts of interest, and in particular financial ties to the insurance industry, contrary to the trial protocol’s promise to adhere to the Declaration of Helsinki, which mandates such disclosures.

Although the open letter was sent to The Lancet in mid-November, editor Richard Horton has not yet responded to our request for an independent review. We are therefore requesting that Queen Mary University of London to provide some of the raw trial data, fully anonymized, under the provisions of the U.K.’s Freedom of Information law.

In particular, we would like the raw data for all four arms of the trial for the following measures: the two primary outcomes of physical function and fatigue (both bimodal and Likert-style scoring), and the multiple criteria for “recovery” as defined in the protocol published in 2007 in BMC Neurology, not as defined in the 2013 paper published in Psychological Medicine. The anonymized, individual-level data for “recovery” should be linked across the four criteria so it is possible to determine how many people achieved “recovery” according to the protocol definition.

We are aware that previous requests for PACE-related data have been rejected as “vexatious.” This includes a recent request from psychologist James Coyne, a well-regarded researcher, for data related to a subsequent study about economic aspects of the illness published in PLoS One—a decision that represents a violation of the PLoS policies on data-sharing.

Our request clearly serves the public interest, given the methodological issues outlined above, and we do not believe any exemptions apply. We can assure Queen Mary University of London that the request is not “vexatious,” as defined in the Freedom of Information law, nor is it meant to harass. Our motive is easy to explain: We are extremely concerned that the PACE studies have made claims of success and “recovery” that appear to go beyond the evidence produced in the trial. We are seeking the trial data based solely on our desire to get at the truth of the matter.

We appreciate your prompt attention to this request.

Sincerely,

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University

Bruce Levin, PhD
Professor of Biostatistics
Columbia University

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University

David Tuller, DrPH
Lecturer in Public Health and Journalism
University of California, Berkeley

Trial by error, Continued: PACE Team’s Work for Insurance Companies Is “Not Related” to PACE. Really?

by

By David Tuller, DrPH

David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.

In my initial story on Virology Blog, I charged the PACE investigators with violating the Declaration of Helsinki, developed in the 1950s by the World Medical Association to protect human research subjects. The declaration mandates that scientists disclose “institutional affiliations” and “any possible conflicts of interest” to prospective trial participants as part of the process of obtaining informed consent.

The investigators promised in their protocol to adhere to this foundational human rights document, among other ethical codes. Despite this promise, they did not tell prospective participants about their financial and consulting links with insurance companies, including those in the disability sector. That ethical breach raises serious concerns about whether the “informed consent” they obtained from all 641 of their trial participants was truly “informed,” and therefore legitimate.

The PACE investigators do not agree that the lack of disclosure is an ethical breach. In their response to my Virology Blog story, they did not even mention the Declaration of Helsinki or explain why they violated it in seeking informed consent. Instead, they defended their actions by noting that they had disclosed their financial and consulting links in the published articles, and had informed participants about who funded the research–responses that did not address the central concern.

“I find their statement that they disclosed to The Lancet but not to potential subjects bemusing,” said Jon Merz, a professor of medical ethics at the University of Pennsylvania. “The issue is coming clean to all who would rely on their objectivity and fairness in conducting their science. Disclosure is the least we require of scientists, as it puts those who should be able to trust them on notice that they may be serving two masters.”

In their Virology Blog response, the PACE team also stated that no insurance companies were involved in the research, that only three of the 19 investigators “have done consultancy work at various times for insurance companies,” and that this work “was not related to the research.” The first statement was true, but direct involvement in a study is of course only one possible form of conflict of interest. The second statement was false. According to the PACE team’s conflict of interest disclosures in The Lancet, the actual number of researchers with insurance industry ties was four—along with the three principal investigators, physiotherapist Jessica Bavington acknowledged such links.

But here, I’ll focus on the third claim–that their consulting work “was not related to the research.” In particular, I’ll examine an online article posted by Swiss Re, a large reinsurance company. The article describes a “web-based discussion group” held with Peter White, the lead PACE investigator, and reveals some of the claims-assessing recommendations arising from that presentation. White included consulting work with Swiss Re in his Lancet disclosure.

The Lancet published the PACE results in February, 2011; the undated Swiss Re article was published sometime within the following year or so. The headline: “Managing claims for chronic fatigue the active way.” (Note that this headline uses “chronic fatigue” rather than “chronic fatigue syndrome,” although chronic fatigue is a symptom common to many illnesses and is quite distinct from the disease known as chronic fatigue syndrome. Understanding the difference between the two would likely be helpful in making decisions about insurance claims.)

The Swiss Re article noted that the illness “can be an emotive subject” and then focused on the implications of the PACE study for assessing insurance claims. It started with a summary account of the findings from the study, reporting that the “active rehabilitation” arms of cognitive behavioral therapy and graded exercise therapy “resulted in greater reduction of patients’ fatigue and larger improvement in physical functioning” than either adaptive pacing therapy or specialist medical care, the baseline condition. (The three intervention arms also received specialist medical care.)

The trial’s “key message,” declared the article, was that “pushing the limits in a therapeutic setting using well described treatment modalities is more effective in alleviating fatigue and dysfunction than staying within the limits imposed by the illness traditionally advocated by ‘pacing.’”

Added the article: “If a CFS patient does not gradually increase their activity, supported by an appropriate therapist, then their recovery will be slower. This seems a simple message but it is an important one as many believe that ‘pacing’ is the most beneficial treatment.”

This understanding of the PACE research—presumably based on information from Peter White’s web-based discussion—was wrong. Pacing is not and has never been a “treatment.” It is also not one of the “four most commonly used therapies,” as the newsletter article declared, since it has never been a “therapy” either. It is a self-help method practiced by many patients seeking the best way to manage their limited energy reserves.

The PACE investigators did not test pacing. Instead, the intervention they dubbed “adaptive pacing therapy” was an operationalized version of “pacing” developed specifically for the study. Many patients objected to the trial’s form of pacing as overly prescriptive, demanding and unlike the version they practiced on their own. Transforming an intuitive, self-directed approach into a “treatment” administered by a “therapist” was not a true test of whether the self-help approach is effective, they argued–with significant justification. Yet the Swiss Re article presented “adaptive pacing therapy” as if it were identical to “pacing.”

The Swiss Re article did not mention that the reported improvements from “active rehabilitation” were based on subjective outcomes and were not supported by the study’s objective data. Nor did it report any of the major flaws of the PACE study or offer any reasons to doubt the integrity of the findings.

The article next asked, “What can insurers and reinsurers do to assist the recovery and return to work of CFS claimants?” It then described the conclusions to be drawn from the discussion with White about the PACE trial—the “key takeaways for claims management.”

First, Swiss Re advised its employees, question the diagnosis, because “misdiagnosis is not uncommon.”

The second point was this: “It is likely that input will be required to change a claimant’s beliefs about his or her condition and the effectiveness of active rehabilitation…Funding for these CFS treatments is not expensive (in the UK, around £2,000) so insurers may well want to consider funding this for the right claimants.”

Translation: Patients who believe they have a medical disease are wrong, and they need to be persuaded that they are wrong and that they can get better with therapy. Insurers can avoid large payouts by covering the minimal costs of these treatments for patients vulnerable to such persuasion, given the right “input.”

Finally, the article warned that private therapists might not provide the kinds of “input” required to convince patients they were wrong. Instead of appropriately “active” approaches like cognitive behavior therapy and graded exercise therapy, these therapists might instead pursue treatments that could reinforce claimants’ misguided beliefs about being seriously ill, the article suggested.

“Check that private practitioners are delivering active rehabilitation therapies, such as those described in this article, as opposed to sick role adaptation,” the Swiss RE article advised. (The PACE investigators, drawing on the concept known as “the sick role” in medical sociology, have long expressed concern that advocacy groups enabled patients’ condition by bolstering their conviction that they suffered from a “medical disease,” as Michael Sharpe, another key PACE investigator, noted in a 2002 UNUMProvident report. This conviction encouraged patients to demand social benefits and health care resources rather than focus on improving through therapy, Sharpe wrote.)

Lastly, the Swiss Re article addressed “a final point specific to claims assessment.” A diagnosis of chronic fatigue syndrome, stated the article, provided an opportunity in some cases to apply a mental health exclusion, depending upon the wording of the policy. In contrast, a diagnosis of myalgic encephalomyelitis did not.

The World Health Organization’s International Classification for Diseases, or ICD, which clinicians and insurance companies use for coding purposes, categorizes myalgic encephalomyelitis as a neurological disorder that is synonymous with the terms “post-viral fatigue syndrome” and “chronic fatigue syndrome.” But the Swiss Re article stated that, according to the ICD, “chronic fatigue syndrome” can also “alternatively be defined as neurasthenia which is in the mental health chapter.”

The PACE investigators have repeatedly advanced this questionable idea. In the ICD’s mental health section, neurasthenia is defined as “a mental disorder characterized by chronic fatigue and concomitant physiologic symptoms,” but there is no mention of “chronic fatigue syndrome” as a discrete entity. The PACE investigators (and Swiss Re newsletter writers) believe that the neurasthenia entry encompasses the illness known as “chronic fatigue syndrome,” not just the common symptom of “chronic fatigue.”

This interpretation, however, appears to be at odds with an ICD rule that illnesses cannot be listed in two separate places—a rule confirmed in an e-mail from a WHO official to an advocate who had questioned the PACE investigators’ argument. “It is not permitted for the same condition to be classified to more than one rubric as this would mean that the individual categories and subcategories were no longer mutually exclusive,” wrote the official to Margaret Weston, the pseudonym for a longtime clinical manager in the U.K. National Health Service.

Presumably, after White disseminated the good news about the PACE results at the web-based discussion, Swiss Re’s claims managers felt better equipped to help ME/CFS claimants. And presumably that help included coverage for cognitive behavior therapy and graded exercise therapy so that claimants could receive the critical “input” they needed in order to recognize and accept that they didn’t have a medical disease after all.

In sum, contrary to the investigators’ argument in their response to Virology Blog, the PACE research and findings appear to be very much “related to” insurance industry consulting work. The claim that these relationships did not represent “possible conflicts of interest” and “institutional affiliations” requiring disclosure under the Declaration of Helsinki cannot be taken seriously.

Update 11/17/15 12:22 PM: I should have mentioned in the story that, in the PACE trial, participants in the cognitive behavior therapy and graded exercise therapy arms were no more likely to have increased their hours of employment than those in the other arms. In other words, there was no evidence for the claims presented in the Swiss Re article, based on Peter White’s presentation, that these treatments were any more effective in getting people back to work.

The PACE investigators published this employment data in a 2012 paper in PLoS One. It is unclear whether Peter White already knew these results at the time of his Swiss Re presentation on the PACE results.

Update 11/18/15 6:54 AM: I also forgot to mention in the story that the three principal PACE investigators did not respond to an e-mail seeking comment about their insurance industry work. Lancet editor Richard Horton also did not respond to an e-mail seeking comment.

An open letter to Dr. Richard Horton and The Lancet

by

Dr. Richard Horton
The Lancet
125 London Wall
London, EC2Y 5AS, UK

Dear Dr. Horton:

In February, 2011, The Lancet published an article called “Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomized trial.” The article reported that two “rehabilitative” approaches, cognitive behavior therapy and graded exercise therapy, were effective in treating chronic fatigue syndrome, also known as myalgic encephalomyelitis, ME/CFS and CFS/ME. The study received international attention and has had widespread influence on research, treatment options and public attitudes.

The PACE study was an unblinded clinical trial with subjective primary outcomes, a design that requires strict vigilance in order to prevent the possibility of bias. Yet the study suffered from major flaws that have raised serious concerns about the validity, reliability and integrity of the findings. The patient and advocacy communities have known this for years, but a recent in-depth report on this site, which included statements from five of us, has brought the extent of the problems to the attention of a broader public. The PACE investigators have replied to many of the criticisms, but their responses have not addressed or answered key concerns.

The major flaws documented at length in the recent report include, but are not limited to, the following:

*The Lancet paper included an analysis in which the outcome thresholds for being “within the normal range” on the two primary measures of fatigue and physical function demonstrated worse health than the criteria for entry, which already indicated serious disability. In fact, 13 percent of the study participants were already “within the normal range” on one or both outcome measures at baseline, but the investigators did not disclose this salient fact in the Lancet paper. In an accompanying Lancet commentary, colleagues of the PACE team defined participants who met these expansive “normal ranges” as having achieved a “strict criterion for recovery.” The PACE authors reviewed this commentary before publication.

*During the trial, the authors published a newsletter for participants that included positive testimonials from earlier participants about the benefits of the “therapy” and “treatment.” The same newsletter included an article that cited the two rehabilitative interventions pioneered by the researchers and being tested in the PACE trial as having been recommended by a U.K. clinical guidelines committee “based on the best available evidence.” The newsletter did not mention that a key PACE investigator also served on the clinical guidelines committee. At the time of the newsletter, two hundred or more participants—about a third of the total sample–were still undergoing assessments.

*Mid-trial, the PACE investigators changed their protocol methods of assessing their primary outcome measures of fatigue and physical function. This is of particular concern in an unblinded trial like PACE, in which outcome trends are often apparent long before outcome data are seen. The investigators provided no sensitivity analyses to assess the impact of the changes and have refused requests to provide the results per the methods outlined in their protocol.

*The PACE investigators based their claims of treatment success solely on their subjective outcomes. In the Lancet paper, the results of a six-minute walking test—described in the protocol as “an objective measure of physical capacity”–did not support such claims, notwithstanding the minimal gains in one arm. In subsequent comments in another journal, the investigators dismissed the walking-test results as irrelevant, non-objective and fraught with limitations. All the other objective measures in PACE, presented in other journals, also failed. The results of one objective measure, the fitness step-test, were provided in a 2015 paper in The Lancet Psychiatry, but only in the form of a tiny graph. A request for the step-test data used to create the graph was rejected as “vexatious.”

*The investigators violated their promise in the PACE protocol to adhere to the Declaration of Helsinki, which mandates that prospective participants be “adequately informed” about researchers’ “possible conflicts of interest.” The main investigators have had financial and consulting relationships with disability insurance companies, advising them that rehabilitative therapies like those tested in PACE could help ME/CFS claimants get off benefits and back to work. They disclosed these insurance industry links in The Lancet but did not inform trial participants, contrary to their protocol commitment. This serious ethical breach raises concerns about whether the consent obtained from the 641 trial participants is legitimate.

Such flaws have no place in published research. This is of particular concern in the case of the PACE trial because of its significant impact on government policy, public health practice, clinical care, and decisions about disability insurance and other social benefits. Under the circumstances, it is incumbent upon The Lancet to address this matter as soon as possible.

We therefore urge The Lancet to seek an independent re-analysis of the individual-level PACE trial data, with appropriate sensitivity analyses, from highly respected reviewers with extensive expertise in statistics and study design. The reviewers should be from outside the U.K. and outside the domains of psychiatry and psychological medicine. They should also be completely independent of, and have no conflicts of interests involving, the PACE investigators and the funders of the trial.

Thank you very much for your quick attention to this matter.

Sincerely,

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University

Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London

Leonard A. Jason, PhD
Professor of Psychology
DePaul University

Bruce Levin, PhD
Professor of Biostatistics
Columbia University

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University

Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley