Trial By Error, Continued: The Dutch Studies (Again!), and an Esther Crawley Bonus

By David Tuller, DrPH

David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.

Wow, the research from the CBT/GET crowd in The Netherlands never ceases to amaze. Like the work of their friends in the U.K., each study comes up with new ways to be bad. It’s almost too easy to poke holes in these things. And yet the investigators appear unable to restrain themselves from making extremely generous over-interpretations of their findings–interpretations that cannot withstand serious scrutiny. The investigators always conclude, no matter what, that cognitive and/or behavioral therapies are effective for treating the disease they usually call chronic fatigue syndrome.

That this so-called science manages to get through peer review is astonishing. That is, unless we assume the studies are all peer-reviewed by other investigators who share the authors’ “unhelpful beliefs” and “dysfunctional cognitions” about ME/CFS and the curative powers of cognitive behavior therapy and graded exercise therapy.

Let’s take a quick look at yet another Dutch study of CBT for adolescents, a 2004 trial published in the BMJ. This one offers a superb example of over-interpretation. The small trial, with 71 participants, had two arms. One group received ten sessions of CBT over five months. The other received…a place on a waiting list for treatment. That’s right–they got nothing. Guess what? Those who got something did better on subjective measures at five months than those who got nothing. The investigators’ definitive conclusion: CBT is an effective treatment for sick teens.

I mean, WTF? It’s not hard to figure out that, you know, offering people some treatment is more likely to produce positive responses to subjective questions than offering them a place on a waiting list. That banal insight must be right in the first chapter of Psychological Research for Dummies. Aren’t these investigators presenting themselves as authorities on human behavior? Have they heard of something called the placebo effect?

Here’s what this BMJ study proved: Ten sessions of something lead to more reports of short-term benefits than no sessions of anything. But ten sessions of what? Maybe ten sessions of poker-playing or ten sessions of watching Seinfeld reruns while holding hands with the therapist and singing “The Girl from Ipanema” in falsetto would have produced the same results. Who knows? To flatly declare that their findings prove that CBT is an effective treatment—without caveats or an iota of caution—is a huge and unacceptable interpretive leap. The paper should never have been published in this form. It’s ridiculous to take this study as some kind of solid “evidence” for CBT.

But from the perspective of the Dutch research group, this waiting-list strategy apparently worked so well that they used it again for a 2015 study of group CBT for chronic fatigue syndrome. In this study, providing CBT in groups of four or eight patients worked significantly better than placing patients on a waiting list and providing them with absolutely nothing. Of course, no one could possibly take these findings to mean that group CBT specifically is an effective treatment—except they did.

When I’m reading this stuff I sometimes feel like I’m going out of my mind. Do I really have to pick through every one of these papers to point out flaws that a first-year epidemiology student could spot?

One big issue here is how these folks piggy-back one bad study on top of another to build what appears to be a robust body of research but is, in fact, a house of cards. When you expose the cracks in the foundational studies, the whole edifice comes tumbling down. A case in point: a 2007 Dutch study that explored the effect of CBT on “self-reported cognitive impairments and neuropsychological test performance.” Using data from two earlier studies, the investigators concluded that CBT reduced self-reported cognitive impairment but did not improve neuropsychological test performance.

Which studies was this 2007 study based on? Well, one of them was the very problematic 2004 study I have just discussed–the one that found CBT effective when compared to nothing. The other was the 2001 study in The Lancet that I wrote about in my last post. As I noted, this Lancet study claimed to be using the CDC criteria for chronic fatigue syndrome, but then waived the requirement that patients have four other symptoms besides fatigue. So it was, in effect, a study of a heterogeneous group of people suffering from at least six months of fatigue.

This case definition—six months of fatigue, with no other symptoms necessary—was used in the PACE trial and is known as the Oxford criteria. It has been discredited because it generates heterogeneous populations of people suffering from a variety of fatiguing illnesses. The results of Oxford criteria studies cannot be extrapolated to those with ME/CFS.

The 2007 study relies on the accuracy and validity of the two studies whose data it incorporates. Since those earlier studies violated basic understandings of scientific analysis, the new study is also bogus and cannot be taken seriously.

The PACE authors themselves have perfected this strategy of generating new bad papers by stacking up earlier bad ones. In November, Trudie Chalder demonstrated her personal flair for this technique as co-author of a systematic review of “attentional and interpretive bias towards illness-related information in chronic fatigue syndrome.” The authors’ conclusion: “Cognitive processing biases may maintain illness beliefs and symptoms in people with CFS.” The proposed solution to that would obviously be some sessions of CBT to correct those pesky cognitive processing biases.

Among other problems, Dr. Chalder and her co-authors included data from Oxford criteria studies. By including in the mix these heterogeneous samples of people suffering from chronic fatigue, Dr. Chalder and her colleagues have invalidated their claim that it is a study of the illness known as chronic fatigue syndrome. Of course, Psychological Medicine, which published this new research gem, is the journal that published—and has consistently refused to correct–the PACE “recovery” paper in which participants could get worse but still meet “recovery” thresholds.

The Dutch branch of the CBT/GET ideological brigade has been centered at Radboud University Nijmegen, home base for many years of two of the movement’s leading lights: Dr. Gijs Bleijenberg and Dr. Hans Knoop. Dr. Knoop recently moved to the University of Amsterdam and is currently a co-investigator of FITNET-NHS with Esther Crawley. Dr. Bleijenberg, on the occasion of his own retirement a few years ago, had this to say about his longtime friend and colleague, PACE investigator Michael Sharpe: “Dear Mike, we know each other nearly 20 years. You have inspired me very much in the way you treated CFS. Thanks a lot!”

Indeed. Dr. Bleijenberg and his Dutch colleagues appear to have learned a great deal from their PACE besties. Dr. Bleijenberg and Dr. Knoop demonstrated their own nimble use of language in the 2011 commentary in The Lancet that accompanied the publication of the first PACE results. I discussed this deceptive commentary at length in a post last year, so I won’t regurgitate the whole sorry argument here. But the Dutch investigators themselves are well aware that their claim that thirty percent of PACE participants met a “strict criterion” for recovery is preposterous.

How do I know that Dr. Bleijenberg and Dr. Knoop know this? Because as I documented in last year’s post, claims in the 2011 commentary contradict and ignore statements they themselves made in a 2007 paper that posed this question: “Is a full recovery possible after cognitive behavioural therapy for chronic fatigue syndrome?” (The answer, of course, was yes. Peter White, the lead PACE investigator, was a co-author of the 2007 paper.) Moreover, Dr. Bleijenberg and Dr. Knoop certainly know that the “strict criterion” they touted included thresholds that some participants had already met at baseline—yet they have still refused to correct this statement.

Given that all of these studies present serious methodological concerns, the Dutch Health Council panel considering the science of ME/CFS should be very, very wary of using them to formulate recommendations. The panel should understand that, within the next few months, peer-reviewed analyses of the original PACE data are likely to be published. (Two such analyses—one by the PACE authors themselves, one by an independent group of patients and academic statisticians–have already been published online, without peer review.) The upcoming papers will demonstrate conclusively that the “benefits” reported by the PACE team were mostly or completely illusory—and were obtained only by methodological anomalies like dramatic and unacceptable changes in outcome measures.

In an open letter to The Lancet posted on Virology Blog last February, dozens of prominent scientists and clinicians condemned the PACE study and its conclusions in harsh terms. In the U.K., the First-Tier Tribunal cited this worldwide dismay about the trial’s egregious lapses while demolishing the PACE authors’ excuses for withholding their data. The studies from the Radboud University crowd and their compatriots all rest on the same silly, unproven hypotheses of dysfunctional thinking, fear of activity, and deconditioning, and are just as intellectually incoherent and dishonest.

Should the Health Council produce a report recommending cognitive and behavioral treatments based on this laughable body of “research,” the organization could become an international joke and suffer enormous long-term reputational damage. The entire PACE paradigm is undergoing a very public unraveling. Everyone can now see what patients have seen for years. Meanwhile, biomedical researchers in the U.S., Norway, and elsewhere are narrowing in on the actual pathophysiology underlying ME/CFS.

It would be a shame to see the Dutch marching backwards to embrace scientific illiteracy and adopt an “Earth-is-flat” approach to reality.

*****

And for a special bonus, let’s now take another quick peek at Dr. Crawley’s work. Someone recently e-mailed me a photo of a poster presentation by Dr. Crawley and three colleagues. This poster was shown at the inaugural conference of the U.K. CFS/ME Research Collaborative, or CMRC, held in 2014. The poster was based on information from the same dataset used for Dr. Crawley’s recent Pediatrics study. As I pointed out two posts ago, that flawed study claimed a surprisingly high prevalence of 2 % among adolescents—a figure that drew widespread attention in media reports.

Dr. Crawley has cited high prevalence estimates to argue for more research into and treatment with CBT and GET. And if these prevalence rates were real, that might make sense. However, as I noted, her method of identifying the illness was specious—she decided, without justification or explanation, that she could diagnose chronic fatigue syndrome through parental and child reports of chronic fatigue, and without information from clinical examinations. In fact, after those who appeared to have high levels of depression were removed, the prevalence fell to 0.6 %–although this lower figure is not the one Dr. Crawley has emphasized.

Despite the high prevalence, however, the same dataset showed that adolescents suffering from the illness generally got better without any treatment at all, according to the 2014 poster presentation. Here’s the poster’s conclusion: “Persistent CFS/ME is rare in teenagers and most teenagers not seen in a clinical service will recovery spontaneously.”

Isn’t that great? Why haven’t I seen these hopeful data before? Although the poster predated this year’s Pediatrics paper, the data about very high rates of spontaneous recovery did not make it into that prevalence study. Moreover, the FITNET-NHS protocol and the recruitment leaflet highlight the claim that few adolescents will recover at six months without “specialist treatment” but most will recover if they receive it. Unmentioned is the highly salient fact that this “specialist treatment” apparently makes no long-term difference.

In reality, the adolescents who recovered spontaneously most likely were not suffering from ME/CFS in the first place. Dr. Crawley certainly hasn’t provided sufficient evidence that any of the children in the database she used actually had it, despite her insistence on using the term. Most likely, some unknown number of those identified as having chronic fatigue syndrome in the Pediatrics paper and in the poster presentation did have ME/CFS. But many or most were experiencing what could only be called a bout of chronic fatigue, for unknown reasons.

It is disappointing that Dr. Crawley did not include the spontaneous recovery rate in the Pediatrics paper or in the FITNET-NHS protocol. In fact, as far as I can tell, these optimistic findings have not been published anywhere. I don’t know the rationale for this decision to withhold rather than publish substantive information. Perhaps the calculation is that public reports of high rates of spontaneous recovery would undermine the arguments for ever-more funding to study CBT and GET? Just a guess, of course.

(Esther–Forgive me if I’m mistaken about whether these data have been published somewhere. I have only seen this information in your poster for the inaugural CMRC conference in 2014. If the data have been peer-reviewed and published, I stand corrected on that point and applaud your integrity.)

Trial By Error, Continued: A Follow-Up Post on FITNET-NHS

By David Tuller, DrPH

David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.

Last week’s post on FITNET-NHS and Esther Crawley stirred up a lot of interest. I guess people get upset when researchers cite shoddy “evidence” from poorly designed trials to justify foisting psychological treatments on kids with a physiological disease. I wanted to post some additional bits and pieces related to the issue.

*****

I sent Dr. Crawley a link to last week’s post, offering her an opportunity to send her response to Dr. Racaniello for posting on Virology Blog, along with my response to her response. So far, Dr. Racaniello and I haven’t heard back—I doubt we will. Maybe she feels more comfortable misrepresenting facts in trial protocols and radio interviews than in addressing the legitimate concerns raised by patients and confronting the methodological flaws in her research. I hope Dr. Crawley knows she will always have a place on Virology Blog to present her perspective, should she choose to exercise that option. (Esther, are you reading this?)

*****

From reading the research of the CBT/GET/PACE crowd, I get the impression they are all in the habit of peer-reviewing and supporting each others’ work. I make that assumption because it is hard to imagine that independent scientists not affiliated with this group would overlook all the obvious problems that mar their studies—like outcome measures that represent worse health than entry criteria, as in the PACE trial itself. So it’s not surprising to learn that one of the three principal PACE investigators, psychiatrist Michael Sharpe, was on the committee that reviewed—and approved—Dr. Crawley’s one-million-pound FITNET-NHS study.

FITNET-NHS is being funded by the U.K.’s National Institute for Health Research. I have no idea what role, if any, Dr. Sharpe played in pushing through Dr. Crawley’s grant, but it likely didn’t hurt that the FITNET-NHS protocol cited PACE favorably while failing to point out that it has been rejected as fatally flawed by dozens of distinguished scientists and clinicians. Of course, the protocol also failed to point out that the reanalyses of the trial data have shown that the findings published by the PACE authors were much better than the results using the methods they promised in their protocol. (More on the reanalyses below.) And as I noted in my previous post, the FITNET-NHS protocol also misstated the NICE guidelines for chronic fatigue syndrome, making post-exertional malaise an optional symptom rather than a required component—thus conflating chronic fatigue and chronic fatigue syndrome, just as the PACE authors did by using the overly broad Oxford criteria.

The FITNET-NHS proposal also didn’t note some similarities between PACE and the Dutch FITNET trial on which it is based. Like the PACE trial, the Dutch relied on a post-hoc definition of “recovery.” The thresholds the FITNET investigators selected after they saw the results were pretty lax, which certainly made it easier to find that participants had attained “recovery.” Also like the PACE trial, the Dutch participants in the comparison group ended up in the same place as the intervention group at long-term follow-up. Just as the CBT and GET in PACE offered no extended advantages, the same was true of the online CBT provided in FITNET.

And again like the PACE authors, the FITNET investigators downplayed these null findings in their follow-up paper. In a clinical trial, the primary results are supposed to be comparisons between the groups. Yet in the follow-up PACE and FITNET articles, both teams highlighted the “within-group” comparisons. That is, they treated the fact that there were no long-term differences between the groups as an afterthought and boasted instead that the intervention groups sustained the progress they initially made. That might be an interesting sub-finding, but to present “within-group” results as a clinical trial’s main outcome is highly disingenuous.

*****

As part of her media blitz for the FITNET-NHS launch, Dr. Crawley was interviewed on a BBC radio program by a colleague, Dr. Phil Hammond. In this interview, she made some statements that demonstrate one of two things: Either she doesn’t know what she’s talking about and her misrepresentations are genuine mistakes, or she’s lying. So either she’s incompetent, or she lacks integrity. Not a great choice.

Let’s parse what she said about the fact that, at long-term follow-up, there were no apparent differences between the intervention and the comparison groups in the Dutch FITNET study. Here’s her comment:

“Oh, people have really made a mistake on this,” said Dr. Crawley. “So, in the FITNET Trial, they were offered FITNET or usual care for six months, and then if they didn’t make a recovery in the usual care, they were offered FITNET again, and they were then followed up at 2 to 3 years, so of course what happened is that a lot of the children who were in the original control arm, then got FITNET as well, so it’s not surprising that at 2 or 3 years, the results were similar.”

This is simply not an accurate description. As Dr. Crawley must know, some of the Dutch FITNET participants in the “usual care” comparison group went on to receive FITNET, and others didn’t. Both sets of usual care participants—not just those who received FITNET—caught up to the original FITNET group. For Dr. Crawley to suggest that the reason the others caught up was that they received FITNET is, perhaps, an unfortunate mistake. Or else it’s a deliberate untruth.

*****

Another example from the BBC radio interview: Dr. Crawley’s inaccurate description of the two reanalyses of the raw trial data from the PACE study. Here’s what she said:

“First of all they did a reanalysis of recovery based on what the authors originally said they were going to do, and that reanalysis done by the authors is entirely consistent with their original results. [Actually, Dr. Crawley is mistaken here; the PACE authors did a reanalysis of “improvement,” not of “recovery”]…Then the people that did the reanalysis did it again, using a different definition of recovery, that was much much harder to reach–and the trial just wasn’t big enough to show a difference, and they didn’t show a difference. [Here, Dr. Crawley is talking about the reanalysis done by patients and academic statisticians.] Now, you know, you can pick and choose how you redefine recovery, and that’s all very important research, but the message from the PACE Trial is not contested; the message is, if you want to get better, you’re much more likely to get better if you get specialist treatment.”

This statement is at serious odds with the facts. Let’s recap: In reporting their findings in The Lancet in 2011, the PACE authors presented “improvement” results for the two primary outcomes of fatigue and physical function. They reported that about 60 percent of participants in the CBT and GET arms reached the selected thresholds for “improvement” on both measures. In a 2013 paper in the journal Psychological Medicine, they presented “recovery” results based on a composite “recovery” definition that included the two primary outcomes and two additional measures. In this paper, they reported “recovery” rates for the favored intervention groups of 22 percent.

Using the raw trial data that the court ordered them to release earlier this year, the PACE authors themselves reanalyzed the Lancet improvement findings, based on their own initial, more stringent definition of “improvement” in the protocol. In this analysis, the authors reported that only about 20 percent “improved” on both measures, using the methods for assessing “improvement” outlined in the protocol. In other words, only a third as many “improved,” according to the authors’ own original definition, compared to the 60 percent they reported in The Lancet. Moreover, in the reanalysis, ten percent “improved” in the comparison group, meaning that CBT and GET led to “improvements” in only one in ten participants—a pretty sad result for a five-million-pound trial.

However, because these meager findings were statistically significant, the PACE authors and their followers have, amazingly, trumpeted them as supporting their initial claims. In reality, the new “improvement” findings demonstrate that any “benefits” offered by CBT and GET are marginal. It is preposterous and insulting to proclaim, as the PACE authors and Dr. Crawley have, that this represents confirmation of the results reported in The Lancet. Dr. Crawley’s statement that “the message from the PACE trial is not contested” is of course nonsense. The PACE “message” has been exposed as bullshit—and everyone knows it.

The PACE authors did not present their own reanalysis of the “recovery” findings—probably because those turned out to be null, as was shown in a reanalysis of that data by patients and academic statisticians, published on Virology Blog. That reanalysis found single-digit “recovery” rates for all the study arms, and no statistically significant differences between the groups. Dr. Crawley declared in the radio interview that this reanalysis used “a different definition of recovery, that was much harder to reach.” And she acknowledged that the reanalysis “didn’t show a difference”—but she blamed this on the fact that the PACE trial wasn’t big enough, even though it was the largest trial ever of treatments for ME/CFS.

This reasoning is specious. Dr. Crawley is ignoring the central point: The “recovery” reanalysis was based on the authors’ own protocol definition of “recovery,” not some arbitrarily harsh criteria created by outside agitators opposed to the trial. The PACE authors themselves had an obligation to provide the findings they promised in their protocol; after all, that’s the basis on which they received funding and ethical permission to proceed with the trial.

It is certainly understandable why they, and Dr. Crawley, prefer the manipulated and false “recovery” data published in Psychological Medicine. But deciding post-hoc to use weaker outcome measures and then refuse to provide your original results is not science. That’s data manipulation. And if this outcome-switching is done with the intent to hide poor results in favor of better ones, it is considered scientific misconduct.

*****

I also want to say a few words about the leaflet promoting FITNET-NHS. The leaflet states that most patients “recover” with “specialist treatment” and less than ten percent “recover” from standard care. Then it announces that this “specialist treatment” is available through the trial—implicitly promising that most of those who get the therapy will be cured.

This is problematic for a host of reasons. As I pointed out in my previous post, any claims that the Dutch FITNET trial, the basis for Dr. Crawley’s study, led to “recovery” must be presented with great caution and caveats. Instead, the leaflet presents such “recovery” as an uncontested fact. Also, the whole point of clinical trials is to find out if treatments work—in this case, whether the online CBT approach is effective, as well as cost-effective. But the leaflet is essentially announcing the result–“recovery”—before the trial even starts. If Dr. Crawley is so sure that this treatment is effective in leading to “recovery,” why is she doing the trial in the first place? And if she’s not sure what the results will be, why is she promising “recovery”?

Finally, as has been pointed out many times, the PACE investigators, Dr. Crawley and their Dutch colleagues all appear to believe that they can claim “recovery” based solely on subjective measures. Certainly any definition of “recovery” should require that participants can perform physically at their pre-sickness level. However, the Dutch researchers refused to release the one set of data—how much participants moved, as assessed by ankle monitors called actometers–that would have proven that the kids in FITNET had “recovered” on an objective measure of physical performance. The refusal to publish this data is telling, and leaves room for only one interpretation: The Dutch data showed that participants did no better than before the trial, or perhaps even worse, on this measure of physical movement.

This FITNET-NHS leaflet should be withdrawn because of its deceptive approach to promoting the chances of “recovery” in Dr. Crawley’s study. I hope the advertising regulators in the U.K. take a look at this leaflet and assess whether it accurately represents the facts.

*****

As long as we’re talking about the Dutch members of the CBT/GET ideological movement, let’s also look briefly at another piece of flawed research from that group. Like the PACE authors and Dr. Crawley, these investigators have found ways to mix up those with chronic fatigue and those with chronic fatigue syndrome. A case in point is a 2001 study that has been cited in systematic reviews as evidence for the effectiveness of CBT in this patient population. (Dr. Bleijenberg, a co-investigator on the FITNET-NHS trial, was also a co-author of this study.)

In this 2001 study, published in The Lancet (of course!), the Dutch researchers described their case definition for identifying participants like this: “Patients were eligible for the study if they met the US Centers for Disease Control and Prevention criteria for CFS, with the exception of the criterion requiring four of eight additional symptoms to be present.”

This statement is incoherent. (Why do I need to keep using words like “incoherent” and “preposterous” when describing this body of research?) The CDC definition has two main components: 1) six months of unexplained fatigue, and 2) four of eight other symptoms. If you abandon the second component, you can no longer refer to this as meeting the CDC definition. All you’re left with is the requirement that participants have suffered from six months of fatigue.

And that, of course, is the case definition known as the Oxford criteria, developed by PACE investigator Michael Sharpe in the 1990s. And as last year’s seminal report from the U.S. National Institutes of Health suggested, this case definition is so broad that it scoops up many people with fatiguing illnesses who do not have the disease known as ME/CFS. According to the NIH report, the Oxford criteria can “impair progress and cause harm,” and should therefore be “retired” from use. The reason is that any results could not accurately be extrapolated to people with ME/CFS specifically. This is especially so for treatments, such as CBT and GET, that are likely to be effective for many people suffering from other fatiguing illnesses.

In short, to cite any findings from such studies as evidence for treatments for ME/CFS is unscientific and completely unjustified. The 2001 Dutch study might be an excellent look at the use of CBT for chronic fatigue*. But like FITNET-NHS, it is not a legitimate study of people with chronic fatigue syndrome, and the Dutch Health Council should acknowledge this fact in its current deliberations about the illness.

*In the original phrasing, I referred to the intervention mistakenly as ‘online CBT.’

Trial By Error, Continued: The New FITNET Trial for Kids

By David Tuller, DrPH

David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.

The past year has been a disaster for proponents of the PACE trial. They have faced growing international resistance to their exaggerated claims that cognitive behavior therapy and graded exercise therapy are effective treatments for chronic fatigue syndrome, also known as ME/CFS. The recent court-ordered release of key trial data has confirmed what was long self-evident: The PACE authors weakened their outcome criteria mid-stream in ways that allowed them to report dramatically better results for “improvement” (in The Lancet in 2011) and “recovery” (in Psychological Medicine in 2013). In refusing to provide the findings per the original protocol methods, or statistical analyses assessing the impact of the many mid-trial changes, or their actual trial data, they were able to hide their disastrous results for five years.

Yet the PACE authors and their allies continue, astonishingly, to defend the indefensible study, cite its findings approvingly, and push forward with ever more research into behavioral and cognitive interventions. The latest case in point: Esther Crawley, a British pediatrician and a highly controversial figure in the ME/CFS community because of her longtime promotion of the CBT/GET approach. On November 1st, the Science Media Centre in London held a press briefing to tout Dr. Crawley’s current venture—FITNET-NHS, a one-million-pound study of online CBT that is now recruiting and seeks to enroll more than 700 adolescents.

Dr. Crawley is a professor of child health at the University of Bristol. She is also currently recruiting for the MAGENTA study of graded exercise therapy for children with the illness. She is a lead player in the U.K. CFS/ME Research Collaborative, an umbrella organization that is sponsoring an ambitious Big Data effort called MEGA, now in the planning stages. While patients and advocates are desperate for the kind of top-notch biomedical and genetic research being proposed, many oppose MEGA precisely because of the involvement of Dr. Crawley and Peter White, the lead PACE investigator. (Dr. White is reportedly no longer involved in MEGA; Dr. Crawley still definitely is.)

The rationale for FITNET-NHS is that many ME/CFS patients live too far from specialists to obtain adequate care. Therefore, CBT delivered through an online portal, along with e-mail communication with a therapist, could potentially provide a convenient answer for those in such circumstances. The SMC press briefing generated widespread and enthusiastic news coverage. The BBC’s breathless online report about the “landmark” study noted that the online CBT “successfully treats two-thirds of children with chronic fatigue syndrome.” According to the BBC story, the intervention was designed “to change the way the children think of the disease.”

The BBC story and other news reports did not mention that the PACE trial–a foundational piece of evidence for the claim that changing people’s thoughts about the disease is the best way to treat it—has been publicly exposed as nonsense and is the subject of a roiling worldwide scientific debate. The stories also didn’t mention a more recent paper by the authors of the 2012 study from the Netherlands that was the source of the BBC’s claim of a “two-thirds” success rate.

The 2012 study, a Dutch version of FITNET, was published in The Lancet. (Why is The Lancet always involved?) In a subsequent paper published in Pediatrics in 2013, the Dutch team reported no differences among their trial participants at long-term follow-up. In other words, as with the PACE trial itself, any apparent advantages conferred by the investigators’ preferred treatment disappeared after the study was over. (More on the Dutch study below.)

The SMC, a purportedly neutral arbiter of science, actually functions as a cheerleader for research about cognitive and behavioral treatments for ME/CFS. Simon Wessely, a founder of the CBT/GET treatment paradigm and a close colleague of the PACE authors, is on the SMC’s board of trustees. The journalist who wrote the BBC story, James Gallagher, sits on the SMC’s advisory committee, so the reporting wasn’t exactly conducted at arm’s length. This reportorial conflict-of-interest was not disclosed in the BBC story itself.

(In fact, the Countess of Mar, a member of the House of Lords and a longtime advocate for ME/CFS patients, has filed a formal complaint with the BBC to protest its biased reporting on FITNET-NHS. In her complaint, she noted that “the BBC coverage was so hyperbolic and it afforded the FITNET trial so much publicity that it was clearly organised as a counter-punch to the anti-PACE evidence which is now gaining world-wide attention.”)

As a treatment for chronic fatigue syndrome, cognitive behavior therapy is grounded in an unproven hypothesis. According to the theory, the cause of patients’ continuing symptoms is a vicious downward spiral generated by false illness beliefs, a fear of engaging in activity, and progressive deconditioning. Whatever the initial viral or other illness that might have triggered the syndrome, patients are presumed to be currently free of any organic disease. Changing their beliefs through CBT, per the theory, will help encourage them to increase their levels of activity and resume their normal lives.

Here’s the rationale for the treatment from the PACE study itself: “CBT was done on the basis of the fear avoidance theory of chronic fatigue syndrome. This theory regards chronic fatigue syndrome as being reversible and that cognitive responses (fear of engaging in activity) and behavioural responses (avoidance of activity) are linked and interact with physiological processes to perpetuate fatigue. The aim of treatment was to change the behavioural and cognitive factors assumed to be responsible for perpetuation of the participant’s symptoms and disability. Therapeutic strategies guided participants to address unhelpful cognitions, including fears about symptoms or activity by testing them in behavioural experiments.”

The goal of this specific form of CBT, therefore, is to reverse the “reversible” illness by helping patients abandon their “unhelpful” beliefs of having a medical disease. This is definitely not the goal of CBT when it is used to help people cope with cancer, Parkinson’s, or other illnesses—no one claims those diseases are “reversible.” That the PACE authors, Dr. Crawley, and their Dutch colleagues promote CBT as a curative treatment and not simply a management or adaptive strategy is clear from their insistence on using the word “recovery”—a term that has no well-defined or universally understood meaning when it comes to this illness but has a very clear meaning to the general public.

While PACE so far remains in the literature, the study has been rejected by dozens of leading clinicians and academics, in the U.S. and elsewhere. Last February, an open letter to The Lancet signed by 42 experts and posted on Virology Blog condemned its egregious flaws, noting that they “have no place in published research.” The study has even been presented as a case study of bad science in graduate epidemiology seminars and at major scientific gatherings.

*****

Like the work of the PACE authors, Dr. Crawley’s research is fraught with misrepresentations and methodological problems. Like them, she routinely conflates the common symptom of chronic fatigue with the illness called chronic fatigue syndrome—a serious error with potentially harmful consequences. (I will mostly use chronic fatigue syndrome in describing the research because that is the term they use.)

Dr. Crawley favors subjective over objective outcomes. In PACE, of course, the objective measures–like a walking test, a step-test for fitness, and employment status—all failed to demonstrate “recovery” or reflect the reported improvements in the two primary subjective outcomes of physical function and fatigue. FITNET-NHS doesn’t even bother with such measures. The primary outcome is a self-report questionnaire assessing physical function, and almost all the secondary outcomes are also subjective.

This is particularly troubling because FITNET-NHS, like PACE, is non-blinded; that is, both participants and investigators know which intervention they are receiving. Non-blinded studies with subjective outcomes are notoriously vulnerable to bias—even more when the intervention itself involves telling participants that the treatment will make them better, as is the case with the kind of cognitive behavior therapy provided for ME/CFS patients.

The FITNET-NHS study protocol states that participants will be identified using the guidelines developed by NICE—the U.K.’s National Institute for Health and Care Excellence. The protocol describes the NICE guidelines as requiring three months of fatigue, plus one or more of nine additional symptoms: post-exertional malaise, difficulty sleeping, cognitive dysfunction, muscle and/or joint pain, headaches, painful lymph nodes, general malaise, dizziness and/or nausea, or palpitations. In other words, according to the protocol, post-exertional malaise is not required to participate in FITNET-NHS; it is clearly identified as an optional symptom. (In the U.K., the illness can be diagnosed at three months in children, rather than at six months.)

But the proposal’s claim to be following the NICE guidelines does not appear to be true. In the NICE guidelines, post-exertional malaise is not an optional symptom. It is required, as an essential element of the fatigue itself. (In addition, one or more of ten other symptoms must also be present.) To repeat: post-exertional malaise is required in the NICE guidelines, but is not required in the description of the NICE guidelines provided in the FITNET-NHS protocol.

By making this subtle but significant shift—a sleight-of-guideline, so to speak—Dr. Crawley and her colleagues have quietly transformed their prospective cohort from one in which post-exertional malaise is a cardinal characteristic of the illness to one in which it might or might not be present. And they have done this while still claiming–inaccurately–to follow NICE guidelines. As currently described, however, Dr. Crawley’s new study is NOT a study of chronic fatigue syndrome, as she maintains, but of chronic fatigue.

As a result, the actual study participants, like the PACE cohort, will likely be a heterogeneous grab bag of kids suffering from fatigue for any number of reasons, including depression–a common cause of exhaustion and a condition that often responds to psychotherapeutic interventions like CBT. Some or even many participants—an unknown number—will likely be genuine ME/CFS patients. Yet the results will be applied to ALL adolescents identified as having that illness. Since those who actually have it suffer from the required symptom of post-exertional malaise, an intervention that encourages them to increase their activity levels, like CBT, could potentially cause harm.

(I suppose it’s possible the FITNET-NHS protocol’s inaccurate description of the role of post-exertional malaise in the NICE guidelines was inadvertent, a case of sloppiness. If so, it would be an extraordinary oversight, given the number of people involved in the study and the enormous implications of the switch. It is curious that this obvious and jarring discrepancy between the NICE guidelines and the FITNESS-NHS description of them was not flagged during the review process, since it is easy to check whether the protocol language accurately reflects the recommendations.)

Yet Dr. Crawley is experienced at this blurring of categories–she did the same in a study she co-authored in the journal Pediatrics, in January of this year. In the study, “Chronic Fatigue Syndrome at Age 16 Years,” she and colleagues reported that almost one in fifty adolescents suffered from the illness—an extremely high rate that attracted widespread media attention. The main conclusion was described like this: “CFS affected 1.9% of 16-year-olds in a UK birth cohort and was positively associated with higher family adversity.”

However, the Pediatrics study is unreliable as a measure of “chronic fatigue syndrome.” It is of note that this paper, like the FITNET-NHS protocol, also appears to have inaccurately presented the NICE guidelines. According to the Pediatrics paper, NICE calls for a CFS diagnosis after three months of “persistent
or recurrent fatigue that is not
the result of ongoing exertion, not substantially alleviated by rest, has resulted in a substantial reduction of activities, and has no known cause.” But this description is incomplete–it omits the NICE requirement that the fatigue must include the specific characteristic of post-exertional malaise in order to render a diagnosis of chronic fatigue syndrome.

In the Pediatrics paper, the determination of illness was based not on clinical examination but on parental reports of children’s unexplained fatigue. In a previous study of 13-year-olds that relied on the same U.K. database, Dr. Crawley and her co-authors referred to the endpoint—appropriately–as “disabling chronic fatigue.” But in this study, they justified changing the endpoint to “chronic fatigue syndrome” by noting that they cross-referenced the parental reports with children’s self-reports of their own fatigue.

Here’s how they explained this shift in nomenclature: “In the earlier study, we were unable to confirm a diagnosis of CFS because we had only parental report of fatigue; hence, chronic disabling fatigue was defined as the study outcome. In the present study, parental and child report of fatigue were combined to identify adolescents with CFS.”

This reasoning is incoherent. A child’s confirmation of a parental report of fatigue cannot be taken to indicate the presence of chronic fatigue syndrome–especially without a clinical examination to rule out other possible conditions. Moreover, neither the parental nor child reports appear to have included information about post-exertional malaise, which is required for a diagnosis of chronic fatigue syndrome—even though the Pediatrics study did not mention this requirement in its description of the NICE guidelines. In fact, the authors provided no evidence or data to support their assumption that a double-report of fatigue equaled a case of chronic fatigue syndrome. (How’d that assumption ever pass peer review, anyway?)

Moreover, the study itself acknowledged that, when those found to be suffering from high levels of depression were removed, the prevalence of what the investigators called chronic fatigue syndrome was only 0.6 %. And since depression is likely to be highly correlated with chronic fatigue as well as with family adversity, it is not surprising that the study found the apparent association between family adversity and chronic fatigue syndrome that the investigators highlighted in their conclusion. That misinterpretation of their data has likely lent support to the widespread but inaccurate belief that the illness is largely or even partly psychiatric in nature.

In any event, the figure of 0.6 % should have been identified as the prevalence of “chronic disabling fatigue, not attributable to high levels of depression.” Without any further clinical data, to identify either 1.9 % or 0.6 % as the prevalence of chronic fatigue syndrome was unwarranted and irresponsible. Although the authors cited the lack of clinical diagnosis as a limitation, this acknowledgement does not excuse their interpretive leap. To call this a study of chronic fatigue syndrome is really misleading–a serious over-interpretation of the data.

In subsequent correspondence, three professors of pediatrics—Marvin Medow and Julian Stewart from New York Medical College, and Peter Rowe from Johns Hopkins–scolded the study authors for identifying the participants as having chronic fatigue syndrome rather than chronic fatigue. They cited this misclassification as the likely source of the reported link between chronic fatigue syndrome and family adversity. In particular, they challenged diagnoses made without benefit of clinical evaluations.

“An important component of the diagnosis is a physician’s history and physical examination to exclude conditions that could explain the fatigue, including hypothyroidism, heart disease, cancer, liver failure, covert drug abuse, medication side effects, gastrointestinal/nutritional, infectious and psychiatric conditions,” they wrote. The Pediatrics paper, concluded the three pediatricians, “should be titled ‘Chronic Fatigue but not Chronic Fatigue Syndrome at Age 16 Years.’”

In response, the study authors agreed that clinical diagnoses would be more accurate. But they did not address the critical issue of why they decided that two reports of chronic fatigue could be used to identify chronic fatigue syndrome.

*****

The conflation of chronic fatigue and chronic fatigue syndrome is a huge problem in ME/CFS research. That’s why a major report last year from the National Institutes of Health declared that the case definition used in PACE—which required only six months of unexplained fatigue but no other symptoms–could “impair progress and cause harm,” and should be “retired” from use. But Dr. Crawley and her colleagues do not seem to have gotten the message.

At the SMC press briefing presenting FITNET-NHS, one of the experts appearing with Dr. Crawley was Dr. Stephen Holgate, the leader of the CFS/ME Research Collaborative and a professor of immunopharmacology at the University of Southampton. According to the BBC report, he praised the new trial as “high-quality research.” This endorsement suggests that Dr. Holgate, like Dr. Crawley, does not appreciate the significance of the distinction between the symptom of chronic fatigue and the illness called chronic fatigue syndrome—a troubling blind spot. It also suggests that Dr. Holgate is unaware or unconcerned that the main support for the use of CBT in this illness, the PACE trial, has been discredited.

Also at the SMC briefing was Paul McCrone, a professor of health economics from King’s College London and a PACE co-author. Dr. McCrone is serving as the chair of FITNET-NHS’ independent steering committee–another unsettling sign. As I have documented on Virology Blog, Dr. McCrone made false claims as lead author in a 2012 PLoS One paper—and those false claims allowed the PACE authors to declare that CBT and GET were cost-effective. They have routinely cited this fraudulent finding in promoting the therapies.

Beyond the problem of conflating “chronic fatigue” and “chronic fatigue syndrome,” Dr. Crawley’s reliance on the Dutch trial suggests that this previous FITNET study warrants a closer look—especially since the BBC and other news outlets cited its robust claims of success in extolling the U.K. version.

The approach to CBT in the Dutch FITNET trial reflects that in the U.K. Of the online intervention’s 21 modules, according to the protocol for the Dutch study, fourteen “focus on cognitive behavioural strategies and include instructions and exercises on how to identify, challenge and change cognitive processes that contribute to CFS.” Of course, experts outside the CBT/GET/PACE bubble understand that ME/CFS is a physiological disease and that faulty “cognitive processes” have nothing to do with perpetuating or contributing to it.

The Dutch study found that those assigned to FITNET reported less fatigue, greater physical function, and greater school attendance than those in the comparison group, who received standard treatment–referred to as “usual care.” And using a composite definition of “recovery,” the study reported that 63% of those in the FITNET group–just shy of two-thirds–“recovered” at six months, compared to just eight percent in the comparison group. But this apparent success masks a much more complicated reality and cannot be taken at face value, for multiple reasons.

First, the subsequent 2013 paper from the Dutch team found no differences in “recovery” between participants in the two groups at long-term follow-up (on average, 2.7 years after starting). Those in the comparison group improved after the trial and had caught up to the intervention group, so the online CBT conferred no extended advantages or benefits. The researchers argued that the therapy was nonetheless useful because patients achieved gains more quickly. But they failed to consider another reasonable explanation for their results.

Those in usual care were attending in-person sessions at clinics or doctors’ offices. Depending on how often they went, how far they had to travel and how sick they were, the transportation demands could easily have triggered relapses and harmed their health. In contrast, those in the FITNET group could be treated at home. Perhaps they improved not from the treatment itself but from an unintended side effect–the sedentary nature of the intervention allowed them more time to rest. The investigators did not control for this aspect of the online CBT.

Second, the “recovery” figure in the Dutch FITNET study was a post-hoc calculation, as the authors acknowledged. The protocol for the trial included the outcomes to be measured, of course, but the authors did not identify before the trial what thresholds participants would need to meet to be considered “recovered.” The entire definition was constructed only after they saw the results—and the thresholds they selected were extremely lenient. Even two of the PACE authors, in a Lancet commentary praising the Dutch study, referred to the “recovery” criteria as “liberal” and “not stringent.” (In fact, only 36% “recovered” under a more modest definition of “recovery,” but the FITNET authors tucked this finding into an appendix and Dr. Crawley’s FITNET-NHS protocol didn’t mention it.)

Now, the fact that “recovery” was a post-hoc measure doesn’t mean it isn’t valid. But anyone citing this “recovery” rate should do so with caveats and some measure of caution. Dr. Crawley has exhibited no such reticence—in a recent radio interview, she declared flatly that the Dutch participants had made a “full recovery.” (In the same interview, she called PACE “a great, great study.” Then she completely misrepresented the results of the recent reanalyses of the PACE trial data. So, you know, take her words for what they’re worth.)

Given the hyperbole about “recovery,” the public is understandably likely to assume that Dr. Crawley’s new “landmark” study will result in similar success. A corollary of that assumption is that anyone who opposes the study’s approach, like so many in the patient and advocacy communities, could be accused of acting in ways that harm children by depriving them of needed treatment. This would be an unfair charge, since the online CBT being offered is based on the questionable premise that the children harbor untrue cognitions about their illness.

Third, the standard treatments received by the usual care group were described like this: “individual/group based rehabilitation programs, psychological support including CBT face-to-face, graded exercise therapy by a physiotherapist, etc.” In other words, pretty much the kinds of “evidence-based” strategies these Dutch experts and their U.K. colleagues had promoted for years as being effective for chronic fatigue syndrome. In the end, two-thirds of those in usual care received in-person CBT, and half received graded exercise therapy. (Many participants in this arm received more than one form of usual care.)

And yet less than one in ten of the usual care participants were found to have “recovered” at six months, according to the 2012 study. So what does that say about the effectiveness of these kinds of rehabilitative approaches in the first place? In light of the superlative findings for online CBT, why haven’t all chronic fatigue syndrome patients in the Netherlands now been removed from in-person treatments and offered this more convenient option? (Dr. Crawley’s FITNET-NHS proposal tried to explain away this embarrassing finding of the Dutch study by suggesting that those providing usual care were not trained to work with this kind of population.)

Finally, the Dutch study did not report any objective measures of physical performance. Although the study included assessments using an actometer—an ankle bracelet that monitors distance moved—the Lancet paper did not mention those results. In previous studies of cognitive and behavioral treatments for ME/CFS, reported improvements on subjective measures for fatigue or physical function were not accompanied by increases in physical movement, as measured by actometer. And in PACE, of course, the investigators dismissed their own objective measures as irrelevant or non-objective—after these outcomes failed to provide the desired results.

In response to correspondence calling for publication of the actometer data, the Dutch investigators refused, noting that “the goal of our treatment was reduction of fatigue and increase in school attendance, not increase in physical activity per se.” This is an inadequate explanation for the decision to withhold data that would shed light on whether participants actually improved in their physical performance as well as in their subjective impressions of their condition. If the actometer data demonstrated remarkable increases in activity levels in the online CBT group, is there any doubt they would have reported it?

In short, the Dutch FITNET study leaves a lot of questions unanswered. So does its U.K. version, the proposed FITNET-NHS. And Dr. Crawley’s recent media blitz—which included a “can’t-we-all-get-along” essay in The New Scientist—did little to quell any of the reasonable qualms observers might have about this latest effort to bolster the sagging fortunes of the CBT/GET/PACE paradigm.

“Patients are desperate for this trial, yet some people are still trying to stop us,” wrote Dr. Crawley in The New Scientist. “The fighting needs to end.”

However, those mysterious and sinister-sounding “some people” cited by Dr. Crawley have very thoughtful and legitimate reasons for questioning the quality of her research. The fighting, as she calls it, is likely to end when Dr. Crawley and her colleagues stop conflating chronic fatigue and chronic fatigue syndrome through the use of loose diagnostic criteria. And when they acknowledge what scientists in the U.S. and around the world now understand: The claim that cognitive and behavioral approaches are effective treatments that lead to “recovery” is based on deeply flawed research.

A Short Postscript:

Several Dutch colleagues have joined Dr. Crawley as part of the FITNET-NHS study. Two of them, Dr. Gijs Bleijenberg from the Radboud University Medical Centre in Nijmegen, and Dr. Hans Knoop from the University of Amsterdam, are among the leaders of the CBT/GET movement in the Netherlands and have collaborated with their U.K. counterparts. Not surprisingly, their work is similarly dodgy.

In a post last year, I dissected a 2011 commentary in The Lancet on the PACE trial, co-authored by Dr. Bleijenberg and Dr. Knoop, in which they argued that 30 percent of the participants in the CBT and GET groups had met “a strict criterion for recovery.” This statement was absurd, since these “strict” thresholds for “recovery” were in fact so lax that participants could get worse during the study and still meet them. Although the problematic nature of the thresholds has been pointed out to Dr. Bleijenberg and Dr. Knoop, they have stood by their nonsensical claim.

Earlier this year, the Dutch parliament asked the Health Council—an independent scientific advisory body—to review the state of evidence related to the illness, including the evidence on treatments like CBT and GET. The Health Council appointed a committee to conduct the review. Among the committee members are Dr. Knoop and colleagues who share his perspective. It remains unclear whether the committee is taking sufficient account of the methodological flaws underpinning the evidence for the CBT/GET paradigm and of the ongoing condemnations of the PACE trial from well-respected scientists. I plan to blog about this situation soon.

Trial By Error, Continued: The Real Data

by David Tuller, DrPH

David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.

‘The PACE trial is a fraud.’ Ever since Virology Blog posted my 14,000-essord investigation of the PACE trial last October, I’ve wanted to write that sentence. (I should point out that Dr. Racaniello has already called the PACE trial a “sham,” and I’ve already referred to it as “doggie-poo.” I’m not sure that “fraud” is any worse. Whatever word you use, the trial stinks.)

Let me be clear: I don’t mean “fraud” in the legal sense—I’m not a lawyer–but in the sense that it’s a deceptive and morally bankrupt piece of research. The investigators made dramatic changes from the methodology they outlined in their protocol, which allowed them to report purported “results” that were much, much better than those they would have been able to claim under their originally planned methods. Then they reported only the better-looking “results,” with no sensitivity analyses to analyze the impact of the changes—the standard statistical approach in such circumstances.

This is simply not allowed in science. It means the reported benefits for cognitive behavior therapy and graded exercise therapy were largely illusory–an artifact of the huge shifts in outcome assessments the authors introduced mid-trial. (That’s putting aside all the other flaws, like juicing up responses with a mid-trial newsletter promoting the interventions under investigation, failing to obtain legitimate informed consent from the participants, etc.)

That PACE suffered from serious methodological deficiencies should have been obvious to anyone who read the studies. That includes the reviewers for The Lancet, which published the PACE results for “improvement” in 2011 after what editor Richard Horton has called “endless rounds of peer-review,” and the journal Psychological Medicine, which published results for “recovery” in 2013. Certainly the deficiencies should have been obvious to anyone who read the trenchant letters and commentaries that patients routinely published in response to the egregious errors committed by the PACE team. Even so, the entire U.K. medical, academic and public health establishments refused to acknowledge what was right before their eyes, finding it easier instead to brand patients as unstable, anti-science, and possibly dangerous.

Thanks to the efforts of the incredible Alem Matthees, a patient in Perth, Australia, the U.K.’s First-Tier Tribunal last month ordered the liberation of the PACE trial data he’d requested under a freedom-of-information request. (The brief he wrote for the April hearing, outlining the case against PACE in great detail, was a masterpiece.) Instead of appealing, Queen Mary University of London, the home institution of lead PACE investigator Peter White, made the right decision. On Friday, September 9, the university announced its intention to comply with the tribunal ruling, and sent the data file to Mr. Matthees. The university has a short window of time before it has to release the data publicly.

I’m guessing that QMUL forced the PACE team’s hand by refusing to allow an appeal of the tribunal decision. I doubt that Dr. White and his colleagues would ever have given up their data willingly, especially now that I’ve seen the actual results. Perhaps administrators had finally tired of the PACE shenanigans, recognized that the study was not worth defending, and understood that continuing to fight would further harm QMUL’s reputation. It must be clear to the university now that its own reputational interests diverge sharply from those of Dr. White and the PACE team. I predict that the split will become more apparent as the trial’s reputation and credibility crumble; I don’t expect QMUL spokespeople to be out there vigorously defending the unacceptable conduct of the PACE investigators.

Last weekend, several smart, savvy patients helped Mr. Matthees analyze the newly available data, in collaboration with two well-known academic statisticians, Bruce Levin from Columbia and Philip Stark from Berkeley.  Yesterday, Virology Blog published the group’s findings of the single-digit, non-statistically significant “recovery” rates the trial would have been able to report had the investigators adhered to the methods they outlined in the protocol. That’s a remarkable drop from the original Psychological Medicine paper, which claimed that 22 percent of those in the favored intervention groups achieved “recovery,” compared to seven percent for the non-therapy group.

Now it’s clear: The PACE authors themselves are the anti-science faction. They tortured their data and ended up producing sexier results. Then they claimed they couldn’t share their data because of alleged worries about patient confidentiality and sociopathic anti-PACE vigilantes. The court dismissed these arguments as baseless, in scathing terms. (It should be noted that their ethical concerns for patients did not extend to complying with a critical promise they made in their protocol—to tell prospective participants about “any possible conflicts of interest” in obtaining informed consent. Given this omission, they have no legitimate informed consent for any of their 641 participants and therefore should not be allowed to publish any of their data at all.)

The day before QMUL released the imprisoned data to Mr. Matthees, the PACE authors themselves posted a pre-emptive re-analysis of results for the two primary outcomes of physical function and fatigue, according to the protocol methods. In the Lancet paper, they had revised and weakened their own definition of what constituted “improvement.” With this revised definition, they could report in The Lancetthat approximately 60 % in the cognitive behavior and graded exercise therapy arms “improved” to a clinically significant degree on both fatigue and physical function.

The re-analysis the PACE authors posted last week sought to put the best possible face on the very poor data they were required to release. Yet patients examining the new numbers quickly noted that, under the more stringent definition of “improvement” outlined in the protocol, only about 20 percent in the two groups could be called “overall improvers.”. Solely by introducing a more relaxed definition of “improvement,” the PACE team—enabled by The Lancet’s negligence and an apparently inadequate “endless” review process–was able to triple the trial’s reported success rate..

So now it’s time to ask what happens to the papers already published. The editors have made their feelings clear. I have written multiple e-mails to Lancet editor Richard Horton since I first contacted him about my PACE investigation, almost a year before it ran. He never responded until September 9, the day QMUL liberated the PACE data. Given that the PACE authors’ own analysis showed that the new data showed significantly less impressive results than those published in The Lancet, I sent Dr. Horton a short e-mail asking when we could expect some sort of addendum or correction to the 2011 paper. He responded curtly: “Mr. Tuller–We have no such plans.”

The editors of Psychological Medicine are Kenneth Kendler of Virginia Commonwealth University and Robin Murray of Kings College London. After I wrote to the journal last December, pointing out the problems, I received the following from Dr. Murray, whose home base is KCL’s Department of Psychosis Studies: “Obviously the best way of addressing the truth or otherwise of the findings is to attempt to replicate them. I would therefore like to encourage you to initiate an attempted replication of the study. This would be the best way for you to contribute to the debate…Should you do this, then Psychological Medicine will be most interested in the findings either positive or negative.”

This was not an appropriate response. I told Dr. Murray it was “disgraceful,” given that the paper was so obviously flawed. This week, I wrote again to Dr. Murray and Dr. Kendler, asking if they now planned to deal with the paper’s problems, given the re-analysis by Matthees et al. In response, Dr. Murray suggested that I submit a re-analysis, based on the released data, and Psychological Medicine would be happy to consider it. “We would, of course, send it out to referees for scientific scrutiny in the same manner as we did for the original paper,” he wrote.

I explained that it was his and the journal’s responsibility to address the problems, whether or not anyone submitted a re-analysis. I also noted that I could not improve on the Matthees re-analysis, which completed rebutted the results reported in Psychological Medicine’s paper. I urged Dr. Murray to contact either Dr. Racaniello or Mr. Matthees to discuss republishing it, if he truly wished to contribute to the debate. Finally, I noted that the peer-reviewers for the original paper had okayed a study in which participants could be disabled and recovered simultaneously, so I wasn’t sure if the journal’s assessment process could be trusted.

(By the way, Kings College London, where Dr. Murray is based, is also the home institution of PACE investigator Trudie Chalder as well as Simon Wessely, a close colleague of the PACE authors and president of the Royal College of Psychiatrists*. That could explain Dr. Murray’s inability or reluctance to acknowledge that the “recovery” paper his journal peer-reviewed and published is meaningless.)

Earlier today, the PACE authors posted a blog on The BMJ site, their latest effort to salvage their damaged reputations. They make no mention of their massive research errors and focus only on their supposed fears that releasing even anonymous data will frighten away future research participants. They have provided no evidence to back up this unfounded claim, and the tribunal flatly rejected it. They also state that only researchers who present  “pre-specified” analysis plans should be able to obtain trial data. This is laughable, since Dr. White and his colleagues abandoned their own pre-specified analyses in favor of analyses they decided they preferred much later on, long after the trial started.

They have continued to refer to their reported analyses, deceptively, as “pre-specified,” even though these methods were revised mid-trial. The following point has been stated many times before, but bears repeating: In an open label trial like PACE, researchers are likely to know very well what the outcome trends are before they review any actual data. So the PACE team’s claim that the changes they made were “pre-specified” because they were made before reviewing outcome data is specious. I have tried to ask them about this issue multiple times, and have never received an answer.

Dr. White, his colleagues, and their defenders don’t yet seem to grasp that the intellectual construct they invented and came to believe in—the PACE paradigm or the PACE enterprise or the PACE cult, have your pick—is in a state of collapse. They are used to saying whatever they want about patients—Internet Abuse! Knife-wielding! Death threats!!–and having it be believed. In responding to legitimate concerns and questions, they have covered up their abuse of the scientific process by providing non-answers, evasions and misrepresentations—the academic publishing equivalent of “the dog ate my homework.” Amazingly, journal editors, health officials, reporters and others have accepted these non-responsive responses as reasonable and sufficient. I do not.

Now their work is finally being scrutinized the way it should have been by peer reviewers before this damaging research was ever published in the first place. The fallout is not going to be pretty. If nothing else, they have provided a great gift to academia with their $8 million** disaster—for years to come, graduate students in the U.S., the U.K. and elsewhere will be dissecting PACE as a classic case study of bad research and mass delusion.

*Correction: The original version of the post mistakenly called the organization the Royal Society of Psychiatrists.

**Correction: The original version of the post stated that PACE cost $8 million, not $6.4 million. In fact, PACE cost five million pounds, so the cost in dollars depends on the exchange rate used. The $8 million figure is based on the exchange rate from last October, when Virology Blog published my PACE investigation. But the pound has fallen since the Brexit vote in June, so the cost in dollars at the current exchange rate is lower.

No ‘Recovery’ in PACE Trial, New Analysis Finds

Last October, Virology Blog posted David Tuller’s 14,000-word investigation of the many flaws of the PACE trial (link to article), which had reported that cognitive behavior therapy and graded exercise therapy could lead to “improvement” and “recovery” from ME/CFS. The first results, on “improvement,” were published in The Lancet in 2011; a follow-up study, on “recovery,” was published in the journal Psychological Medicine in 2013.

The investigation by Dr. Tuller, a lecturer in public health and journalism at UC Berkeley, built on the impressive analyses already done by ME/CFS patients; his work helped demolish the credibility of the PACE trial as a piece of scientific research. In February, Virology Blog posted an open letter (link) to The Lancet and its editor, Richard Horton, stating that the trial’s flaws “have no place in published research.” Surprisingly, the PACE authors, The Lancet, and others in the U.K. medical and academic establishment have continued their vigorous defense of the study, despite its glaring methodological and ethical deficiencies.

Today, I’m delighted to publish an important new analysis of PACE trial data—an analysis that the authors never wanted you to see.  The results should put to rest once and for all any question about whether the PACE trial’s enormous mid-trial changes in assessment methods allowed the investigators to report better results than they otherwise would have had. While the answer was obvious from Dr. Tuller’s reporting, the new analysis makes the argument incontrovertible.

ME/CFS patients developed and wrote this groundbreaking analysis, advised by two academic co-authors. It was compiled from data obtained through a freedom-of-information request, pursued with heroic persistence by an Australian patient, Alem Matthees. Since the authors dramatically weakened all of their “recovery” criteria long after the trial started, with no committee approval for the redefinition of “recovery,” it was entirely predictable that the protocol-specified results would be worse. Now we know just how much worse they are.

According to the new analysis, “recovery” rates for the graded exercise and cognitive behavior therapy arms were in the mid-single-digits and were not statistically significant. In contrast, the PACE authors managed to report statistically significant “recovery” rates of 22 percent for their favored interventions. Given the results based on the pre-selected protocol metrics for which they received study approval and funding, it is now up to the PACE authors to explain why anyone should accept their published outcomes as accurate, reliable or legitimate.

The complete text of the analysis is below. A pdf is also available (link to pdf).

***

A preliminary analysis of ‘recovery’ from chronic fatigue syndrome in the PACE trial using individual participant data

 

Wednesday 21 September 2016

Alem Matthees (1), Tom Kindlon (2), Carly Maryhew (3), Philip Stark (4), Bruce Levin (5).

1. Perth, Australia. alem.matthees@gmail.com
2. Information Officer, Irish ME/CFS Association, Dublin, Ireland.
3. Amersfoort, Netherlands.
4. Associate Dean, Mathematical and Physical Sciences; Professor, Department of Statistics; University of California, Berkeley, California, USA.
5. Professor of Biostatistics and Past Chair, Department of Biostatistics, Mailman School of Public Health, Columbia University, New York, USA.

Summary

The PACE trial tested interventions for chronic fatigue syndrome, but the published ‘recovery’ rates were based on thresholds that deviated substantially from the published trial protocol. Individual participant data on a selection of measures has recently been released under the Freedom of Information Act, enabling the re-analysis of recovery rates in accordance with the thresholds specified in the published trial protocol. The recovery rate using these thresholds is 3.1% for specialist medical care alone; for the adjunctive therapies it is 6.8% for cognitive behavioural therapy, 4.4% for graded exercise therapy, and 1.9% for adaptive pacing therapy. This re-analysis demonstrates that the previously reported recovery rates were inflated by an average of four-fold. Furthermore, in contrast with the published paper by the trial investigators, the recovery rates in the cognitive behavioural therapy and graded exercise therapy groups are not significantly higher than with specialist medical care alone. The implications of these findings are discussed.

Introduction

The PACE trial was a large multi-centre study of therapeutic interventions for chronic fatigue syndrome (CFS) in the United Kingdom (UK). The trial compared three therapies which were each added to specialist medical care (SMC): cognitive behavioural therapy (CBT), graded exercise therapy (GET), and adaptive pacing therapy (APT). [1] Henceforth SMC alone will be ‘SMC’, SMC plus CBT will be ‘CBT’, SMC plus GET will be ‘GET’, and SMC plus APT will be ‘APT’. Outcomes consisted of two self-report primary measures (fatigue and physical function), and a mixture of self-report and objective secondary measures. The trial’s co-principal investigators are longstanding practitioners and proponents of the CBT and GET approach, whereas APT was a highly formalised and modified version of an alternative energy management approach.

After making major changes to the protocol-specified “recovery” criteria, White et al. (2013) reported that when using “a comprehensive and conservative definition of recovery”, CBT and GET were associated with significantly increased recovery rates of 22% at 52-week follow-up, compared to only 8% for APT and 7% for SMC [2]. However, those figures were not derived using the published trial protocol (White et al., 2007 [3]), but instead using a substantially revised version that has been widely criticised for being overly lax and poorly justified (e.g. [4]). For example, the changes created an overlap between trial eligibility criteria for severe disabling fatigue, and the new “normal range”. Trial participants could consequently be classified as recovered without clinically significant improvements to self-reported physical function or fatigue, and in some cases without any improvement whatsoever on these outcome measures. Approximately 13% of participants at baseline simultaneously met the trial eligibility criteria for ‘significant disability’ and the revised recovery criteria for normal self-reported physical function. The justification given for changing the physical function threshold of recovery was apparently based on a misinterpretation of basic summary statistics [5,6], and the authors also incorrectly described their revised threshold as more stringent than previous research [2]. These errors have not been corrected, despite the publishing journal’s policy that such errors should be amended, resulting in growing calls for a fully independent re-analysis of the PACE trial results [7,8].

More than six years after data collection was completed for the 52-week follow-up, the PACE trial investigators have still not published the recovery rates as defined in the trial protocol. Queen Mary University of London (QMUL), holder of the trial data and home of the chief principal investigator, have also not allowed access to the data for others to analyse these outcomes. Following a Freedom of Information Act (FOIA) request for a selection of trial data, an Information Tribunal upheld an earlier decision from the Information Commissioner ordering the release of that data (see case EA/2015/0269). On 9 September 2016, QMUL released the requested data [9]. Given the public nature of the data release, and the strong public interest in addressing the issue of “recovery” from CFS in the PACE trial, we are releasing a preliminary analysis using the main thresholds set in the published trial protocol. The underlying data is also being made available [10], while more detailed and complete analyses on the available outcome measures will be published at a later date.

Methods

Measures and criteria

Using the variables available in the FOIA dataset, ‘recovery’ from CFS in the PACE trial is analysed here based on the main outcome measures described by White et al. (2013) in the “cumulative criteria for trial recovery” [2]. These measures are: (i) the Chalder Fatigue Questionnaire (CFQ); (ii) the Short-Form-36 (SF-36) physical function subscale; (iii) the Clinical Global Impression (CGI) change scale; and (iv) the Oxford CFS criteria. However, instead of the weakened thresholds used in their analysis, we will use the thresholds specified in the published trial protocol by White et al. (2007) [3]. A comparison between the different thresholds for each outcome measure is presented in Table 1.

table 1

Where follow-up data for self-rated CGI scores were missing we did not impute doctor-rated scores, in contrast to the approach of White et al., because the trial protocol stated that all primary and secondary outcomes are “either self-rated or objective in order to minimise observer bias” from non-blinded assessors. We discuss the minimal impact of this imputation below. Participants missing any recovery criteria data at 52-week follow-up were classified as non-recovered.

Statistical analysis

White et al. (2013) conducted an available-case analysis which excluded from the denominators of each group the participants who dropped out [2]. This is not the recommended practice in clinical trials, where intention-to-treat analysis (which includes all randomised participants) is commonly preferred. An available-case analysis may overestimate real-world treatment effects because it does not include participants who were lost to follow-up. Attrition from trials can occur for various reasons, including an inability to tolerate the prescribed treatment, a perceived lack of benefit, and adverse reactions. Thus, an available-case analysis only takes into account the patients who were willing and able to tolerate the prescribed treatments. Nonetheless, both types of analyses are presented here for comparison. We present a preliminary exploratory analysis of the frequency and percentage of participants meeting all the recovery criteria in each group, based on the intention-to-treat principle, as well as the available-case subgroup.

Neither the published trial protocol [3] nor the published statistical analysis plan [11] specified a method for determining the statistical significance of the differences in recovery rates between treatment groups. In their published paper on recovery, White et al. (2013) presented logistic regression analyses for trial arm pairwise comparisons, adjusting for the baseline stratification variables of treatment centre, meeting CDC CFS criteria, meeting London ME criteria, and having a depressive illness [2]. However, it has been shown that logistic regression may be an inappropriate method of analysis in the context of randomised trials [12]. While Fisher’s exact test would be preferable, a more rigorous approach would also take into account stratification variables, which unfortunately were not part of the available FOIA dataset. Nonetheless, there is reason to believe that the effect of including these stratification variables would be minimal on our analyses: the stratification variables were approximately evenly distributed between groups [1], and attempting to replicate the previously published [2] odds ratios and 95% confidence intervals using logistic regression, but without stratification variables, yielded very similar results to the ones previously published (see Table 3).

We therefore present recovery rates for each group and compare the observed rates for each active treatment arm with those of the SMC arm using Fisher’s exact tests. The confidence intervals for recovery rates in each group and comparative odds ratios are exact 95% confidence intervals using the point probability method [13]. For sake of direct comparison with results published by White et al. (2013), we also present results of logistic regression analysis which included only the treatment arm as a predictor variable, with conventional approximate 95% confidence intervals.

Results

For our analysis of ‘recovery’ in the PACE trial, full data were available for 89% to 94% of participants, depending on the treatment group and outcome measure. Percentages are calculated for both intention-to-treat, and on an available-case basis. Imputing the missing self-rated CGI scores with doctor-rated CGI scores made no difference to the intention-to-treat analysis, as there were no participants with missing self-rated CGI scores with an assessor rating of 1, required for recovery; in the available-case analysis, the only effect this had was to decrease the CBT denominator by 1, and the assessor score for that participant was 3, “a little better”, therefore non-recovered. Table 2 provides the results and Figure 1 compares our recovery rates with those of White et al. (2013):

table 2

figure 1

The CBT, GET, and APT groups did not demonstrate a statistically significant advantage over the SMC group in any of the above analyses, nor an empirical recovery rate that would generally be considered adequate (the highest observed rate was 7.7%). In the intention-to-treat analysis, the exact p value for the three degree of freedom chi-squared test for no overall differences amongst the four groups was 0.14. In the available-case analysis, the p value was 0.10. Given the number of comparisons, a correction for multiple testing might be appropriate, but as none of the uncorrected p values were significant at the p<0.05 level, this more conservative approach would not alter the conclusion. Our findings therefore contradict the conclusion of White et al. (2013), that CBT and GET were significantly more likely than the SMC group to be associated with ‘recovery’ at 52 weeks [2]. However, the very low recovery rates substantially decrease the ability to detect statistically significant differences between groups (see the Limitations section). The multiple changes to the recovery criteria had inflated the estimates of recovery by approximately 2.3 to 5.1 -fold, depending on the group, with an average inflation of 3.8-fold.

Limitations

Lack of statistical power

When designing the PACE trial and determining the number of participants needed, the investigators’ power analyses were based not on recovery estimates but on the prediction of relatively high rates of clinical improvement in the additional therapy groups compared to SMC alone [3]. However, the very low recovery rates introduce a complication for tests of significance, due to insufficient statistical power to detect modest but clinically important differences between groups. For example, with the CBT vs. SMC comparison by intention-to-treat, a true odds ratio of 4.2 would have been required to give Fisher’s exact test 80% power to declare significance, given the observed margins. If we assume SMC has a probability of 3.1%, an odds ratio of 4.2 would have conferred a recovery probability of 11.8%, which was not achieved in the trial.

We believe that for our preliminary analysis it was important to follow the protocol-specified recovery criteria, which make more sense than the revised thresholds. For example, the former required level of physical function would suggest a ‘recovered’ individual could at least do most normal activities, but may have limitations with a few of the items on the SF-36 health survey, such as vigorous exercise, walking up flights of stairs, or bending down. The revised threshold that White et al. (2013) used meant that a ‘recovered’ individual could have remained limited on four to eight out of ten items depending on severity. We found that when using the revised recovery criteria, 8% (7/87) of the ‘recovered’ participants still met trial eligibility criteria for ‘significant disability’.

Weakening the recovery thresholds increases statistical power to detect group differences because it makes the event (i.e. ‘recovery’) rates more frequent (i.e. less close to zero) but it also leads to the inclusion of patients who still, for example, have significant illness-related restrictions in physical capacity as per SF-36 physical function score. We argue that if significant differences between groups cannot be detected in sample sizes of approximately n=160 per group, then this may indicate that CBT and GET simply do not substantially increase recovery rates.

Lack of data on stratification variables

In order to increase the chance of being granted or enforced, the FOIA request asked for a ‘bare minimum’ set of variables, as asking for too many variables, or for variables that may be judged to significantly increase the risk of re-identification of participants, would have decreased the chance that the FOIA request would be granted. This was a reasonable compromise given that QMUL had previously blocked all requests for the protocol-specified recovery rates and the underlying data to calculate them. Some non-crucial variables are therefore missing from the dataset acquired under the FOIA but there is reason to believe that this would have little effect on the results.

Allocation of participants in the PACE trial was stratified [1]: “The first three participants at each of the six clinics were allocated with straightforward randomisation. Thereafter allocation was stratified by centre, alternative criteria for chronic fatigue syndrome and myalgic encephalomyelitis, and depressive disorder (major or minor depressive episode or dysthymia), with computer-generated probabilistic minimisation.”

This means that testing for statistical significance assuming simple randomisation results in p- values that are approximate and effect-size estimates that might be biased. The FOIA dataset does not contain the stratification variables. While the lack of these variables may somewhat alter the estimated treatment effects and the p-values or confidence levels, we expect the differences to be minor, a conclusion that is supported by Table 3 below. Table 1 of the publication of the main trial results (White et al., 2011) shows that the stratification variables were approximately evenly distributed between groups [1]. We have replicated the rates of “trial recovery” as previously published by White et al. (2013) [2]. We also attempted to replicate their previously reported logistic regression, without the stratification variables, and the results were essentially the same (see Table 3), suggesting that the adjustments would not have a significant impact on the outcome of our own analysis of recovery.

table 3

If QMUL or the PACE trial investigators believe that further adjustment is necessary here to have confidence in the results, then we invite them to present analyses that include stratification variables or release the raw data for those variables without unnecessary restrictions.

Lack of data on alternative ME/CFS criteria

For the same reasons described in the previous subsection, the FOIA dataset does not contain the variables for meeting CDC CFS criteria or London ME (myalgic encephalomyelitis) criteria. These were part of the original definition of recovery, but we argue that these are superfluous because:

(a) While our definition of recovery is less stringent without the alternative ME/CFS criteria, these additional criteria had no significant effect on the results reported by White et al. (2013) [2]). (b) The alternative ME/CFS criteria used in the trial had some questionable modifications [14], that have not been used in any other trial, thus seriously limiting cross-trial comparability and validation of their results. (c) The Oxford CFS criteria are the most sensitive and least specific (most inclusive) criteria, so those who fulfil all other aspects of the recovery criteria would most likely also fail to meet alternative ME/CFS criteria. (d) All participants were first screened using the Oxford CFS criteria as this was the primary case definition, whereas the additional case criteria were not entry requirements [1].

Discussion

It is important that patients, health care professionals, and researchers have accurate information about the chances of recovery from CFS. In the absence of definitive outcome measures, recovery criteria should set reasonable standards that approach restoration of good health, in keeping with commonly understood conceptions of recovery from illness [15]. Accordingly, the changes made by the PACE trial investigators after the trial was well under way resulted in the recovery criteria becoming too lax to allow conclusions about the efficacy of CBT and GET as rehabilitative treatments for CFS. This analysis, based on the published trial protocol, demonstrates that the major changes to the thresholds for recovery had inflated the estimates of recovery by an average of approximately four-fold. QMUL recently posted the PACE trial primary ‘improvement’ outcomes as specified in the protocol [16] and that also showed a similar difference between the proportion of participants classified as improved compared to the post-hoc figures previously published in the Lancet in 2011 [1]. It is clear from these results that the changes made to the protocol were not minor or insignificant, as they have produced major differences that warrant further consideration.

The PACE trial protocol was published with the implication that changes would be unlikely [17], and while the trial investigators describe their analysis of recovery as pre-specified, there is no mention of changes to the recovery criteria in the statistical analysis plan that was finalised shortly before the unblinding of trial data [11]. Confusion has predictably ensued regarding the timing and nature of the substantial changes made to the recovery criteria [18]. Changing study endpoints should be rare and is only rarely acceptable; moreover, trial investigators may not be appropriate decision makers for endpoint revisions [19,20]. Key aspects of pre-registered design and analyses are often ignored in subsequent publications, and positive results are often the product of overly flexible rules of design and data analysis [21,22].

As reported in a recent BMJ editorial by chief editor Fiona Godlee (3 March 2016), when there is enough doubt to warrant independent re-analysis [23]: “Such independent reanalysis and public access to anonymised data should anyway be the rule, not the exception, whoever funds the trial.” The PACE trial provides a good example of the problems that can occur when investigators are allowed to substantially deviate from the trial protocol without adequate justification or scrutiny. We therefore propose that a thorough, transparent, and independent re-analysis be conducted to provide greater clarity about the PACE trial results. Pending a comprehensive review or audit of trial data, it seems prudent that the published trial results should be treated as potentially unsound, as well as the medical texts, review articles, and public policies based on those results.

Acknowledgements

Writing this article in such a brief period of time would not have been possible without the diverse and invaluable contributions from patients, and others, who chose not to be named as authors.

Declarations

AM submitted a FOIA request and participated in legal proceedings to acquire the dataset. TK is a committee member of the Irish ME/CFS Association (voluntary position).

References

1. White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O’Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; PACE trial management group. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011 Mar 5;377(9768):823-36. doi: 10.1016/S0140-6736(11)60096-2. Epub 2011 Feb 18. PMID: 21334061. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3065633/

2. White PD, Goldsmith K, Johnson AL, Chalder T, Sharpe M. Recovery from chronic fatigue syndrome after treatments given in the PACE trial. Psychol Med. 2013 Oct;43(10):2227-35. doi: 10.1017/S0033291713000020. PMID: 23363640. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3776285/

3. White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R; PACE trial group. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol. 2007 Mar 8;7:6. PMID: 17397525. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2147058/

4. A list of articles by David Tuller on ME/CFS and PACE at Virology Blog. http://www.virology.ws/mecfs/

5. Kindlon T, Baldwin A. Response to: reports of recovery in chronic fatigue syndrome may present less than meets the eye. Evid Based Ment Health. 2015 May;18(2):e5. doi: 10.1136/eb-2014-101961. Epub 2014 Sep 19. PMID: 25239244. http://ebmh.bmj.com/content/18/2/e5.long

6. Matthees A. Assessment of recovery status in chronic fatigue syndrome using normative data. Qual Life Res. 2015 Apr;24(4):905-7. doi: 10.1007/s11136-014-0819-0. Epub 2014 Oct 11. PMID: 25304959. http://link.springer.com/article/10.1007%2Fs11136-014-0819-0

7. Davis RW, Edwards JCW, Jason LA, et al. An open letter to The Lancet, again. Virology Blog. 10 February 2016. http://www.virology.ws/2016/02/10/open-letter-lancet-again/

8. #MEAction. Press release: 12,000 signature PACE petition delivered to the Lancet. http://www.meaction.net/press-release-12000-signature-pace-petition-delivered-to-the-lancet/

9. Queen Mary University of London. Statement: Disclosure of PACE trial data under the Freedom of Information Act. 9 September 2016 Statement: Release of individual patient data from the PACE trial. http://www.qmul.ac.uk/media/news/items/smd/181216.html

10. FOIA request to QMUL (2014/F73). Dataset file: https://sites.google.com/site/pacefoir/pace-ipd_foia-qmul- 2014-f73.xlsx Readme file: https://sites.google.com/site/pacefoir/pace-ipd-readme.txt

11. Walwyn R, Potts L, McCrone P, Johnson AL, DeCesare JC, Baber H, Goldsmith K, Sharpe M, Chalder T, White PD. A randomised trial of adaptive pacing therapy, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome (PACE): statistical analysis plan. Trials. 2013 Nov 13;14:386. doi: 10.1186/1745-6215-14-386. PMID: 24225069. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4226009/

12. Freedman DA. Randomization Does Not Justify Logistic Regression. Statistical Science. 2008;23(2):237–249. doi:10.1214/08-STS262. https://arxiv.org/pdf/0808.3914.pdf

13. Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. 3rd ed. Hoboken, N.J: J. Wiley; 2003. 760 p. IBSN: 978-0-471-52629-2. (Wiley series in probability and statistics). http://au.wiley.com/WileyCDA/WileyTitle/productCd-0471526290.html

14. Matthees A. Treatment of Myalgic Encephalomyelitis/Chronic Fatigue Syndrome. Ann Intern Med. 2015 Dec 1;163(11):886-7. doi: 10.7326/L15-5173. PMID: 26618293.

15. Adamowicz JL, Caikauskaite I, Friedberg F. Defining recovery in chronic fatigue syndrome: a critical review. Qual Life Res. 2014 Nov;23(9):2407-16. doi: 10.1007/s11136-014-0705-9. Epub 2014 May 3. PMID: 24791749. http://link.springer.com/article/10.1007%2Fs11136-014-0705-9

16. Goldsmith KA, White PD, Chalder T, Johnson AL, Sharpe M. The PACE trial: analysis of primary outcomes using composite measures of improvement. 8 September 2016. http://www.wolfson.qmul.ac.uk/images/pdfs/pace/PACE_published_protocol_based_analysis_final_8th_Sept_201 6.pdf

17. BMC editor’s comment on [Protocol for the PACE trial] (Version: 2. Date: 31 January 2007) http://www.biomedcentral.com/imedia/2095594212130588_comment.pdf

18. UK House of Lords. PACE Trial: Chronic Fatigue Syndrome/Myalgic Encephalomyelitis. 6 February 2013. http://www.publications.parliament.uk/pa/ld201213/ldhansrd/text/130206-gc0001.htm

19. Evans S. When and how can endpoints be changed after initiation of a randomized clinical trial? PLoS Clin Trials. 2007 Apr 13;2(4):e18. PMID 17443237. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1852589/

20. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010 Mar 23;340:c869. doi: 10.1136/bmj.c869. PMID: 20332511. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2844943

21. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011 Nov;22(11):1359-66. doi: 10.1177/0956797611417632. Epub 2011 Oct 17. PMID: 22006061. http://pss.sagepub.com/content/22/11/1359.long

22. Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HL, Kievit RA. An Agenda for Purely Confirmatory Research. Perspect Psychol Sci. 2012 Nov;7(6):632-8. doi: 10.1177/1745691612463078. PMID: 26168122. http://pps.sagepub.com/content/7/6/632.full

23. Godlee F. Data transparency is the only way. BMJ 2016;352:i1261. (Published 03 March 2016) doi: http://dx.doi.org/10.1136/bmj.i1261 http://www.bmj.com/content/352/bmj.i1261

Trial By Error, Continued: My Questions for Lancet Editor Richard Horton

By David Tuller, DrPH

In January, I posted a list of the questions I still wanted to ask the PACE authors, who have repeatedly refused my requests to interview them about their ethically and methodologically challenged study. Richard Horton, editor of The Lancet, has similarly declined to talk with me, ignoring my e-mails seeking comment for the initial investigation, posted on Virology Blog last October, as well as for several follow-up articles. Now Dr. Horton has doubled-down on his efforts to keep a lid on the controversy by rejecting a letter that he personally solicited—a major breach of professional courtesy to the 43 well-regarded researchers and clinicians who signed it.

As Dr. Racaniello explained this week at Virology Blog, he submitted the letter on behalf of the group in March, in response to an express invitation from Dr. Horton. The invitation came right after Virology Blog posted an open letter, based on my investigation, that outlined the trial’s major missteps. Dr. Racaniello presumed from the wording of Dr. Horton’s invitation that the letter would, in fact, be published, as did the other signatories. On Monday, having been dissed by The Lancet, Dr. Racaniello finally posted the letter on PubMed Commons. He also called the PACE trial “a sham.” (I’ve called it “a piece of crap.” I might also have referred to it somewhere as “doggie-poo,” but I’m not sure.)

In rejecting the letter that he himself solicited, Dr. Horton certainly appeared to be trying to squelch the growing public controversy over PACE and its recommendations that graded exercise therapy and cognitive behavior therapy are effective treatments for chronic fatigue syndrome (or myalgic encephalomyelitis, CFS, ME, CFS/ME, or ME/CFS, or some other name). But The Lancet’s effort to shield PACE is doomed, not only because the study is so bad but because the emerging science presents a completely different portrait of the illness. On Monday, a paper in Proceedings of the National Academy of Sciences reported distinct patterns of metabolites in the plasma of ME/CFS patients—an important finding that, if confirmed, could finally lead to diagnostic tests. The PNAS paper and other recent research support the conclusion of reports last year from both the Institute of Medicine and National Institutes of Health: ME/CFS is a devastating physiological disease.

Back in January, Columbia statistics professor Andrew Gelman blogged about the harm Dr. Horton was already inflicting on his journal by not addressing the serious questions that serious critics were raising about PACE. The longstanding claim of the PACE authors, The Lancet and the trial’s other defenders—that the opponents were a small cabal of irrational, dangerous, and anti-psychiatry patients—has been exposed as false. The PACE authors, The Lancet and their colleagues wielded this narrative for years to discredit those challenging the trial. To their dismay, this tactic is no longer working.

The Lancet’s decision to reject the Virology Blog letter will only compound the journal’s growing reputational damage over the issue. It also seems deeply short-sighted, in light of last month’s powerful court decision ordering Queen Mary University of London, the professional home of principal PACE investigator Peter White, to release the raw trial data. That would allow others to determine whether the PACE investigators altered their outcome assessments strategies to produce results more likely to get published in The Lancet and other journals. (The answer should not surprise anyone except those in extreme stages of denial.)

The decision involved a freedom-of-information request filed two years ago by Alem Matthees, an Australian patient. Since the published results did not include the results per the assessment methods outlined in the PACE trial protocol, Matthees wanted the data necessary to calculate those results for the two primary outcomes of fatigue and physical function, as well as for the original definition of “recovery.” Last October, the Information Commissioner’s Office, an independent agency, found that QMUL had no grounds for refusing to provide the data. QMUL appealed that ruling to the First-Tier Tribunal, which issued the recent decision.

The U.K. medical-academic-media establishment has wholly endorsed the PACE trial’s unreliable findings and accepted the authors’ unsubstantiated claims that they have been subjected to a concerted campaign of threats and harassment. In contrast, the tribunal demonstrated a refreshing unwillingness to play along. In robust language, the tribunal smacked down the specious arguments raised by the university in its attempt to shield the data from public disclosure.

The chance that any participant could or would be identified from the anonymized data was “remote,” the tribunal found. The scenarios envisioned ed by QMUL’s data security expert, who sketched out far-fetched strategies that “activist” patients might pursue to re-identify and then harass trial participants, were “grossly exaggerated” and “a considerable amount of supposition and speculation,” wrote the tribunal. In fact, noted the tribunal, the only incident of “harassment” proven by QMUL’s experienced legal team was that someone somewhere once heckled Trudie Chalder, a principal PACE investigator who also testified at the tribunal hearing. (I also have some thoughts on Dr. Chalder’s testimony, but will hold those for another time.)

In contrast to the QMUL portrait of PACE opponents, the tribunal cited Virology Blog’s open letter to The Lancet as evidence of a robust scientific debate, noting that “the identity of those questioning the research…was impressive.” The tribunal also noted that QMUL’s decision to share data with friendly researchers but not with others had created the impression that it was acting out of self-interest, not principle. “There is a strong public interest in releasing the data given the continued academic interest so long after the research was published and the seeming reluctance for Queen Mary University to engage with other academics they thought were seeking to challenge their findings,” declared the tribunal in the decision.

The PACE authors, QMUL, Dr. Horton, and The Lancet are stonewalling the obvious, at the expense of millions of sick patients. Although Dr. Horton will never grant me an interview, I want to highlight some of the questions I have about his actions, claims and thoughts, in case someone else gets the chance to talk with him. This list of questions is certainly not exhaustive, but it’s a decent start.

So, Dr. Horton–Here’s what I’d like to ask you:

1) Do you agree that the invitation you sent to Dr. Racaniello certainly implied, even if it didn’t explicitly promise, that The Lancet would publish the letter? Since the letter submitted by Dr. Racaniello, on behalf of himself and 42 other experts, reflected the points made in the Virology Blog open letter that triggered your invitation, what changed your mind about whether it added something to the debate? Since you personally solicited the letter from Dr. Racaniello and his colleagues, do you feel you should have sent him a personal apology, rather than leaving your correspondence editor, Audrey Ceschia, to answer for your behavior?

2) In your invitation to Dr. Racaniello, you noted that the PACE authors would have a chance to respond, alongside the published letter. That was a fair plan. When did that plan of offering them a response morph into the plan of offering them a role in discussions about whether to publish the critical letter in the first place? What impact did their views have on your decision? Did the PACE authors argue, as they have in the past, that they have already answered all these criticisms?

This repeated claim that they have answered all questions is simply untrue. They have never explained, for example, how it is possible to be disabled and “within normal range” on an indicator simultaneously, and why 13 % of their participants were already “within normal range” on one or both primary outcome sat baseline. When anyone asks legitimate questions, they evade, ignore or misstate the issues—including in the correspondence following The Lancet’s 2011 paper. (This pattern of non-response is clear from their non-responsive responses to the charges raised in my Virology Blog investigation, and my rebuttal of their non-responses.)

3) What’s your reaction to the First-Tier Tribunal’s decision ordering the release of the PACE trial data? Do you agree with the tribunal’s observation, referring to Virology Blog’s February open letter to you and The Lancet, that the roster of scientists and researchers now publicly questioning the methodology and findings of PACE is “impressive”?

4) Do think QMUL should spend more public money to appeal the decision?

If QMUL decides to appeal, do you think this will fuel the already-widespread assumption that PACE had null findings per the original protocol methods of assessment?

5) The PACE interventions, as described in The Lancet, are based on the premise that deconditioning rather than any pathological process perpetuates the illness, and that increased activity and a new psychological mind-set will fix the problem. The descriptions of the interventions categorically exclude the possibility of a continuing organic disease as the cause. Do you think this portrait of the illness squares with the view emerging from this week’s study in PNAS and other recent research, including last year’s reports from the Institute of Medicine and the National Institutes of Health?

6) The IOM report identified “exertion intolerance”—the prolonged relapses patients often suffer after minimal activity–as the core symptom of the illness. Yet a key aspect of the PACE rehabilitative interventions, GET and CBT, is urging patients to increase their activity and to interpret a resurgence of symptoms as a transient event, not a sign of deterioration. Given the IOM’s focus on “exertion intolerance” as the central phenomenon, isn’t the PACE approach contraindicated?

7) Does it bother you that you published a paper in which 13% of the sample had already, at baseline, met the outcome thresholds for one or both primary measures? These outcome thresholds, which represented worse health than the entry criteria, were variously defined as being “within normal range” (the Lancet paper), “back to normal” (Dr. Chalder’s statement at the press conference for the Lancet paper), and “a strict criterion for recovery” (the Lancet commentary by colleagues of the PACE authors). Can you point me to any other studies published in The Lancet, or anywhere, in which positive outcome scores represented worse health than entry criteria?

8) Does it bother you personally that the PACE authors did not inform you or your editorial staff that a significant minority of patients were already “within normal range” on at least one primary outcome at baseline? (I presume they didn’t mention it to you because, well, it’s hard to imagine you would have published the paper if you or anyone there had been told about or noticed the inexplicable overlap in the entry criteria and the post-hoc “normal range” thresholds.)

9) During a 2011 Australian radio interview not long after The Lancet published the first PACE results, you said the following about the trial’s critics: “One sees a fairly small, but highly organised, very vocal and very damaging group of individuals who have I would say actually hijacked this agenda and distorted the debate so that it actually harms the overwhelming majority of patients.” Given that the First-Tier Tribunal expressed a different perspective on the stature and credibility of those criticizing PACE, do you still agree with your 2011 characterization of the trial’s opponents?

10) During the same interview, you stated that the PACE trial had undergone “endless rounds of peer review.” Yet the trial was also “fast-tracked” to publication, as indicated on the version of the article in the ScienceDirect database. Can you explain the mechanics of “fast-tracking” a paper to publication while simultaneously subjecting it to “endless rounds of peer review”? How long was the fast-track process for the PACE paper, and how many actual rounds of review did the paper undergo during that endless period?

11) Can you explain why the Lancet’s endless peer review process did not catch the most criticized aspect of the paper—the very obvious fact that participants could be simultaneously disabled enough for entry yet already “within normal range”/”back to normal”/”recovered” on the primary outcomes? Can you explain why the reviewers did not request the authors to provide either the original results promised in the protocol or else sensitivity analyses to assess the impact of the mid-trial changes they introduced?

12) Do you think it was appropriate for the PACE investigators to publish a mid-trial newsletter that promoted the therapies under study and included glowing testimonials from earlier participants about their excellent outcomes? Can you point to other published studies that featured such mid-trial dissemination of personal testimonials and explicit descriptions of outcomes? The PACE authors have stated that the newsletter testimonials did not identify participants’ trial arms and therefore could not have created any bias. Do you agree with this novel and creative argument that influencing all remaining participants in a trial in a positive direction is not a form of bias?

13) Did The Lancet’s peer review process include an evaluation of the PACE trial’s consent forms, given the authors’ explicit promise in the protocol to abide by the Declaration of Helsinki? The Declaration of Helsinki requires investigators to disclose “any possible conflicts of interest” not just to journals but to prospective participants. Yet the PACE consent forms did not disclose the authors’ close financial and consulting ties with the insurance industry. Do you agree this omission violates their protocol promise, and that given this violation the PACE authors failed to obtain legitimate informed consent from their participants? Without legitimate informed consent, did the PACE authors have the right to publish their findings in The Lancet and other journals? What should happen to the PACE papers already published, since the authors do not appear to have legitimate informed consent from participants?

14) Who do you think should be held responsible for the $8,000,000 in U.K. government funds wasted on the PACE trial? Who should be held responsible for the harm it has caused? What responsibility, if any, does The Lancet bear for the debacle?

Once Again, Lancet Stumbles on PACE

Last February, Virology Blog posted an open letter to The Lancet and its editor, Dr. Richard Horton, describing the indefensible flaws of the PACE trial of treatments for ME/CFS, the disease otherwise known as chronic fatigue syndrome (link to letter). Forty-two well-regarded scientists, academics and clinicians put their names to the letter, which declared flatly that the flaws in PACE “have no place in published research.” The letter called for a completely independent re-analysis of the PACE trial data, since the authors have refused to publish the results they outlined in their original protocol. The letter was also sent directly to Dr. Horton.

The open letter was based on the extensive investigative report written by David Tuller, the academic coordinator of UC Berkeley’s joint program in journalism and public health, which Virology Blog posted last October (link to report). This report outlines such egregious failings as outcome thresholds that overlapped with entry criteria, mid-trial promotion of the therapies under investigation, failure to provide the original results as outlined in the protocol, failure to adhere to a specific promise in the protocol to inform participants about the investigators’ conflicts of interest, and other serious lapses.

Virology Blog first posted the open letter in November, with six signatories (link to letter). At that time, Dr. Horton’s office responded that he would reply after returning from “traveling.” Three months later, we still had not heard back from Dr. Horton–perhaps he was still “traveling”–so we decided to republish it with many more people signed on.

The day the second open letter was posted, Dr Horton e-mailed me and solicited a letter from the group. (He did not explain where he had been “traveling” for the previous three months.) Here’s what he wrote: “Many thanks for your email. In the interests of transparency, I would like to invite you to submit a letter for publication–please keep your letter to around 500 words. We will then invite the authors of the study to reply to your very serious allegations.”

Dr. Horton’s e-mail clearly indicated that the letter would be published, with the PACE authors’ response to the charges raised; there was no equivocation or possibility of misinterpretation. In good faith, we submitted a letter for publication the following month, with 43 signatories this time, through The Lancet’s online editorial system (see the end of this article for a list of those who signed the letter). After several months with no response, we learned only recently by checking the online editorial system that The Lancet had flatly rejected the letter, with no explanation. No one contacted me to explain the decision or why we were asked to spend time creating a letter that The Lancet clearly had no intention of publishing.

I wrote back to Dr. Horton, pointing out that his behavior was highly unprofessional and requesting an explanation for the rejection. I also asked him if he was in the habit of soliciting letters from busy scientists and researchers that his journal had no actual interest in publishing. I further asked if the journal planned to reconsider this rejection, in light of the recent First-Tier Tribunal decision, which demolished the PACE authors’ bogus reasons for refusing to provide data for independent analysis.

Dr. Horton did not himself apologize or even deign to respond. Instead, Audrey Ceschia, the Lancet’s correspondence editor, replied, explaining that the Lancet editorial staff decided, after discussing the matter with the PACE authors, that the letter did not add anything substantially new to the discussion. She assured us that if we submitted another letter focused on the First-Tier Tribunal decision, it would be “seriously” considered. I’m not sure why she or Dr. Horton think that any such assurance from The Lancet is credible at this point.

The reasons given for the rejection are clearly specious. The letter for publication reflected the matters addressed in the open letter that prompted Dr. Horton’s invitation in the first place, and closely adhered to his directive  to outline our “serious allegations”. If outlining these allegations was not considered publication-worthy by The Lancet, it is incomprehensible to us why Dr. Horton solicited the letter in the first place. Perhaps it was just an effort to hold off further criticism for a period of months while we awaited publication of the letter, unaware of the journal’s intention to reject it. It is certainly surprising that The Lancet appears to have given the PACE authors some power to determine what letters appear in the journal itself.

Dr. Tuller’s investigation, based on the groundbreaking analyses conducted by many savvy patients and advocates since The Lancet published the first PACE results in 2011, has effectively demolished the credibility of the findings. So has a follow-up analysis by Dr. Rebecca Goldin, a math professor at George Mason University and director of Stats.org, a think tank co-sponsored by the American Statistical Association. In short, the PACE study is a sham, with meaningless results. In this case, the emperor truly has no clothes. Dr. Horton and his editorial team at The Lancet are stark naked.

Yet the PACE study remains in the literature. Its recommendation of treatments that are potentially harmful to patients–specifically, graded exercise therapy and cognitive behavior therapy, both designed specifically to increase patients’ activity levels–remains highly influential.

Of particular concern, the PACE findings have laid the groundwork for the MAGENTA study, a so-called “PACE for kids” that will be testing graded exercise therapy in children and adolescents. A feasibility study, sponsored by Royal United Hospitals Bath NHS Foundation Trust, is currently recruiting participants. It is, of course, completely unacceptable that any study should justify itself based on the uninterpretable findings of the PACE trial. The MAGENTA trial should be halted until the PACE authors have done what the First-Tier Tribunal ordered them to do–release their raw data and allow others to analyze it according to the outcomes specified in the PACE trial protocol.

Today, because of the urgency of the issue, we are posting on PubMed Commons the letter that The Lancet rejected. That way readers can judge for themselves whether it adds anything to the current debate.

Please note that the opinions in this blog post are mine only, not those of any of the other signers of the Lancet letter, listed below

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, New York

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, California

Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, England, United Kingdom

Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, Illinois

Bruce Levin, PhD
Professor of Biostatistics
Columbia University
New York, New York

Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley
Berkeley, California

******

Dharam V. Ablashi, DVM, MS, Dip Bact
Scientific Director – HHV-6 Foundation
Former Senior Investigator
National Cancer Institute, NIH
Bethesda, Maryland

James N. Baraniuk, MD
Professor, Department of Medicine
Georgetown University
Washington, D.C.

Lisa F. Barcellos, PhD, MPH
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California
Berkeley, California

Lucinda Bateman MD PC
MECFS and Fibromyalgia clinician
Salt Lake City, Utah

Alison C. Bested MD FRCPC
Clinical Associate Professor of Hematology
University of British Columbia
Vancouver, British Columbia, Canada

John Chia, MD
Clinician/Researcher
EV Med Research
Lomita, California

Lily Chu, MD, MSHS
Independent Researcher
San Francisco, California

Derek Enlander, MD, MRCS, LRCP
Attending Physician
Mount Sinai Medical Center, New York
ME CFS Center, Mount Sinai School of Medicine
New York, New York

Mary Ann Fletcher, PhD
Schemel Professor of Neuroimmune Medicine
College of Osteopathic Medicine
Nova Southeastern University
Professor Emeritus, University of Miami School of Medicine
Fort Lauderdale, Florida

Kenneth Friedman, PhD
Associate Professor of Pharmacology and Physiology (retired)
New Jersey Medical School
University of Medicine and Dentistry of NJ
Newark, New Jersey

Robert F. Garry, PhD
Professor of Microbiology and Immunology
Tulane University School of Medicine
New Orleans, Louisiana

Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, Virginia

David L. Kaufman, MD,
Medical Director
Open Medicine Institute
Mountain View, California

Susan Levine, MD
Clinician, Private Practice
Visiting Fellow, Cornell University
New York, New York

Alan R. Light, PhD
Professor, Department of Anesthesiology
Department of Neurobiology and Anatomy
University of Utah
Salt Lake City, Utah

Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, Virginia

Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs
Solve ME/CFS Initiative
Los Angeles, California

James M. Oleske, MD, MPH
Francois-Xavier Bagnoud Professor of Pediatrics
Senator of RBHS Research Centers, Bureaus, and Institutes
Director, Division of Pediatrics Allergy, Immunology & Infectious Diseases
Department of Pediatrics
Rutgers – New Jersey Medical School
Newark, New Jersey

Richard N. Podell, M.D., MPH
Clinical Professor
Rutgers Robert Wood Johnson Medical School
New Brunswick, New Jersey

William Satariano, PhD
Professor of Epidemiology and Community Health
University of California, Berkeley
Berkeley, California

Paul T Seed MSc CStat CSci
Senior Lecturer in Medical Statistics
King’s College London, Division of Women’s Health
St Thomas’ Hospital
London, England, United Kingdom

Charles Shepherd, MB BS
Honorary Medical Adviser to the ME Association
London, England, United Kingdom

Christopher R. Snell, PhD
Scientific Director
WorkWell Foundation
Ripon, California

Nigel Speight, MA, MC, BChir, FRCP, FRCPCH, DCH
Pediatrician
Durham, England, United Kingdom

Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, California

Eleanor Stein, MD FRCP(C)
Assistant Clinical Professor
University of Calgary
Calgary, Alberta, Canada

John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, California

Ronald G. Tompkins, MD, ScD
Summer M Redstone Professor of Surgery
Harvard University
Boston, Massachusetts

Rosemary Underhill, MB BS.
Physician, Independent Researcher
Palm Coast, Florida

Dr Rosamund Vallings MNZM, MB BS
General Practitioner
Auckland, New Zealand

Michael VanElzakker, PhD
Research Fellow, Psychiatric Neuroscience Division
Harvard Medical School and Massachusetts General Hospital
Boston, Massachusetts

Mark Vink, MD
Family Physician
Soerabaja Research Center
Amsterdam, The Netherlands

Prof Dr FC Visser
Cardiologist
Stichting CardioZorg
Hoofddorp, The Netherlands

William Weir, FRCP
Infectious Disease Consultant
London, England, United Kingdom

John Whiting, MD
Specialist Physician
Private Practice
Brisbane, Australia

Marcie Zinn, PhD
Research Consultant in Experimental Neuropsychology, qEEG/LORETA, Medical/Psychological Statistics
NeuroCognitive Research Institute, Chicago
Center for Community Research
DePaul University
Chicago, Illinois

Mark Zinn, MM
Research consultant in experimental electrophysiology
Center for Community Research
DePaul University
Chicago, Illinois

TWiV 397: Trial by error

Journalism professor David Tuller returns to TWiV for a discussion of the PACE trial for ME/CFS: the many flaws in the trial, why its conclusions are useless, and why the data must be released and re-examined.

You can find TWiV #397 at microbe.tv/twiv, or listen below.

Click arrow to play
Download TWiV 397 (67 MB .mp3, 93 min)
Subscribe (free): iTunesRSSemailGoogle Play Music

Become a patron of TWiV!

An open letter to PLoS One

PLoS One
1160 Battery Street
Koshland Building East, Suite 100
San Francisco, CA 94111

Dear PLoS One Editors:

In 2012, PLoS One published “Adaptive Pacing, Cognitive Behaviour Therapy, Graded Exercise, and Specialist Medical Care for Chronic Fatigue Syndrome: A Cost-Effectiveness Analysis.” This was one in a series of papers highlighting results from the PACE study—the largest trial of treatments for the illness, also known as ME/CFS. Psychologist James Coyne has been seeking data from the study based on PLoS’ open-access policies, an effort we support.

However, as David Tuller from the University of California, Berkeley, documented in an investigation of PACE published last October on Virology Blog, the trial suffered from many indefensible flaws, as patients and advocates have argued for years. Among Dr. Tuller’s findings: the main claim of the PLoS One paper–that cognitive behavior therapy and graded exercise therapy are cost-effective treatments–is wrong, since it is based on an erroneous characterization of the study’s sensitivity analyses. The PACE authors have repeatedly cited this inaccurate claim of cost-effectiveness to justify their continued promotion of these interventions.

Yet the claim is not supported by the evidence, and it is not necessary to obtain the study data to draw this conclusion. The claim is based solely on the decision to value the free care provided by family and friends as if it were compensated at the level of a well-paid health care worker. Here is what Dr. Tuller wrote last October about the PLoS One paper and its findings:

        The PLoS One paper argued that the graded exercise and cognitive behavior therapies were the most cost-effective treatments from a societal perspective. In reaching this conclusion, the investigators valued so-called  “informal” care—unpaid care provided by family and friends–at the replacement cost of a homecare worker. The PACE statistical analysis plan (approved in 2010 but not published until 2013) had included two additional, lower-cost assumptions. The first valued informal care at minimum wage, the second at zero compensation. 

       The PLoS One paper itself did not provide these additional findings, noting only that “sensitivity analyses revealed that the results were robust for alternative assumptions.”

Commenters on the PLoS One website, including [patient] Tom Kindlon, challenged the claim that the findings would be “robust” under the alternative assumptions for informal care. In fact, they pointed out, the lower-cost conditions would reduce or fully eliminate the reported societal cost-benefit advantages of the cognitive behavior and graded exercise therapies. 

        In a posted response, the paper’s lead author, Paul McCrone, conceded that the commenters were right about the impact that the lower-cost, alternative assumptions would have on the findings. However, McCrone did not explain or even mention the apparently erroneous sensitivity analyses he had cited in the paper, which had found the societal cost-benefit advantages for graded exercise therapy and cognitive behavior therapy to be “robust” under all assumptions. Instead, he argued that the two lower-cost approaches were unfair to caregivers because families deserved more economic consideration for their labor.

        “In our opinion, the time spent by families caring for people with CFS/ME has a real value and so to give it a zero cost is controversial,” McCrone wrote. “Likewise, to assume it only has the value of the minimum wage is also very restrictive.”

       In a subsequent comment, Kindlon chided McCrone, pointing out that he had still not explained the paper’s claim that the sensitivity analyses showed the findings were “robust” for all assumptions. Kindlon also noted that the alternative, lower-cost assumptions were included in PACE’s own statistical plan.

      “Remember it was the investigators themselves that chose the alternative assumptions,” wrote Kindlon. “If it’s ‘controversial’ now to value informal care at zero value, it was similarly ‘controversial’ when they decided before the data was looked at, to analyse the data in this way. There is not much point in publishing a statistical plan if inconvenient results are not reported on and/or findings for them misrepresented.”

Given that Dr. McCrone, the lead author, directly contradicted in his comments what the paper itself claimed about sensitivity analyses having confirmed the “robustness” of the findings under other assumptions, it is clearly not necessary to scrutinize the study data to confirm that this central finding cannot be supported. Dr. McCrone has not responded to e-mail requests from Dr. Tuller to explain the discrepancy. And PLoS One, although alerted to this problem last fall by Dr. Tuller, has apparently not yet taken steps to rectify the misinformation about the sensitivity analyses contained in the paper.

PLoS One has an obligation to question Dr. McCrone about the contradiction between the text of the paper and his subsequent comments, so he can either provide a reasonable explanation, produce the actual sensitivity analyses demonstrating “robustness” under all three assumptions outlined in the statistical analysis plan, or correct the paper’s core finding that CBT and GET are “cost-effective” no matter how informal care is valued.  Should he fail to do so, PLoS One has an obligation itself to correct the paper, independent of the disposition of the issue of access to trial data.

We appreciate your quick response to these concerns.

Sincerely,

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University

Rebecca Goldin, Ph.D.
Professor of Mathematical Sciences
George Mason University

Bruce Levin, PhD
Professor of Biostatistics
Columbia University

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University

Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley

An open letter to The Lancet, again

On November 13th, five colleagues and I released an open letter to The Lancet and editor Richard Horton about the PACE trial, which the journal published in 2011. The study’s reported findings–that cognitive behavior therapy and graded exercise therapy are effective treatments for chronic fatigue syndrome–have had enormous influence on clinical guidelines for the illness. Last October, Virology Blog published David Tuller’s investigative report on the PACE study’s indefensible methodological lapses. Citing these problems, we noted in the letter that “such flaws have no place in published research” and urged Dr. Horton to commission a fully independent review.

Although Dr. Horton’s office e-mailed that he would respond to our letter when he returned from “traveling,” it has now been almost three months. Dr. Horton has remained silent on the issue. Today, therefore, we are reposting the open letter and resending it to The Lancet and Dr. Horton, with the names of three dozen more leading scientists and clinicians, most of them well-known experts in the ME/CFS field.

We still hope and expect that Dr. Horton will address–rather than continue to ignore–these critical concerns about the PACE study.

****

Dr. Richard Horton

The Lancet
125 London Wall
London, EC2Y 5AS, UK

Dear Dr. Horton:

In February, 2011, The Lancet published an article called “Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomized trial.” The article reported that two “rehabilitative” approaches, cognitive behavior therapy and graded exercise therapy, were effective in treating chronic fatigue syndrome, also known as myalgic encephalomyelitis, ME/CFS and CFS/ME. The study received international attention and has had widespread influence on research, treatment options and public attitudes.

The PACE study was an unblinded clinical trial with subjective primary outcomes, a design that requires strict vigilance in order to prevent the possibility of bias. Yet the study suffered from major flaws that have raised serious concerns about the validity, reliability and integrity of the findings. The patient and advocacy communities have known this for years, but a recent in-depth report on this site, which included statements from five of us, has brought the extent of the problems to the attention of a broader public. The PACE investigators have replied to many of the criticisms, but their responses have not addressed or answered key concerns.

The major flaws documented at length in the recent report include, but are not limited to, the following:

*The Lancet paper included an analysis in which the outcome thresholds for being “within the normal range” on the two primary measures of fatigue and physical function demonstrated worse health than the criteria for entry, which already indicated serious disability. In fact, 13 percent of the study participants were already “within the normal range” on one or both outcome measures at baseline, but the investigators did not disclose this salient fact in the Lancet paper. In an accompanying Lancet commentary, colleagues of the PACE team defined participants who met these expansive “normal ranges” as having achieved a “strict criterion for recovery.” The PACE authors reviewed this commentary before publication.

*During the trial, the authors published a newsletter for participants that included positive testimonials from earlier participants about the benefits of the “therapy” and “treatment.” The same newsletter included an article that cited the two rehabilitative interventions pioneered by the researchers and being tested in the PACE trial as having been recommended by a U.K. clinical guidelines committee “based on the best available evidence.” The newsletter did not mention that a key PACE investigator also served on the clinical guidelines committee. At the time of the newsletter, two hundred or more participants—about a third of the total sample–were still undergoing assessments.

*Mid-trial, the PACE investigators changed their protocol methods of assessing their primary outcome measures of fatigue and physical function. This is of particular concern in an unblinded trial like PACE, in which outcome trends are often apparent long before outcome data are seen. The investigators provided no sensitivity analyses to assess the impact of the changes and have refused requests to provide the results per the methods outlined in their protocol.

*The PACE investigators based their claims of treatment success solely on their subjective outcomes. In the Lancet paper, the results of a six-minute walking test—described in the protocol as “an objective measure of physical capacity”–did not support such claims, notwithstanding the minimal gains in one arm. In subsequent comments in another journal, the investigators dismissed the walking-test results as irrelevant, non-objective and fraught with limitations. All the other objective measures in PACE, presented in other journals, also failed. The results of one objective measure, the fitness step-test, were provided in a 2015 paper in The Lancet Psychiatry, but only in the form of a tiny graph. A request for the step-test data used to create the graph was rejected as “vexatious.”

*The investigators violated their promise in the PACE protocol to adhere to the Declaration of Helsinki, which mandates that prospective participants be “adequately informed” about researchers’ “possible conflicts of interest.” The main investigators have had financial and consulting relationships with disability insurance companies, advising them that rehabilitative therapies like those tested in PACE could help ME/CFS claimants get off benefits and back to work. They disclosed these insurance industry links in The Lancet but did not inform trial participants, contrary to their protocol commitment. This serious ethical breach raises concerns about whether the consent obtained from the 641 trial participants is legitimate.

Such flaws have no place in published research. This is of particular concern in the case of the PACE trial because of its significant impact on government policy, public health practice, clinical care, and decisions about disability insurance and other social benefits. Under the circumstances, it is incumbent upon The Lancet to address this matter as soon as possible.

We therefore urge The Lancet to seek an independent re-analysis of the individual-level PACE trial data, with appropriate sensitivity analyses, from highly respected reviewers with extensive expertise in statistics and study design. The reviewers should be from outside the U.K. and outside the domains of psychiatry and psychological medicine. They should also be completely independent of, and have no conflicts of interests involving, the PACE investigators and the funders of the trial.

Thank you very much for your quick attention to this matter.

Sincerely,

Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University

Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London

Leonard A. Jason, PhD
Professor of Psychology
DePaul University

Bruce Levin, PhD
Professor of Biostatistics
Columbia University

Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University

Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley

****

Dharam V. Ablashi, DVM, MS, Dip Bact
Scientific Director, HHV-6 Foundation
Former Senior Investigator
National Cancer Institute, NIH
Bethesda, Maryland

James N. Baraniuk, MD
Professor, Department of Medicine,
Georgetown University
Washington, D.C.

Lisa F. Barcellos, PhD, MPH
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California
Berkeley, California

Lucinda Bateman, MD
Medical Director, Bateman Horne Center
Salt Lake City, Utah

David S. Bell, MD
Clinical Associate Professor of Pediatrics
State University of New York at Buffalo
Buffalo, New York

Alison C. Bested MD FRCPC
Clinical Associate Professor of Hematology
University of British Columbia
Vancouver, British Columbia, Canada

Gordon Broderick, PhD
Director, Clinical Systems Biology Group
Institute for Neuro Immune Medicine
Professor, Dept of Psychology and Neuroscience
College of Psychology
Nova Southeastern University
Miami, Florida

John Chia, MD
Clinician/Researcher
EV Med Research
Lomita, California

Lily Chu, MD, MSHS
Independent Researcher
San Francisco, California

Derek Enlander, MD, MRCS, LRCP
Attending Physician
Mount Sinai Medical Center, New York
ME CFS Center, Mount Sinai School of Medicine
New York, New York

Mary Ann Fletcher, PhD
Schemel Professor of Neuroimmune Medicine
College of Osteopathic Medicine
Nova Southeastern University
Professor Emeritus, University of Miami School of Medicine
Fort Lauderdale, Florida

Kenneth Friedman, PhD
Associate Professor of Pharmacology and Physiology (retired)
New Jersey Medical School
University of Medicine and Dentistry of NJ
Newark, New Jersey

David L. Kaufman, MD,
Medical Director
Open Medicine Institute
Mountain View, California

Nancy Klimas, MD
Professor and Chair, Department of Clinical Immunology
Director, Institute for Neuro-Immune Medicine
Nova Southeastern University
Director, GWI and ME/CFS Research, Miami VA Medical Center
Miami, Florida

Charles W. Lapp, MD
Director, Hunter-Hopkins Center
Assistant Consulting Professor at Duke University Medical Center
Charlotte, North Carolina

Susan Levine, MD
Clinician, Private Practice
New York, New York
Visiting Fellow, Cornell University
Ithaca, New York

Alan R. Light, PhD
Professor, Department of Anesthesiology and Department of Neurobiology and Anatomy
University of Utah
Salt Lake City, Utah

Sonya Marshall-Gradisnik, PhD
Professor and Co-Director
National Centre for Neuroimmunology and Emerging Diseases
Griffith University
Queensland, Australia

Peter G. Medveczky, MD
Professor, Department of Molecular Medicine, MDC 7
College of Medicine
University of South Florida
Tampa, Florida

Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs
Solve ME/CFS Initiative
Los Angeles, California

James M. Oleske, MD, MPH
Francois-Xavier Bagnoud Professor of Pediatrics
Senator of RBHS Research Centers, Bureaus, and Institutes
Director, Division of Pediatrics Allergy, Immunology & Infectious Diseases
Department of Pediatrics
Rutgers – New Jersey Medical School
Newark, New Jersey

Richard N. Podell, M.D., MPH
Clinical Professor
Rutgers Robert Wood Johnson Medical School
New Brunswick, New Jersey

Charles Shepherd, MB, BS
Honorary Medical Adviser to the ME Association
London, United Kingdom

Christopher R. Snell, PhD
Scientific Director
WorkWell Foundation
Ripon, California

Nigel Speight, MA, MB, BChir, FRCP, FRCPCH, DCH
Pediatrician
County Durham, United Kingdom

Donald Staines, MBBS MPH FAFPHM FAFOEM
Professor and Co-Director
National Centre for Neuroimmunology and Emerging Diseases
Griffith University
Queensland, Australia

Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, California

Eleanor Stein, MD FRCP(C)
Assistant Clinical Professor
University of Calgary
Calgary, Alberta, Canada

John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, California

Ronald G. Tompkins, MD, ScD
Summer M Redstone Professor of Surgery
Harvard University
Boston, Massachusetts

Rosemary Underhill, MB BS.
Physician, Independent Researcher
Palm Coast, Florida

Dr Rosamund Vallings MNZM, MB BS
General Practitioner
Auckland, New Zealand

Michael VanElzakker, PhD
Research Fellow, Psychiatric Neuroscience Division
Harvard Medical School & Massachusetts General Hospital
Boston, Massachusetts

William Weir, FRCP
Infectious Disease Consultant
London, England

Marcie Zinn, PhD
Research Consultant in Experimental Neuropsychology, qEEG/LORETA, Medical/Psychological Statistics
NeuroCognitive Research Institute, Chicago
Center for Community Research
DePaul University
Chicago, Illinois

Mark Zinn, MM
Research consultant in experimental electrophysiology
Center for Community Research
DePaul University
Chicago, Illinois