David Tuller

Trial By Error: Did the IBS Trial Really Show that Web-Based CBT Offered Significant Clinical Effectiveness?

24 June 2021 by David Tuller Leave a Comment

By David Tuller, DrPH

I wrote some posts last year about the ACTIB trial–a major study of telephone-delivered and web-delivered cognitive behavior therapy (TCBT and WCBT) for irritable bowel syndrome (IBS). Contrary to how the results have been framed by those with reputational and financial interests in promoting them, the study demonstrated that WCBT did not provide clinically significant benefits over treatment as usual on its two primary outcomes. At least that’s how I interpret the data.

Both of these primary outcome measures were self-reported—a symptom severity scale and a measure of impact on work and social functioning. (More on these below.) As no less an authority than the Journal of Psychosomatic Research has recently noted, these sorts of outcomes are at particular risk of bias when blinding is not rigorous. In this case, with blinding not possible given the nature of the interventions, the results should already be assumed to be infused with an unknown amount of bias. In other words, marginal or modest reported benefits would be well within the range expected given the bias inherent in an unblinded study relying on subjective outcomes.

In January of 2020, Mahana Therapeutics, a San-Francisco start-up, announced that it had licensed the rights to the web-based program. In the trial, the program was called Regul8; Mahana rechristened it as Parallel. (Does anyone get the new name? Parallel to what?) I noted in blog posts and in letters to Mahana officials and advisers that the company was making inflated claims about the purported benefits of the program.

Since then, the US Food andDrug Administration has approved it for marketing–although a product like this is not assessed by anything approaching the (purportedly) rigorous standards used for pharmaceuticals. It has also received a CE mark in the UK, which apparently allows it to be sold there as well. (Given Brexit, the implications of the CE mark for European sales are not fully clear to me–or, I presume, to anyone.) This is a very minimal regulatory bar. According to the relevant UK government agency, “the CE mark is not a quality mark, nor a guarantee that the product meets all of the requirements of relevant EU product safety law.”

More recently, the same research team has published cost-effectiveness data from the IBS study. My friend and colleague Keith Geraghty, a research fellow at the University of Manchester, has tweeted about this new paper and some of its assumptions. I haven’t looked at it yet in depth, but what I don’t get is how an intervention can be cost-effective if it isn’t really clinically effective. Cost-effective at what?

The abstract of the new cost-effectiveness study starts with the following sentence: “Telephone therapist delivered CBT (TCBT) and web-based CBT (WCBT) have been shown to be significantly more clinically effective than treatment as usual (TAU) at reducing IBS symptom severity and impact at 12 months in adults with refractory IBS.”

It is important to note that the results for TCBT and WCBT in the IBS study were not the same. TCBT scored better overall on primary and secondary measures. That is not surprising. The greater personal contact could potentially lead to greater therapeutic effects; it could also lead to greater bias in participant responses. Teasing the two phenomena apart would be challenging.

But Mahana is marketing WCBT, not TCBT—and key academic investigators have financial interests in the product. The statement about the interventions being “significantly more clinically effective than treatment as usual” in the new cost-effectiveness paper prompted me to take another look at ACTIB’s 12-month results for the two co-primary outcomes. (Secondary outcomes are secondary outcomes; they should not be advanced as proving success when primary outcomes generate null results.) And when it comes to “reducing IBS symptom severity and impact at 12 months,” the above sentence from the cost-effectiveness paper does not seem to be consistent with the specific WCBT data for the primary outcomes.

**********

Two co-primary outcomes for the IBS study

The two co-primary outcomes were the 12-month results for the IBS-Symptom Severity Scale (IBS-SSS) and the Work and Social Adjustment (WASA) scale. In previous posts, I focused on the results for the first–a 500-point scale in which higher scores represent more severe symptoms. A reduction of at least 50 points represents what is considered a clinically significant reduction in symptom severity for an individual. In this trial, at 12 months, the WCBT group had a mean score of 35.2 points lower than the TAU arm. That is quite a bit less than the 50-point difference considered to be clinically significant for an individual.

Despite this 50-point threshold, the investigators seem to have created something of a statistical out. Here’s what they wrote: “IBS-SSS is widely used in IBS studies and a 50-point within-participant change from baseline regarded as clinically significant. We used a 35-point between-group change as our minimal clinically important difference (MCID) assuming some improvement in TAU arm (15 points placebo response) as in the MIBS trial.”

Given that the 35.2-point finding for the WSAS in the WCBT arm squeaks past the 35-point threshold designated by the investigators, they might consider these results to represent clinical significance. But it is unclear why they should reduce the MCID target for their intervention by 15 points because of a presumed placebo response–even if a previous trial reported such a finding.

The TAU condition is meant to replicate, more or less, the usual care that patients receive. To be considered effective, the intervention should demonstrate a clinically significant benefit when compared with TAU—not when compared with baseline. In fact, the first sentence of the abstract of the new cost-effectiveness paper specifically references the comparison with TAU—not with baseline numbers. After all, clinical trials are designed to assess the effects of interventions over and above changes in those who don’t receive them.

To be fair, the investigators also reported that a greater percentage of those who answered the IBS-SSS at 12 months in the WCBT arm reported an improvement of at least 50 points, compared to the percentage in the TAU arm. But 30% of the participants did not provide 12-month responses for the IBS-SSS, so their responses are unknown. That means interpreting these data can be challenging. Perhaps those who received the intervention were more inspired to report positive results than those who received TAU alone, even if the latter also felt they’d improved to the same extent. We simply don’t know.

In other words, these reported findings do not take into account the large amount of missing data. In contrast, the designated primary outcomes—the differences in the mean scores of the arms at 12 months–were specifically adjusted to address that issue.

The other co-primary outcome measure, the WSAS, is a 40-point scale to assess how an illness or medical condition impacts work and social domains. As with the IBS-SSS, lower scores represent better results. In the IBS study, the differences in mean scores at 12 months between the intervention arms and the TAU arms at 12 months were modest—a 3.5-point difference for TCBT and a 3-point difference for WCBT. The paper did not include an MCID for that measure, unlike for the IBS-SSS.

However, members of the same research team have recently published a study of CBT as a treatment for so-called “persistent physical symptoms” (PPS), a category that includes IBS; in fact, the trial recruited a substantial number of participants from gastroenterology. In this instance, the investigators designated the WSAS as the sole primary outcome and further identified a difference of 3.6-points as the MCID for that measure. As reported, the benefits for the intervention fell well below the MCID for the measure.

If the MCID of a 3.6-point difference on the WSAS is applied to the IBS study findings—and I see no reason why it shouldn’t be, especially given the overlap in research teams and patient populations–the conclusion is clear. Neither the TCBT nor the WCBT provided clinically significant benefits over TAU on that measure.

Yet the investigators still claim in the first sentence of the abstract of their new cost-effectiveness paper that both TCBT and WCBT were shown to have a clinically significant effect on reducing symptom severity and impact. And the WCBT program is being marketed with the same argument. Given that the co-primary outcomes of the major study of WCBT found no clinically significant reduction of symptom severity and no clinically significant impact on improving work and social adjustment at 12 months, these claims are certainly open to question.

Trial By Error: Null Outcomes Presented as Success in Yet Another CBT Trial from Prof Trudie Chalder

19 June 2021 by David Tuller 3 Comments

By David Tuller, DrPH

Trudie Chalder, a professor of cognitive behavior therapy (CBT) at King’s College London, has recently published yet another high-profile paper: the main results for “efficacy” from a trial of CBT for patients with so-called “persistent physical symptoms” (PPS) in secondary care. As usual with this group of investigators, things haven’t turned out well. But despite null results for the primary outcome, Professor Chalder and her like-minded colleagues have cast the findings in a positive light in their article, published in Psychological Medicine.

(Psychological Medicine also published the bogus 2013 “recovery” paper from the PACE team; Professor Chalder was one of the three lead investigators for this classic of likely research misconduct, in which participants could get worse on the primary outcomes and still be deemed. to be “recovered.” When I complained to the editors about it a few years ago, I was advised to replicate the PACE trial; instead, I wrote a letter to the journal that demanded an immediate retraction and garnered more than 100 signatories.)

The new study, called the PRINCE Secondary Trial, is separate from another trial of people with persistent physical symptoms in primary care—the PRINCE Primary Trial. Both are part of the ongoing campaign by Professor Chalder and her colleagues to provide an evidentiary base to justify the expansion of psychological services to anyone suffering from PPS, a category also frequently referred to as “medically unexplained symptoms” (MUS). For Professor Chalder and her colleagues, PPS and MUS include chronic fatigue syndrome, irritable bowel syndrome, fibromyalgia, and pretty much anything else that resists easy clinical assessment and diagnosis and could be inferred to be psychologically driven and/or perpetuated by experts predisposed toward such an interpretation.

First, let’s note that PRINCE Secondary is an unblinded trial relying on self-reported outcomes—a study design fraught with potential and actual bias. This is not just the opinion of people who dislike the PACE trial and believe that Psychological Medicine publishes a lot of crap research. As I’ve noted, the current editor-in-chief of the Journal of Psychosomatic Research, along with his two predecessors, published an editorial earlier this year in which they clearly indicated that subjective outcomes were subject to enormous bias in studies that were not rigorously blinded. (That hasn’t stopped the journal from continuing to publish such problematic research, such as recent output from Professor Chalder’s PACE colleague, Professor Peter White.)

On the basis of this laudable stance from the Journal of Psychosomatic Research, any positive findings from PRINCE Secondary would be suspect from the start. But that’s not even at issue here, given the null results for the primary outcome—the Work and Social Adjustment Scale (WSAS) at 52-weeks. (The invaluable blog CBT Watch has published a critique of the trial.)

PRINCE Secondary, according to the trial protocol, was “an RCT designed to evaluate the efficacy and cost-effectiveness of a transdiagnostic cognitive behavioural intervention for adults with PPS in secondary care.” The intervention—”therapist-delivered transdiagnostic” CBT, or TDT-CBT—was offered in addition to standard medical care (SMC). It was specifically developed to address the following concerns, as outlined in the published paper: “Patients with PPS can develop unhelpful cognitions and behaviour which can consequently lead to a reduction in daily functioning, reduced quality of life, and an increased susceptibility towards developing depression and anxiety.” The comparison arm received SMC alone.

The protocol touted the trial as a major initiative: “The PRINCE Secondary study will be the first trial worldwide to address the efficacy and cost-effectiveness of a manual-based, transdiagnostic, approach…If it proves to be efficacious, this treatment approach could significantly improve overall functioning in patients with PPS and may lead to substantial long-term economic benefits to the NHS.”

That last point is important. Professor Chalder and many of her colleagues have promoted the expansion of the National Health Service’s Improving Access to Psychological Therapies program to people with MUS. After this publication, it will be hard to cite PRINCE Secondary as proof that CBT for MUS is an “efficacious” treatment–on the contrary, the results undermine any such claim. But that probably won’t make Professor Chalder or any other members of the CBT ideological brigades question their own assertions about their favored interventions. [In research, “efficacy” and “efficacious” refer to how interventions perform in controlled studies like clinical trials; “effectiveness” and “effective” refer to how interventions perform in the real world.]

**********

Let’s forget about “efficacy” and cite “helpfulness” instead

The protocol for the PRINCE Secondary Trial, published in BMC Pyschiatry, clearly stated: “Efficacy will be assessed by examining the difference between arms in the primary outcome Work and Social Adjustment Scale (WSAS) at 52 weeks after randomisation.” The trial itself notes that the WSAS “was chosen as the primary outcome because the focus of therapy was on targeting processes which might result in a reduction of the impact of symptoms.” In other words, the TDT-CBT was aimed specifically at influencing the cognitive and behavioral factors that were presumed to be preventing PPS patients from full engagement in their work and social lives.

Ok, then. After conducting their due diligence and assessing all the earlier studies and possible outcome measures in the process of developing an authoritative protocol, the investigators determined how they wanted everyone to definitively measure the impact of their intervention. The WSAS is a 40-point scale. The investigators calculated that a difference of 3.6 points or more on the scale would be considered clinically significant. That is, any change of less than 3.6 points would be of insignificant clinical benefit to an individual—it would be an essentially meaningless blip that would not translate into a noticeable improvement.

At 52 weeks, the mean WSAS score of those who received the intervention was only 1.48 points lower than those did not. (Lower WSAS scores represent improvement.) The p-value was 0.139—far from the 0.05 threshold needed to be considered statistically significant. So the WSAS 52-week findings were both clinically and statistically insignificant. Moreover, the entire confidence interval range (-3.44 to 0.48) fell below the designated 3.6-point threshold for clinical significance. These are pretty unequivocal results. They are definitely not useful to those seeking to promote the use of CBT as a treatment for PPS and MUS.

That’s why the conclusion of the paper’s abstract is so striking, and so bizarre. An abstract’s conclusion is what many people who scan a paper will likely remember the most. Here is the entire conclusion of this abstract: “We have preliminary evidence that TDT-CBT + SMC may be helpful for people with a range of PPS. However, further study is required to maximise or maintain effects seen at end of treatment.”

Anyone who takes the time to review the paper should be mystified by this conclusion. This full-scale trial was approved because a lot of earlier research, as outlined in the protocol, had produced ample “preliminary evidence” of the kind mentioned in the conclusion. The protocol, unless I misread it, did not propose to produce more “preliminary evidence” that the TDT-CBT intervention “may be helpful.” PRINCE Secondary was presented in the protocol and received funding based on the notion that it would produce hard data about “the efficacy and cost-effectiveness” of the intervention. (The Psychological Medicine paper did not include the “cost-effectiveness” data.)

It should be noted that “helpfulness” is not the same as “efficacy” and is not defined in the protocol or the trial itself. An intervention might be “helpful” in some way as a supportive strategy while having no “efficacy” as an actual treatment. In this trial, the method of assessing the “efficacy” of the treatment was clearly designated; the results did not achieve that metric, so the treatment cannot be described as “efficacious.” As a vague stand-in, “helpfulness” sounds positive but can mean more or less anything—as it seems to here.

In the paper, the investigators designate eight secondary outcomes. They tout marginal improvements in three of them as indicating possible “helpfulness.” But the results suggest at best the following: Giving people eight sessions of encouragement and attention could prompt them to upgrade their answers by one step or two on some—but not most—questionnaires, compared to those who do not receive eight weeks of such encouragement and attention. That’s it. Expansive interpretations of “helpfulness” are not justified.

Let’s examine these secondary results in a bit more detail. The first is the WSAS at 20 weeks, which reported a 2.41-point difference between the groups. This is still below the 3.6-point threshold for being a clinically significant difference. And as expected because of the bias inherent with self-reported outcomes in unblinded studies, even this minimal apparent effect was not maintained by the 52-week point. (The WSAS at 20 weeks was not in fact listed as a secondary outcome in the protocol.)

Other results are also unimpressive. Five of the eight listed secondary outcomes did not produce statistically significant findings. Two others did. The intervention group achieved a 1.51-point difference from the comparison group on the 30-point Patient Health Questionnnaire 15 and a 0.55-point difference on the 9-point Global Clinical Impression scale. These minimal reported improvements do not provide convincing evidence in favor of the intervention, since they are well within the range of responses that one might expect from the kind of bias noted by the editors of the Journal of Psychosomatic Research in an unblinded study with subjective outcomes.

*******

The problem of multiple comparisons

And there’s another issue here. When authors engage in multiple comparisons, they increase the likelihood of obtaining some results that reach statistical significance by chance. To make up for that, it is common to correct or adjust the results with standard statistical steps—and the Bonferroni correction is the most well-known. Yet the PRINCE investigators don’t like that approach. It is too stringent for their needs, so they decided not to do it. Here’s what they say:

“Throughout this paper, we present unadjusted p values. Methods for adjusting the family-wise error by methods such as the Bonferroni correction are known to be conservative. However, if one were to use a method that controlled the false-discovery rate such as the Benjamini–Hochberg procedure then the differences on PHQ-15, WSAS at 20 weeks and CGI remained statistically significant and would therefore be considered as discoveries after correction for all nine outcomes (eight secondary plus primary outcome).“

My creative interpretation of this statement: Our findings are weak but they’re even weaker than they appear to be as reported. That’s why we didn’t bother to calculate and present p-values that took into account and corrected for how many tests we ran to try to find statistically significant results. Also, the standard method to correct for multiple tests in a study like this is really, really tough to get through, so we’re not going to use it. But we can assure you that a correction method we like better still lets us call these results “discoveries!” (We’re not presenting those corrected results, but trust us on this one–the study is a success.)

An abstract’s conclusion should at least make an effort to incorporate the findings for the primary outcome—the main results of interest. The conclusion from Chalder and colleagues should have forthrightly noted that the intervention was not efficacious. In this context, to prioritize exceedingly modest unadjusted results from a minority of secondary outcomes over the null results of the primary outcome is an insult to readers. This tactic for hiding bad news reeks of desperation—like a bald man’s comb-over.

Furthermore, it is disingenuous for investigators to assert that these meager data represent “preliminary evidence” for the intervention when the primary outcome was a bust. The call for further research to study how to maintain or extend these effects is unwarranted. The intervention failed to produce the predicted and desired effects. That’s the only credible interpretation of these disastrous results.

Peer reviewers and journal editors are supposed to act as safeguards against misrepresentations. In this case, the system failed. A conclusion that does not mention the null results for the primary outcome is unacceptable—and it should have been unpublishable.

Trial By Error: #MEAction’s Chronic Illness Survey; Patients’ Research Informs CDC’s Long-COVID Advice

15 June 2021 by David Tuller 4 Comments

By David Tuller, DrPH

New survey to assess links between chronic conditions

Jaime Seltzer, #MEAction’s Director of Scientific & Medical Outreach, has been pretty busy lately. This week, she appeared on a panel at the World Health Organization’s Long COVID webinar—alongside no less than the distinguished and always affable Professor Sir Simon Wessely. Seltzer has also just been named to the Independent Advisory Group for Cochrane’s new exercise therapy review. And she recently launched, in cooperation with Dr Mady Hornig, an epidemiologist at Columbia University, the Chronic Illness Survey Adventure (Symptom Cluster Characterization in Complex Chronic Disease), an effort to examine possible links and overlaps between ME/CFS and long COVID, POTS, hEDS, and MCAS.

Trial By Error: More Disinformation from Professor White in Journal of Psychosomatic Research

10 June 2021 by David Tuller 3 Comments

By David Tuller, DrPH

The Journal of Psychosomatic Research seems to be suffering from some sort of identity crisis. Earlier this year, the editor and his two immediate predecessors published an admirable editorial in which they noted the serious risk of bias in subjective outcomes in studies that are not rigorously blinded. Yet the journal’s editorial advisory board is studded with investigators who specialize in unblinded studies that rely solely on subjective outcomes—in particular, Professors Peter White and Michael Sharpe, two of the lead PACE investigators, and Professor Per Fink, their Danish colleague.

Trial By Error: Norway Rejects New Clinical Trial of Woo-Woo Lightning Process

7 June 2021 by David Tuller 10 Comments

By David Tuller, DrPH

In a welcome display of scientific acumen, Norwegian research ethics authorities have rejected a proposed study of the woo-woo called the Lightning Process as a treatment for ME/CFS. Since Norway generally appears to be a hotbed of biopsychosocial thinking, this excellent decision is a bit of a surprise. It follows a heated public debate about the issue over the last year, both before and after regional ethics authorities granted initial approval to the study in November.

(Dagbladet, a popular daily publication, ran an opinion piece of mine last June about the issue. In April, Codastory.com, an international news site, published my lengthy investigation of the Lightning Process, which highlighted the proposed Norwegian study.)

Trial By Error: Journal Editorial Calls for Caution in Exercise-Based Rehab Programs for Long COVID

15 May 2021 by David Tuller

By David Tuller, DrPH

The Journal of Orthopaedic & Sports Physical Therapy recently published an editorial called “Humility and Acceptance: Working Within Our Limits With Long COVID and Myalgic Encephalomyelitis/ Chronic Fatigue Syndrome.” The editorial emerged from a collaboration between rehabilitation specialists and patients, mostly from Canada. It confronts the conundrum of whether those experiencing what is called Long COVID should be advised to exercise, explicitly linking the issue to the history of misguided interventions for ME/CFS and the symptom of post-exertional malaise.

Trial By Error: More Letters about Professor Anthony David’s Mis-Citations of Key Study on Costs of MUS

11 May 2021 by David Tuller

By David Tuller, DrPH

I have been in correspondence with the journal Psychological Medicine in my efforts to get it to correct an undisputed factual era related to the cost of so-called “medically unexplained symptoms” (MUS). After three weeks, the journal’s co-editor-in-chief, Professor Robin Murray, finally alerted me on May 4th that the authors “have agreed” to a correction–as if they were making a concession to me rather than ensuring the accuracy of the medical literature.

Trial By Error: My Letters to Psychosomatics Journal About Prof White’s Misleading GETSET Paper

6 May 2021 by David Tuller

By David Tuller, DrPH

In early April, I wrote about a study published in the Journal of Psychosomatic Research—a one-year follow-up of the GETSET trial of self-help graded exercise therapy for ME/CFS. The investigators had previously reported short-term benefits for the intervention. In this new paper, despite no benefits of the intervention over regular care, the team reported success because the intervention arm did not score worse than it had at 12 weeks. Needless to say, this is not the proper way to report clinical trial results.

Trial By Error: In Guardian Column, Professor Pariante Parrots Standard Biopsychosocial Nonsense

1 May 2021 by David Tuller

By David Tuller, DrPH

On Tuesday (four days ago), The Guardian published an opinion piece from Professor Carmine Pariante titled “Long Covid is very far from ‘all in the mind’–but psychology can still help us to treat it.” The article is the latest from a member of the biopsychosocial ideological brigades to demonstrate what has long been apparent—those who proclaim that others are falling into an unhelpful mind-body dichotomy are using this argument to deflect attention from the failings of research into their favored interventions. Professor Pariante is a psychiatrist based at King’s College London, the home of Professor Trudie Chalder, Professor Sir Simon Wessely, and other leading lights of the biopsychosocial firmament, so he is likely steeped constantly in a brew of self-reaffirming but unwarranted assertions.

Trial By Error: Professor David’s Third Mis-Citation of Seminal Study of “Medically Unexplained Symptoms”

26 April 2021 by David Tuller

By David Tuller, DrPH

*April is crowdfunding month at Berkeley. I conduct this project as a senior fellow in public health and journalism at the university’s Center for Global Public Health. If you would like to support the project with a donation to Berkeley (tax-deductible for US taxpayers), here’s the place: https://crowdfund.berkeley.edu/project/25504

I have recently blogged about the multiple mis-citations of a seminal study involving so-called “medically unexplained symptoms” (MUS). The 2010 study, Bermingham et al, found that the amount spent by the National Health Service on working-age people who were assessed as “somatising” accounted for around 10% of what was spent on that population. Since the study was published more than a decade ago, experts in MUS have routinely misrepresented it by asserting that these costs accounted for 10% of total NHS expenditures—in effect more than tripling their apparent financial impact.