Trial By Error: Revisiting Mahana and Irritable Bowel Syndrome…

By David Tuller, DrPH

As the world continues to wrestle with the coronavirus epidemic, President Trump is calling on us here in the US to get back to work. So I decided to start seeking answers again from Mahana Therapeutics, which announced in January that it had licensed a web-based cognitive behavior therapy program for irritable bowel syndrome from King’s College London.

According to the results of the major study that road-tested the program, the benefits in symptom severity at 12 months over treatment-as-usual were statistically significant but clinically insignificant, and at 24 months they were neither statistically nor clinically significant. In other word, the web-based program proved to be pretty much of a dud. Yet Mahana falsely promoted these results as “dramatic” and “potentially game-changing” on its website.

Apparently, those words have recently been removed–at least, I didn’t see them this morning when I checked the Mahana website. That’s a positive step. Yet the website continues to include misleading statements. This morning, I sent Robert Paull, Mahana’s CEO and co-founder, the following letter.

**********

Dear Mr Paull:

In February, before coronavirus swamped everything, I sent you two letters concerning Mahana Therapeutics’ expansive but unjustified claims about its new web-based cognitive behavior therapy program for irritable bowel syndrome. I did not hear back. I also reached out to three of Mahana’s gasteroenterology advisers. None of them responded.

To recap: Mahana recently announced that it had licensed the web-based CBT program from King’s College London. My concerns involved statements in a press release about the deal and on Mahana’s  website that clearly exceeded the data from the relevant study. The press release described the web-based program’s impacts on symptom severity as “substantial” and “durable.” The website called  them “dramatic” and “potentially game-changing.”

As I have repeatedly pointed out, these descriptions cannot be justified. At 12 months, the reported benefits of the web-based program over treatment-as-usual on the study’s measure of symptom severity were statistically significant but clinically insignificant. At 24 months, these reported benefits were neither statistically nor clinically significant. Given these weak results for the core indicator, it is hard to understand why Mahana decided to license the product in the first place.

It now appears that Mahana has changed the description of the purported benefits on its website and no longer characterizes them as “dramatic” and “potentially game-changing.” I am pleased to see that the company has paid attention to my concerns. However, the site now states that “patients enrolled in a minimal-contact digital CBT program experienced significant and clinically meaningful reduction in the severity of their IBS.”

This is an empty statement, since it would literally be true if only two patients in the web-based arm had achieved this “significant and clinically meaningful reduction” in symptom severity. It could also easily be said about the treatment-as-usual arm, since clearly some patients in that arm also achieved “significant and clinically meaningful reduction” in symptom severity. The relevant question is whether the program provides clinically meaningful benefits over and above what is achieved through treatment-as-usual–and the main results for the symptom severity scale indicate that it does not.

The two data points you still include on the website about these purported benefits are not the central findings for the IBS symptom severity measure. To cite them in this manner is highly misleading. I have previously explained this on Virology Blog, but I guess I need to do so again.

According to Mahana’s website, “66% of patients reported significant and clinically meaningful reduction in the severity of their IBS.”

It is true that 66% of those in the web-based CBT arm who responded at 12 months had a reduction in the scores on the IBS Symptom Severity Scale of 50 or more points. (A reduction of fifty or more points is considered clinically significant.) But it is not true that most of those changes can be attributed to the web-based program, which is what the statement implies. Mahana does not mention that 44% of those in the treatment-as-usual arm who reported at 12 months also had a reduction in scores of 50 or more on the same scale. Given those numbers, it seems likely that many or most of the 66 % in the web-based arm could have reported those improvements anyway.

Moreover, the site does not make clear that only 70% of the study sample provided data at 12 months. We can’t know what the final results for the remaining 30% would have been. That means we have no idea how those who dropped out from the web-based program arm felt about the intervention or whether it helped them. They were, in epidemiological terms, “lost to follow-up.”

In any event, the study’s main analysis of the symptom severity scores used an “intention-to-treat” strategy, which tries to account for this sort of missing data. At 12 months, as I have already noted, the mean score for the web-based group was found to be 35.2 points lower than for the treatment-as-usual group—quite a bit less than the 50-point difference that would represent a clinically significant improvement for an individual.

Mahana’s website also states“On average, reduction in IBS severity was twice that of patients receiving medical care as usual.”

Again true, and again misleading. When improvements are small, improvements that are twice the size are also pretty small. Just because something doubles does not automatically mean the change is of much significance. The more telling statistic is often not the relative difference between groups–the kind cited in the above claim about average reductions of IBS severity–but the absolute difference. In this case, as I’ve already noted, the absolute difference in score between the means of the groups was 35.2 points–well under the 50 points that would represent a clinically significant change for an individual on that scale.

Unfortunately for Mahana, this product cannot be accurately marketed as having any proven clinically meaningful impacts on symptom severity beyond treatment-as-usual. Perhaps more rigorous scrutiny of the actual study results would have been warranted before the deal was made.

The coronavirus epidemic certainly heightens the potential appeal of effective web-based therapies, but that does not change the indisputable facts about this specific program. While the current situation has obviously delayed my ability to focus on this matter, President Trump is now urging us to start getting back to work, and I am heeding his call.

As we slowly emerge from our current state of paralysis, I plan to resume my efforts to highlight the difference between Mahana’s promotional claims and the actual study results. I would, of course, appreciate hearing from you.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

 

Comments on this entry are closed.

  • Alicia Butcher Ehrhardt, PhD 15 April 2020, 2:14 pm

    In my opinion, limited as it has to be under the circumstances, trying to change a scam artist into a legitimate businessperson isn’t possible.

    They can change their site as much as they want, but, as with the CBT and GET people, they are basically scientific frauds trying to defend the indefensible. And there is no accountability possible. Crediting them with good intentions won’t stop them; they just keep trying to weasel out.

    It would be better to take it up with those who hire them and pay for their outrageously ridiculous ‘programs’ out of tax money. And for that it may be better to let their insupportable claims remain on their websites.

    I’m glad it’s you jousting with this, and in such a polite way. I couldn’t do what you do. Thanks for being on the side of the good guys.

  • CT 16 April 2020, 7:29 am

    Covid-19 – not so much a ‘black swan’ event as a black vulture opportunity?

    See this tweet -https://twitter.com/AliceSibelli/status/1241072341247045634 by Dr Alice Sibelli, Clinical Innovation Researcher at Mahana Therapeutics, and make up your own mind.

    I suspect that many people may suffer mental health problems as a direct or indirect consequence of the coronavirus crisis, but that is exactly why we need misleading claims about ‘MUS’ highlighted now, before diagnostic overshadowing becomes the default setting and millions more patients are misdiagnosed with mental health disorders for their very real physical health complaints on the back of the much-hyped anticipated surge in mental illness.

  • Lady Shambles 16 April 2020, 8:51 am

    I couldn’t agree with ‘CT’ more.

  • Lucie Webb 16 April 2020, 9:36 am

    IBS is yet another label – it doesn’t merit the nomenclature of a diagnosis – thats get stuck on women’s medical records more than men’s (2/2.5 to 1 ratio). My experience of gastroenterologists in the NHS is that they are worse than useless. When I lost a 1/4 of my body weight in a few weeks, 5 years ago, the gastroenterologist, registrar!, I saw in Bracknell refused to accept that I had a gut infection. Why did I have to send stool specimens in a aeroplane across the Atlantic to a lab in the States to confirm the infection? And why did I have to get help from the Australians who told me the best medication to eradicate the commensurate organisms that had taken a turn towards the dark side in my gut? At the JR in Oxford another gastroenterologist said he would have prescribed Metronidazole had I not already treated myself. The Australians had told me that Metronidazole would probably make me worse.
    I believe that my autonomic nervous system stopped sending electronic and hormonal messaging at optimal levels and this caused my digestive system to stop working. At the same time viruses that’d been kept in check reappeared. I have had blood tests done at Acumen Labs, Devon, that show mitochondrial damage arising from exposure to various substances but including cetrimide which used to be used in antiseptics. Suboptimal energy levels prevent cells working efficiently.
    There are too many psychologists and psychiatrists in this world and not enough TMJ dentists in the mould of Farrand Robson, dec’d. Gastroenterologists who cosy up to the mental health brigade are heading towards a cup-de-sac.
    Y’all keep taking those deep breaths now.

  • steve hawkins 16 April 2020, 1:43 pm

    Sooner or later KCL is going to have to think of its reputation and stop allowing promotion of quackery from its premises, and under its name.