Trial By Error: More on the Mahana Mess…

If you appreciate this piece and/or my work in general, here’s a link to my current UC Berkeley crowdfunding effort: https://crowdfund.berkeley.edu/project/20159

**********

By David Tuller, DrPH

Today I sent a third letter to Dr Mel Heyman, a prominent gastroenterologist at the University of California, San Francisco. Dr Heyman is one of Mahana Therapeutics’ science advisers. I’ve been trying to understand why the company decided to license a web-based cognitive behavior therapy program for irritable bowel syndrome. After all,  a major study documented that the program did not produce clinically significant benefits in symptom severity over treatment-as-usual.

As I have previously noted, Mahana has implicitly acknowledged my criticisms by removing its laughable description of the symptom severity reductions as “dramatic” and “potentially game-changing.” But marketing strategy isn’t the problem if a program doesn’t perform as promised. At this point, Mahana’s science advisers need to make it clear whether they believe this program has clinically significant benefits  in reducing symptom severity or not.

**********

Dear Dr Heyman–

In February, before coronavirus swamped everything, I sent you two letters concerning Mahana Therapeutics’ expansive claims about its new web-based cognitive behavior therapy program for irritable bowel syndrome. I also reached out to other Mahana gastroenterology advisers as well as Rob Paull, the chief executive officer. (I have cc’d Mr Paull on this letter, as well as Vincent Racaniello, the Columbia microbiologist who hosts my posts on Virology Blog.)

Now I’m trying to get back to work.

Let’s recap: Mahana announced in January that it had licensed the web-based CBT program from King’s College London. My concerns involved statements in a press release and on Mahana’s website that clearly exceeded the data from the relevant study. The press release described the web-based program’s impacts on symptom severity as “substantial” and “durable.” The website called them “dramatic” and “potentially game-changing.”

As I have repeatedly pointed out, these descriptions cannot be justified. At 12 months, the reported benefits of the web-based program over treatment-as-usual on the study’s measure of symptom severity were statistically significant but clinically insignificant. At 24 months, these reported benefits were neither statistically nor clinically significant. Given the weak results for this core indicator, it is hard to understand why Mahana decided to license the product.

It now appears that Mahana has changed the website’s description of the program’s purported benefits and no longer characterizes them as “dramatic” and “potentially game-changing.” I am pleased to see this change. However, the site currently states that “patients enrolled in a minimal-contact digital CBT program experienced significant and clinically meaningful reduction in the severity of their IBS.”

This is an empty statement, since it would be true if only two patients in the web-based arm had achieved this “significant and clinically meaningful reduction” in symptom severity. It could also easily be said about the treatment-as-usual arm, since at least two patients in that arm also achieved “significant and clinically meaningful reduction” in symptom severity. The relevant question is whether the program provides clinically meaningful benefits over and above what is achieved through treatment-as-usual–and the main results for the symptom severity scale suggest that it does not.

The two data points Mahana includes on the website about these purported benefits are not the central findings for IBS symptom severity. To cite them in this manner is highly misleading, as I have previously explained on Virology Blog.

According to Mahana’s website, “66% of patients reported significant and clinically meaningful reduction in the severity of their IBS.”

It is true that 66% of those in the web-based CBT arm who responded at 12 months had a reduction in the scores on the IBS Symptom Severity Scale of 50 or more points–the threshold for clinical significance. But it is not true that most of those changes can be attributed to the web-based program, which is what the statement implies. Mahana does not mention that 44% of those in the treatment-as-usual arm who reported at 12 months also had a reduction in score of 50 or more on the same scale. Given those numbers, it seems likely that many or most of the 66% in the web-based arm could have reported those improvements anyway.

Moreover, the site does not make clear that only 70% of the study sample provided data at 12 months. We can’t know what the final results for the remaining 30%–those considered “lost to follow-up”–would have been. That means we have no idea how those who dropped out from the web-based program arm felt about the intervention or whether it helped them. At 12 months, as I have already noted, the mean score for the web-based group in the intention-to-treat analysis was 35.2 points lower than for the treatment-as-usual group—quite a bit less than the 50-point difference that is considered a clinically significant improvement.

Mahana’s website also states“On average, reduction in IBS severity was twice that of patients receiving medical care as usual.”

Again true, and again misleading. When improvements are small, improvements that are twice the size are also small. Just because something doubles does not mean the change is of much or any clinical significance. The more telling statistic is often not the relative difference between groups–the kind cited in the above claim about average reductions of IBS severity–but the absolute difference. In this case, as I’ve already noted, the absolute difference in score between the means of the groups was 35.2 points–well under the 50 points that is considered a clinically significant change.

Unfortunately for Mahana, this web-based CBT program cannot be accurately marketed as having clinically meaningful impacts on symptom severity beyond treatment-as-usual. The coronavirus epidemic certainly heightens the potential appeal of effective web-based therapies, but the data from this study cannot be twisted to mean what they don’t.

Dr Heyman, do you stand by the way in which Mahana is presenting the symptom severity findings for the web-based program, or do you find the selective use of data problematic? Can you explain why Mahana decided to license a program with such minimal reported benefits over treatment-as-usual in reducing symptom severity? How did your own association with Mahana begin? Finally, is your advisory position compensated?

I would, of course, appreciate hearing from you.

Best–David

David Tuller, DrPH

Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

 

Comments on this entry are closed.

  • CT 25 April 2020, 4:30 am

    If individuals want to spend their money on a program like this then I guess that’s up to them, but the NHS surely can’t justify forking out on this web-based program for IBS. David, perhaps you could also write to Harpreet Sood, who is apparently NHS England’s Associate Chief Clinical Information Officer and one of Mahana Therapeutics’ ‘Leaders in Business’ (https://www.mahanatx.com/about)?

  • Richard Vallee 25 April 2020, 2:42 pm

    I’m wondering if economics isn’t starting to play a significant role here. That the efforts to license some of this work comes because the entire premise of the BPS model is to save money, which it systematically fails at, actually increasing total costs, and some people who have been sponsoring this are starting to demand that some of the promised actually make it back in some form, whatever form.

    Whether in its official form within IAPT or as commercial programs, the entire premise of this approach is that it is cost-effective. So far it’s been nothing but expenses and zero benefits. The expenses may be rather small but this is starting to become problematic with the IAPT expansion, the numbers start adding up very quickly at this scale, and anyway given enough time even small expenses add up to quite a lot. We are 3 decades into this and at some point empty promises start to wear thin.

    There is basically no justification for this licensing deal in any way that relates to health care. It is not and has nothing to do with providing health benefits to anyone. It seems to be a stop-gap attempt at showing there can be some returns on investment, a promise that has wildly over-promised while delivering absolutely nothing but failure.

  • CT 26 April 2020, 5:51 pm

    Correction to my previous comment-
    Dr Harpreet Sood is no longer NHS England’s Associate Chief Clinical Information Officer (since last Spring) but does have other roles including being a Board member of Health Education England.

  • Shakota 1 May 2020, 1:08 pm

    In general, I distrust most doctors who, in their professional practice, also do outside of clinical practice various things to augment their income such as a myriad of drug trials, a cosmetic arm connected to their practice or co-development with the sponsorship of pharma, a so-called patient learning website.

    Why? Because as their patient I can never be sure whether their clinical advice to me is sound. Is it coming from a profit motive (one the cash generating arms of their practice) or is it good clinical practice.