By David Tuller, DrPH
As the world continues to wrestle with the coronavirus epidemic, President Trump is calling on us here in the US to get back to work. So I decided to start seeking answers again from Mahana Therapeutics, which announced in January that it had licensed a web-based cognitive behavior therapy program for irritable bowel syndrome from King’s College London.
According to the results of the major study that road-tested the program, the benefits in symptom severity at 12 months over treatment-as-usual were statistically significant but clinically insignificant, and at 24 months they were neither statistically nor clinically significant. In other word, the web-based program proved to be pretty much of a dud. Yet Mahana falsely promoted these results as “dramatic” and “potentially game-changing” on its website.
Apparently, those words have recently been removed–at least, I didn’t see them this morning when I checked the Mahana website. That’s a positive step. Yet the website continues to include misleading statements. This morning, I sent Robert Paull, Mahana’s CEO and co-founder, the following letter.
**********
Dear Mr Paull:
In February, before coronavirus swamped everything, I sent you two letters concerning Mahana Therapeutics’ expansive but unjustified claims about its new web-based cognitive behavior therapy program for irritable bowel syndrome. I did not hear back. I also reached out to three of Mahana’s gasteroenterology advisers. None of them responded.
To recap: Mahana recently announced that it had licensed the web-based CBT program from King’s College London. My concerns involved statements in a press release about the deal and on Mahana’s website that clearly exceeded the data from the relevant study. The press release described the web-based program’s impacts on symptom severity as €œsubstantial€ and €œdurable.€ The website called them €œdramatic€ and €œpotentially game-changing.€
As I have repeatedly pointed out, these descriptions cannot be justified. At 12 months, the reported benefits of the web-based program over treatment-as-usual on the study’s measure of symptom severity were statistically significant but clinically insignificant. At 24 months, these reported benefits were neither statistically nor clinically significant. Given these weak results for the core indicator, it is hard to understand why Mahana decided to license the product in the first place.
It now appears that Mahana has changed the description of the purported benefits on its website and no longer characterizes them as “dramatic” and “potentially game-changing.” I am pleased to see that the company has paid attention to my concerns. However, the site now states that “patients enrolled in a minimal-contact digital CBT program experienced significant and clinically meaningful reduction in the severity of their IBS.”
This is an empty statement, since it would literally be true if only two patients in the web-based arm had achieved this “significant and clinically meaningful reduction” in symptom severity. It could also easily be said about the treatment-as-usual arm, since clearly some patients in that arm also achieved “significant and clinically meaningful reduction” in symptom severity. The relevant question is whether the program provides clinically meaningful benefits over and above what is achieved through treatment-as-usual–and the main results for the symptom severity scale indicate that it does not.
The two data points you still include on the website about these purported benefits are not the central findings for the IBS symptom severity measure. To cite them in this manner is highly misleading. I have previously explained this on Virology Blog, but I guess I need to do so again.
According to Mahana’s website, €œ66% of patients reported significant and clinically meaningful reduction in the severity of their IBS.€
It is true that 66% of those in the web-based CBT arm who responded at 12 months had a reduction in the scores on the IBS Symptom Severity Scale of 50 or more points. (A reduction of fifty or more points is considered clinically significant.) But it is not true that most of those changes can be attributed to the web-based program, which is what the statement implies. Mahana does not mention that 44% of those in the treatment-as-usual arm who reported at 12 months also had a reduction in scores of 50 or more on the same scale. Given those numbers, it seems likely that many or most of the 66 % in the web-based arm could have reported those improvements anyway.
Moreover, the site does not make clear that only 70% of the study sample provided data at 12 months. We can’t know what the final results for the remaining 30% would have been. That means we have no idea how those who dropped out from the web-based program arm felt about the intervention or whether it helped them. They were, in epidemiological terms, €œlost to follow-up.€
In any event, the study’s main analysis of the symptom severity scores used an €œintention-to-treat€ strategy, which tries to account for this sort of missing data. At 12 months, as I have already noted, the mean score for the web-based group was found to be 35.2 points lower than for the treatment-as-usual group, quite a bit less than the 50-point difference that would represent a clinically significant improvement for an individual.
Mahana’s website also states: €œOn average, reduction in IBS severity was twice that of patients receiving medical care as usual.€
Again true, and again misleading. When improvements are small, improvements that are twice the size are also pretty small. Just because something doubles does not automatically mean the change is of much significance. The more telling statistic is often not the relative difference between groups€“the kind cited in the above claim about average reductions of IBS severity€“but the absolute difference. In this case, as I’ve already noted, the absolute difference in score between the means of the groups was 35.2 points€“well under the 50 points that would represent a clinically significant change for an individual on that scale.
Unfortunately for Mahana, this product cannot be accurately marketed as having any proven clinically meaningful impacts on symptom severity beyond treatment-as-usual. Perhaps more rigorous scrutiny of the actual study results would have been warranted before the deal was made.
The coronavirus epidemic certainly heightens the potential appeal of effective web-based therapies, but that does not change the indisputable facts about this specific program. While the current situation has obviously delayed my ability to focus on this matter, President Trump is now urging us to start getting back to work, and I am heeding his call.
As we slowly emerge from our current state of paralysis, I plan to resume my efforts to highlight the difference between Mahana’s promotional claims and the actual study results. I would, of course, appreciate hearing from you.
Best–David
Comments are closed.