By David Tuller, DrPH
Tom Chiversâ€™ terrific article on the Lightning Process and Professor Esther Crawleyâ€™s SMILE trial in the Archives of Disease in Childhood has received a lot of attention and comment. I wanted to respond to the short sections in which Professor Crawley seeks to justify her methodological choices. Here are the relevant passages:
In the highest-quality medical trials, subjects are â€œblindedâ€ â€“ they donâ€™t know whether theyâ€™re getting the treatment thatâ€™s being tested, or what itâ€™s being tested against. It helps stop the results being biased in favour of the treatment. If you canâ€™t blind the trial, said [Professor Jonathan] Edwards, then itâ€™s important to ensure that you measure something that canâ€™t be affected by patientsâ€™ perceptions. â€œYou can have an unblinded trial and measure everyoneâ€™s blood sodium concentration at the end,â€ he said. â€œThey canâ€™t do anything to their sodium concentration, so it doesnâ€™t matter if they know whether theyâ€™re getting the treatment or not.
â€œAnd the other way around is fine: If you blind everything so they patients donâ€™t know if theyâ€™ve had the treatment, then you can use a subjective measure. But you canâ€™t have an unblinded trial and a subjective outcome.â€ But the SMILE trial was unblinded, and Edwards also pointed out that the primary outcome that was measured was changed from an objective measure, school attendance, to a questionnaire. Edwards said such self-reported measures are often prone to bias, as subjects give the answers they think they are expected to give. For that reason, he believes, the trial is â€œuselessâ€.
In response, Crawley said “all the outcomes were collected as planned, but children didnâ€™t like our recommended primary outcome, school attendance, so we used disability.” She added that the primary outcome measure change was made, and reported, before results were collected.
The problems were, in some eyes, made worse by the fact that the methods of the Lightning Process involve making people say that they feel better. â€œAll this trial shows is that if you tell people to say theyâ€™re better, and then ask them if theyâ€™re better, theyâ€™ll say theyâ€™re better,â€ said Edwards. â€œIt’s a textbook case of how not to design a trial.â€
He claims the SMILE trialâ€™s results also undermine the PACE trial â€“ which also used an unblinded trial with subjective outcomes â€“ by showing that “the same techniques can get you the same answer for a completely quack therapy based on complete nonsense like standing on pieces of paper and telling your disease to stopâ€â€¦
Dorothy Bishop, a professor of developmental neuropsychology at the University of Oxford, told BuzzFeed News she was also concerned about the â€œwisdom of running a trial [into something] that doesnâ€™t seem to have much scientific basis and is commercial, because if you find a result you end up giving huge kudos to something that may not deserve itâ€.
â€œI donâ€™t want to come down like a ton of bricks on Esther Crawley because I think sheâ€™s doing her best,â€ she said, but she was concerned about a â€œa mega-placebo effectâ€.
Crawley told BuzzFeed News it was possible that there was some placebo effect involved, but that the questionnaires she used in the trial asked questions about how far you can walk and how much school you attended, rather than simply whether people felt better. She added that self-reported school attendance lined up very well with the schoolsâ€™ records of attendance.
The concerns expressed by Professor Jonathan Edwards go to the heart of much of the criticism about studies from the biopsychosocial groupâ€”they are open-label studies that rely on subjective findings. This is a recipe for producing bias, as Dr. Edwards explains, or a â€œmega-placebo effect,â€ per Professor Bishopâ€™s wordsâ€”especially since the Lightning Process itself involves telling participants that they can get well if they follow the programâ€™s precepts.
Professor Dorothy Bishopâ€™s presence in the BuzzFeed article is an interesting development. She allowed herself to get roped into the Science Media Centreâ€™s efforts to hype the study, offering cautious praise along with other experts. Perhaps her statements to BuzzFeed represent a recognition that she squandered some of her reputational capital to promote what is clearly bogus research. She should certainly have been embarrassed by her participation in the dog-and-pony show hosted by the SMC, which never seems to miss a chance to highlight new research from members of the biopsychosocial ideological brigades.
In her response to BuzzFeed, Professor Crawley downplays the enormous placebo problem. She acknowledges only that â€œit was possibleâ€ there could be â€œsomeâ€ placebo effect. She then appears to suggest she has addressed the issue because the questionnaires asked people to estimate numbers, not just qualities of feeling. This is a silly and unconvincing answer. (To be clear, this is Professor Crawleyâ€™s response as conveyed by the reporter, and my assumption is that it is an accurate representation of what she said. But I did not talk to Professor Crawley myself.)
As experienced researchers know, people can be notoriously unreliable at estimating and reporting the metrics of personal experienceâ€”things like how far they walk, how much they eat, how often they use condoms, how much time they spend reading to their kids. It is not just that they forget; they are also influenced by other factors, like a desire to report improvements, a desire to please the researchers, or a desire to not admit that they ate another chocolate bar or skipped another day or school.
So contrary to Professor Crawleyâ€™s implication, just because a questionnaire asks people how far they can walk does not make it an objective measure. It remains a self-reported measure subject to a huge amount of potential biasâ€”including the effect of repeated messages that the intervention, in this case the Lightning Process, will make them better. Thatâ€™s why objective measures, when available, are a critical means of assessing the accuracy of self-reported responses.
In fact, in some past studies of behavioral and psychological interventions for ME/CFS, participants have worn ankle monitors to measure movement over the course of days or a week. In these studies, participants have reported subjective improvements even as the ankle monitors have demonstrated no evidence of increased movement. In the now-debunked PACE trial, the objective measuresâ€”how far people could walk, how fit they were, whether they returned to work, and whether they received benefitsâ€”all failed to match subjective reports of success.
Given this context, I was interested at Professor Crawleyâ€™s claim in BuzzFeed that schoolsâ€™ official records of attendance matched the self-reported attendance figures. In the protocols for both the feasibility trial and the full trial, the investigators promised to seek access to these records. But the Archives paper did not mention these records at all, for unexplained reasons.
From Professor Crawleyâ€™s statement, it now appears that the investigators did in fact access such data. If so, why were they not included in the paper? In publicly citing these records to affirm the legitimacy of the studyâ€™s self-reported findings while failing to include them in the Archives paper, Professor Crawley has raised further questions about the trialâ€™s claims and its methodological lapses.
Professor Crawley also states that the change in the primary outcome measure was made before results were collected. This is true. But it is also true that the change was made after results were collected. Patients in the feasibility trial were recruited starting in September 2010, meaning the first 12-month results would have been in September 2011.
The feasibility study recruited participants through mid-2012. The ethics committee approved both the extension of the feasibility trial into the full trial and the swapping of the outcome measures after that. Then the remaining participants in the full sample were recruited. That means that results were collected both before and after the primary measure change was made. Professor Crawleyâ€™s statement is accurate as far as it goes, but it is also incomplete and ambiguous and therefore highly misleading.
Moreover, it is interesting that, according to Professor Crawleyâ€™s statement to BuzzFeed, the investigators changed the outcome measures based on what the children thought. It is rather unusual to delegate such key decision-making to trial participants. In any event, it is worth noting that the change purportedly favored by the kids in the study coincidentally allowed the Archives paper to make positive claims for its primary outcome.
What if the children had preferred to retain school attendance at six months as the primary measure? Would Professor Crawley and her colleagues have followed their advice, and reported null results for their primary outcome? Perhaps Iâ€™m being too cynical, but that seems very unlikely.