Trial By Error: Professor Crawley’s Bogus BuzzFeed Claims

By David Tuller, DrPH

Tom Chivers’ terrific article on the Lightning Process and Professor Esther Crawley’s SMILE trial in the Archives of Disease in Childhood has received a lot of attention and comment. I wanted to respond to the short sections in which Professor Crawley seeks to justify her methodological choices. Here are the relevant passages:

In the highest-quality medical trials, subjects are “blinded” – they don’t know whether they’re getting the treatment that’s being tested, or what it’s being tested against. It helps stop the results being biased in favour of the treatment. If you can’t blind the trial, said [Professor Jonathan] Edwards, then it’s important to ensure that you measure something that can’t be affected by patients’ perceptions. “You can have an unblinded trial and measure everyone’s blood sodium concentration at the end,” he said. “They can’t do anything to their sodium concentration, so it doesn’t matter if they know whether they’re getting the treatment or not.

“And the other way around is fine: If you blind everything so they patients don’t know if they’ve had the treatment, then you can use a subjective measure. But you can’t have an unblinded trial and a subjective outcome.” But the SMILE trial was unblinded, and Edwards also pointed out that the primary outcome that was measured was changed from an objective measure, school attendance, to a questionnaire. Edwards said such self-reported measures are often prone to bias, as subjects give the answers they think they are expected to give. For that reason, he believes, the trial is “useless”.

In response, Crawley said “all the outcomes were collected as planned, but children didn’t like our recommended primary outcome, school attendance, so we used disability.” She added that the primary outcome measure change was made, and reported, before results were collected.

The problems were, in some eyes, made worse by the fact that the methods of the Lightning Process involve making people say that they feel better. “All this trial shows is that if you tell people to say they’re better, and then ask them if they’re better, they’ll say they’re better,” said Edwards. “It’s a textbook case of how not to design a trial.”

He claims the SMILE trial’s results also undermine the PACE trial – which also used an unblinded trial with subjective outcomes – by showing that “the same techniques can get you the same answer for a completely quack therapy based on complete nonsense like standing on pieces of paper and telling your disease to stop”…

Dorothy Bishop, a professor of developmental neuropsychology at the University of Oxford, told BuzzFeed News she was also concerned about the “wisdom of running a trial [into something] that doesn’t seem to have much scientific basis and is commercial, because if you find a result you end up giving huge kudos to something that may not deserve it”.

“I don’t want to come down like a ton of bricks on Esther Crawley because I think she’s doing her best,” she said, but she was concerned about a “a mega-placebo effect”.

Crawley told BuzzFeed News it was possible that there was some placebo effect involved, but that the questionnaires she used in the trial asked questions about how far you can walk and how much school you attended, rather than simply whether people felt better. She added that self-reported school attendance lined up very well with the schools’ records of attendance.

**********

The concerns expressed by Professor Jonathan Edwards go to the heart of much of the criticism about studies from the biopsychosocial group—they are open-label studies that rely on subjective findings. This is a recipe for producing bias, as Dr. Edwards explains, or a “mega-placebo effect,” per Professor Bishop’s words—especially since the Lightning Process itself involves telling participants that they can get well if they follow the program’s precepts.

Professor Dorothy Bishop’s presence in the BuzzFeed article is an interesting development. She allowed herself to get roped into the Science Media Centre’s efforts to hype the study, offering cautious praise along with other experts. Perhaps her statements to BuzzFeed represent a recognition that she squandered some of her reputational capital to promote what is clearly bogus research. She should certainly have been embarrassed by her participation in the dog-and-pony show hosted by the SMC, which never seems to miss a chance to highlight new research from members of the biopsychosocial ideological brigades.

In her response to BuzzFeed, Professor Crawley downplays the enormous placebo problem. She acknowledges only that “it was possible” there could be “some” placebo effect. She then appears to suggest she has addressed the issue because the questionnaires asked people to estimate numbers, not just qualities of feeling. This is a silly and unconvincing answer. (To be clear, this is Professor Crawley’s response as conveyed by the reporter, and my assumption is that it is an accurate representation of what she said. But I did not talk to Professor Crawley myself.)

As experienced researchers know, people can be notoriously unreliable at estimating and reporting the metrics of personal experience—things like how far they walk, how much they eat, how often they use condoms, how much time they spend reading to their kids. It is not just that they forget; they are also influenced by other factors, like a desire to report improvements, a desire to please the researchers, or a desire to not admit that they ate another chocolate bar or skipped another day or school.

So contrary to Professor Crawley’s implication, just because a questionnaire asks people how far they can walk does not make it an objective measure. It remains a self-reported measure subject to a huge amount of potential bias—including the effect of repeated messages that the intervention, in this case the Lightning Process, will make them better. That’s why objective measures, when available, are a critical means of assessing the accuracy of self-reported responses.

In fact, in some past studies of behavioral and psychological interventions for ME/CFS, participants have worn ankle monitors to measure movement over the course of days or a week. In these studies, participants have reported subjective improvements even as the ankle monitors have demonstrated no evidence of increased movement. In the now-debunked PACE trial, the objective measures—how far people could walk, how fit they were, whether they returned to work, and whether they received benefits—all failed to match subjective reports of success.

Given this context, I was interested at Professor Crawley’s claim in BuzzFeed that schools’ official records of attendance matched the self-reported attendance figures. In the protocols for both the feasibility trial and the full trial, the investigators promised to seek access to these records. But the Archives paper did not mention these records at all, for unexplained reasons.

From Professor Crawley’s statement, it now appears that the investigators did in fact access such data. If so, why were they not included in the paper? In publicly citing these records to affirm the legitimacy of the study’s self-reported findings while failing to include them in the Archives paper, Professor Crawley has raised further questions about the trial’s claims and its methodological lapses.

Professor Crawley also states that the change in the primary outcome measure was made before results were collected. This is true. But it is also true that the change was made after results were collected. Patients in the feasibility trial were recruited starting in September 2010, meaning the first 12-month results would have been in September 2011.

The feasibility study recruited participants through mid-2012. The ethics committee approved both the extension of the feasibility trial into the full trial and the swapping of the outcome measures after that. Then the remaining participants in the full sample were recruited. That means that results were collected both before and after the primary measure change was made. Professor Crawley’s statement is accurate as far as it goes, but it is also incomplete and ambiguous and therefore highly misleading.

Moreover, it is interesting that, according to Professor Crawley’s statement to BuzzFeed, the investigators changed the outcome measures based on what the children thought. It is rather unusual to delegate such key decision-making to trial participants. In any event, it is worth noting that the change purportedly favored by the kids in the study coincidentally allowed the Archives paper to make positive claims for its primary outcome.

What if the children had preferred to retain school attendance at six months as the primary measure? Would Professor Crawley and her colleagues have followed their advice, and reported null results for their primary outcome? Perhaps I’m being too cynical, but that seems very unlikely.

Comments on this entry are closed.

  • Lois Addy 17 January 2018, 2:00 am

    my first thought was, since when has what kids or participants thought made a difference? I can see how school attendance may not be a great indicator if say kids who never made it to school carried on not doing so, but maybe were able to do more at home. but why not a walking test? something they could all do at beginning and end.

    I am getting fed up with the sheer unprofessionalism of the way the results of these trialsis being presented. Is there no best practice guidelines for them to follow? for both the trial AND the results? and what’s with this moving hte goalposts half way through? does this happen for other illnesses? for other psychologists?

    is it happening NOW post all the kerfuffle over PACE and subsequent results released? Also why does it take this many years for results to be issued? trial was 2011 to 2012 and it’s now 2017? what? that’s 5 years!!!!

  • Alicia Butcher Ehrhardt 17 January 2018, 2:59 am

    A walking test is just as affected by bias as any other – unless they measure to exhaustion, and exhaustion is something measurable, say, by blood oxygen content. And it is well known that doing things to exhaustion is very bad for ME/CFS patients, who will pay for it for days afterward. Many of us can do something, but know darn well we shouldn’t! When we do it anyway, we pay the price.

    Objective measurements, please. Things that can’t be affected by beliefs or attention.

  • CureCFS 17 January 2018, 5:18 am

    It is amazing that standing on pieces of paper and telling your disease to stop can cure a disease with a 100% gene expression match with Systemic Inflammatory Response Syndrome. Perhaps the Lightning Process, cognitive behavioral therapy, and graded exercise therapy would be good treatments for sepsis? #sarcasm

    It is amazing that top medical journals such as the Lancet would publish unblinded studies with subjective outcome measures. Perhaps Esther Crawley actually did us a favor by showing the world that using the same techniques can get you the same answer as the PACE trial for a completely quack therapy.

    So… would it be possible for someone to get the school attendance records from the school and see if they match the self-reported attendance?

  • Sleepyblondie 17 January 2018, 5:43 am

    Agreed. And that doesn’t take I to account post exertional exhaustion sufferered afterwards

  • Sleepyblondie 17 January 2018, 5:46 am

    It’s just appalling that her university continue to support her, and that journals pub,ish her work. Does she have any knowledge of effective research methodology? Seems not. Does she have any understanding of ME? Seems not.

  • Brenda Vreeswijk 17 January 2018, 6:02 am

    What was she trying to accomplish in the first place?? She knows the trial was biased to begin with…then she changed outcomes because of the children didn’t like the outcomes.
    What do you think will happen if you Gaslight people…”you feel better…now you feel a lot better” It was nothing more than brainwashing and has nothing to do with science…just like her other work. She can say what she want’s but she’s doing the trials for her own personal gain…she not there to help children. She’s a liability to children and young adults..she should be forced to step down and not be able to do more harm.
    She’s a disgrace for her profession and a health hazard to children…well in my opinion.

  • Jane Neunistiva 17 January 2018, 6:07 am

    “children didn’t like our recommended primary outcome, school attendance, so we used disability.” I can’t believe a researcher can make such a statement without being ridiculed by the whole scientific community.

  • Fiona 17 January 2018, 7:16 am

    “Moreover, it is interesting that, according to Professor Crawley’s statement to BuzzFeed, the investigators changed the outcome measures based on what the children thought. It is rather unusual to delegate such key decision-making to trial participants. In any event, it is worth noting that the change purportedly favored by the kids in the study coincidentally allowed the Archives paper to make positive claims for its primary outcome.

    What if the children had preferred to retain school attendance at six months as the primary measure? Would Professor Crawley and her colleagues have followed their advice, and reported null results for their primary outcome? Perhaps I’m being too cynical, but that seems very unlikely.”

    David, I think you’re bang on target there!

  • Jo Best 17 January 2018, 7:22 am

    School attendance was a poor primary outcome in any case because it doesn’t measure how well the child felt during the school day or their ability to take part in the day’s learning and other activities, or if they had to go home early because of illness or if they crashed once at home after school. Moreover, participants in the SMILE trial may have forced themselves to go to school so as not to appear to ‘fail’ the ‘treatment’ test. Most children want to please their parents and authority figures, such as their specialist doctor and the Lightning Process practioners in the case of the SMILE trial and there is in-built coercion in the LP by telling participants that ‘the process’ will work if they (the participants) apply it correctly and telling them to deny their symptoms and to say they feel well even when they don’t. That the SMILE trial was allowed to go ahead in the feasibility stage, let alone then allowed to transform into a full trial, is a national disgrace. I feel it really is a case of institutionalised abuse of the rights, safety and dignity of sick children.

  • pinklil 17 January 2018, 9:07 am

    “Crawley said “all the outcomes were collected as planned, but children didn’t like our recommended primary outcome, school attendance, so we used disability.” ”

    My query would be: at what point and how was this information received by Crawley without ‘unblinding’ her admittedly completely unblinded study? Ie: Crawley likes to live the pretence this was blinded, but how, within that pretence, were the wishes of the children made known to her? Quite apart from the absurdity that i) children might have this opinion at all (unless they were genuinely concerned their own questionnaire response on attendance wouldn’t match the actual record which would presumably bring into question the usefulness of the intervention) ii) surely few children/ teenagers actually concern themselves with ‘primary end points’ at all ? and iii) since when did any researcher ever in whole history of medical research listen to the subjects of a research project and change their endpoints mid-trial because the subjects ‘didn’t like them’? … how was Crawley made aware of this ‘dislike’? She must have had access to these children to know this (if it was actually true in the first place..and we know Crawley and truth are rare bedfellows) in which case she would also have had the opportunity to access information about the success , or otherwise, of the intervention before adjusting her endpoints..wouldn’t she?
    However you look at this statement (above) it’s both bizarre and rife with ethical questions around the protocol of the trial in terms of what actually happened in use (as opposed to what might officially be written down as the ‘protocol’ before the trial started.)

  • jimells 17 January 2018, 10:38 am

    Since children are experts at trial design let’s give *them* the research money. Their results couldn’t possibly be worse than Professor Crawley’s. Teenagers would likely come up with some novel ideas if they had half a chance and a little encouragement.

  • Jan Wade 17 January 2018, 10:55 am

    Esther Crawley is harmful & a danger to all living beings.

  • Jan Wade 17 January 2018, 10:57 am

    Lawsuits please

  • Jan Wade 17 January 2018, 10:58 am

    For goodness sakes please bring on the lawyers

  • Jan Wade 17 January 2018, 11:01 am

    Yup. As Jimells pointed out earlier, it’s a key aspect of a lawsuit known as “DISCOVERY”

  • Sue Wilson 17 January 2018, 11:43 am

    My 18 and 14 know nothing about outcome measures!

  • Sue Wilson 17 January 2018, 11:44 am

    My 18 and 14 year old know nothing about outcome measues. Wouldn’t like them to be making major decisions about such an important trial!

  • Alex Young 17 January 2018, 12:34 pm

    Yes, presence at school does not mean learning occurs.

    To give an example from my own own history, as an adult. As a mild patient I had to attend lectures early in the day, and later in the day, for my B.Sc.. In between lectures I would go to the library, put my head down in a study cubicle, and rest for hours. That enabled me to persist. Participation in school activities might be largely avoided by children even with attendance. What happened to their grades? Extra-curricula activities? How many had to repeat a grade? These are deeply personal issues to children, and I think much of it might be exempt from an FOI for a bunch of different reasons, and this will also apply to legal discovery.

    I am also concerned that asking children about outcome measures might be heavily subject to bias. How and what you tell them might have a huge impact on their response.

  • C'est la même 17 January 2018, 5:31 pm

    They’ve never asked adults to review the commonly used PROMs and other outcome measures so it is extremely unusual that they’d suddenly ask children such.

  • C'est la même 17 January 2018, 5:33 pm

    This is true, the 6 minute walking distance is not a reliable measure of exercise capability in an unblinded trial.

    Actigraphy and 2-day CPET are reliable measures (though the latter is not suitable for all patients).

  • C'est la même 17 January 2018, 5:41 pm

    I disagree, have them complete the Chalder Fatigue Scalem, the Short-Form Health
    Survey Physical Function Subscale (SF-36 PF), the Hospital Anxiety and Depression Scale, the Spence Children’s Anxiety Scale and the Euroqol 5D.

    Then ask them how useful, accurate and reliable they think these questionnaires are?

    Ask them how accurate they think these questionnaires would be during a therapy that is designed to change how you think about the experience/perception of symptoms?

  • Alicia Butcher Ehrhardt 17 January 2018, 5:53 pm

    Both require an enormous effort from the participant. I would not risk such for me, except if my whole disability determination had come to that. i KNOW I would crash, and I know it would be a long one.

    First, you have to get to where they do a test. This requires so much planning – including naps, transportation, food, support, dressing – that I would be exhausted long before I got there.

    Then you have to do a test, surrounded by people (!) who think nothing of making you wait an extra half hour or whatever, in a brightly-lit and noisy environment. On unfamiliar territory and equipment. Then, you have to somehow get home. And do it again the next day? Seriously?

    I leave the house as little as I can – because all that extra stuff normals take for granted is a huge energy suck for us.

    And I’m fairly functional! If I am very careful, I can sit up, use a computer, and even write fiction! But energy expended never comes back.

  • Guido den Broeder 17 January 2018, 7:38 pm

    School attendence is also subjective. The child will attend more often when it thinks it’s supposed to because of the treatment.

  • Eli 18 January 2018, 12:36 am

    You can get an objective measure of school attendance, but as a proxy for overall health there still might be problems with bias. As an outcome measure though, it seems likely to be less prone to problems with bias than just a questionnaire about physical functioning.

  • Carolyn Wilshire 18 January 2018, 12:37 am

    “From Professor Crawley’s statement, it now appears that the
    investigators did in fact access such data. If so, why were they not
    included in the paper? In publicly citing these records to affirm the
    legitimacy of the study’s self-reported findings while failing to
    include them in the Archives paper, Professor Crawley has raised further
    questions about the trial’s claims and its methodological lapses.”

    What an excellent point you make here, David. This is an admission that attendance based on the official school records was indeed collected. Why on earth would it not be reported? Its totally fine to include measures “the children like”. But why on earth should that imply that the original protocol specified ones should be binned?

    A trial protocol constitutes a promise as to how you’re going to analyse your data. Its purpose is to prevent people selecting the outcomes that show the results in the best light. There is no point at all to a trial protocol is you are free to change those outcomes for whatever reason you fancy. Any justification for change must involve some factor that could have been anticipated at the time of writing the protocol (for example, failure of a particular test or technology). Deciding to change outcomes because “Its what the children like” simply does not cut it!

  • Olivia Beatty 18 January 2018, 3:28 am

    The people on the ethics committee who approved this nonsensical trial need to be taken to task. What other rubbish have they approved?

  • acacia1 18 January 2018, 3:28 am

    The PACE trial investigators were well aware of patient limitations and used the lack of walking to exhaustion as an excuse for poor results in the 6 mwt: ‘Due to concerns about patients with CFS coping with physical exertion, no encouragement was given to
    participants as they performed the test, by contrast to the way this test is usually applied (Guyatt et al.therapy, graded exercise therapy, and specialist medical
    1984; American Thoracic Society, 2002). Rather than encouragement, we told participants, ‘You should walk continuously if possible, but can slow down or stop if you need to.’

  • Guido den Broeder 18 January 2018, 9:43 am

    * Pointing to my comment *

  • Barry 18 January 2018, 1:07 pm

    Thanks again David.

    “Professor Crawley’s statement is accurate as far as it goes, but it is also incomplete and ambiguous and therefore highly misleading” …

    Really glad you are highlighting this, because EC does it time and time again. Partial truths can lead people to unwittingly infer such gross untruths from them. No accident that in a court of law you have to tell “The truth, the whole truth, …”.

  • Barry 18 January 2018, 1:15 pm

    But EC effectively asked them what kind of measures they preferred. What kind of an investigator asks their participants how they would like their efforts to be measured anyway, and uses that as a justification? I think it’s eye wash. Wish I’d been a fly on the wall.

    Subjective bias does not mean you have to know about the outcome measures. It just means there are all manner of reasons where a person’s perceptions can mislead them, and often do, as well as other pressures to conform etc.

    But if trial investigators have good clues as to which outcomes give most favourable results, and are then allowed to change which outcome measures they are going to use, they can cherry pick what they like. It’s a bit like peeking at the answers during an exam.

  • Barry 18 January 2018, 1:24 pm

    “Deciding to change outcomes because “Its what the children like” simply does not cut it!”

    In fact it would be farcical/hilarious if it were not so serious.

  • Seven 19 January 2018, 10:12 pm

    Also, if she understood ME she would know I can walk 6min then walk again If I do not do other stuff. I can increase my walking (at expense of other activities) so unless they capture the overall activity this is all so useless. This people REALLY do not even know the disease. Is very scary.

    They should have a job interview for specialist by the patients: How much does a doctor understand the specialty they want to go to! Imagine the grades for EC.= -10 from me, not only does she not understand it, she gets it all wrong, then pretends she understand, and then damages the patient even further.
    It isn’t that hard, My 3 year old got it perfectly. Why can’t she???

  • Emma Lloyd 21 January 2018, 2:15 am

    Does this mean the children where told the goal of the study and its outcome were for them to attend more school? Even a slight bias caused by the belief that they may get better and attend more school could have an effect. as in “wow they think i may be able to go to school more, this must have the chance of working”

    Add this to all the other biasing done during this trial and you could get these kids to say anything. Especially if you had an agenda.