Trial By Error: A Recap of the School Absence Study

By David Tuller, DrPH

Last week I broke the news that Bristol University is conducting an independent investigation of a number of studies that were exempted from ethical review on the grounds that they qualified as “service evaluation.” Because the issues involved are confusing and complex, I thought it would be helpful to repost here part of my initial investigation of one of those exempted studies–the 2011 school absence paper published in BMJ Open.

That paper–“Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics”–turned out to be one of at least 11 studies that were exempted from ethical review based on a single 2007 opinion from the local research ethics committee. The school absence study included a formal hypothesis, generalizable conclusions, and in-person data collection activities. In fact, the title is itself a generalizable conclusion.

Since all three characteristics are hallmarks of “research” requiring ethical review, the presence of any one of them should have disqualified the paper from being published as service evaluation. This problem should have been obvious to any experienced investigator and journal editor. I continue to be perplexed at why BMJ Open not only published the paper in the first place but has continued to undermine its own reputation and credibility by consistently defending its indefensible decision.

In my initial post on the school absence study, I outlined some of the core issues:

*Professor Esther Crawley and co-authors claimed a 2011 study in BMJ Open was exempt from ethical review because it involved the routine collection of data for “service evaluation.” Yet the 2011 study was not an evaluation of routine clinical service provision–it was designed to road-test a new methodology to identify undiagnosed CFS/ME patients among students with records of chronic absence.

*To support the claim that the study was exempt from ethical review, Professor Crawley and co-authors cited a 2007 research ethics committee opinion that had nothing to do with the data-collection activities described in the 2011 paper.

*For the 2011 study, school letters were sent to families of 146 students, inviting them to meet with Professor Crawley. In the end, only 28 were identified as having CFS/ME–meaning more than 100 families of students without CFS/ME received potentially disconcerting letters inviting them to a medical meeting about a sensitive issue. This type of pilot program is beyond the scope of what many would consider to be service evaluation.

*A pre-publication reviewer, noting the data collection activities described in the paper, raised serious questions about the lack of ethical review. In her response, Professor Crawley did not provide satisfactory answers to the concerns raised by the reviewer, but BMJ Open published the paper anyway, without ethical review.

**********

The last point is particularly significant, since it demonstrates not only problems with the paper itself but also BMJ Open’s editorial negligence in agreeing to publish it without ethical review. It should be noted that the peer reviewer in question was Professor Matthew Hotopf, a professor of general hospital psychiatry at Kings College London—a colleague of those whose work I have criticized. Given his association with members of the CBT/GET ideological brigades, it is surprising that his concerns do not appear to have been taken more seriously.

Here is what I wrote:

In fact, a reviewer invited by BMJ Open to comment on the draft of the 2011 paper appeared perturbed at the lack of ethical review and approval. In his comments, he expressed surprise that the REC would have considered the outreach aspects of the study to be service evaluation exempt from ethical approval, rather than research requiring it.

Here’s what he wrote: “It is understandable that the REC might see use of routine data from the existing clinical service as not being research…but it is surprising that they did not see the surveillance component as research. Children who are unknown to services were being contacted using information from their schools and it seems to me that there are significant issues of confidentiality and data protection which, in my experience as a researcher and one time REC member, I am surprised the REC did not think amount to research. Assuming the REC was fully aware of these issues, and still made a decision that the work was not research, then it would be unfair to oppose publication on those grounds, but the authors should make a fuller explanation, and in the interest of openness might want to make their application to the REC and subsequent correspondence available with the publication.”

The reviewer’s statement clearly presumed that Professor Crawley and her colleagues had filed an application and corresponded with the appropriate REC about the specific set of activities involved in this school absence study. Given that presumption, he made a reasonable request–that they should publish the REC application and correspondence along with the paper. The reviewer also requested the authors to provide more information about what the families were told and how their consent to participate in the project was obtained.

A pre-publication review like that should raise red flags with editors–at least enough for them to ensure that the authors provide acceptable answers. That apparently did not happen in this case. In response to the reviewer’s comments, Professor Crawley did not cite any correspondence with the REC about the school absence study under discussion, perhaps because no such correspondence existed. Nor did she offer much detail on what information was provided to families and on how consent was obtained. And she did not mention that she was not relying on recent REC correspondence about this specific study, as the reviewer presumed, but on an REC opinion about a much narrower method for additional data collection from four years earlier.

Instead, Professor Crawley explained in her response that “the specialist service has been advised that ethical approval for routine collection and analysis of service data is not required.” She did not provide a legitimate explanation for why this pilot program qualified as “routine collection and analysis of service data” when it involved outreach to families whose children were not already enrolled in the clinical service, including many families whose children did not even have CFS/ME. Instead, she pointed out that she is a community pediatrician and that the children were seen in school clinics, although it is not immediately evident why these facts should have exempted the study from ethical review.

In her response, Professor Crawley also referenced top-level admiration for her work. “The project has been of great interest to the Department of Education who included it last year as an exemplar in their training for attendance officers in the UK,” she wrote. But whether or not education officials were impressed with Professor Crawley and her work was of course irrelevant to the question at hand, which was whether the study should have been considered research or service evaluation. It was not clear why Professor Crawley included this point, except perhaps to suggest that she had well-placed supporters.

According to Professor Crawley’s response, she sought further assurance that the data collection involved was indeed part of service evaluation. Here’s what she wrote: “We checked with the co-ordinator for the local REC that recording outcomes on school based clinics run by school nurses is part of service evaluation (and therefore does not require a submission to Ethics) and they have agreed that it is.”

This statement is confusing. The clinics described in this study cannot reasonably be defined as nurse-run school-based clinics. According to the paper, they were set up specifically so Professor Crawley could meet with the children and families identified through the pilot program she developed. Professor Crawley is not a school nurse. The paper’s description of the meetings with families indicated the presence of “a member of school staff,” not a school nurse. Professor Crawley’s own response to the pre-publication reviewer indicated that this staff member was “usually the attendance officer.”

In fact, the study itself stated that “it would be of interest to evaluate whether school nurses, rather than doctors, can undertake the initial assessments in school clinics.” In other words, the way Professor Crawley personally collected data for this pilot program had little or nothing to do with the provision of routine care in nurse-run school clinics.

In any event, Professor Crawley provided no documentation of this exchange with the unnamed “co-ordinator for the local REC.” Nor did she provide any details of what this unnamed local coordinator was told. Was the coordinator told simply that this was data collection made during routine nurse-run school clinics? Or was the coordinator told that this was a pilot program to identify previously undiagnosed patients using their school absence records?

Furthermore, was the coordinator told that families would be sent letters inviting them to meet with a community pediatrician at the school, in the likely presence of the school attendance officer but not a school nurse? Was the coordinator told that the students might then be recruited as patients into the clinical service run by the community pediatrician? Was the coordinator told that most of the families identified and impacted by this recruitment process turned out not to have children with CFS/ME? Absent further documentation of this apparently critical exchange, Professor Crawley’s second-hand reassurance that the unnamed local coordinator agreed with her interpretation of events is utterly meaningless.

**********

Needless to say, BMJ Open ignored Professor Hotopf’s concerns and the self-evident facts of the matter and published the paper despite the lack of ethical review. Given that my scrutiny of the study ultimately led to the current investigation of multiple papers from the Bristol team, it is probably fair to say that BMJ Open made the wrong decision. I wonder how much longer we will have to wait for those involved in producing and publishing this mess to demonstrate enough honesty and integrity to acknowledge their serious errors in judgement.

{ 9 comments… add one }
  • Scott Simpson 3 February 2019, 10:17 am

    Worth repeating that final sentence:

    “I wonder how much longer we will have to wait for those involved in producing and publishing this mess to demonstrate enough honesty and integrity to acknowledge their serious errors in judgement.”

    We have already waited so long that it is reasonable to conclude they lack integrity.

  • Peter Trewhitt 3 February 2019, 10:56 am

    Robust ethical aproval systems are vital to well conducted research involving human subjects.

    However, it would appear incontrovertible that this School Absence Study used a letter relating to a previous questionnaire based service evaluation to avoid any ethical scrutiny. Over and above the breach of normal procedures there seem to be specific issues in this research that should warrant careful ethical consideration:

    (i) As mentioned above, the participating schools informed a health worker(s) (ie Prof Crawley) of the children’s school attendence records. It needs to be clear that the schools had informed consent to divulge this information and that there was a choice to refuse.

    (ii) All participants were non school attenders, a situation that could subject parents/guardians to legal sanctions and could result in child protection issues. It therefore needs to be established beyond reasonable doubt that any consents were freely given.

    (iii) All participants were subject to a medical assesment regardless of whether there were any prior health concerns or not. Given it was not expected that all children would have ME/CFS and that it turned out the majority did not, a majority of the participating children were subject to an unnecessary medical assesment.

    (iv) Those children identified by Prof Crawley as having CFS were then seen by her specialist clinical service whose main interventions are GET and CBT, controversial treatments. Could the families in these circumstances (see ii above) be assumed to give free consent, without clear information that refusal had no sanctions?

    I am not say that this research was conducted unethically, but that the ethical sensitivities of this specific research were such that these questions needed to be addressed openly with the conclusions and necessary safeguards officially recorded.

  • Wendy Boutilier 3 February 2019, 12:45 pm

    I continue to be perplexed at how Esther Crawley is authorized to freely use pseudo science on children with ME. She should have been slapped with a restraining order long ago. The same goes for any organization that supports any kind of psychological therapies including those high profile ME & CFS advocates who don’t understand the “babble” around it but promote it as treatment no matter what it’s called; MUS, GET, Mandatory CBT, Deconditioning, etc……..

  • Couch Turnip 3 February 2019, 3:53 pm

    It seems that there’s little desire by those in authority to uphold ethical principles anymore, not even when sick and vulnerable children are involved. What a sad state of affairs it is when personal gain and corporate reputation come first.

  • Marion Horrod 3 February 2019, 6:25 pm

    Can someone explain to me what the value of peer review is when the reviewer’s opinion is then ignored by the publication on whose behalf s/he carried out the review?

  • Justin Reilly 4 February 2019, 1:29 am

    Wow! If even Prof. Hotopf thinks the avoidance of ethical review was problematic, it’s got to be really bad!!

  • Couch Turnip 4 February 2019, 5:27 am

    I’m reminded of a comment made by Janet Wisley OBE a few years ago, when she was Chief Executive of the HRA. At the 2016 Academy of Medical Sciences Forum Annual Lecture entitled “What can research do to improve productivity in the NHS?”, she remarked from the audience to the effect that the HRA wanted people to ask them difficult questions and that those who came to them thinking that they couldn’t do something would be helped to find a way. Prof Sir Simon Wessely who was on the panel then affirmed that this was true. So how does the HRA, which is tasked with trying to ensure that research ethics are upheld in the UK, help researchers to find a way exactly? That’s what I’d like to know. And did the Bristol REC wave through the school absence study, as Hotopf seems to suggest they might have? This really requires a proper investigation, because there are others besides Crawley who have serious questions to answer here.

  • jimells 4 February 2019, 9:26 am

    I hope I’m wrong but I expect any “independent” investigation short of a civil complaint handled by very competent solicitors will be a whitewash.

    I had noticed some time ago that Crawley has kept a low profile since her promotion. I speculate that the promotion, along with an order to stuff a sock in it, was Step One. Step Two is an investigation that concludes “Nothing to see hear. Move along.” I have a vague recollection that Regius Professor Sir Simon actually used that phrase, but I can’t remember where.

    What we really need is the full release of all PACE trial data to someone who will properly analyze it. Reports from self-identified PACE trial participants suggest the data will reveal massive harm to some participants. The data will also show that the trial investigators knew about the harm and covered it up. That is why they have spent so much money and effort (at some cost to their reputation) to hide the data.

    Nearly everybody knows this is the case, but nobody wants to say it, probably out of fear of being labelled a nutter or the worst one of all, a “conspiracy theorist”.

    Personally, I am way past being worried about a label. I just want to see the truth exposed. And the Wessely School put out of business.

Leave a Comment