By David Tuller, DrPH
Last week I broke the news that Bristol University is conducting an independent investigation of a number of studies that were exempted from ethical review on the grounds that they qualified as service evaluation. Because the issues involved are confusing and complex, I thought it would be helpful to repost here part of my initial investigation of one of those exempted studies–the 2011 school absence paper published in BMJ Open.
That paper–Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics–turned out to be one of at least 11 studies that were exempted from ethical review based on a single 2007 opinion from the local research ethics committee. The school absence study included a formal hypothesis, generalizable conclusions, and in-person data collection activities. In fact, the title is itself a generalizable conclusion.
Since all three characteristics are hallmarks of research requiring ethical review, the presence of any one of them should have disqualified the paper from being published as service evaluation. This problem should have been obvious to any experienced investigator and journal editor. I continue to be perplexed at why BMJ Open not only published the paper in the first place but has continued to undermine its own reputation and credibility by consistently defending its indefensible decision.
In my initial post on the school absence study, I outlined some of the core issues:
*Professor Esther Crawley and co-authors claimed a 2011 study in BMJ Open was exempt from ethical review because it involved the routine collection of data for service evaluation. Yet the 2011 study was not an evaluation of routine clinical service provision€“it was designed to road-test a new methodology to identify undiagnosed CFS/ME patients among students with records of chronic absence.
*To support the claim that the study was exempt from ethical review, Professor Crawley and co-authors cited a 2007 research ethics committee opinion that had nothing to do with the data-collection activities described in the 2011 paper.
*For the 2011 study, school letters were sent to families of 146 students, inviting them to meet with Professor Crawley. In the end, only 28 were identified as having CFS/ME€“meaning more than 100 families of students without CFS/ME received potentially disconcerting letters inviting them to a medical meeting about a sensitive issue. This type of pilot program is beyond the scope of what many would consider to be service evaluation.
*A pre-publication reviewer, noting the data collection activities described in the paper, raised serious questions about the lack of ethical review. In her response, Professor Crawley did not provide satisfactory answers to the concerns raised by the reviewer, but BMJ Open published the paper anyway, without ethical review.
**********
The last point is particularly significant, since it demonstrates not only problems with the paper itself but also BMJ Open€™s editorial negligence in agreeing to publish it without ethical review. It should be noted that the peer reviewer in question was Professor Matthew Hotopf, a professor of general hospital psychiatry at Kings College London, a colleague of those whose work I have criticized. Given his association with members of the CBT/GET ideological brigades, it is surprising that his concerns do not appear to have been taken more seriously.
Here is what I wrote:
In fact, a reviewer invited by BMJ Open to comment on the draft of the 2011 paper appeared perturbed at the lack of ethical review and approval. In his comments, he expressed surprise that the REC would have considered the outreach aspects of the study to be service evaluation exempt from ethical approval, rather than research requiring it.
Here€™s what he wrote: It is understandable that the REC might see use of routine data from the existing clinical service as not being research€¦but it is surprising that they did not see the surveillance component as research. Children who are unknown to services were being contacted using information from their schools and it seems to me that there are significant issues of confidentiality and data protection which, in my experience as a researcher and one time REC member, I am surprised the REC did not think amount to research. Assuming the REC was fully aware of these issues, and still made a decision that the work was not research, then it would be unfair to oppose publication on those grounds, but the authors should make a fuller explanation, and in the interest of openness might want to make their application to the REC and subsequent correspondence available with the publication.
The reviewer€™s statement clearly presumed that Professor Crawley and her colleagues had filed an application and corresponded with the appropriate REC about the specific set of activities involved in this school absence study. Given that presumption, he made a reasonable request€“that they should publish the REC application and correspondence along with the paper. The reviewer also requested the authors to provide more information about what the families were told and how their consent to participate in the project was obtained.
A pre-publication review like that should raise red flags with editors€“at least enough for them to ensure that the authors provide acceptable answers. That apparently did not happen in this case. In response to the reviewer€™s comments, Professor Crawley did not cite any correspondence with the REC about the school absence study under discussion, perhaps because no such correspondence existed. Nor did she offer much detail on what information was provided to families and on how consent was obtained. And she did not mention that she was not relying on recent REC correspondence about this specific study, as the reviewer presumed, but on an REC opinion about a much narrower method for additional data collection from four years earlier.
Instead, Professor Crawley explained in her response that the specialist service has been advised that ethical approval for routine collection and analysis of service data is not required. She did not provide a legitimate explanation for why this pilot program qualified as routine collection and analysis of service data when it involved outreach to families whose children were not already enrolled in the clinical service, including many families whose children did not even have CFS/ME. Instead, she pointed out that she is a community pediatrician and that the children were seen in school clinics, although it is not immediately evident why these facts should have exempted the study from ethical review.
In her response, Professor Crawley also referenced top-level admiration for her work. The project has been of great interest to the Department of Education who included it last year as an exemplar in their training for attendance officers in the UK, she wrote. But whether or not education officials were impressed with Professor Crawley and her work was of course irrelevant to the question at hand, which was whether the study should have been considered research or service evaluation. It was not clear why Professor Crawley included this point, except perhaps to suggest that she had well-placed supporters.
According to Professor Crawley€™s response, she sought further assurance that the data collection involved was indeed part of service evaluation. Here€™s what she wrote: We checked with the co-ordinator for the local REC that recording outcomes on school based clinics run by school nurses is part of service evaluation (and therefore does not require a submission to Ethics) and they have agreed that it is.
This statement is confusing. The clinics described in this study cannot reasonably be defined as nurse-run school-based clinics. According to the paper, they were set up specifically so Professor Crawley could meet with the children and families identified through the pilot program she developed. Professor Crawley is not a school nurse. The paper€™s description of the meetings with families indicated the presence of a member of school staff, not a school nurse. Professor Crawley€™s own response to the pre-publication reviewer indicated that this staff member was usually the attendance officer.
In fact, the study itself stated that it would be of interest to evaluate whether school nurses, rather than doctors, can undertake the initial assessments in school clinics. In other words, the way Professor Crawley personally collected data for this pilot program had little or nothing to do with the provision of routine care in nurse-run school clinics.
In any event, Professor Crawley provided no documentation of this exchange with the unnamed co-ordinator for the local REC. Nor did she provide any details of what this unnamed local coordinator was told. Was the coordinator told simply that this was data collection made during routine nurse-run school clinics? Or was the coordinator told that this was a pilot program to identify previously undiagnosed patients using their school absence records?
Furthermore, was the coordinator told that families would be sent letters inviting them to meet with a community pediatrician at the school, in the likely presence of the school attendance officer but not a school nurse? Was the coordinator told that the students might then be recruited as patients into the clinical service run by the community pediatrician? Was the coordinator told that most of the families identified and impacted by this recruitment process turned out not to have children with CFS/ME? Absent further documentation of this apparently critical exchange, Professor Crawley€™s second-hand reassurance that the unnamed local coordinator agreed with her interpretation of events is utterly meaningless.
**********
Needless to say, BMJ Open ignored Professor Hotopf€™s concerns and the self-evident facts of the matter and published the paper despite the lack of ethical review. Given that my scrutiny of the study ultimately led to the current investigation of multiple papers from the Bristol team, it is probably fair to say that BMJ Open made the wrong decision. I wonder how much longer we will have to wait for those involved in producing and publishing this mess to demonstrate enough honesty and integrity to acknowledge their serious errors in judgement.
Comments are closed.