Trial By Error: Bristol Investigating Crawley Papers

By David Tuller, DrPH

In the last year, I have raised multiple concerns about studies from the University of Bristol with the UK Health Research Authority, among others. The HRA is the National Health Service arm that oversees approvals for research in England; the agency’s role is to ensure that research receives a favorable opinion from an ethics committee before it begins.

Yesterday, the agency confirmed to me that, after reviewing the matter, it has raised questions with University of Bristol officials about a number of these studies and has asked them to investigate the situation. The HRA is now awaiting the outcome of that investigation.

In other words, some of Professor Crawley’s work is now under formal scrutiny. This is a serious inquiry, although of course it is difficult to predict what the results will be, given the sorry history of this whole mess. I would expect this investigation to take at least a couple of months, and perhaps more. It’s not always clear to me how things work in the UK, but people can presumably make their own assessments about whether the HRA would have chosen to request an investigation if it believed everything in these studies were kosher.

This is not the first indication that my efforts to draw attention to these problematic studies are possibly bearing fruit. Last June, Archives of Disease in Childhood slapped an editor’s note on the Lighting Process study it published in 2017. (Archives of Disease in Childhood is a BMJ journal.) The investigators claimed that the study was prospectively registered even though 56 out of 100 participants were recruited before trial registration–meaning any claims that it was prospective were self-evidently untrue. It is hard to understand how experienced researchers could have committed such an elemental error.

Moreover, the investigators swapped primary and secondary outcomes after more than half the participants had been recruited, thereby biasing their reported results. Yet the paper failed to disclose these details and presented the study as if it were, in fact, prospective. BMJ, like all top medical journals, has a strict, long-standing policy that it will not consider for publication any studies that are not prospectively registered. And undisclosed outcome-swapping more than half-way through a study is unacceptable by any standard.

Here’s the relevant part of the editor’s note: “Since publication, the study has been criticised for failing to meet ICMJE and BMJ policy on trial registration and for not fully adhering to CONSORT guidance on trial reporting. The journal has been criticised for not detecting these issues during editorial and peer review. We have acknowledged these comments and reviewed our processes in relation to this paper and relating to EQUATOR guidance in general. In addition, we have received clarifications from the authors which are under editorial consideration.”

It is noteworthy that the editor’s note does not refute the concerns I have raised. The statement is most easily interpreted as an admission that these concerns are in fact valid; otherwise there would be no need for a period of “editorial consideration” to consider the “clarifications” provided by the authors. Yet the editor’s note is not linked to or visible from the paper itself. No one reading the paper would have any idea such an editor’s note existed unless they happened to click on the “responses” page. And that means they would not know that significant questions about the study’s methodological and ethical choices had been raised–and “acknowledged” by the journal itself, as the statement indicates.

It is unacceptable that this critical editor’s note has been sequestered away in a section where few people are likely to see it. It is also unacceptable that this period of “editorial consideration” has now lasted more than seven months, with no end or resolution in sight. It takes only a few minutes to check the relevant documentation and determine what happened here. Why hasn’t this matter been addressed and resolved already?

This stonewalling creates the strong impression that the editorial team is more concerned with avoiding reputational damage than protecting children’s health. The same is true of BMJ’s unwillingness to address the problems with the 2011 school absence study in BMJ Open, which was exempted from ethical review on the false grounds that it was service evaluation and not research. BMJ Open not only defended publication of the paper without ethical review but, in the process of doing so, made false claims about the paper’s methodology while simultaneously accusing me of disseminating misinformation.

**********

Fiona Godlee, BMJ’s editorial director, is aware that the situation is problematic and needs to be dealt with. She suggested as much in a note she sent me in October in response to various open letters I had sent her and other BMJ editors. Again, it is noteworthy that she mounted no defense or refutation of the points I have raised about studies published in her journals.

I haven’t mentioned this note previously because, for one reason, Dr Godlee indicated that she was anxious to get to the bottom of things. So I thought I might give her the benefit of the doubt. Yet I have seen no evidence that BMJ has taken further steps since then. So there’s no particular reason not to share our exchange at this point:

Here’s what Dr Godlee wrote on October 12:

Dear David, I am conscious that I have not directly responded to your recent emails. I wanted to let you know that we are working closely with…the HRA…and are doing our best to resolve the situation. There are matters beyond our immediate control that make this challenging as I am sure you understand…You were concerned that the notice on the ADC [Archives of Disease in Childhood] paper was not prominent enough. We are looking into whether this can be made more prominent–our systems don’t make this entirely easy! Please be assured that we are actively working on the various aspects of this case and will do all we can to reach a timely resolution. We will also do our best to keep you informed of progress. All best wishes, Fiona Godlee

**********

I answered Dr Godlee’s note a few days later. (I have deleted some passages from my response.) Here’s what I wrote:

Dear Fiona—

Thank you for your note. I very much appreciate hearing from you directly. I apologize in advance for this long message, but I have been trying to get BMJ to address these issues for some time and I therefore have a few things to explain…I’m pleased that things seem to be progressing, given the troubling nature of the methodological and ethical violations that I have documented.

I have frankly been rather surprised at the way the editors involved with both the school absence and Lightning Process studies have responded to date. Since last fall, I have written more than 20 blog posts about these two studies. (Here is a link to all my Virology Blog posts about ME/CFS, including those: http://www.virology.ws/mecfs/.) No one should have to invest this amount of time and energy to convince reputable publications that they need to address such obvious irregularities. I’m afraid that this protracted and wearying process has created the impression that your editorial team has been more concerned about protecting reputations than about the health and wellbeing of vulnerable children. If BMJ’s attitude toward this mess is now shifting, that is certainly a positive development.

Since I am neither an epidemiologist nor a statistician but an investigative reporter with a public health doctorate from Berkeley, I have vetted my concerns about PACE and the other studies I have criticized with colleagues at the University of California, Columbia, Harvard and elsewhere. As you can see from the credentials of the experts who signed the various open letters to The Lancet and Psychological Medicine as well as Archives of Disease in Childhood and BMJ Open, eminent researchers and scientists in the US and elsewhere are genuinely shocked at the deficiencies of all these studies. From our perspective, the situation is a real-life version of “the emperor has no clothes,” and that description would seem to apply to much if not most of those occupying high-level positions in the UK medical and academic establishments.

[paragraph deleted]

The main reason these matters need to be dealt with as soon as possible is that information about the Lightning Process and school absence studies has been widely disseminated, at least in part because both papers were promoted by the good folks at the Science Media Centre. This research has the potential to influence policy, if it hasn’t already. That means children have been or are being impacted by BMJ studies that should never have been published in the first place.

[Paragraph deleted]

Since you have now opened up a dialogue with me, I should let you know—in the interests of transparency–that I am working on more open letters to you…The first is a request for an apology from BMJ Open for its actions in response to inquiries about the school absence study. In particular, I am owed an apology for the journal’s decision to call my reporting “inaccurate” and “misrepresentative” in its statement to COPE—not to mention the fact that this COPE statement included the untrue claim that the data used in the study were collected anonymously. Since anyone who read the study would immediately know that children and their families were interviewed in person, I am curious why BMJ Open editors thought the COPE statement was an appropriate description of what happened. (More on this issue below.)

Now to the points you have raised in your letter. You suggest that I am concerned that the editor’s note in Archives of Disease in Childhood is not “prominent enough.” That is not exactly so. My concern is that it is not “prominent” at all. This editor’s note has been posted where no one is likely see it unless they are looking for it. Furthermore, I am confused by the statement that BMJ’s systems “don’t make this entirely easy.” Why would that be? When BMJ journals need to issue a correction, isn’t it appended directly to the article in question? If so, why is that possible to do with a correction but somehow complicated to do with an editor’s note? (In fact, my main problem with the editor’s note is that Dr Brown chose to post it rather than retract the study as soon as he had confirmed the facts–more on that below.)

In addition, I am afraid I do not understand why BMJ feels that its actions are constrained by “matters beyond [y]our immediate control.” This argument does not seem to apply to either of the studies in question.

Regarding the school absence paper…I do not see why BMJ Open should need more time to acknowledge that it failed in its basic oversight function by publishing the school absence study without ethical review….

Regarding the Lightning Process study, the editor’s note makes clear that Archives of Disease in Childhood has received a response from the investigators and that the matter is now under “editorial consideration.” The editor’s note was published four months ago. How much “editorial consideration” is needed to determine that this paper violated core scientific principles? After all, the company’s own policy states that “in accordance with the ICMJE Recommendations, BMJ will not consider reports of clinical trials unless they were registered prospectively before recruitment of any participants.” The resolution of this issue does not therefore seem to be dependent on factors beyond your immediate control. BMJ Open needs no further information to take action, unless the journal has for some reason outsourced its editorial decision-making to other parties.

Around the time I raised these questions about the Lightning Process study, BMJ Open published a paper on the importance of requiring prospective registration for all trials being considered for publication. To highlight this point, the former editor-in-chief of BMJ Open tweeted that “prospective registration of #clinicaltrials isn’t red tape: it’s done for sound ethical and scientific reasons.” Dr Groves is right, of course. One corollary of that principle is that any cases in which the opposite occurred should be addressed immediately for the same “sound ethical and scientific reasons.” Moreover, the Lightning Process investigators omitted crucial details from the published paper, which was written up as a prospective trial and linked to an ISRCTN registration that falsely categorized the trial as prospective…

[Three paragraphs deleted]

Under the circumstances, it is hard to understand why Archives of Disease in Childhood would consider any resolution but retraction—along with a full accounting of what happened and why the journal failed to notice these problems. In fact, retraction is the only solution that will allow Archives of Disease in Childhood to maintain that it has adhered to BMJ’s unequivocal policy about prospective registration. As I suggested in my most recent open letter to Dr Brown, I hope BMJ now understands the importance of asking editors and peer reviewers to read not just drafts of papers but supporting documentation—trial registration, research ethics committee applications, and so on. Otherwise BMJ journals will likely experience more such embarrassing episodes.

[Much of the following information, about my request for an apology, was also part of an open letter to Dr Godlee on this issue, so regular readers might find these passages familiar.]

Now I’ll address my request for an apology related to BMJ Open’s defense of the school absence study—the subject of my next open letter to you. Since last year, BMJ Open has repeatedly obfuscated the issues and denied any problems. This happened first when a reader contacted the journal with valid complaints. After the journal brushed her off, she contacted me. I quickly verified that the study was indeed research that required ethical review. After all, the paper included a formal hypothesis, the title itself expressed a generalizable conclusion, and the lead investigator personally interviewed participants. These features are all well-known hallmarks of research, not service evaluation.

I then contacted Dr Groves. It seemed from the start that she hoped the whole matter would go away or that I would lose interest or…well, I don’t really know what she thought. She stated in correspondence with me that BMJ Open itself had reviewed the matter and had apparently satisfied itself that the study was appropriately categorized as service evaluation—despite the fact that it was published under the heading of “research.” I noted in response that this internal BMJ Open review could not have been very thorough. Any editor who took ten minutes to read the study should have known immediately that it should never have been exempted from ethical review.

In something of a scholarly version of Rashomon, the journal then presented its own misleading account to COPE. In response, COPE suggested that BMJ Open should ascertain how the data were collected to determine whether the study could qualify as service evaluation. At that point, BMJ Open apparently chose to close the COPE case without further action. In a subsequent letter to me, Dr Groves cited the COPE statement as evidence that BMJ Open had taken the issue seriously—without mentioning the journal’s apparent failure to follow COPE’s advice.

In February, Professor Vincent Racaniello, a Columbia microbiologist and host of Virology Blog, sent an open letter to BMJ Open, signed by multiple colleagues from Berkeley and elsewhere, to draw attention to the journal’s negligence. In her response, Dr Groves suggested that we submit a letter for publication. Professor Racaniello and I saw no reason to do so, since it seemed likely that the journal would publish our letter, along with a letter from the Bristol investigators, and then wash its hands of the matter. This approach would have allowed BMJ Open to present itself as a neutral arbiter of an unresolved dispute. But of course the facts are indisputable. BMJ Open published a research paper that exempted itself from ethical review on false grounds, and the journal has continued to defend this indefensible decision.

In her final communication, Dr Groves explained that a letter from us would allow BMJ Open to request a detailed response to our concerns from the investigators. We did not understand why a publishable letter from us was needed in order for the journal to seek adequate explanations from the investigators. Naturally, we declined to participate in BMJ Open’s apparent effort to extricate itself from this thorny situation while evading responsibility for publishing the paper in the first place.

I only appealed to…the Health Research Authority when it became clear that BMJ Open had no interest in admitting fault and that Archives of Disease in Childhood was taking an inordinately long time to investigate the Lightning Process study. I am concerned that the school absence study could lead to further efforts by investigators at Bristol and elsewhere to interview children and their parents without ethical review. But I am especially worried that the continuing delay in addressing the issues with the Archives of Disease in Childhood study could have potentially harmful consequences for the very ill children whose desperate parents have sought out Lightning Process practitioners based on BMJ’s seal-of-approval. As I have repeatedly pointed out, the Lightning Process founder has claimed that he has been able to “step into other people’s bodies over the years to assist them in their healing with amazing results.” Now BMJ has endowed his current magical modality—a goulash of neuro-linguistic programming, osteopathy, self-affirmations, and life-coaching–with an aura of scientific credibility.

As I wrote in one of my open letters, enough is enough. It is time for BMJ to take stock, acknowledge its mistakes, and do what it can to make things right. I apologize again for the length of this letter, but these things needed to be said. I look forward to further communications from you about BMJ’s progress in coming to a timely resolution of these matters. (I have just arrived in London today and will be in England until October 30. I would be happy to meet with you at your convenience if that would be helpful.)

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

{ 18 comments… add one }
  • Peter Trewhitt 31 January 2019, 7:19 am

    That the HRA is now investigating this group of studies is at last a step forward. Thank you David for your clear analysis and persistence in dealing with what amounts to repeated research malpractice.

    That the University of Bristol should collude in covering up this and seek to stifle your reporting through exerting pressure on your University is disappointing but perhaps not completely surprising. However that the BMJ group of journals should also seek to deny their own failure to recognise these issues when pointed out to them and to repeatedly obfuscate and prevaricate is totally unacceptable.

    There is obviously a significant failure of good governance in both British Universities and in British medical journals. Hopefully your reporting of these egregious research and publishing practices will improve how ME/CFS is dealt with by British academia, and also prompt others to ask if this incompetence is confined to one group of researchers and their supporters or if it is more widespread.

  • Scott Simpson 31 January 2019, 7:55 am

    Score 1 for the good guys.

    David, I believe you wrote this rhetorically:

    “It is hard to understand how experienced researchers could have committed such an elemental error.”

    As it is obvious these researchers did not make an error.

    They intentionally deceived for their personal gain and at the direct expense of the health of people with ME.

  • Couch Turnip 31 January 2019, 8:04 am

    The problem seems to be that whenever people raise genuine issues of concern about any BPS research (like David and others have done), the medical establishment/BPS School/SMC/academic institutions etc automatically decry these as harassment and/or abuse or simply refuse to respond. This seems to have been a line that was adopted as an effective strategy a long time ago by the Wessely School. Now at least the HRA doesn’t seem to be following suit, which gives us hope, but the BMJ and COPE seem to lack that objectivity. I’m rather surprised that we haven’t seen such an counter-attack since the ME debate in the House of Commons when so many MPs (27 wasn’t it?) spoke about the suffering and predicament of their constituents, that surely must have put the wind up them. After all, it’s become a familiar pattern for them to attack with these accusations when they have no other defence. But perhaps they’re following the blanking strategy this time round too.

  • jimells 31 January 2019, 10:10 am

    Dr Tuller: “BMJ Open needs no further information to take action, unless the journal has for some reason outsourced its editorial decision-making to other parties.”

    Imaginary BMJ reply: “Well, now that you mention it, ummm, well, you do know that the Wessely School and their paymasters are skilled at retaliation…”

  • Sheila Campbell 31 January 2019, 10:30 am

    Hopeful news, thanks so very much indeed for all your amazing hard work, slogging away against the brick wall of the UK scientific establishment, which over the years has been shown to have precious little concern for scientific principles or people with ME.

  • Richard Vallee 31 January 2019, 10:37 am

    “There are matters beyond our immediate control that make this challenging as I am sure you understand”

    This is puzzling and needs clarification as it makes no sense in the circumstances. It seems to imply that their hands are tied, which is inexplicable and only raises further concerns. The issues raised are all within BMJ’s own rules and guidelines.

    The only reasonable interpretation is political influence, which would raise serious problems with the journal’s editorial independence. How common is that? Inversely, if this is exceptional, why?

    And in another line of questioning, why did it seem expected that the matter would just go away on its own? This is the opposite of what one would expect from a medical journal being presented with serious flaws in its own editorial process.

    The research itself was poor and of exceedingly low quality, it absolutely does not merit this much protection, or any such protection at all. Especially in consideration that this is typical of Crawley’s work, which always seems to necessitate a lot of institutional water-carrying and goal-widening-to-the-entire-field.

  • Andy Dearden 31 January 2019, 11:50 am

    Sounds like the British establishment (& medical establishment) closing ranks and wanting to avoid offending anybody – hoping that it will go away by publishing an exchange of letters – or putting an editors note somewhere where people won’t see it (remind me now – where did they put the editors note about the cochrane review?).

    But as you rightly point out, these publications impact on policy -the parliamentary briefing papers for last week’s debate demonstrate how the academic publications get filtered and ministers are told that criticisms ‘have been answered’ by the authors in leading journals.
    This stuff is not without victims and we need to keep pressing on until this stuff is recognised for the pseudoscience that it is. Let’s hope that the folks at the HRA are able to put patients interests ahead of a quiet life for themselves & their colleagues.

    Well done David, keep up the good work!

  • Matthew 31 January 2019, 12:26 pm

    Great tenacity david. There seems to be no end of lapses of proffesionalism anywhere that the subjects are pwme. Much appreciation, and may you continue to expose these practices.

  • Maggie Wallace 31 January 2019, 1:29 pm

    Wow! Best non Birthday Present I’ve had in all of my life, and I’m 70! Thank you!

  • Richard Smallfield 31 January 2019, 1:55 pm

    Thanks again, David for your tenacity.

    It’s outrageous and astounding that you should have to keep banging your head against a brick wall for so long in an attempt to get medical journals to follow their own methodological and ethical guidelines.

    I can’t help wondering if they are just trying to wear you down until you are forced to give up, having exhausted your time and energy.

  • Lois Addy 31 January 2019, 7:10 pm

    I feel heartened by this. I don’t feel it’s just me/cfs, I feel there’s systemic failures in how research is undertaken, the measures changed mid trial and how papers are presented in a way that twists the data, and that publications don’t have sufficiently robust procedures to check this. I do hope that the work you have been doing leads to a wholesale improvement in industry standards so that unscientific trials become a thing of the past (or at least less prevalent)

  • Alicia Butcher Ehrhardt 1 February 2019, 3:04 am

    I’m glad you’re on top of this; I can’t even read the details any more. There was a time when I was a research physicist…

    We (the ME/CFS community) are incredibly lucky to have you fighting this ‘mess’ of non-scientific garbage.

    Thank you.

  • Kelly McLellan 1 February 2019, 9:21 am

    For the sake of clarity and openness, could you please give an indication as to why you have withheld to deleted paragraphs?

  • David Tuller 2 February 2019, 11:37 am

    In response to Kelly’s question, I deleted some passages from my response to Dr Godlee because they included either confidential information or statements that for now I prefer not to publish.

  • Kelly McLellan 3 February 2019, 10:36 am

    Thanks David!

Leave a Comment