Trial By Error: The NICE “Topic Expert” Reports

By David Tuller, DrPH

My first recent freedom of information request to the National Institute for Health and Care Excellence (NICE) was for information about the experts consulted in the current process of reviewing CG53, the 2007 guidance for the illness the agency calls chronic fatigue syndrome/myalgic encephalomyelitis. In its response, the agency explained that seven topic experts had been consulted in the process of preparing the surveillance document, which recommended leaving the guidance as is.

(I have previously written about the NICE review process on CG53 here, here and here. My e-mail exchange with Sir Andrew Dillon, chief executive of the NICE guidance executive, is described here.)

NICE, of course, declined to reveal the names of the seven topic experts, although it reported that four of them were members of the original 2007 guidance development committee. NICE also told me, as it told the ME Association, that the panel included three psychiatrists, two neurologists, a pediatrician, and a patient. NICE did not explain why the panel included more psychiatrists than other specialists, but the selection of topic experts certainly provides insight into NICE’s perspective on the illness.

My next FOI request was for any reports prepared by these topic experts–properly anonymized, of course. I recently received them and have asked for them to be posted on Phoenix Rising. If this is the full extent of the input provided by the topic experts, it seems like NICE made a remarkably anemic effort to seek outside advice. And it sought that advice from a relatively narrow spectrum of professional opinion, especially given the enormous complexity of the disease.

In its e-mail to me, NICE explained that the topic experts provided “three types of contributions.” Each person was asked to fill out two questionnaires. The first questionnaire sought suggestions on new studies or research that could impact diagnostics, interventions and other aspects of the guidance. The second sought input about two specific studies and whether the surveillance report should contain in-depth commentaries on them. These questionnaire responses were due last December. The questionnaires were sent to me as separate documents.*

(The previous sentence has been corrected. The sentence originally read: “The questionnaires, which were often only partially filled out, were sent to me as separate documents.”)

In March of this year, NICE surveyed the same experts about three 2015 U.S. reports—from the Agency for Healthcare Research and Quality, the Institute of Medicine (now the National Academy of Medicine), and the CFS/ME Advisory Committee to the Department of Health and Human Services. NICE wanted to know about the possible impact of these reports on diagnostic criteria and other aspects of CG53. Not all of the topic experts answered. The responses were all sent to me as part of a single document.

Strangely, the first questionnaire sent to all seven topic experts included this sentence in the first paragraph: “As a member of the committee that developed this guideline we welcome your views on any areas that need updating.” That suggests that all seven topic experts consulted were, in fact, members of the committee that developed the 2007 guidance–even though NICE had already informed me that only four of the seven were on the 2007 committee.

I have no idea whether NICE misinformed me that only four of the seven were on the original committee, or whether the wrong form was sent three topic experts who were not actually on the 2007 guidance committee. Whatever. NICE obviously has glitches in its fact-checking department. I hope that problem gets worked out soon.

The documents I received include some interesting and irritating statements from the experts, but nothing particularly surprising. Two of the six medical experts (#4 and #7) expressed modest concerns that recent events had rendered the current guidance somewhat out-of-date. For example, #4 cited the “considerable on-going public and media interest in this disorder” and advised NICE to have “a current and up to date view on this controversial area.” The other four (#1, #2, #3, #6) found that the research supported the current recommendations.

The patient in the group (#5) expressed personal opinions about the difficulties patients face as well as concerns about case definition–but did not fill out the questionnaires provided by NICE and did not make recommendations related to the guidance. #5 attached two additional statements from other ME patients to his/her/their submission to NICE; together they included many references to the emerging science and to critiques of the PACE trial.*

(The previous sentence has been corrected–originally I wrote that NICE did not send me these two additional attachments as part of the FOI response, but it turned out I didn’t notice them in the set of documents.)

Expert #6, an apparent CBT/GET hardliner, had particularly firm opinions about what was what: “This guidance has stood the test of time well, and has been instrumental in shifting thinking about the common problem of CFS/ME and medically unexplained symptoms (functional) symptoms in general. I hope that a new guidance on management of functional disorders will extend this process further.”

In response to the CFSAC suggestion that it was important to focus on biomarkers and diagnostic tests, #6 wrote this: “In the UK I think we would see what the HHS CFSAC see as a failure to undertake rigorous research, more as a failure of the biological model to explain the condition adequately. However, I can understand that an alternative reaction to the failure of biological models to explain the condition is to try and define a subset of patients with the condition who appear to share a common biomarker. This approach has failed so far.”

Of the six medical professionals, #7 adopted the most moderate view, suggesting that “some aspects of practice have moved on considerably so this guideline needs at least partial review in relation to interventions, diagnostics and prognostics.”

Moreover, #7 made some key suggestions about the guideline development group that should undertake these critical revisions: “It is important that the chair is someone who has no previous connection to CFS/ME practice or research. That person should not be a mental health professional if the guideline is going to be accepted by NHS clinics and patient organizations. However, I think there should be a mental health professional on the guideline development group.”

#7 was also the only one to offer specific concerns about PACE. Here’s what he/she/they wrote in discussing the research since the guidance came out in 2007: “There are some larger and more definitive UK RCTs of treatments recommended for CFS/ME by NICE in 2007 but these are controversial and one in particular PACE in over 600 patients, published in the Lancet, has been subject to legal challenge on the grounds that after the study was started the primary outcome was changed.”

That comment isn’t exactly accurate and it does not indicate a full awareness of the facts or of the widespread rejection of the PACE trial by the larger scientific community. But it’s more skepticism and concern than shown by his/her/their colleagues, who mostly toed the standard line and responded as you’d predict.

In response to safety questions, #3 provided more evidence—if any were needed—that the Cochrane reviews of GET and CBT continue to be major obstacles. Here’s what #3 wrote: “Whilst the patient community remain concerned about safety, recent Cochrane reviews suggest exercise therapy is safe and patients are more likely to improve with treatment.”

So Cochrane says it’s safe, and that means the patients are wrong. And yet the reviews include the studies featuring the methodological flaws that patients—and the worldwide scientific community—are challenging in the first place. Reviews can’t provide good evidence if they’re based on bad studies, of course, but #3 did not point this out.

In the second questionnaire, the Dutch FITNET trial was one of two being considered as the subject of an in-depth commentary in the surveillance report. The questionnaire briefly described the study and asked for input. Here’s what NICE included in its description of FITNET in the questionnaire:

“CBT is already recommended by CG53 and this study shows it can be delivered via the internet, indicating an alternative mode of delivery for this treatment for this age group. Although the study was from the Netherlands, a UK trial is underway. There were some issues with the study (blinding was not possible due to nature of interventions, and usual care varied because the quality and quantity of CBT differed according to local availability and was often combined with other treatments such as GET). These issues would be discussed in full in the commentary.”

It must be pointed out that lack of blinding is not just some small problem with the Dutch FITNET study. The lack of blinding combined with the reliance on subjective outcomes renders FITNET, and the bulk of studies from the CBT/GET ideological brigades, incapable of providing reliable or valid evidence of treatment efficacy. Yet the topic experts generally agreed with NICE’s positive assessment of FITNET and largely failed to address the study’s serious flaws in their comments.

Among those problems was that the “usual care” group did very poorly. Unlike in PACE, usual care was not just a few meetings with a clinician. In FITNET, the usual care group mostly received CBT or GET, or both—in other words, the treatments already recommended for the illness in the Netherlands as well as in CG53. And yet apparently these treatments largely failed to work. That raises this critical question: If CBT and GET are performing so poorly, why are they the standard of care in the Netherlands, not to mention the U.K.?

In its comments about the study in the questionnaire, NICE fudged this enormous problem by suggesting that the usual care in the study “varied” in quality—suggesting that the treatments performed poorly because they were not optimally delivered. That’s a very convenient and self-serving explanation, when the obvious and simpler answer is that the therapies simply do not work as promised.

Citing this study, Professor Esther Crawley of Bristol University has received approval to inflict the FITNET approach on British children. But NICE, Professor Crawley, the topic experts and the Dutch investigators themselves have not actually considered one obvious reason why the patients assigned to the FITNET intervention might have reported better subjective outcomes. For sick patients, it would involve much less exertion to have online CBT at home than to run around town attending multiple in-person sessions. It seems logical that patients treated while in front of their laptops and not forced to travel weekly to clinic might do better than those in “usual care” for that reason alone.

In the document compiling the responses whether the 2015 U.S. reports should impact the diagnostic criteria used to define the illness in the NICE guidance, this comment from #1 jumped out as particularly uninformed: “I am unaware of any concerns about the inclusion criteria of trials in CFS.” Wow. Given the enormous international controversy over the range of case definitions that have been developed for the illness, it is hard to know how to respond to such a statement.

Not surprisingly, this person expressed satisfaction with the criteria outlined in CG53: “I do not see any need to change the diagnostic criteria at present. From a clinical perspective, CG53 are pragmatic and useful criteria. In my clinical experience in a CFS/ME clinic over the 10 years since the guidelines were published, no concerns have ever been expressed by patients attending the clinic about the diagnostic criteria used in CG53.”

And here’s what #6, the CBT/GET hardliner, had to say about the U.S.-based reports: “I am suspect [sic] that there is a tendency in the USA to push towards an entirely biological explanation for the condition, whereas in the UK there is an increasing acceptance amongst patients and clinicians alike of a model that includes CFS/ME in the umbrella of functional neurological disorders, i.e. that it is an emotionally driven disorder.”

Ok, then. The notion that CFS/ME is “an emotionally driven disorder” is certainly a popular concept in specific academic and medical circles in the U.K. But #6 does not cite any data to support the claim that there is an “increasing acceptance” of such an idea among British patients and clinicians. That this pet theoretical project of a powerful clique of psychiatrists (and some others) is now presented to NICE as having popular appeal and clinical legitimacy is distressing.

And that this set of comments is the best NICE could drum up in seeking external guidance for this critical review process is deeply worrying.

Comments on this entry are closed.

  • Jane M.

    Thank you for your ongoing effort, David! I’m from Germany and the NICE guidelines are the main reference for our “tiredness” guidelines. Ours will only be updatet if NICE will be updatet. Since NICE is so influential beyond the UK, this work is essential for pwME in many countries.

  • Jane M.

    Hi Trish, I believe the IOM even looked at 9000 studies.

  • Sean

    Very intelligent strategy by David Tuller.

    Getting the reports by the “experts” into the public domain. Their comments are ludicrous.

    It would take some time to go through the FOI & ICO processes to get the names.

  • deboruth

    There are workable biomarkers US experts use and also ones identified by UK experts like Ramsay, Beha
    n. But to come up with ones US agencies will endorse requires are least $100 million which certain bureaucrats will always sabotage. One needs a sequence of miracles.

  • Steve Hawkins

    It seems as clear as day that the NICE has made no effort to be advised by actual experts in the disease for which the guideline is intended, but has simply gone back to the same clique of self appointed owners of the whole set of diseases for which there has been insufficient research to understand the physical cause. It’s as if they had decided to do a guideline on epilepsy but only invited homoeopaths!

    This is an unbelievably unprofessional way to go about the setting of any kind of national standard of care, and NICE themselves are, thus, failing in their duty of care for the whole patient population. It ought to be obvious to any judge in a judicial review of the guideline setting process chosen by NICE, that it was unfit for purpose and incapable–by incompetence or design–of coming up with a true representation of the facts of the disease and it’s optimum management.

    NICE itself, really does have to go. It’s making a laughing stock of UK medical science.

  • Pingback: Trial By Error: The NICE “Topic Expert” Reports (FOI) | WAMES (Working for ME in Wales)()

  • Trish Davis

    Thanks, Jane, I’ve edited my post.

  • Leigh

    They told me this just before these documents were published so maybe the arm I was speaking to didn’t realise they could do that. I have replied asking if they can publish with the names redacted.

  • George

    7 mental health UK ‘experts”, no references to publications, generalisations, incomplete questionaire, the same people looking at reviewing the guidelines as that wrote them in 2007…..versus a USA panel of experts from a wide range of disciplines, fully referenced and researched views…..

  • Pingback: Trial By Error: The NICE “Topic Expert” – The York ME Community()

  • nancyblake