Google Flu Trends is not accurate

18 May 2010

Google Flu Trends uses analysis of large numbers of search queries to track influenza-like illness in a population. The idea is that the frequency of certain queries correlates with the percentage of physician visits in which a patient presents with influenza-like symptoms. Google claims that it can accurately estimate the level of weekly influenza activity in each region of the United States. But a recent study shows that Google Flu Trends is not as accurate at estimating rates of laboratory-confirmed influenza as surveillance carried out by the CDC.

Google Flu Trends and CDC surveillance results were compared for the period of  2003 – 2008. As reported at the 2010 American Thoracic Society Conference, the greatest deviation of Google Flu Trends from CDC surveillance occurred during the 2003-04 influenza season. That year was characterized by early and frequent influenza activity, many pediatric deaths, and heavy focus by the news media.

The main reason for the discrepancy is likely the fact that influenza may only account for 20-70% of influenza-like illnesses. The remainder are caused by other viruses that produce similar clinical syndromes. The authors of the study concluded that “Internet search behavior is likely different during anomalous seasons such as during 2003-4…during periods of intense media interest or unexpected influenza activity such as the 2009 H1N1 influenza pandemic, Google Flu Trends may be least accurate at estimating influenza activity.”

Google Flu Trends does provide an estimate of influenza activity more quickly and cheaply than can be achieved in a diagnostic laboratory. But in this case, cheaper and quicker means less accurate.

Ginsberg, J., Mohebbi, M., Patel, R., Brammer, L., Smolinski, M., & Brilliant, L. (2008). Detecting influenza epidemics using search engine query data Nature, 457 (7232), 1012-1014 DOI: 10.1038/nature07634

Tagged as: , , , ,

  • BrianHanley

    Well, in part that depends on how you define accuracy.

    If it is defined as time accuracy then Google Flutrends is more accurate. If it is defined as accuracy of estimates of infection numbers that's probably CDC. But the latter is a bit of a tautology because we have defined CDC as the standard of what is correct. Influenza numbers are notoriously hard to pin down.

    My personal view is that I suspect Google Flutrends has a bit of 20-20 hindsight. It isn't practical to set searches fixed, so it needs to conform itself as much as possible to CDC's figures by tweaking its criteria. I'm not suggesting it is invalid, I'm just saying that's how it works. So it depends what you want. If you want to know what is probably happening right now then look at Google Flutrends if you are in the developed world (with apologies to Rosling for implicitly calling the others developing). Outside the developed world it's a question.

    I think we will eventually have instruments that moms can buy to test their kids for a few bucks, and if those can be tied into the cell phone network, then we will have another alternative. Until then, Google Flutrends will be our best early warning.

  • http://www.virology.ws profvrr

    Accurate means laboratory-confirmed influenza. I don't see how any
    google based survey of ILI could be more accurate than finding
    influenza viral RNA in your nose. But if you want a general idea of
    what's happening then Flu Trends is ok. Until the day when you walk in
    front of a sensor in your home and it delivers your infection status.

  • BrianHanley

    Spoken like a virologist. :-)

    There's an “of course” to that, but until we have those gizmos (which I have worked on and tried to push for) that diagnose what's in your nose that isn't possible. So any measurements we make lag behind. Accurate can also mean finding secondary indicators correlating with the incidence of infection. In much of the world the only statistics that can be collected are of ILI. Lab tests just don't get done, so diagnosis is presumptive. That's clinical medicine. That's most of it in the USA. The CDC program is a sampling.

    From a clinical public health viewpoint it makes a lot of sense to use secondary indicators historically, those have been clinical “presumptive diagnoses” by physicians, RNs, PAs and public health personnel. The Google searches are arguably as close a diagnostic as a physician's observations of the same symptoms as the google queries. Those queries originate with the patient or someone close to them. (Generally a mother.) Essentially, the patient is answering a question without being asked.

    Looking at the paper, Google Flutrends leads lab diagnostics significantly. That's inevitable because of “queuing time” consisting of time to get the patient in, collect the sample, send it off, test it, report it. If I had reviewed it I would have asked if they could also compare with clinical ILI reports to compare apples with apples. But I don't think I would have required it. Mostly I like the paper. It's useful I think.

  • BrianHanley

    Spoken like a virologist. :-)

    There's an “of course” to that, but until we have those gizmos (which I have worked on and tried to push for) that diagnose what's in your nose that isn't possible. So any measurements we make lag behind. Accurate can also mean finding secondary indicators correlating with the incidence of infection. In much of the world the only statistics that can be collected are of ILI. Lab tests just don't get done, so diagnosis is presumptive. That's clinical medicine. That's most of it in the USA. The CDC program is a sampling.

    From a clinical public health viewpoint it makes a lot of sense to use secondary indicators historically, those have been clinical “presumptive diagnoses” by physicians, RNs, PAs and public health personnel. The Google searches are arguably as close a diagnostic as a physician's observations of the same symptoms as the google queries. Those queries originate with the patient or someone close to them. (Generally a mother.) Essentially, the patient is answering a question without being asked.

    Looking at the paper, Google Flutrends leads lab diagnostics significantly. That's inevitable because of “queuing time” consisting of time to get the patient in, collect the sample, send it off, test it, report it. If I had reviewed it I would have asked if they could also compare with clinical ILI reports to compare apples with apples. But I don't think I would have required it. Mostly I like the paper. It's useful I think.

  • http://www.virology.ws profvrr

    Yes – after I wrote the comment I realized how our positions reflect
    the type of science we do. You are correct, of course, in that the
    viral detection is always in retrospect. Until gizmos are available
    then sampling methods such as Google Flu Trends are the best we have
    for predicting. As the paper says, we just need to realize that the
    data might not be entirely correct.

  • BrianHanley

    I have done both actually, and things in between. I think we agree. Depending how one looks at it, I think that the virology is harder to do. I do wonder if Google leads physician ILI reports by as much as it leads confirmations. I was a little surprised the lead wasn't longer than a couple of weeks. A good home diagnostic might or might not lead Google, I'm not sure. It could lead if it was cheap enough because moms tend to be fanatic about the health of their kids and young children are a primary demographic. (Although much of their ILI is adenoviruses, rotaviruses, cold viruses and probably combinations of them.) I don't think it would lag Google though, because Google reflects first appearance of symptoms. One should keep in mind that first appearance of clear symptoms lags infection by around half a week.

  • Pingback: Does Marketing Research Need To Be Perfect? Sometimes Close Is OK. | Train of Thoughts, Ottawa ON

  • Guest

    This assumes that CDC’s data is reflective of reality; I have seen no indication of this.