The filtering of medical evidence has clearly failed

Let me tell you about this fabulous new test – it’s far better than the old tests, I’ve got a decade’s worth of research publications to show you, plus it’s a floor wax and a dessert topping.

Who am I?  Why, I’m simply an academic clinician, advocating for this test because it’s the next great thing to help us tailor treatments to patients.  Does it matter that I’m on the unofficial payroll of the manufacturer of the test equipment?

That is the question, sadly, physicians and policy-makers are asked to make with recurring regularity when trying to translate clinical research to patient care.  To make these choices, we rely on layers of experts and statisticians to do the heavy lifting in the review process.  This starts at the initial process of institutional review, filters up through pilot studies, then small publications, and finally, systematic reviews and publications in the most prominent medical journals.  By the time the state of the art reaches these most prominent journals, it ought to be possible that we can put unfailing faith in its veracity.

This is decidedly not the case.  A couple weeks ago the Journal of the American Medical Association published a “Clinical Evidence Synopsis” covering the use of procalcitonin measurements to promote antibiotic stewardship.  This synopsis strongly endorses the use of procalcitonin assays to guide antibiotic prescribing in pneumonia and summarizes 14 positive studies in support of its recommendations.  I hardly count myself an expert in this area, but I recalled at the minimum a couple recent studies showing procalcitonin-guided strategies were ineffective or harmful.

To cut a long story short, the JAMA article is written by three physicians who have financial conflicts of interest with the manufacturer of the procalcitonin assay.  The evidence they summarize is a Cochrane Review that counts themselves and a total of eight authors who declare COI with the manufacturer.  On top of that, 11 of the 14 positive studies included in the Cochrane Review are published by authors who declare COI with the manufacturer.  I reviewed each of these articles, and they count amongst themselves trials performed in heterogenous settings with small sample sizes (as few as 8!), high risks of bias, and significant limitations.

The filtering of medical evidence has clearly failed on multiple levels to reach this point.  Practicing clinicians have neither the free time to comb through the original articles, nor likely the institutional access that allows them to even download them without incurring substantial costs.  They are dependent on prominent journals to be vigilant against distortion in a fashion most would find excessive.  JAMA, with a qualified circulation in excess of 300,000 and one of the highest impact factors in medicine, should be one of those gatekeepers.

We need to expect and demand better of these respected institutions to better reflect their mission “to promote the science and art of medicine and the betterment of the public health.”

Ryan Radecki is an emergency physician and AHRQ Patient Safety and Quality Fellow.  He blogs at Emergency Medicine Literature of Note and can be found on Twitter @emlitofnote.

email

Comments are moderated before they are published. Please read the comment policy.

  • http://twitter.com/aztezcan Ayse Tezcan

    JAMA has published several articles with questionable merits recently. I am not sure if the quality of research or the the review process is going down or I am becoming more cynical as I get alder and wiser, I seem to notice more and more of these issues with many reputable journals.

  • trinu

    Unfortunately many in the medical research field seem to be under the mistaken impression that sample size alone is enough to determine the quality of a study. The two primary questions when assessing the quality of any study should be whether the variables analyzed by the study answer the question and whether the methodology was sound. All too often these criteria are not met, but the study is accepted as strong evidence anyways just because of sample size.

  • ninguem

    Only a small minority of physicians are AMA members.

    I fail to understand why people think the organization…….or its journals……deserve credibility.

    • http://www.facebook.com/shirie.leng Shirie Leng

      I agree with you. The AMA has failed to advocate for physicians in the area of cost containment and EMR as well.

  • Daniel W

    I completely agree. Thank you for being a voice to highlight this issue.

  • http://twitter.com/FerkhamPasha Ferkham pasha

    Informative article

  • meyati

    The problem is the bias of the researchers- and yes-JAMA- etc. should realize that there is bias.