Bias and error are rampant in medical literature

A heretic essay in JAMA by Vinay Prasad (Northwestern of Chicago), Adam Cifu (U. of Chicago) and John Ioannidis (Stanford) should be required reading for every medical student, resident, and to pass any board certification exam in any specialty … in my humble opinion.

John Ioannidis became one of my personal heroes with the publication of his great paper in PLoS Medicine, “Why Most Published Research Findings Are False,” and its followup Atlantic magazine profile (“Lies, Damn Lies, and Medical Science“). Rather than being nihilistic, Ioannidis’s essay is an elegant statistical proof that makes the simple point that bias and error are inevitable and rampant in the published body of medical literature.

Here they put away their statistical software and ask us, doctor-to-doctor, to simply consider “When To Abandon Ship,” i.e., to identify and permanently shelve those medical practices that though familiar, are in reality failed or useless to patients (but not to doctors’, hospitals’, and industry’s balance sheets). True to previous form, they eschew formal “journalspeak” and use their same refreshingly unadorned and fearless prose:

How many established standards of medical care are wrong? It is not known. Medical practice has evolved out of centuries of theorizing, personal experiences, bits of evidence, expert consensus, and diverse conflicts and biases. Rigorous questioning of long-established practices is difficult. There are thousands of clinical trials, but most deal with trivialities or efforts to buttress the sales of specific products. Given this conundrum, it is possible that some entire medical subspecialties are based on little evidence. Their disappearance probably would not harm patients and might help salvage derailed health budgets. However, it is unlikely that specialists would support trials testing practices that constitute their main source of income. Instead, the research community performs studies of modest incremental value without even knowing whether the basic standards of care are appropriate.

They use examples of stenting for stable coronary artery disease (which comprised 85% of stenting procedures until the aptly-named COURAGE trial showed it didn’t help), vertebroplasty, and estrogen replacement therapy, among others. A highly-cited previous article by these authors suggested that when followup randomized trials are conducted on a major accepted therapy, they refute standard practice about half the time.

Authors call for greater barriers to implementation of new devices, drugs, and interventional procedures, and the requirement of large randomized trials to establish benefits before wide implementation of new practices. (Industry-supported advocacy groups figured out long ago how to counter these efforts, though: 1. Organize patients with the condition. 2. Make the public relations case to politicians and the media that regulation is stifling innovation that could help people who are suffering. Voila! Restrictions on funding for unproven medical technologies disappear.)

It’s a trip through the looking glass of our profession’s solipsism: the endemic self-interest, self-protection and self-promotion that has guided the centuries-long evolution of medical practice and pervades it today — including all our professional societies, high-minded journals, jealously guarded reputations, and industry relationships. None of us wants outsiders (other specialists, policymakers, patients) to look too closely or skeptically at what we do – and the essential prerequisite to preventing that is to not look too closely, ourselves. If something we’re used to doing doesn’t seem to work well, or we just don’t know, there’s always tradition, habit, or patient expectations to fall back on.

But are these barriers to self-examination just rationalizations, proxies for what is in reality a desire to maintain the veil of secrecy that we know will help preserve our incomes, professional turf, and what remains of our respect from the public? The final irony is that the more energetically the veil is maintained, the less respect we earn or deserve as society’s last trusted protectors of our patients’ interests in health.

Matthew Hoffman is a fellow in pulmonary and critical care who blogs at

Submit a guest post and be heard on social media’s leading physician voice.

Comments are moderated before they are published. Please read the comment policy.

  • ER Jedi


  • Anonymous

    Well said!  I have thought for a long time that the medical profession has needed to start asking questions and really regulating itself.  Until physicians take a good, hard look at what their profession has become, we will have no change or reform!

  • TheresaWillett MDPhD

    I would love to know the most effective ways to truly teach critical thinking and assessment of published trials to trainees. The usual journal club format is so dry, and I fear that the principles of understanding limitations and bias can get lost in the statistics.

  • Bruce Ramshaw

    This is an important issue, however the interpretation by Ioannidis and others is incomplete.  If we were to apply the same thinking to the example of antibiotic use before abdominal surgery, we would abandon that practice completely (after 10 years of that mandate and near 100% compliance, there has been no reduction in surgical site infection).  A more complete understanding of this issue is possible with the understanding of complex adaptive systems science.  People will group into clusters.  When we identify which patient groups would benefit from preoperative antibiotics and which will not, then we can have a much more value-based understanding of medical treatments (large prospective randomized studies are inadequate to determine this- we will need to apply complex systems science and utilize principles of clinical quality improvement to determine how to improve value for our medical therapies).  There is an excellent TED talk by Malcolm Gladwell on YouTube (search- Malcolm Gladwell and spaghetti sauce) that describes how this understanding was applied to the consumer food industry a few decades ago.  We desperately need to apply this kind of understanding to medicine.

  • Anonymous

    Thank you very much!  I have personal knowledge of this “bias and error” from an ORIF surgery I had 6 years ago.  Doctors love this surgery, but it has some very severe side effects, nerve damage, carpal tunnel, loss of grip etc.  Is this the best way to repair a broken arm?  Maybe not, at least in my case.  I wish my surgeon had explained all of this to me.  If he had just told me where the incision was going to be, I might have had some warning.  If I break my arm again, I will reset it, just like last time, apply a splint and leave it alone.  If the bone comes through the skin, I might request some antibiotics…  With the severity of the adverse events and the necessity for more than one surgery, it doesn’t seem worth it TO ME!  Add in the crna who is also trying to justify their six figure salary by giving all and sundry a sedation drug called Versed, whether you need it or not, you have the perfect storm of medical care.  Versed creates all kinds of side effects all by itself, and yet, I constantly read where this drug is “safe.”   It is NOT safe for a lot of patients.  Where are the unbiased studies for this type of surgery?  Why is nobody looking into the problems with Versed?   Seems to me that bias and error IS rampant in medical literature. I seem to find articles which gloss over any bad outcomes and focus on only those who have a good(?) outcome.  Then we have all these medical device people extolling the virtues of their product, along with Roche pushing their drugs.   I have seen it while researching what the Sam Hill happened to me and why.  My surgeon and crna are in complete denial over the whole thing regardless of the poor outcome.  It’s amazing, maddening, infuriating etc.  The final touch?  The “patient relations” nurse claimed that it was MY attitude which caused the surgical problems and the extreme reaction to Versed.  So it’s all my fault because I was upset over the outcome.  Denial in its most extreme form.  Who taught this woman this hocus pocus?

  • maryhirzel

     Strange this piece is getting so little attention, when it’s probably one of the most important I’ve seen on KevinMD.

    The millions of children, worldwide, injured and then discarded by the medical profession, as “acceptable collateral damage” of the vaccination program, and their parents, abused as “anti-vaxers” and enemies of the people, need more physicians critically examining the studies ( touted as “proof” vaccines never did anything but good.

    You have no idea.

    For us, and our children, the “last trusted protectors” died long ago.

  • jamesp

    Here Here!!!

    I have been railing for years how so many studies proclaim “Risk factor X may cause problem Y.”

    May cause?   MAY?     What good does that do anyone???  How much money and effort is wasted in such speculation?  I want to tell those authors, “come back when you’re sure!”

Most Popular