Bias and error are rampant in medical literature


A heretic essay in JAMA by Vinay Prasad (Northwestern of Chicago), Adam Cifu (U. of Chicago) and John Ioannidis (Stanford) should be required reading for every medical student, resident, and to pass any board certification exam in any specialty … in my humble opinion.

John Ioannidis became one of my personal heroes with the publication of his great paper in PLoS Medicine, “Why Most Published Research Findings Are False,” and its followup Atlantic magazine profile (“Lies, Damn Lies, and Medical Science“). Rather than being nihilistic, Ioannidis’s essay is an elegant statistical proof that makes the simple point that bias and error are inevitable and rampant in the published body of medical literature.

Here they put away their statistical software and ask us, doctor-to-doctor, to simply consider “When To Abandon Ship,” i.e., to identify and permanently shelve those medical practices that though familiar, are in reality failed or useless to patients (but not to doctors’, hospitals’, and industry’s balance sheets). True to previous form, they eschew formal “journalspeak” and use their same refreshingly unadorned and fearless prose:

How many established standards of medical care are wrong? It is not known. Medical practice has evolved out of centuries of theorizing, personal experiences, bits of evidence, expert consensus, and diverse conflicts and biases. Rigorous questioning of long-established practices is difficult. There are thousands of clinical trials, but most deal with trivialities or efforts to buttress the sales of specific products. Given this conundrum, it is possible that some entire medical subspecialties are based on little evidence. Their disappearance probably would not harm patients and might help salvage derailed health budgets. However, it is unlikely that specialists would support trials testing practices that constitute their main source of income. Instead, the research community performs studies of modest incremental value without even knowing whether the basic standards of care are appropriate.

They use examples of stenting for stable coronary artery disease (which comprised 85% of stenting procedures until the aptly-named COURAGE trial showed it didn’t help), vertebroplasty, and estrogen replacement therapy, among others. A highly-cited previous article by these authors suggested that when followup randomized trials are conducted on a major accepted therapy, they refute standard practice about half the time.

Authors call for greater barriers to implementation of new devices, drugs, and interventional procedures, and the requirement of large randomized trials to establish benefits before wide implementation of new practices. (Industry-supported advocacy groups figured out long ago how to counter these efforts, though: 1. Organize patients with the condition. 2. Make the public relations case to politicians and the media that regulation is stifling innovation that could help people who are suffering. Voila! Restrictions on funding for unproven medical technologies disappear.)

It’s a trip through the looking glass of our profession’s solipsism: the endemic self-interest, self-protection and self-promotion that has guided the centuries-long evolution of medical practice and pervades it today — including all our professional societies, high-minded journals, jealously guarded reputations, and industry relationships. None of us wants outsiders (other specialists, policymakers, patients) to look too closely or skeptically at what we do — and the essential prerequisite to preventing that is to not look too closely, ourselves. If something we’re used to doing doesn’t seem to work well, or we just don’t know, there’s always tradition, habit, or patient expectations to fall back on.

But are these barriers to self-examination just rationalizations, proxies for what is in reality a desire to maintain the veil of secrecy that we know will help preserve our incomes, professional turf, and what remains of our respect from the public? The final irony is that the more energetically the veil is maintained, the less respect we earn or deserve as society’s last trusted protectors of our patients’ interests in health.

Matthew Hoffman is a fellow in pulmonary and critical care who blogs at

Submit a guest post and be heard on social media’s leading physician voice.


View 7 Comments >

Most Popular

✓ Join 150,000+ subscribers
✓ Get KevinMD's most popular stories