Skip to content
  • About
  • Contact
  • Contribute
  • Book
  • Careers
  • Podcast
  • Recommended
  • Speaking
  • All
  • Physician
  • Practice
  • Policy
  • Finance
  • Conditions
  • .edu
  • Patient
  • Meds
  • Tech
  • Social
  • Video
    • All
    • Physician
    • Practice
    • Policy
    • Finance
    • Conditions
    • .edu
    • Patient
    • Meds
    • Tech
    • Social
    • Video
    • About
    • Contact
    • Contribute
    • Book
    • Careers
    • Podcast
    • Recommended
    • Speaking

Did the NEJM publish a bad study about checklists?

Josh Herigon, MPH
Education
March 30, 2014
41 Shares
Share
Tweet
Share

Recently, a study in the New England Journal of Medicine called into question the effectiveness of surgical checklists for preventing harm. Atul Gawande, one of the original researchers demonstrating the effectiveness of such checklists and author of a book on the subject, quickly wrote a rebuttal on the Incidental Economist. He writes, “I wish the Ontario study were better,” and I join him in that assessment, but want to take it a step further.

Gawande first criticizes the study for being underpowered. I had a hard time swallowing this argument given they looked at over 200,000 cases from 100 hospitals. I had to do the math. A quick calculation shows that given the rates of death in their sample, they only had about 40% power (we conventionally aim for a power of 80% or better.) Then I became curious about Gawande’s original study. They achieved better than 80% power with just over 7,500 cases. How is this possible?

The most important thing I keep in mind when I think about statistical significance, other than the importance of clinical significance, is that not only does it depend on the sample size, but also the baseline prevalence and the magnitude of the difference you are looking for. In Gawande’s original study, the baseline prevalence of death was 1.5%. This is substantially higher than the 0.7% in the Ontario study. When your baseline prevalence approaches the extremes (i.e. 0% or 50%) you have to pump up the sample size to achieve statistical significance.

So, Gawande’s study achieved adequate power because their baseline rate was higher and the difference they found was bigger. The Ontario study would have needed a little over twice as many cases to achieve 80% power.

This raises an important question: Why didn’t the Ontario study look at more cases?

The number of cases in a study is dictated by limitations in data collection. Studies are generally limited by the manpower they can afford to hire and the realistic time limitations of conducting a study. However, studies that use existing databases are usually not subject to these constraints. While creating queries to extract data is often tricky, once you have setup your extraction methodology it simply dumps the data into your study database. You can extend or contract the time period for data collection by simply changing the parameters of your query. Modern computing power means there are few limitations on the sizes of these study databases and the statistical methodologies we can employ. Simply put, the Ontario study (which relied on “administrative health data,” read: “existing data”) easily could have doubled the number of cases in their study.

Exactly how did they define their study group? As Gawande points out in his critique, the Ontario study relied on this bizarre 3-month window before and after checklist implementation at individual hospitals. Why 3 months? Why not 6 or 12 or 18? They even write in their methods: “We conducted sensitivity analyses using different periods for comparison.”

They never give the results of these sensitivity analyses or provide sound justification for the choice of a 3-month period. Three months not only keeps their power low, but it fails to account for secular trends. Maybe something like influenza was particularly bad in the post-checklist period, leading to more deaths despite effective checklist use. Maybe a new surgical technique or tool was introduced, like DaVinci robots, or many new, inexperienced surgeons were hired that increased mortality. In discussing their limitations, they address this:

Since surgical outcomes tend to improve over time, it is highly unlikely that confounding due to time-dependent factors prevented us from identifying a significant improvement after implementation of a surgical checklist.

I will leave it to you to decide if you think this is an adequate explanation. I’m not buying it.

Gawande concludes that this study reflects a failure of implementation of using checklists, rather than a failure of checklists themselves. I’m inclined to agree.

Ultimately, I don’t wonder why this study was published; bad studies are published all the time (hence the work of John Ioannidis). I wonder why this study was published in the New England Journal of Medicine. NEJM is supposed to be the gold standard for academic medical research. If they print it, you should be confident in the results and conclusions. Their editors and peer reviewers are supposed to be the best in the world. The Ontario study seems to be far below the standards I expect for NEJM.

I think their decision to accept the paper hinged on the fact that this was a large study that showed a negative finding on a subject that has been particularly hot over the past few years. Nobody seemed to care that this was not a particularly well-conducted study; this is the sadness that plagues the medical research community. Be a critical reader.

 Josh Herigon is a medical student who blogs at mediio.

Prev

Understanding the uproar over Zohydro ER

March 30, 2014 Kevin 5
…
Next

Hospital medicine doctors are key to improving patient satisfaction

March 31, 2014 Kevin 0
…

Tagged as: Hospital-Based Medicine, Surgery

Post navigation

< Previous Post
Understanding the uproar over Zohydro ER
Next Post >
Hospital medicine doctors are key to improving patient satisfaction

More by Josh Herigon, MPH

  • a desk with keyboard and ipad with the kevinmd logo

    The threat of technology to proper patient care

    Josh Herigon, MPH
  • a desk with keyboard and ipad with the kevinmd logo

    How social media will merge with electronic medical records

    Josh Herigon, MPH
  • a desk with keyboard and ipad with the kevinmd logo

    Why medical education needs to evolve away from memorization

    Josh Herigon, MPH

More in Education

  • How Tratak yoga reshaped my USMLE Step 2 prep

    Dr. Nikita Mehdiratta
  • The role of income in medical school acceptance

    Carter Do
  • Balancing tension and kindness in medical education

    Chloe N. L. Lee, MD, MPH
  • I want to be a doctor who can provide care for women: What states must I rule out for my medical education?

    Nandini Erodula
  • A chance encounter in Chicago: lessons in compassionate medicine

    Emily S. Hagen
  • Business education’s role in preventing physician practice decline

    Curtis G. Graham, MD
  • Most Popular

  • Past Week

    • Challenging the diagnosis: dehydration or bias?

      Sydney Lou Bonnick, MD | Physician
    • A teenager’s perspective: the pressing need for mental health days in schools

      Ruhi Saldanha | Conditions
    • Understanding reproductive rights: complex considerations

      Anonymous | Physician
    • COVID-19 unleashed an ongoing crisis of delirium in hospitals

      Christina Reppas-Rindlisbacher, MD, Nathan Stall, MD, and Paula Rochon, MD | Conditions
    • Air quality alert: Reducing our carbon footprint in health care

      Shreya Aggarwal, MD | Conditions
    • When physicians are disrespected [PODCAST]

      The Podcast by KevinMD | Podcast
  • Past 6 Months

    • Medical gaslighting: a growing challenge in today’s medical landscape

      Tami Burdick | Conditions
    • I want to be a doctor who can provide care for women: What states must I rule out for my medical education?

      Nandini Erodula | Education
    • Balancing opioid medication in chronic pain

      L. Joseph Parker, MD | Conditions
    • Unveiling excessive medical billing and greed

      Amol Saxena, DPM, MPH | Policy
    • The erosion of patient care

      Laura de la Torre, MD | Physician
    • I’m a doctor, and I almost died during childbirth

      Bayo Curry-Winchell, MD | Physician
  • Recent Posts

    • When physicians are disrespected [PODCAST]

      The Podcast by KevinMD | Podcast
    • Addressing dual diagnosis needs in addiction treatment

      Susan Hertz Berrick, EdD | Conditions
    • The essence of health narratives, including poetry

      Arthur Lazarus, MD, MBA | Physician
    • Discover the power of patience

      Diane W. Shannon, MD, MPH | Physician
    • Doctors rediscover joy in practicing medicine, on their own terms

      Kim Downey, PT | Physician
    • End-of-life ethics and antibiotic use [PODCAST]

      The Podcast by KevinMD | Podcast

Subscribe to KevinMD and never miss a story!

Get free updates delivered free to your inbox.


Find jobs at
Careers by KevinMD.com

Search thousands of physician, PA, NP, and CRNA jobs now.

Learn more

View 3 Comments >

Founded in 2004 by Kevin Pho, MD, KevinMD.com is the web’s leading platform where physicians, advanced practitioners, nurses, medical students, and patients share their insight and tell their stories.

Social

  • Like on Facebook
  • Follow on Twitter
  • Connect on Linkedin
  • Subscribe on Youtube
  • Instagram

CME Spotlights

From MedPage Today

Latest News

  • Novavax's Updated COVID Shot Authorized by FDA
  • SBRT Noninferior to Conventional RT for Intermediate-Risk Prostate Cancer
  • Mixed Bag for Early Metformin in Gestational Diabetes
  • FDA Advisors to Consider DFMO Maintenance for High-Risk Neuroblastoma in Kids
  • Adding Tirzepatide to Basal Insulin Cuts HbA1c in Poorly Controlled T2D

Meeting Coverage

  • SBRT Noninferior to Conventional RT for Intermediate-Risk Prostate Cancer
  • Mixed Bag for Early Metformin in Gestational Diabetes
  • Adding Tirzepatide to Basal Insulin Cuts HbA1c in Poorly Controlled T2D
  • Low Relapse Rates With Twice-Yearly Schizophrenia Treatment
  • Menopause Can Negatively Affect Women's Careers
  • Most Popular

  • Past Week

    • Challenging the diagnosis: dehydration or bias?

      Sydney Lou Bonnick, MD | Physician
    • A teenager’s perspective: the pressing need for mental health days in schools

      Ruhi Saldanha | Conditions
    • Understanding reproductive rights: complex considerations

      Anonymous | Physician
    • COVID-19 unleashed an ongoing crisis of delirium in hospitals

      Christina Reppas-Rindlisbacher, MD, Nathan Stall, MD, and Paula Rochon, MD | Conditions
    • Air quality alert: Reducing our carbon footprint in health care

      Shreya Aggarwal, MD | Conditions
    • When physicians are disrespected [PODCAST]

      The Podcast by KevinMD | Podcast
  • Past 6 Months

    • Medical gaslighting: a growing challenge in today’s medical landscape

      Tami Burdick | Conditions
    • I want to be a doctor who can provide care for women: What states must I rule out for my medical education?

      Nandini Erodula | Education
    • Balancing opioid medication in chronic pain

      L. Joseph Parker, MD | Conditions
    • Unveiling excessive medical billing and greed

      Amol Saxena, DPM, MPH | Policy
    • The erosion of patient care

      Laura de la Torre, MD | Physician
    • I’m a doctor, and I almost died during childbirth

      Bayo Curry-Winchell, MD | Physician
  • Recent Posts

    • When physicians are disrespected [PODCAST]

      The Podcast by KevinMD | Podcast
    • Addressing dual diagnosis needs in addiction treatment

      Susan Hertz Berrick, EdD | Conditions
    • The essence of health narratives, including poetry

      Arthur Lazarus, MD, MBA | Physician
    • Discover the power of patience

      Diane W. Shannon, MD, MPH | Physician
    • Doctors rediscover joy in practicing medicine, on their own terms

      Kim Downey, PT | Physician
    • End-of-life ethics and antibiotic use [PODCAST]

      The Podcast by KevinMD | Podcast

MedPage Today Professional

An Everyday Health Property Medpage Today
  • Terms of Use | Disclaimer
  • Privacy Policy
  • DMCA Policy
All Content © KevinMD, LLC
Site by Outthink Group

Did the NEJM publish a bad study about checklists?
3 comments

Comments are moderated before they are published. Please read the comment policy.

Loading Comments...