Skip to content
  • About
  • Contact
  • Contribute
  • Book
  • Careers
  • Podcast
  • Recommended
  • Speaking
  • All
  • Physician
  • Practice
  • Policy
  • Finance
  • Conditions
  • .edu
  • Patient
  • Meds
  • Tech
  • Social
  • Video
    • All
    • Physician
    • Practice
    • Policy
    • Finance
    • Conditions
    • .edu
    • Patient
    • Meds
    • Tech
    • Social
    • Video
    • About
    • Contact
    • Contribute
    • Book
    • Careers
    • Podcast
    • Recommended
    • Speaking

We need more science in hospital quality measures

Peter Pronovost, MD, PhD
Policy
January 6, 2017
291 Shares
Share
Tweet
Share

If you understand statistics and possess the intestinal fortitude to examine a ranking methodology, you will recognize that it involves ingredients that have to be recombined, repackaged and renamed. It’s messy, like sausage-making.

This is not to say that the end product — hospital rankings — are distasteful. Patients deserve valid, transparent and timely information about quality of care so they can make informed decisions about whether and where to receive care. Ratings organizations like U.S. News & World Report work hard to create valid, unbiased hospital rankings out of imperfect data and measures. But the recipe needs to be right.

Patient safety indicators, a set of measures that reflect the incidence of various kinds of harm to hospital patients, is one ingredient I believe should be left out. These data are derived not from clinical records but from administrative codes in the bills sent to the federal Centers for Medicare and Medicaid Services, or CMS. Despite being broadly used in hospital ranking programs and pay-for-quality programs, patient safety indicators are notoriously inaccurate: They miss many harms while reporting false positives.

In a peer-reviewed paper published this spring in Medical Care, Johns Hopkins colleagues and I concluded that of 21 patient safety indicators, none can be considered scientifically valid.

Yet U.S. News wrote in a recent blog post that the inaccuracy of these measures might not pose such a vexing problem when it comes to comparing hospitals. If the frequency and degree of inaccuracies is similar across hospitals, according to this argument, then patient safety indicators can show how hospitals stack up against one another. If coding accuracy between hospitals is significantly different, however, that raises the question of whether these data should be used at all.

Recent research, as well as my hospital’s own experiences in improving our coding, would suggest that we should not assume that hospitals’ coding practices are relatively uniform. Coding accuracy, coding practices and patient and hospital characteristics can skew many different kinds of data sent to CMS.

A 2014 Cleveland Clinic study, for instance, found that differences in coding of severe pneumonia cases could result in more than 28 percent of hospitals being assigned the wrong mortality rating by CMS. In another study, published this year, researchers reported that in small rural hospitals with no stroke unit or team, diagnosis codes matched the clinical record in only 60% of ischemic stroke cases, while in large metropolitan hospitals with a stroke unit or team, the codes matched nearly 97% of the time. These variations may affect comparisons between hospitals but also alter reimbursement for stroke patients, the authors write.

Finally, hospitals deemed the highest quality by measures such as accreditation and better process and outcome performance are penalized more than five times as frequently for hospital-acquired conditions by CMS as hospitals scoring the worst. The researchers suggested that high-performing hospitals may simply look harder for adverse outcomes, and therefore find them more often. We experienced this first hand at The Johns Hopkins Hospital: After implementing a best practice for routine ultrasound screening for blood clots, the number of clots that we found increased tenfold.

I have no reason to believe that patient safety indicator coding is somehow an exception to this unevenness. At Johns Hopkins, we have reduced by 75% the number of patient safety indicator incidents that we report, saving millions in unimposed penalties and improving our public profile. About 10% of the improvement resulted from changes in clinical care. The other 90% resulted from documentation and coding that was more thorough and accurate. Other hospitals may not have the resources to take on this complex effort, or they may be unaware that their coding accuracy is a problem.

To their credit, U.S. News editors announced in late June that they were reducing the weight of the patient safety indicators-based patient safety score from 10% of a hospital’s overall score to 5%. They also removed a particularly problematic patient safety indicator that tabulates the incidence of pressure ulcers.

We must not be satisfied with measures that only give us relative performance — how hospitals compare to one another. We need to have absolute measures of performance: How often are patients harmed? How often is a desired outcome achieved?

Certainly, it’s good to know which hospitals or surgeons have better complication rates than others for hip replacements. But as a patient, don’t you also want to know the absolute complication rate so you can decide whether to have surgery in the first place? If you’re planning a prostatectomy, don’t you want to know how frequently your surgeon’s patients suffer from impotence afterwards? If you’re a hospital leader, don’t you want to know how your organization is progressing toward eliminating infections?

If health care used valid and reliable measures and audited the data hospitals provide — just as we audit financial data — hospitals would not have to get into the so-called coding game. And physicians might engage in quality improvement rather than be put off by the drive to look good. In the end, patients deserve quality measures that are more science and less sausage-making.

Peter Pronovost is an anesthesiologist and director, Armstrong Institute for Patient Safety and Quality.  He blogs at Voices for Safer Care. This article originally appeared in U.S. News & World Report. 

Image credit: Shutterstock.com

Prev

One way to train gritty doctors: Don't allow them to quit

January 6, 2017 Kevin 2
…
Next

Don't engage in fishbowl emergency medicine

January 6, 2017 Kevin 1
…

Tagged as: Hospital-Based Medicine

Post navigation

< Previous Post
One way to train gritty doctors: Don't allow them to quit
Next Post >
Don't engage in fishbowl emergency medicine

More by Peter Pronovost, MD, PhD

  • Explore the behavioral factors behind antibiotic misuse

    Peter Pronovost, MD, PhD
  • Revamp health regulations to reduce cost and improve patient safety

    Peter Pronovost, MD, PhD
  • How peer-to-peer review helps hospitals

    Peter Pronovost, MD, PhD

Related Posts

  • Quality measures have gotten ahead of the science of quality measurement

    Peter Ubel, MD
  • When quality measures interfere with good care

    Michael McCutchen, MD, MBA
  • Don’t judge when trainees use dating apps in the hospital

    Austin Perlmutter, MD
  • Why quality reports for hospitals and doctors are interesting but flawed

    Mark Kelley, MD
  • Is social media a friend or foe of science?

    Michael Joyce, MD
  • How hospitals prepare for hurricanes

    Daniel B. Hess, PhD

More in Policy

  • Pediatricians grapple with guns in America, from Band-Aids to bullets

    Tasia Isbell, MD, MPH
  • Health care wins, losses, and lessons

    Robert Pearl, MD
  • Maximizing care amidst provider shortages: the power of measurement-based care

    Tom Zaubler, MD
  • Unveiling excessive medical billing and greed

    Amol Saxena, DPM, MPH
  • Chronic health issues and homelessness

    Michele Luckenbaugh
  • The impact of certificate of need laws on rural health care

    Jaimie Cavanaugh, JD and Daryl James
  • Most Popular

  • Past Week

    • Reigniting after burnout: 3 physician stories

      Kim Downey, PT | Physician
    • I’m a doctor, and I almost died during childbirth

      Bayo Curry-Winchell, MD | Physician
    • I’m tired of being a distracted doctor

      Shiv Rao, MD | Tech
    • Inside the grueling life of a surgery intern

      Randall S. Fong, MD | Physician
    • Ketamine for mental health conditions: What every primary care physician needs to know

      Carlene MacMillan, MD & L. Alison McInnes, MD | Meds
    • Using the language of art to create work-life balance

      Sarah Samaan, MD | Physician
  • Past 6 Months

    • Medical gaslighting: a growing challenge in today’s medical landscape

      Tami Burdick | Conditions
    • Balancing opioid medication in chronic pain

      L. Joseph Parker, MD | Conditions
    • I want to be a doctor who can provide care for women: What states must I rule out for my medical education?

      Nandini Erodula | Education
    • Mourning the silent epidemic: the physician suicide crisis and suggestions for change

      Amna Shabbir, MD | Physician
    • Reigniting after burnout: 3 physician stories

      Kim Downey, PT | Physician
    • 1 in 4 attempt suicide: the persecution of autistic physicians

      Patricia Celan, MD | Physician
  • Recent Posts

    • Using the language of art to create work-life balance

      Sarah Samaan, MD | Physician
    • Levamisole is good for your dog, but bad for your cocaine

      Robert Killeen, MD | Meds
    • Physician autonomy and patient interactions in corporate health care

      Michele Luckenbaugh | Conditions
    • PSA screening: What you need to know [PODCAST]

      The Podcast by KevinMD | Podcast
    • Vague criteria can lead to misdiagnosis and prison

      L. Joseph Parker, MD | Conditions
    • U.S. maternal mortality crisis: a deep dive

      Alan Lindemann, MD | Conditions

Subscribe to KevinMD and never miss a story!

Get free updates delivered free to your inbox.


Find jobs at
Careers by KevinMD.com

Search thousands of physician, PA, NP, and CRNA jobs now.

Learn more

View 7 Comments >

Founded in 2004 by Kevin Pho, MD, KevinMD.com is the web’s leading platform where physicians, advanced practitioners, nurses, medical students, and patients share their insight and tell their stories.

Social

  • Like on Facebook
  • Follow on Twitter
  • Connect on Linkedin
  • Subscribe on Youtube
  • Instagram

CME Spotlights

From MedPage Today

Latest News

  • Black Patients at Higher Risk for Physical Restraint in the ED
  • Compassion Is a Learnable Skill
  • When a Video Visit Leads to Real Connection
  • Smaller Lesions, Better Baseline Visual Acuity Bode Well for Submacular Hemorrhage
  • COVID Conspiracies Return; Norovirus on the Hiking Trail; 2nd Pig Heart Transplanted

Meeting Coverage

  • Loneliness Needs to Be Treated Like Any Other Health Condition, Researcher Suggests
  • Stopping Medical Misinformation Requires Early Detection
  • AI Has an Image Problem in Healthcare, Expert Says
  • Want Better Health Outcomes? Check Out What Other Countries Do
  • ERS Roundup: Cell Transplant Boosts Lung Function in COPD Patients
  • Most Popular

  • Past Week

    • Reigniting after burnout: 3 physician stories

      Kim Downey, PT | Physician
    • I’m a doctor, and I almost died during childbirth

      Bayo Curry-Winchell, MD | Physician
    • I’m tired of being a distracted doctor

      Shiv Rao, MD | Tech
    • Inside the grueling life of a surgery intern

      Randall S. Fong, MD | Physician
    • Ketamine for mental health conditions: What every primary care physician needs to know

      Carlene MacMillan, MD & L. Alison McInnes, MD | Meds
    • Using the language of art to create work-life balance

      Sarah Samaan, MD | Physician
  • Past 6 Months

    • Medical gaslighting: a growing challenge in today’s medical landscape

      Tami Burdick | Conditions
    • Balancing opioid medication in chronic pain

      L. Joseph Parker, MD | Conditions
    • I want to be a doctor who can provide care for women: What states must I rule out for my medical education?

      Nandini Erodula | Education
    • Mourning the silent epidemic: the physician suicide crisis and suggestions for change

      Amna Shabbir, MD | Physician
    • Reigniting after burnout: 3 physician stories

      Kim Downey, PT | Physician
    • 1 in 4 attempt suicide: the persecution of autistic physicians

      Patricia Celan, MD | Physician
  • Recent Posts

    • Using the language of art to create work-life balance

      Sarah Samaan, MD | Physician
    • Levamisole is good for your dog, but bad for your cocaine

      Robert Killeen, MD | Meds
    • Physician autonomy and patient interactions in corporate health care

      Michele Luckenbaugh | Conditions
    • PSA screening: What you need to know [PODCAST]

      The Podcast by KevinMD | Podcast
    • Vague criteria can lead to misdiagnosis and prison

      L. Joseph Parker, MD | Conditions
    • U.S. maternal mortality crisis: a deep dive

      Alan Lindemann, MD | Conditions

MedPage Today Professional

An Everyday Health Property Medpage Today
  • Terms of Use | Disclaimer
  • Privacy Policy
  • DMCA Policy
All Content © KevinMD, LLC
Site by Outthink Group

We need more science in hospital quality measures
7 comments

Comments are moderated before they are published. Please read the comment policy.

Loading Comments...