Digging deeper into the Commonwealth Fund health rankings

The Commonwealth Fund just published its fourth Mirror, Mirror on the Wall study comparing the U.S. health care system with other countries, and as in all previous studies, we ranked as the absolutely worst health care system in the developed world, bar none.

Yikes.

The Commonwealth Fund studied many health care domains, and we didn’t rank in first place for anything. The best we managed to do is place a lackluster third in the subcategory of effective care. The United Kingdom, on the other hand, with its socialized medicine system, took first place in almost every category, and the Swiss came in second.  That’s almost enough to drive a proud American into deep despair, and as the report bluntly states, “The claim that the United States has ‘the best health care system in the world’ is clearly not true.” To add insult to injury, ours is also clearly the most expensive system in the world, and no, that doesn’t count as being #1 for something.

Digging deeper into the Commonwealth Fund health rankings

The authors of the Commonwealth Fund report are gracefully doing their best to cheer us up and give us hope, by pointing out that “[s]ince the data in this study were collected, the U.S. has made significant strides adopting health information technology and undertaking payment and delivery system reforms spurred by the Affordable Care Act.” It may be okay to hope that the next Mirror, Mirror report will show us moving up a couple of notches, instead of continuing to be the laughing stock of all developed nations. So how do we go about improving our scores? Adopting health IT is obviously the first thing, and then we need to “encourage more affordable access and more efficient organization and delivery of health care, and allow investment in preventive and population health measures.” Sounds like a plan.

Except, one thing in that picture looks very peculiar. The UK, the poster child of frugal and immaculate perfection, scored almost as bad as we did in the only domain that can be regarded as an outcome: health. The bon vivant French people, with the worst access to care and horrific patient-centeredness, seem to enjoy the healthiest lives of all (and Jefferson is finally vindicated). Looking further, it seems that Sweden, where care is of abysmal quality, but most equitable and efficient, came in second in healthy lives and third overall. Can something even be simultaneously of low quality and very efficient? Can a country have dangerous, ineffective care, like Norway, and still be ranked comfortably in the middle of the pack? For inquiring minds of the confused variety, the study provides more granular data points to peruse, so let’s dive in.

Over at the Incidental Economist, Dr. Aaron Carroll is warning us to stay away from “Zombie arguments defending the U.S. health care system.” Fair enough. Let’s not worry about the U.S. system, or any system, and let’s even hold back on questioning the much too flawless results of this or that system. Let’s just look at the data. There are four major domains in the study: quality, access, efficiency, equity and healthy lives. Without splitting hairs, healthy lives can be considered an outcome of efforts in all other domains, but of course, it shouldn’t be, and the study authors acknowledge that the health care system is “just one of many factors, including social and economic well-being, that influence the health of a nation.” Completely agree. In which case, it is unclear to me why healthy lives measures are factored into the rankings of health care systems, straight up with no weighting or adjustments.

Let’s dig in a little deeper.  The quality domain is divided into four subdomains: effective care, safe care, coordinated care and patient-centeredness. Without debating this particular definition of quality, let’s look at how effectiveness is measured on two axes, preventive care and chronic care, each one assessed based on a series of data points. So for example, the first three prevention measures are:

  1. the ease of printing out lists of patients due for preventive care
  2. patients who received preventive care reminders
  3. patients routinely sent computerized reminders for preventive and routine care

I would call this triple dipping, because the only measure that actually counts here is whether patients received reminders or not, and how they responded, which was not measured at all. Whether it is easy to “print out” lists, or whether people are bombarded with computer calls that nobody picks up the phone for, is irrelevant.

The U.S. was ranked 3rd for patients receiving reminders and 7th for the other two useless measures. The UK ranked 1st for the useless measures and 5th for the mildly pertinent measure. For the remaining preventive measures, dealing with lifestyle advice provided by physicians to their patients, the U.S. ranked 1st and 2nd overall. To assess effectiveness, I would have expected perhaps a ratio of reminders sent, to reminders acted upon by patients, or at least reminders received, instead of an average score for those two, plus some strange measure about printing lists to paper.

The chronic care portion of the effectiveness subdomain illustrates yet another logical flaw in the study. Similar to the preventive care measures, here too the U.S. scores decently on actual chronic care activities, and poorly on ease of producing lists. But the bigger issue is the one measure evaluating cost barriers to adherence, and as expected the U.S. scored poorly on affordability, which is what this measure is all about. It may be fine to blast the U.S. system for being expensive, but to say that we are paying too much for a bad system, while assessing badness based on the system being expensive, is circular logic that should have no place in serious scientific conversation.

Another, rather perplexing methodology flaw, is the many repetitive questions with conflicting answers that were nevertheless dutifully added as is to the averages. Questions of this type are routinely included in surveys to validate answers, but are not meant to be independent data points. For example, how is it possible to have a bad score on primary care docs receiving discharge summaries in general, coupled with a good score on receiving discharge summaries in a timely manner? You can keep on digging if you are so inclined, but my general impression is that the data in the analyzed surveys are neither sufficient, nor pertinent enough to allow for meaningful rankings of national health care systems. Let’s also note that all data is derived from surveys. Even if surveys would be classified as objective observation, which they are not, how can we infer causality from a narrow observational study?

And here is my biggest problem with these rankings and the subsequent conclusions drawn by the Commonwealth Fund, i.e. more computerization, more preventive care, more population management, or in other words more corporate and data driven health care. The study authors are basing their findings on a subset of indicators subjectively selected by the survey designers. What about the heaps and troves of other indicators that may also be pertinent to these findings? For instance, the average primary care panel size in the UK is a little over half the average panel in the U.S., and most primary care is delivered in small private practice, by primary care physicians who are better paid than UK specialists. Are those things pertinent to the UK stellar performance on all study domains? I don’t know, and neither does anybody else until a proper study is conducted.

Looking under the lamp post for lost keys is not a scientific method of inquiry, and when we can’t find said keys, it is not proper to blame the low wattage of the lamp. There is nothing in those surveys supporting the conclusions and recommendations put forward by this report, other than faith and preconceived opinions which were neither validated nor disproved by survey responders. There is no indication that the U.S. is an outlier in health information technology, preventive care or population management, and there is zero indication that these factors are the most salient factors in the performance of a health care system.

Does the fact the U.S. is the only country in this cohort where poor people are segregated away into special insurance plans that pay doctors and hospitals below cost, have anything to do with our poor numbers for equity and access? Does the for-profit nature of our system affect the exorbitance of our costs and hence all study domains? Are these things perhaps a tad more important than the ease of generating and printing out lists of patients? We may never know …

It is proper to observe, as the Commonwealth study does, that all other countries have universal health care, while the U.S. does not. It may even be logical to assume that such a huge systemic difference must in some ways adversely affect our outcomes. But it is nothing short of perplexing to conclude that the remedy consists of mixing a tiny bit of our overpriced (and yes, best in the world) medicine, with lots of corporate run analytic dashboards, followed by universal administration of this homeopathic concoction to innocent people.

Margalit Gur-Arie is founder, BizMed. She blogs at On Healthcare Technology.

Comments are moderated before they are published. Please read the comment policy.

  • John C. Key MD

    No one should get excited about the Commonwealth or any other ranking system…there’s just too much disparity in the parameters. We all know that you can construct the matrix to yield any result that you wish to show. The agenda of the rankers is more in disrepute than the order of the ranked.

    • http://onhealthtech.blogspot.com Margalit Gur-Arie

      That is true. However these reports make their way into the mainstream media in the form of headlines only, or as citations from a respected non-profit foundation, to create a sense of fear, uncertainty and doubt in the public, and to justify the urgency of taking some sort of action, any action, since anything is better than the current “broken” system.
      I also suspect that a significant number of well meaning “experts”, who should know better, are just taking the assertions of this series of reports at face value because the Commonwealth Fund is indeed very well respected, and people rarely feel the need to double check anything it publishes.

  • QQQ

    “The UK, the poster child of frugal and immaculate perfection,”

    I guess the The Commonwealth Fund didn’t want people to see some of the other issues the U.K. had in this past!

    ———————————————————————————————————–

    “UK healthcare on brink of collapse – NHS regulator”

    http://rt.com/news/uk-nhs-health-crisis-049/
    ————————————————————————————————————

    “Britain’s Health Care System: Its Problems Continue”

    http://news.investors.com/ibd-editorials/030613-647023-british-health-care-workers-wont-recommend-their-workplaces.htm
    ————————————————————————————————————

    “NHS ‘can’t cope’ with multi-disease patients”

    http://www.bbc.com/news/health-18007951

    ————————————————————————————————————-

    “Public Health Care in UK Faces Major Problems”

    http://stanfordreview.org/article/public-healthcare-in-uk-faces-major-problems/

    ————————————————————————————————————-

    “British National Health Service in Trouble?”

    http://www.outsidethebeltway.com/british-national-health-service-in-trouble/

    ————————————————————————————————————–

  • QQQ

    The European Example: Rethinking Government-Run Health Care

    by David Kinkade

    Mar 12, 2014

    Since the launch of Obamacare, Americans have begun to witness the
    reality of what more government intervention in the health care system
    means—burdensome mandates, tax increases, punitive fines, and heavy
    regulations.

    Yet at the same time Obamacare has created an expanded federal role in
    health care, other countries that have been operating government-run
    health programs for decades are furiously seeking to reform their
    dysfunctional and financially unsustainable systems before they unravel.

    http://www.freeenterprise.com/health-care/european-example-rethinking-government-run-health-care

  • EmilyAnon

    Margalit, your article here has been cited on Medpage Today. Nice.

    • http://onhealthtech.blogspot.com Margalit Gur-Arie

      Thanks, Emily….

  • LeoHolmMD

    Great dissection of the report. I imagine there are loads of bureaucrats trying to figure out how to get these metrics up, while continuing to reinforce some of the worst aspects of our system.

  • http://onhealthtech.blogspot.com Margalit Gur-Arie

    Thanks for reading, even if humor was a bit lacking in this one :-) I promise to do better next time….
    Yes, the Commonwealth Fund is very well respected, and I guess that is precisely why I am expecting that their methodology should live up to that respect….
    I think you have hit the ball squarely in the middle with the notion that simplification is what we need most. But simplification means that there will be very little opportunity left to extract money from the system for lots of folks that are currently carrying the cash out with wheelbarrows while lecturing us that we are spending too much on health care… go figure….
    I am doing what I can for independent primary care, particularly of the Solo type, but this is looking more and more like a losing battle that no one else cares about…..except maybe (maybe) the few remaining solo docs…. :-)

  • http://onhealthtech.blogspot.com Margalit Gur-Arie

    I like the OECD reports a lot http://www.oecd.org/health/
    Health Affairs had a nice issue dedicated to other health systems back in 2010, so it is a bit dated but still very good (August 2010;

    Volume 29,
    Issue 8 Lessons From Around The World)
    The Commonwealth Fund has a very comprehensive report they issue periodically, with the last one being “International Profiles of Health Care Systems, 2013″
    Also the World Health Organization (WHO) has plenty of data as well.

  • Patient Kit

    Great post, as usual, Margalit. Even the best, most respected researchers need their research vetted by someone like you. And I say that as someone who spent the last 10 years prior to my layoff as a professional research analyst. You’re posts are always awesome.

    • http://onhealthtech.blogspot.com Margalit Gur-Arie

      Thanks Kit. I must have picked the digging habit from my dog somehow, or maybe he learned from me :-)

  • PaulC

    Another excellent analysis by Margalit!

  • Karen Ronk

    I am curious if the surveys that were referred to included actual patients or just institutions? Having had a brief experience with the NHS in the UK, my impression is that patients there are more accepting of what we might consider unacceptable in terms of care here in the US. If you have only know national health care, as many in Europe and Scandinavia have, how would you know the difference? That is not to say, however, that we don’t have a very dysfunctional health care system here that needs a lot of work.

Most Popular