Performance measuring tools aren’t up to the task

Performance measuring tools arent up to the task

I was invited to give a talk about patient satisfaction at a recent grand rounds. I have written previously that satisfaction is a pretty low bar, and so I spoke instead about the patient experience.

I opened my talk with the cartoon above.

As I anticipated (and intended), it got a few chuckles from the audience, and I pointed out that it is funny because it resonates – we already recognize that we often provide a poor experience for patients.

I went on to explain that we are all being evaluated by our patients in unscientific ways and that we should embrace soliciting a more authentic voice of our patients through surveys, so that we can improve the experience we provide them. I also pointed out that hospitals are already subject to penalties or bonuses from CMS based on their standardized patient survey results and that the same will soon be true for physicians as well.

After my remarks, I received an email from someone who had been there. It included a blog post that implied that it was a fool’s errand to try to evaluate the performance of physicians, and likened it to trying to catch “a cloud with a butterfly net.” Patient surveys were said to be particularly useless, since patients may “like” an incompetent physician. I replied (and believe) that patient surveys are certainly insufficient to evaluate physician performance, but still tell us something important about a particular dimension of care.

I was still thinking about that when I came across two recent articles in the New England Journal of Medicine. Both discuss the effort underway by CMS to measure physician performance and tie individual physician payment to it, and both made compelling cases that this effort is deeply flawed. And, frankly, both left me feeling a bit depressed.

I do believe it is important to evaluate physician performance. It is a basic professional obligation, and we can’t improve if we don’t measure. But the current state of the art doesn’t seem up to the task. In fact, the objective measures of physician performance remind me of the old joke about the drunk who looked for his car keys under the lamp post — not because he dropped them there, but because the light was better. We are measuring things not because they are important, but because we can measure them.

So where does this leave us? Rather than throw in the towel, I believe we should:

  • continue to collect and report publicly some objective measures of physician performance, recognizing that we have very few that really make sense
  • continue to survey patients about their experience, and strive to improve it
  • move toward measuring — and holding physicians accountable for — patient outcomes, instead of adherence to physician performance standards.

In the end, it is how patients do and how they experience care that really count.

Ira Nash is a cardiologist who blogs at Auscultation.

email

Comments are moderated before they are published. Please read the comment policy.

  • Dr. Drake Ramoray

    I’m gonna be brief because I’m about to start seeing some non-compliant diabetics. First one came to me with an A1c of 16, which admittedly I didn’t think was possible and for the patient to remain out of the hospital. His A1c is now 10, progress of sorts.

    When medicine starts holding physicians accountable, correction, when third party beancounters start holding physicians accountable for patient outcomes, then nobody will want to take care of sick patients. I’m happy to take care of sick patients, and some of my best breakthroughs and differences I have made have been in non-compliant patients, but I will have to think twice about doing so for any exended period of time if I’m gonna get paid less to care for sick patients than healthy patients. A small thyroid only concierge practice is looking better every day.

    • NewMexicoRam

      Reminds me of the old story about how the ancient Chinese treated their physicians.
      The physician would live as long as the patient lived. If the patient died, so did the doctor.

    • buzzkillerjsmith

      16 really rings the bell. And I thought you were an endocrinologist! Perhaps, doctor, you should have a discussion about diet and walking a bit more (assuming no double amputations, of course). Just sayin’. :)

      • Dr. Drake Ramoray

        I once had a patient with an A1c of 14 that dropped to 6.5 with diet,exercise and Metformin monotherapy. Quitting the 4 liters of Mountain Dew a day probably helped. Don’t underestimate the effects of Red Neck Pep.

        One of the best moments in is care was stopping the beta blocker that a cardiologist had put him on for SVT. (From the caffeine).

    • buzzkillerjsmith

      Taking care of sick pts is pretty expensive and kinda hard Taking care of healthy pts is cheaper and results in pts that are, uh, healthier. At least this is how I reckon the logic goes.

  • southerndoc1

    “I do believe it is important to evaluate physician performance. It is a basic professional obligation, and we can’t improve if we don’t measure”
    Bullshit.
    What other “profession” is subjected to this kind of nonsense? Lawyers? Accountants? Priests?
    Performance evaluation is about one thing and one thing only: reducing payments to physicians.
    Have to add that I love your conclusion that, even though performance evaluation is basically worthless, we should keep doing it. That makes a WHOLE lot of sense.

    • azmd

      The problem is that lawyers, priests and accountants are not paid by the government and by large corporate entities the way we are.
      Corporate CEOs have their performance measured by metrics with meaning to the stockholder, and so it is not surprising that they would expect the same principles to be able to apply to us. Look what has happened to the teaching profession…

    • NewMexicoRam

      Your last line sums it up very well.
      Of course we need to treat our patients well, as people.
      And I have patients that certainly don’t want to treat me as a person, but I try to best to be nice anyway.
      But, we can’t let patients’ idea of satisfaction be the determinant of our pay, other than will they keep seeing their doctor.
      Satisfaction and good quality medical care are not the same thing.

  • rtpinfla

    I disagree that publicly reporting ANY objective measure of physician “performance” until somebody can come up with one that is actually meaningful. A highly competent physician who takes complicated high risk cases will suffer if his “performance” reflects poorer outcomes relative to the less competent physician who doesn’t take on those harder cases. In a case like this, putting out data that inaccurately shows a patient that Dr. X has worse outcomes than Dr. Y does a disservice to both patient and physician. Physicians will learn that the only way to not be penalized will be to cherry pick their patients- leaving the very patients that need the best doctors out in the cold.

    Patient satisfaction and physician performance are 2 completely different measurements. And what does “performance” really mean? Is it RVU production? Is it complication rate? Is it the number of patients with an elevated LDL that are prescribed a statin? (a measurement parameter which has mercifully changed only recently).
    A high performing physician can score very low on patient satisfaction and vice versa. A physician that opts to not prescribe a certain medication based on a patients unique circumstances will be labeled as “not practicing within the standard of care” even if the decision is the correct one medically speaking. So which measure is more important and who decides how to weigh these measures? What is the intent of all this measuring anyways? It sure isn’t to help patients find the “best” doctor and will do very little weed out the very few bad ones.
    I agree that any good physician will find ways to evaluate his/her performance and always strive to get better. Arbitrary and unwieldy criteria- that can never take into account each patient’s unique situation – counterproductive. This mistake is amplified when it is tied to financial compensation.

    • NewMexicoRam

      This reminds me of a story I heard once:

      There are 2 dentists in town. One has perfect shining teeth, the other a mouth of cavities and foul plaque.

      Which dentist do you go to?

      The answer: Just think about which dentist each dentist sees as a patient.

  • Ira Nash

    For those of you (see below) who want to wait until there are perfect measures of physician performance before publicly reporting anything, all I can say is “good luck with that.” I am sure that some hotel owners hate trip advisor and some restaurant owners hate open table, but the train has left the station. You and I are ALREADY being rated by our patients through completely unreliable, unscientific sites like vitals.com and healthgrades. The only way to drive out bad information is with good information, and the only way to get better is to know how good you are now.
    Sorry that southerndoc1 thinks this is bull****, but do you think that all doctors are already as good as they can get? would you send your family member to any random physician? Do you really want to be held to the standards of accountants?
    You also misread my conclusion. I don’t think it is worthless, I think it is really important, which is why we need to do a better job of it.

    • Dr. Drake Ramoray

      With all due respect you are comparing apples and oranges. Information left on other websites while inaccurate, does not require me to spend thousands of dollars on a computer system that doesn’t talk to any other computer system, and involve adhering to metrics of dubious value (cholesterol being the most recent) and I have made the case for A1c as a metric in the elderly on other posts.

      Furthermore, none of the thought leaders have answered about how thing physician reimbursement from these goals doesn’t negatively impact physician access. Is have recently seen an unfortunate young man who has limited use of his mental faculties, and is a paraplegic, following a MVA. I have a good rapport with the family, we are limiting his therapy to two shots a day so that he can live at home and his family can assist him. It became apparent that he is sneaking sweets at home. His A1c remains above goal.

      So here I am, it’s 2017. I can get paid less to see him. I can dismiss him as a patient. I can demand that he go to a standard MDI therapy but then he probably can’t stay at home and will have to go to a nursing home. The next nearest Endo is over an hour away, and after that it’s 2-3 hours. I already can’t bill Medicare for consults and now you want me to get paid less because of a negative selection bias for my patients as a specialist.

    • Dr. Drake Ramoray

      This may turn out to be a double post as Disqus ate my first.

      With all due respect you are comparing apples and oranges. Vitals.com is no different than bad word of mouth should I upset a patient. Perhaps more publicly so but it does not involve me spending thousands of dollars on a copmuter system that slows me down and then grades me on dubious metrics such as LDL cholesterol goals and A1c levels in Octogenarians.

      Furthermore, vitals.com is not directly tied to my reimbursement
      Furthermore, no thought leader on this forum, or really ever for that matter, has answered how they don’t think that these pay for performance schemes won’t hurt patient access. My group is the only Endo for 50 miles or so and then then ext closes is in the over 100 miles range. I have recently seen an unfortunate young manwith limited faculties and CKD following an MVA whom we have selected sub-optimal pre-mix insulin because that way his family members can administer him his medicatioon and he can stay at home. This stratgey and his living situation make it basicalyimpossible to acheive an A1c of less than 7. (He also sneaks sweets but hey he is paraplegic plus stuck at home all day).

      So when we change to a pay for performance scheme, do I push for multiple daily injections and potentialy force the patient in a nursing home? Do I continue my good rapport with the patient, sub-optimal blood sugars, and his current quality of life but continued to be financially penalized for doing so? Perhaps, I dismiss him from the practice? I have the worst of the worst in terms of diabetics in my community. Shall I pick up and leave to the affluent suburbs where it’s easier to cherry pick healthy patients and make more money?

      Why can’t any of you take what you are arguing for to the next logical step?

    • southerndoc1

      Where to begin?

      1. “You and I are ALREADY being rated by our patients through completely unreliable, unscientific sites”

      Actually, I’m being rated by the most scientific, reliable method available: in an area rife with doctors (3 local med schools), I have a full waiting room every day and a six month waiting list for new patients (family practice). Contrary to your assertion that improvement can only come through measurement, my staff and I work every day to improve the experience of our patients. Check out the really interesting research showing that highly motivated individuals such as physicians actually perform less well when they are being measured.

      2. “would you send your family member to any random physician?”

      No, but that would probably be a better method than choosing doctors on the basis of satisfaction surveys: all the data shows that patients who are “satisfied” undergo more procedures, have more complications, and have poorer outcomes.

      3. “Do you really want to be held to the standards of accountants?”
      My experience with the accountants I’ve hired over the years (none of them chosen by looking at satisfaction surveys) has been outstanding. They have been dedicated, open, and completely ethical. I pay them and I have 100% confidence that they are working for me, not for the government, insurers, or outside investors. I think physicians would do well to look to accountants and others to begin taking back our standards of professionalism.

      Sorry to be so critical, but your post comes across as the thoughts of someone who has drunk too deeply of the Big Corp Med Koolaid,.

    • rtpinfla

      I think what southern is referring to as bull—is not patient satisfaction surveys such as vitals or healthgrades. Savvy docs already know how to manipulate these or should know (hint: only give your really happy patients a paper that invites them to rate you online).
      But to advocate for objective measures that “don’t make any sense” is simply crazy. This is especially true when insurance companies or other entities will use this data to their advantage- and don’t bet for a minute they won’t do just that. Is it happening? Yes, but that doesn’t make it OK. And measuring patient outcomes rather than guideline adherence runs into the exact same problem.
      I think the idea of physician self improvement through patient feed back is important. The issue at hand is when such data is used the wrong way by the wrong people and intentionally or unintentionally adversely affect care.

    • LeoHolmMD

      Lets say 10 years ago one of these “physician performance” mongers wanted to choose getting a yearly PSA as one of their performance measures. Easy, measurable, reportable…now also of no benefit and possibly harmful. Thats the problem with going forward with bad ideas from central planners with questionable outcomes. That is what gets pushed. Real people that I have to look at in the eye can get harmed. Real resources that could go to things the patients are concerned about go to hounding performance measures. I welcome things like Angies List etc. Those are patient driven. Payers have an entirely different agenda: Not paying.

      • NewMexicoRam

        Yes, yes, yes!
        In some ways I feel like we have gone back to the 50′s and 60′s, in a way. The doctor will be TELLING the patient what to do.
        I don’t practice that way. Personally, I find it works best to function as a counselor or advisor to the patient, going through options, risks, benefits, and then giving a personal opinion–but the patient is the one who decides.

  • southerndoc1

    If they’re anything like the insurers I deal with, now they’ll want to know why you didn’t order PSAs on these patients.

    • LeoHolmMD

      Don’t forget to get that LDL down less than 100. What is sad is that physicians who understand the surrogate endpoint game, overtreatment and disease mongering can call BS on something long before the rest of the “thought leaders” can. Someone may have saved the medical system millions by being a “poor performing physician”. Unfortunately, there is no code for “I told you so”. I could retire by now.

  • Dr. Drake Ramoray

    V15.81 is the code for non-compliance. Patients don’t like it very much if you use it though.

  • Suzi Q 38

    I agree.
    I fairly sure that patients are not chosen randomly.
    They are chosen based on how well the visit went.

    I decided to save the evaluation from a mistake free visit, and use it for a different doctor who was a jerk.

    I turned it in with all negative reviews on each question, and nothing happened, LOL.

    It is all a huge “joke.”

  • Rob Burnside

    Satisfaction surveys are not unique to health care. They’re becoming ubiquitous, and I believe it’s a result of our transition to a service economy. In effect, “process” has become “product.” Unfortunately, we’re being asked to make what amounts to, in many cases, a subjective judgment–especially in written or computer surveys where reading and comprehension ability are variable yet paramount. I rarely participate, because the probability of error and ambiguity of results are too high, as are the stakes for those being rated. It may be wrong, but I’ve always judged doctors initially by the number of patients in their waiting rooms. The more patients I see, the more confidence I have.

    • DoubtfulGuest

      Yes, but how is the waiting room *painted*, RB? ;) I also have rarely participated because I don’t know what’s being done with that information. Would even genuine perfect scores be viewed as suspect under certain conditions?

      • Rob Burnside

        Good paint..er…point DG! What did Fernando say? Oh yes, Darling, it’s how you look that counts! You can take it from there….

        • DoubtfulGuest

          I was sort of concerned from Dr. Fielding’s post about these results being linked to staff bonuses? I wonder what response rate they need for these to count? I’d honestly give perfect or near perfect scores to all of my (*ugh*) “providers” these days. But what is the effect when I do or don’t turn in a survey? And who would I ask about this at my particular hospital? Would my asking cause problems if the patient satisfaction folks then looked up who I see at that hospital? It’s easier to just shred the darn things.

          • Rob Burnside

            Yes, DG, shredder fodder for sure if we can’t say something good. I haven’t seen Dr. Fielding’s post, but there you have the other side of the coin. In not participating, are we denying–or helping deny–a bonus someone may need and deserve? It’s a quandary.

Most Popular