How judging doctors by the numbers may be meaningless

How do you judge how good a doctor is? By personal interaction? By what relatives and friends say? By whether he or she is on time when you go for your visit? By doctor rating websites on the Internet? By patient satisfaction surveys conducted by doctors themselves or rating agencies?

Or do you do it by the numbers? The federal government and health plans are increasingly issuing periodic “report cards” containing “objective” evidence of how well doctors are doing in achieving “quality” goals.

How can one argue with this approach, which may become the basis for “pay for performance” programs? After all, if you can’t measure something how can you judge it? This is the cornerstone of management “science.”

The answer, according to Danielle Ofri, MD, PhD in the New England Journal of Medicine,, is that these numbers and these reports are often meaningless to individual physicians.

Here is how she, an internist at New York University School of Medicine and Bellevue Hospital explains her reaction.

“The quarterly ‘report card’ sits on my desk. Only 33% of my patients with diabetes have glycosylated hemoglobin levels that are at goal. Only 44% have cholesterol levels at goal. A measly 26% have blood pressures at goal. All my goals are well below my institution’s targets.”

“It’s hard not to feel like a failure when the numbers are so abysmal. We’ve been getting these reports for 2 years now, and my numbers never budge. It’s wholly dispiriting.”

She is beginning to wonder if these numbers really mean anything. Are they a valid measure of physician competence? How would patients react to them? Should the numbers be used to judge and compensate doctors?

One thing she does know. “These statistics cannot possibly capture the totality of what it means to take good care of patients. They merely measure what is easy to measure.”

So what does she do with the numbers?

“I don’t even bother checking the results anymore. I just quietly push the reports under my pile of unread journals, phone messages, insurance forms, and prior authorizations. It’s too disheartening, and it chips away at whatever is left of my morale. Besides, there are already five charts in my box – real patients waiting to be seen – and I need energy for them.”

Judging doctors by the numbers may be the metaphoric equivalent of whistling in the wind. It sounds good, but it is largely meaningless in the total scheme of things. When I think of these numbers as a means of judging physicians, I’m reminded of Mark Twain’s comment about Richard Wagner’s music, “It’s not as bad as it sounds.” Similarly, numbers for judging doctors are not as bad as they make doctors sound.

Richard Reece is the author of Obama, Doctors, and Health Reform and blogs at medinnovationblog.

Submit a guest post and be heard.

email

Comments are moderated before they are published. Please read the comment policy.

  • http://fertilityfile.com IVF-MD

    Nice post. I agree about not judging doctors by standards that are set by some committee.

    Imagine how much better it would be to involve the patient’s own wishes in all this. Rather than base ratings on things that are chosen by someone on the outside, let things be free to be the way all things in life are when describing a voluntary consensual interaction between two parties. If the patient is happy, let that be the top criterion. Then let all these other “objective” criteria be secondary.

    How would you feel about taking a doctor whom patients hate and who gives bad care and giving him praise and rewards simply because his patients’ HbA1c levels are better than the national average?

  • http://www.aneurysmsupport.com/ Mike

    While I would certainly consider the recommendations, or lack of, from a trusted friend or relative, with me, it still boils down to how I personally relate to the doctor and my own opinion and, even, gut instinct. These “report cards” may, in some way, actually offer a partial measure of physician’s competence but I really doubt they tell the whole story. No matter how skilled a physician is, if the patient will not cooperate and do their part they will never see cholesterol levels at the goal. Good healthcare is a two way street.

  • Kim Lucas

    Recently read a study that showed that the lower a patients socioeconomic status is the lower they rate their physician. Poverty is the real enemy. the impoverished are less healthy and less able to negotiate the healthcare system. Poor health is a result of poverty but then keeps the vicious cycle going. Medicine is not the solution however, education is.

    • http://Www.Twitter.com/alicearobertson Alice

      Could you try to remember where you read about this study and who financed it? In the age of Freakonomics (still shaking my head at a few of their assumptions based on stats that are often preconceived then a study set up to prove whatever the notion was that needed fulfilled).

      I hang out in the inner-city, and find that, although, they do often feel entitled, or judged I really doubt that the supposed outcome of that study would stand up if tested.

      Our gifted doctor said it is the rich that complain and are arrogant, and many doctors do not like treating them…and some gloat that they do, indeed, treat them less well. Did someone finance a study on this? And what doctor in their right mind would dare answer a question truthfully like that when it could end their career?

      Overall, in my unstudied, ignorant world the poor do not have as high an expectation as the rest of us on doctors. Of course, I do make jokes about “Ghetto Justice”. It’s blatant, honest, in-your-face, style that causes me to pause and sometimes smile and ponder……..because more often than not it is raw honesty in an explosive state.
      Maybe we should just listen and not care what sociology-economic scale they are on?

      Just thinking aloud really, and wondering if a study like this was meant to tell doctors to treat the poor with more care because if you don’t…..they may run online and give you a bad rating? Hmm………

      • Carol

        I agree with Alice. I’m a pharmacist and not a physician, but I’ve worked in a very rich neighborhood and a very, very poor one. In my experience, the wealthy were rude, impatient and demanding. The needed to get back to their book clubs and soccer camps, and wanted things done yesterday. The poor were much more appreciative and less rude. If you went out of your way even a little, they would appreciate it. Of course, those are generalizations and there were exceptions in both cases.

        In my experience anyway, the wealthy are used to fast, concise service, so they demand it. The poor are used to being talked down to and ignored, so they appreciate it when you treat them like a human.

        A physician might have a different experience, but a general physician friend of mine in the poor neighborhood says similar.

    • http://Www.Twitter.com/alicearobertson Alice

      Kim …please be patient with me….I know this is off-topic….but logically ….hmmm….I understand your statement about education being the answer….but surely you are addressing poverty and not patient-feedback? Even an illiterate knows if they were treated well by a doctor? Education in all it’s privilege and glory has helped mankind and the medical establishment….but it has not helped increase empathy or caring for the human spirit. There are just some life lessons the schools cannot teach….only the school of life can teach us what I consider the most valuable of all my lessons.

  • Marc Gorayeb, MD

    Physician skill and effectiveness is multi-dimensional. It may be futile to try to quantify it. Characterize a physician by measuring one dimension, and you risk trivializing other dimensions that have equal or greater importance. And the act of measurement itself may alter the physician’s effectiveness in unintended ways; for example, by inducing the physician to reduce his or her roster of patients with complex problems.

  • http://drpullen.com Edward Pullen MD

    It is very possible to both be compassionate, attentive, and caring, yet achieve strong metrics in care of problems like diabetes. Chronic disease management and patient focus are not exclusive goals. You can resent “outside rule makers” grading your care, but don’t think that geting as many diabetics to optimal control parameters as is possible is not an admirable goal.

  • http://fightcolorectalcancer.org/research_news Kate Murphy

    One measure of a doctor’s impact on patients is how well his or her office is run. A doctor can provide fine one-to-one patient care, be compassionate and smart, and still fail patients if telephone calls are dropped, appointments for further testing or specialist visits take weeks to get scheduled, receptionists are confused or unpleasant, and things are just generally inefficient.

  • Charles Cohn

    Culd it be that these stats are as much an indicator of patient compliance as they are of physician competence?

  • DrMom1952

    I agree totally with the article written by Danielle Ofri, MD, PhD
    in NEJM August 10 2010. As a primary care MD, taking care of large numbers of chronic disease pts, this so called “rating of quality” for physicians is absurd. The majority of the time the reason my patients HgbA1c is not at goal or their BP readings are not at guideline recommendations HAS NOTHING TO DO WITH ME, THE PHYSICIAN.
    I cannot go home with my patients to force them to eat correctly, lose weight, exercise, stop smoking, take their medications as directed, or as too often is the case, buy their medications for them (since they can’t afford them) so they can take them. And if they get depressed over their life circumstances and become noncompliant with recommended care, don’t come in for follow up despite repeated calls, and so have “bad numbers”, how is this representative of my quality of care and competence as a physician?
    It is not, and this type rating using certain lab values, or whether a pt gets their opthalmology screening done, etc is a very very flawed system that fails to measure much of anything about physician competency or quality of care.
    Why not measure the insurance companies on how many of their enrollees go off meds due to increased copays, prior authorization requirements, or people who lose insurance coverage due to increased premium costs?
    Now those are entities that definitely affect disease outcomes.
    I am fed up with these meaningless measures that do nothing to improve clinical outcomes.;
    A primary care MD

  • gzuckier

    Yeah, but….. taking the Devil’s Advocate role again, I have no doubt there are surgeons (just because the field makes a clear example here) out there who are very patient-centered and make a real effort and care for every patient and the patient feels that, and yet their coordination is such that they probably shouldn’t be allowed to handle sharp objects. And similarly for all fields of medicine; proper attention to the patient is one factor, but sheer professional competence is another, which the patient is not always equipped to accurately assess.

  • Blaine Carmichael, PA-C

    This is why there are two very important components to
    practicing medicine-the science of medicine which is empirical and quantitative by its very nature. There is a specific symptom or symptoms where the clinician attempts to explain thru the “scientific process” the cause of the condition — that has to be balanced with the human
    aspect or the Art of medicine. The Art of practicing medicine is processing and factoring in – the human element which consists of emotions, feelings, believes, outlook and one’s culture.

    It is the personality of the provider – the very essence of who you are as a person and how well you connect and communicate with people that determines your success in the art of medicine. The issue is on these studies is how do they quantify and measure this very personal essence
    of a provider as perceived by patients? One can in theory score 100% on A1C education and follow up with a pt-but as you know the pt may never establish a connection with the provider-and ultimately trust, credibility, compliance with that pt may never be realized. Therefore these snap shots and dash board reports can by the very nature of this
    complex interaction -only measure a narrow subset of a providers competence – the measurement tool is not sensitive or board enough to capture and tell the whole story. Ultimately the government has to select a neutral and factual methodology to try to capture this very complex human interaction-thus it is limited and flawed from the outset.

    What this really attempts to accomplish -in my humble opinion- is how does the government determine how a limited resource (health care funding) is best utilized to provide the broadest coverage possible for the limited money available to citizens. They have to select some
    system and method don’t they? So the most defensibly system to administer and justify is one based on empirical science which is very black and white. Unfortunately this minimizes the human relationship component – which ultimately might be as important (or in some cases more important) as the science behind our profession!

    Blaine Carmichael, PA-C

    • http://clinician1.com Bob Blumm, PA-C

      I submit this article that I wrote a few months ago for the benefit of Dr. Danielle Ofri, MD, Ph.D from NYU and Bellveiw in reference to her publicized comments in the NEJM. I empathize with her conclusions, as she demonstrates the pain of a medical provider that is or has burned out due to poor compliance and a heavy load or responsibility. This art that we call medicine, is an unending challenge and requires a strong deliberate response from all of us. Danielle, i hope that you have the ability to heal and that your colleagues surround you with support at this time of disillusionment. Bob
      A Lesson From the Tides

      By Robert M. Blumm, MA, PA-C, DFAAPA

      Patients are confronted daily by their health care providers with news that ultimately will affect their health. You’ve seen this on TV, when the “doctor” walks in the exam room, where the patient sits in a johnny coat, and says, “You have diabetes,” or “You have heart disease,” or “You have cancer.”

      In real life, as on the screen, the initial impression is that these patients are sad or distraught about the diagnosis and the prognosis. These patients leave your office and, because they feel hopeless, they may light up a cigarette, stop by the burger joint and “super size it” and then go to a barroom and have a few shots of good old Irish whiskey. They try to drown out that sick, hopeless feeling with the same ingredients that may have created the illness.

      Good Intentions, But…

      The second stanza of this familiar song is when these patients decide that they will change. And change they do! They look at the portion size and leave 25% of their food behind to be used as a doggie bag. They suck on mints instead of smoking a cigarette, or they order club soda with a twist of lemon so as to not attract attention to the fact that they are converting from their alcohol habit. They go to the mall and buy a treadmill, and then to a book store and spend another hundred dollars on self-help books and diet cookbooks. They get support from loved ones and hear things such as, “I know you love me and the children because you’re taking positive steps to wellness.”

      Three months pass, and during that time they walk past McDonald’s and have an urge, but instead of a Big Mac they have a Filet-O-Fish sandwich without the fries and with a diet soda. They go out with friends and have just one cigarette and are amazed at how lightheaded it makes them. At the bar, they order a Campari and soda instead of scotch. When they arrive home, they undress and hang the clothes on the treadmill that now sits in the corner, neatly folded. The health-focused books they purchased a few months back are neatly stacked, forming a nightstand for the new restaurant guide that will be a source of instant pleasure. They look at their loved ones’ faces and try very hard not to recognize the pain that may be inscribed on there or hear the words of encouragement, since for some reason they no longer have the same effect.

      Life’s Ebb & Flow

      Have you ever sat on the rocks near the seashore and just observed the waves as they continuously approach the shore? Sometimes they flow softly and quietly, and sometimes, during a storm, they make an angry thrashing sound as they land on the beach. Occasionally, they are tsunami-force and grab everyone’s attention. Still, they continue to come, never changing their approach. At high tide and at low tide, the waves never cease to move in their fixed direction toward shore.

      This makes me think of myself as a human being and as a patient. It makes me think of the patients I have spoken to, as well as those whom you speak to every day. Studies show that we don’t change habits easily, and when we do, it usually is for short periods. The waves’ increasing roar is distorted by the sounds of life, and we cease to hear it any longer until the tsunami comes, and either we leave this world prematurely or our patients meet that fate.

      Sad, isn’t it, that we have ears to hear but lack the internal incentive to follow the dictates of wisdom? It’s even sadder for those we leave behind, because regardless of our accomplishments in life, we have failed at one of the apparently simplest tasks: self-control. I remember my mother-in-law telling me that the epitaph on my headstone will read, “He killed himself with his fork.” No, I never forgot it, but neither have I made the lifestyle changes that would prove her wrong.

      So what is the take-home message? Where is the doggie bag? You and I encounter our patients daily, give them the sound advice that will keep them well or perhaps alleviate their disease, yet we are faced with the fact that they return three months later with a hemoglobin A1C of 8.5 and demonstrate no change. Looking at myself, when the scale reads 250, do I just admit defeat, or do I determine to change, even for a while? What is our responsibility?

      Like the tides approaching the shore, we need to be consistent in our warning and hope that a small percentage of our patients have a metamorphosis. For our families, we need to ask them for forgiveness, tell them we hear their concerns and that we will make a determined attempt. Although nature rarely changes the course of the waves or the direction of the seas, it sometimes surprises us. Let us continue to believe that we may change the destiny of our patients despite the daily disappointments. This is our calling.