Why Watson makes board certification exams obsolete in medicine

Like most everyone else, I took a break from my evening chores the past few nights, and watched Jeopardy!

IBM’s super-computer, Watson, was taking on Jeopardy! phenoms Ken Jennings and Brad Rutter.  [SPOILER] The computer won handily.

After watching, I began to study for my upcoming board re-certification exam.

But, then, I wondered.  Why?

Why Watson makes board certification exams obsolete in medicine

In the New York Times, applications for Watson are taking center stage.  In the field of medical diagnosis, for instance,

[IBM] plans to announce that it will collaborate with Columbia University and the University of Maryland to create a physician’s assistant service that will allow doctors to query a cybernetic assistant. The company also plans to work with Nuance Communications Inc. to add voice recognition to the physician’s assistant, possibly making the service available in as little as 18 months.

“I have been in medical education for 40 years and we’re still a very memory-based curriculum,” said Dr. Herbert Chase, a professor of clinical medicine at Columbia University who is working with I.B.M. on the physician’s assistant. “The power of Watson- like tools will cause us to reconsider what it is we want students to do.”

Soon, physicians will have easy access to a Watson-like computer with vast medical knowledge, where they can simply ask  a natural language question, and the computer will come back with a medically accurate answer.

Why, then, are we still relying on  a “memory-based curriculum,” where doctors still need to recall endless amount of facts on closed board exams?

If I don’t know the answer when I’m with a patient in the exam room, I look it up, or ask someone who does.  In this era of patient safety and emphasis on reducing medical errors, it doesn’t make much sense to rely on rote memory to practice medicine.

Watson antiquates closed board exams.  Instead of sitting in a testing room, doctors should be evaluated on how well they can find the necessary information — not how well they can recall something they memorized.  Board certification tests should be open-book, or, at least, provide the resources ready for physicians if they can’t recall an obscure medical fact.  Just like real life.

After all, Watson will ensure that decision aids and vast medical knowledge will be readily available at doctors’ fingertips.  Board certification entities need to recognize this reality and update the way they test physicians.

 is an internal medicine physician and on the Board of Contributors at USA Today.  He is founder and editor of KevinMD.com, also on FacebookTwitterGoogle+, and LinkedIn.

Comments are moderated before they are published. Please read the comment policy.

  • http://fertilityfile.com IVF-MD

    Over the years, my sessions with medical students have evolved with less pimping about random trivia and more about clinical problem-solving and critical thinking. It makes sense. Why memorize dosages when you can look that up? Instead, spend your energy understanding what to do clinically and WHY.

  • http://www.thecentralline.org Graham

    I’ve got to disagree, Kevin. This may be true in office-based specialties like yours, but in several fields (most notably Emergency Medicine, Anesthesia, OB, Surgery), there is not time to look up particular information. Obviously a focused clinical question in a stable patient is appropriate–and I think in this way Watson-like queries will be very useful to all of us–but when a patient is tanking or they lose their airway, there’s not even time to open up your iPhone.

    • jsmith

      If you are an office-based practice, looking up every little thing is a good way to stay past 9 pm. You gotta know the material. Keep the boards the way they are.

      • http://www.talktoyourunconscious.wordpress.com BobBapaso

        Once I learned how to put in an airway, and I haven’t forgotten. We don’t need boards at all. Maybe a licensing sub-test after residency and then regular focused CME. If you are a person who wants to do the right thing you should be OK till dementia sets in.

        I remember the things I use, and a few big things; the rest gradually slips away. But it’s in my CME file if I need it. I’d trust that file more than Watson.

  • Brian

    I had similar thoughts as I watched Jeopardy! these past few days. It’s worth noting, however, that this technology is far from ready for prime-time in medicine. As an example, Watson’s answer in Final Jeopardy! was laughable (though it sagely only wagered a small amount); in the context of a game show, this can safely be filed under the heading of “Computers say the darndest things”, but in medicine, such a “cute” mistake could cost a patient dearly.

    For this reason, I think it is extraordinarily important that medical students be expected to retain a certain volume of information, foundational material (biochem, physio, etc.), and be tested on their mastery of those subjects. The computer can only ever be (at least, for the foreseeable future) an aid.

  • Dr Chris

    I would have agreed with Kevin until last week, when our computer service when down last week for 36 hours. My manuals were out of date, I couldn’t look anything up. My phone needed replacing-not the best data source. And I realized what would happen with a big power outage after an earthquake.

  • http://www.myfamilyhealthguide.com MyFamily Health Guide

    I still think it’s far more useful to assess knowledge via a standardized test; docs will always need to know the basics first. I think the main issue is how to test that knowledge; the AAFP board exam has a nice mixture of multiple answer questions plus interactive case studies where you need to evaluate, order tests, made a treatment plan and act on it. I find this to be a much more effective way to assess a doctor’s overall skill.


    Great. Obama Care will now require us to each purchase a Watson so that we can be reimbursed.

  • charlie dickens


    Did anyone ever believe the BC exams were about gaining physician quality??!! They are a huge money making scam for the boards, and now with this maintenance of certification (MC) we are held hostage to info. that is obsolete.
    I cannot believe anyone in AAFP/ ABFM / STFM or whatever organization believes the stuff that come out of their own mouths…The FP specialty is dying, and they come up with MC as the answer…

    Apologies for pontificating here…

    • horseshrink

      Am wondering if re-certification again is really worth it. Since my last recert, requirements/expense have gone up considerably. Yuck. I wrote ABPN to express my displeasure. The weeds of requirement by countless agencies choke my professional pleasure.

      If I won’t lose much income by leaving board recerts behind, I’m going to focus my energies elsewhere. As it is, I am very active in perusing literature … through problem solving or by subscription. I don’t feel a need to pay many additional thousands to others to prove it.

  • http://www.MDWhistleblower.blogspot.com Michael Kirsch, M.D.

    Best of luck with the boards. One of the fascinating topics you will learn is your own unique palm venous pattern, which you will demonstrate each time you enter the sanctified examination chamber.

  • doc99

    It’s time to play Beat The Reaper!

  • http://thehappyhospitalist.blogspot.com The Happy Hospitalist

    Since board exams were implemented, health care costs have skyrocketed and patients have become sicker.

    One must conclude if we want to reduce costs and make patients less sick, we need to get rid of boards.

    • pj


  • Wayne Ruth, M.D.

    Actually, it seems to me that it is time for a hybrid approach. The boards should be interactive with answers to queries responded to immediately, indicating whether correct or incorrect, providing data to support the correct response.

    The process should be liberated from the closed exam center and it should be iterative. If you miss a question then the topic should be revisited later in the testing process…miss it again, you get a debit, get enough debits, fail. If you fail, you should have the opportunity to attend educational program and re-test before losing certification.

    The test should be “open book” and allow using whatever resources you choose to answer the question. In other words, you have to know what you don’t know. The whole closed exam concept is antiquated and needs to be reconsidered, in my humble, non-academic physician’s opinion.

    We are facing massive shortages of physicians. We need to encourage and educate, not penalize, committed providers.

  • Nic

    My 2 cents worth.

    I am an MD… with no memory. Severe ADHD. I can see how the computer out performs me on many things and I have no problems with that….HOWEVER I’ll take on the PC any day on “the big picture” though. My gut feeling on patients.

    My question is: why do we keep “pimping” ? Testing people mainly on what they know, not How they solve problem is real situations ?

    There was an ER doc comment about: I don’t have time to look things up when a patient is loosing an airway.Cmon That is a TINY part of the picture.

    Having been a field MD, on the ambulances, helicopters in Europe 30 years ago, having to look drugs unknown to me, weird side effects etc. I know that, once that tube was in, I would have had the time to query the PC.

  • Nic

    I think it is extraordinarily important that medical students be expected to retain a certain volume of information, foundational material (biochem, physio, etc.), and be tested on their mastery of those subjects.

    Sure man. The Krebs cycle. a good Doctor must know the Krebs cycle and be tested on it, and not one of these puny multiple choices questions, but an empty sheet of paper that they must fill out…

  • Nic

    More seriously: Is the answer to exponential explosion of knowledge to memorize more, to make studies longer ?

    Pehaps we can rethink medecine a bit. How much anatomy of the hand should a shrink know ?

    • Blake

      And yet we lament the fragmentation of medicine.

      • Nic

        Well, Surgeons and Barbers were once the same profession

        • pj

          Thank you Nic- We PCP Docs have WAY too much emphasis on irrelevant stuff- Why do the public and the folks who run healthcare in the US like NP’s/PA’s so much?

          Could it be, they spend lots of time learning clinically helpful info incl. how to interact w/patients and others in the system, and less time on rote memorization of mind numbing things like the krebs cycle?

          Not saying I agree w/the current trends- just pointing out an explanantion for them.

  • http://www.consentcare.net Martin Young

    Memorizing detail for exams is one thing. Remembering them in the long term is another. I am good at the former, and appalling at the latter. I don’t think that makes me a poorer doctor – good doctoring is far more dependent on things other than medical facts. Without computers and easy access to data I would however have a problem! I agree, too much emphasis is placed on exam results.

  • MIS Prof

    In general, I’d say don’t go down the path of eliminating the board exams (although modifying them for interactivity does sound intriguing … how about testing responses in a virtual reality situation for part of the exam? VR gives you a controlled environment).

    This reminds me of college students who often say to me ‘I don’t need to learn this stuff … I can always Google it.’

    Right. Like there will always be time to look it up and the equipment and Internet connection will always be there. Like you’ll never need the whole framework to interpret the facts of a situation and craft a solution on the fly. Like you’ll never need to learn the basics to then be able to learn the advanced material. (Wish I had a penny for every student who couldn’t solve a mid-level difficulty quantitative decision-making problem in a reasonable time frame because they couldn’t do simple arithmetic.)

    I know the college student situation has limited applicability, but I think that’s where the ‘ending the boards’ road will lead.

  • medstudent

    Maybe it is because I was an engineering undergrad, where all tests were open book, note etc or maybe I am just tired of second year, but the amount of memorization needed to pass tests and do well on step 1 is ridiculous. And almost none of it involves critical thinking. And now whenever I see patients with preceptors I worry more about the doc asking me questions about random facts and than actually connecting with the patient and learning to solve their specific problem based on their specific needs and personality.

    I feel like I am being turned into a dumbed down computer that is just expected to regurgitate symptoms of diseases, drug names, expected lab values, virulance factors of bacteria, etc.. I hope this isn’t the case for me and for the rest of the of the profession, because we really should be the ones coming up with creative and new ideas to deliver medicine, to better the health of all our patients and to fix this crazy system we have. These answers arent going to be found using mnemonics and Watson. And right now, our system doesnt teach medical students how to come up with solutions either.

    • horseshrink

      Would have reacted with disdain to this a couple of decades ago … but, nowadays there’s too much information that changes too rapidly within even one specialty for me to absorb.

      I comb news from the AMA, APA, Journal Watch, The Medical Letter, WSJ and NYT, and am amazed at the astonishing, swirling, vast, evolving picture in which I practice. My college and med school textbooks are already becoming historical curiosities!

      I rarely open books anymore. That’s not the information I want. I open Google, PubMed, eMedicine …

  • Kevin

    How about this: we can provide medical texts to everyone at zero marginal cost. Most of everything else doctors know is learned during residency, because 1) we evolved for millions of years to learn by doing and 2) “use it or lose it” applies just as readily to medical knowledge as it does to everything else.

    So we should get rid of the time/money cost as a barrier to entry into the medical profession. If you can pass the boards, you can be my doctor, regardless of whether you have a medical degree. Let’s stop pretending that the medical profession isn’t just extracting money rents from the public, who don’t follow a typical demand curve when it comes to their health.

    In other words, the rarity (and thus, value) of medical expertise can and should decrease in the Internet Age, and to suggest otherwise is just narcissistic.

    • Anil

      I think we are on the same page on a lot of things,
      1.)Medicine is learnt by doing and not passing Boards.
      Its obvious why people ask you , whether you are a practicing(doing medicine) physician, or they may ask you, where do you practice, nobody asks you, are you a Boards taking/learning physician.
      The Boards have reduced Doctor’s to test taking machines, if you are a good test taker its easy for you, but if you are a poor test taker its going to be tough especially with time restraints. In real practcie that does’nt have anything to do with good patient care. If you are good physician you willl look up and get back and give the best care to your patient, end of story. Where did the testd come back again to haunt you. Even the medical student forgets what he had prepared on the step 2 exam when he has to sit for step 3. He goes back to step 2 to cover his base and then depends on few test review books and aids and residency practice to take step3.Its like beating around the bush while flinging dollars into the bush i.eThe Boards).

  • jack ga patuto, MD

    i have to agree with anil……watson is just an information machine……it cannot reason. that is a specialty that all physicians aquire during years of practice.
    then, where would the 2nd opinion go? just ask watson?
    niether board certification,nor, watsonization, does a physician make!!!
    medicine takes extreme dedication ,reasoning and compassion—-something that watson does not have.
    besides, we, as physicians heal humans. humans are also controlled by emotions, something that watson does not have.
    where would watson go to for a 2nd opinion? watson only gives back what it is fed.
    i have been a dr. without borders and internationl physician for more than 20 years. i also teach academics…i have learned from my mentors and from patients. i am now a mentor myself and someday i will be a patient.
    i want to know that my doctor someday will make the same decisions for me that i have made for my patients—from my experience and from my heart…
    but, then again, this is my opinion………thankyou

  • GBC

    TO “IVF-MD” and Graham – and most others: There was just a case in New Mexico where a Cosmetic Surgeon was accused of printing directions for certain procedures from the internet and taping to the cabinets in the O.R. so he could follow them during surgeries.

  • Nic

    I fail to see the point youre trying to make…
    Does he do good quality procedures or dismal ones.

    It tuns out the guy is grossly negligent etc.
    Internet instructions have nothing to do with that.
    Furthermore, I fail to see how it ties in with Watson

  • Dave Miller

    I am forced to disagree with Dr. Pho on this one. Watson may indeed “have all the answers” but it has no intuition, no wisdom borne of experience, no little hairs on the back of its neck to suggest that it lacks a crucial piece of information. Indeed, like any computer, its output is simply based on a highly sophisticated set of processes and calculations performed on its input. As with all computers, “garbage in, garbage out” still applies. You leave out one crucial piece of information (and the computer has no way of knowing whether you have or not) and you will likely get the wrong answer. Humans, with our ability to reason and intuit, can often surmise that we are missing that crucial piece of information and act accordingly.

    Board exams do indeed suck but I’ll take that over relying on a computer any day.

  • http://retiredbicycle.blogspot.com Allison

    As a non-medical person, I’d love to have access to Watson. At present I’m unable to obtain medical care since all of the primary care facilities are booked out for 4 weeks, and I’m unwilling to sit and wait in Urgent Care for 8 hours and be exposed to who knows what. It would be great to know if my cough and sore throat are viral or bacterial. Maybe Watson could cut to the chase and prescribe for me if it’s necessary. Somethings gotta give on the unavailability of doctors, PAs or NPs, and Watson might be a good front line triage mechanism. I wonder how soon it will be until it can run the DaVinci robot.

    • horseshrink

      Be careful what you ask for.

      You might get it.

Most Popular