Problems with the Multiple-Mini Interview for medical school

Some medical schools have altered their admissions process by replacing the traditional applicant interview with the Multiple-Mini Interview (MMI). MMI resembles speed-dating: applicants rotate through numerous interview stations, where they act out scenarios and solve puzzles, sometimes alone and sometimes in groups. A July New York Times article presented a good overview of MMI, as implemented by Virginia Tech Carillon.

As you might expect, schools that have adopted MMI (UCLA and UC Davis among them) maintain its superiority to the traditional interview. For example, Stanford administrators praised the school’s MMI program in a recent editorial:

Considering that as future physicians, candidates for medical school admission will be interacting with patients with diverse personalities and communication styles, we believe that it is essential to identify those skilled at interacting with multiple types of communicators…Scenarios are designed to probe candidates’ ability to reason; to describe and support a particular point of view; or to analyze and discuss an ethical dilemma. There are no “right answers”; the effective communication of critical thinking skills is of the essence.

I’ve interviewed under both formats. While I see strengths in both, I see many reasons why it would be a mistake for MMI to entirely replace the traditional interview:\

Only the traditional interview deters applicants from embellishing their credentials. Interviewers often try to confirm that elements of an application are truthful: if an applicant claims fluency in Spanish, an interviewer might introduce himself in Spanish. And for good reason–there is clear evidence that some medical students cheat. For example, researchers at Brigham & Women’s Hospital established that at least 1 in 20 applicants to their most competitive residency programs had plagiarized their personal statements. MMI refuses to evaluate applicants in the context of their application, a huge drawback that encourages cheating.

MMI questions can be found out ahead of time, and MMI can probably be coached. Many schools purchase a bank of MMI questions from McMaster University in Canada and reuse their questions day to day and year to year. Some schools request that interviewees sign non-disclosure agreements (NDAs) to keep the MMI scenarios they witness a secret. Not all applicants abide by the NDAs and a well-connected student would have little trouble finding out the questions. For that matter, a publicly available fact-sheet put out by McMaster lays out two of the MMI scenarios that I encountered on the interview trail.My impression is that MMIs are coachable: most of the stations involved extemporaneous acting, and having taken improvisational acting as an undergraduate was a tremendous help. After all, if MMI weren’t coachable, why would schools need interviewees to sign NDAs? I’ve already noticed that colleges are putting on “mock MMIs” for their pre-meds. The coachability of MMI will increasingly limit its ability to objectively evaluate interviewees.

The main research studies on MMI are not as relevant as they might appear. The main studies on MMI come from McMaster University (the school that profits by licensing MMI questions, which strikes me as a substantial conflict-of-interest). Researchers asked applicants undergoing traditional interviews to volunteer to participate in a trial MMI. Neither the student’s performance in the MMI nor his decision whether to participate would be considered in the admissions process. The study looked within this sample of volunteers and found that higher scores on the MMI moderately correlated with better evaluations on clinical clerkships. Universities cite this study as evidence that MMI ought to replace the traditional interview.The fallacy in citing this study is that it only examined applicants who performed well enough on the traditional interview to be admitted. The study does not tell us that MMI by itself is better than a traditional interview at assessing candidates. Rather, it suggests that succeeding on both correlates more strongly to good clerkship evaluations than does succeeding on the traditional interview alone. It’s not clear that MMI alone is any better than the traditional interview, and indeed, MMI alone may be much worse. The study is analogous to my asking, “of the girls who I would have an excellent time with on a long date (traditional interview), does how much I enjoy talking to them at a cocktail party (MMI) correlate to how good of a couple we would make?” Even if the answer is yes, it doesn’t mean that when I seek out mates I should abandon dates and only attend cocktail parties.

I agree with the Stanford administrators that the traditional interview and MMI test different skill sets: the interview requires depth and is a one-on-one exploration of someone’s person and character; the MMI assesses how well someone communicates and improvises in different situations. It makes sense that the best doctors have both of these skills, but I have not seen evidence that one set of skills is better than the other.

In sum, I’m not convinced that MMI ought to replace traditional interviews. MMI’s utility is unproven. Should more schools adopt MMI, I would expect students to adapt by obtaining schools’ questions ahead of time and by practicing their acting skills. Lastly, because MMI does not evaluate an interviewee in the context of his application, it offers additional temptation to embellish one’s qualifications.

MMI might be a great supplement to the traditional application process, but it strikes me as a poor replacement.

“Reflex Hammer” is a medical student who blogs at The Reflex Hammer.

Submit a guest post and be heard on social media’s leading physician voice.

email

Comments are moderated before they are published. Please read the comment policy.

  • http://www.facebook.com/rfdbbb Robert Bowman

    Academic focus is great, but is it practical? Admissions committees are attempting to be consistent. The MMI is yet another reaction to poor consistency plaguing admissions in multiple components. The fact is that grades (raw grades problematic), MCAT scores (McGaghie History MCAT, JAMA, Fairtest), statements (original?), and secondaries all have poor consistency with regard to prediction of physician or medical student outcomes. MMI attempts to improve consistency between interviewer findings, but this is only a part of admissions decisions.

    Other questions should be raised such as do committees treat all candidate the same in each component. Such internal consistency regarding the use of data is not only important in selecting the best candidates, it is important in accomplishing the school mission and in avoiding legal problems. Commonly members are willing to overlook some areas to admit those considered to be stronger in others.

    Mostly committees, advised by legal counsel, are unwilling to study admissions or they study admissions in traditional ways without considering important areas such as time management or controlling for factors during This results in poorly conceived research such as use of MCAT scores, race, ethnicity, and other markers when MCAT conflicts with race and ethnicity variables (should not include both in the same equation), the real reasons for Hispanic and Asian higher failure rates are most probably language issues not included in the study, and the study ignores nearly twice the failure rate for males. The study also uses questionable markers for income (Andriole, JAMA) What is happening in admissions is a progressive increase in highest income most urban children of professionals – those least likely to be aware of the basic needs of most Americans. This is made worse by only 68% of US graduate medical education positions filled by US born (90% of US pop). Translated this means those most exclusive from the US and other nations are replacing those most normal. And a major reason is that the US has been falling behind in multiple dimensions for decades. The 65% of US born children left behind further in higher education and medical school admission is just one indicator.

    • http://www.facebook.com/people/Terence-Ivfmd-Lee/1523282856 Terence Ivfmd Lee

      Before I can agree or disagree with you that grades, MCATs, statements and secondaries all have “poor consistency with regard to prediction of physician or medical student outcomes”, I’d be curious as to what measures of outcomes we are talking about. Maybe just simple graduation rate would be a reasonable measure of med student outcome. But what is the measure of good physician outcome? Is it based on ratings and reviews that the actual patients give? To me, that would be one good ultimate measurement. However, it seems that the data would be very hard to collect. We could poll 1000 patients in the general population as to how good their doctor is. Then we could go back and do statistical analysis on their GPA’s, MCAT’s, etc. Only then could we assess how good these criteria are at selecting the best candidates.

      • http://www.facebook.com/rfdbbb Robert Bowman

        It is difficult to compare outcomes. One of the few is in JAMA about communication skills by Tamblyn. Unfortunately this study found out years too late that the bottom 25% in communication skills areas had multiple times greater adverse events. Also this was examined 2 years into medical school.

        To evaluate clinicians common sense would indicate that it takes clinicians observing over a period of time aided by objective testing such as OSCEs. Those not demonstrating the communication skills would be subject to redirection in career.

        The ability to evaluate people over a short period such as a day or two can be demonstrated by Southwest Airlines, but an even shorter period of time and a shorter interviewer contact time is very limited. When medical schools make the commitment to collect a few hundred datapoints every 3 – 6 months for years as with top corporate efforts, they might improve in any number of areas.

  • http://twitter.com/TheReflexHammer Reflex Hammer

    Robert,
    I agree that the admissions process is flawed in the ways you described. And yet, is MMI helping? Is it more consistent and objective than other measures? It may well exacerbate disparities in ethnicity and income.

  • http://www.facebook.com/apobd Tofayel Ahmed Mallick

    Most of medical school want to admit smart student. So every applicant should gather proper knowledge of admissions process.

  • Michal Haran

    I think that an inherent problem with all the systems, is that no one has really defined what are the traits that make a good physician. Being a grade A medical student is probably not a requisite nor a guarantee of that. I personally think that the traits that make a good physician are very complex and seemingly contradictory and are basically an edifice of character you gradually have to build within yourself. 
    One very important trait which is not looked for and even discouraged in most interviewing systems is humbleness. I have learned through years of practice that being able to talk and know what to say and how is quite important, but being able to truly listen is even more so. 
    Very few physicians are capable (or realize the importance) of being knowledgeable and confident in front of their patient and yet being humble enough to learn from them. 
    most physician-patient relationships do not require rapid dating skills (unless you confine yourself to being an ER doctor) but rather a gradual and much deeper relationship that creates a true partnership over days months and years, not one of a car-dealer trying to sell a used car. 
    I too find the MMI very problematic because I am concerned that future physicians will not only be arrogant with regards to their knowledge and skills, but also see themselves as super-communicators. 
    As it is, many physicians think they have a better understanding of their patient’s illness and the way they should respond to it, so what will happen when they also see themselves as possessing superior humanistic skills? 

Most Popular