Warning to Pre-Med Students: Standardized Situational Judgement Tests Coming to a Medical School Near You!

Standardized Situational Judgment Tests, or SJTs

The Association of American Medical Colleges (AAMC) is currently conducting research for the development of a new standardized Situational Judgment Test for medical school entrants. Right now, they’re carrying out trials in a handful of schools and hoping to come up with a major new part of med school admissions that we could see rolled out across North America within the next few years. In this article, we’re going to explore why the AAMC are working on their new test and, more importantly, what it means for you, the candidate.

What exactly is a Situational Judgment Test, anyway? 

Put simply, situational judgement tests are aptitude tests designed to test non-cognitive abilities like problem solving, decision-making, and interpersonal skills. Usually, candidates are shown a video or a written prompt and asked how they would respond to the scenario shown. They have been around since the 1940s (yes, pretty old technology), but in recent years, medical schools have seized on them as a way to test candidates: they hope to find out how you might act as a real doctor; they want to see if you’ll respond to their “people skills” training; and they believe that these tests are popular with candidates. Since medical training puts such a heavy emphasis on developing your ability to work with others and treat your patients with empathy and care in tough situations, the situational judgement test seems like the perfect way to separate hundreds of applicants with great GPAs and MCAT scores! Indeed, if you’ve already been getting ready for the admissions process, you might be familiar with situational judgement tests like CASPer and multiple mini interview (MMI), which are already used by dozens of schools across America.

If CASPer and MMI (and plenty of others!) already exist, why is the AAMC investing money in creating a new test? 

We at BeMo suspect that there are some big reasons behind this. The AAMC is a big organization with plenty of influence over the admissions process nationwide, so the SJT they produce might be in use by just about every school in the country—think of it like an MCAT for cognitive skills. Moreover, CASPer and MMI are run by private for-profit companies. Producing an in-house test through this research is likely a great way for the AAMC to own and cut costs and the middle men in the admissions process.

Why use the situational judgement tests at all? Why not just assess candidates on the hard work they’ve put into scoring top grades at university or college and in the MCAT? Why not just ask me in an interview why I’m a great candidate?! 

These are really important questions. The ever changing med school admissions process can be infuriating for great candidates fighting to make themselves stand out from the crowd.

Well, there is some evidence that situational judgement tests are an effective tool in assessing candidates’ interpersonal abilities. Some scientific reviews published show that situational judgement tests do outperform personality tests and traditional interviews at this. But better doesn’t mean the best. There is very little evidence of situational judgement tests’ long-term effectiveness (and the evidence that does exist is sparse at best). In fact, we at BeMo know that barring a small increase in testing accuracy, the situational judgement test is surrounded by myths that are debunked by the scientific evidence:

Myth number 1: “Situational judgement tests aren't affected by coaching.” 

False! The evidence shows that situational judgement tests might be less susceptible to coaching than other forms of assessment, based on the coaching methods designed by the study authors, but their conclusions are not inclusive of all coaching programs since the researchers failed to test their hypothesis with other existing coaching programs. (In fact, we have evidence that situational judgement tests like CASPer and MMI are MORE susceptible to coaching with specific training and that's why our programs are backed by our money back guarantee.) This is something that situational judgement test proponents keep missing. They are trying to create something that's "immune to coaching" not realizing that all human behavior is shaped by experience and learning, especially personality traits. Importantly, this way of thinking is dangerous and perhaps discriminatory. It's utterly false to imply that certain people are born with personal characteristics and those that are not born with them, will never learn them. You see the fundamental flaw in this reasoning?

Myth number 2: “The situational judgement test is more fair to racialized and other minority candidates.” 

False! Time and again, research shows that white test takers outperform Black, Hispanic and Asian test takers. Wealthier candidates have the chance to develop interpersonal skills tested by situational judgement tests through unpaid volunteering and work experience, and an educational background that offers plenty of opportunities outside the classroom. When BeMo is striving to make admissions tests fairer, introducing another flawed assessment method isn’t great news for minority candidates. This is why BeMo has started development of a scientifically sound and fair applicant screening tool. In fact a significant portion of our profit goes into the research and development of this tool. 

If the situational judgement tests only has negligible benefits, why use it? Why doesn’t the AAMC use its clout to develop something brand new? What is the REAL motivation chasing after an old technology that remains unproven?

Simple. With thousands of applicants every year—and an acceptance rate as low as one in fifty—med schools need a cheap way to sort their candidates. And situational judgement tests, which can be administered offsite in assessment centers with minimal expense, are just that: cheap. Reading personal statements, meeting candidates, and running traditional interviews, on the other hand, is time consuming and very costly.

What does this mean for you, the candidate?

Fundamentally, this means more preparation and a whole new system to learn—and possibly even saying goodbye to personal statements altogether. But don’t worry. We have you covered!

We understand that med school admissions committees want to place too much emphasis and ascribe far too much accuracy to standardized tests like situational judgement tests. What, really, is the difference between an 8 and a 9 on a CASPer or an MMI station? A headache. An attack of anxiety. A misplaced sentence or two in your introduction or conclusion. That’s no way to assess students who have spent hundreds of hours on shadowing placements, lab projects and achieving an outstanding GPA across four years of undergraduate education using such tests. You can be coached to minimize the chance of making these kinds of errors in SJT performance (even if the assessment still isn’t fair!).

Plenty of mid-tier schools, moreover, will continue to look for well-rounded candidates. Jacobs School of Medicine at Buffalo U, for example, state this outright in their selection criteria: “Many medical schools use computer cutoffs to whittle down their applicant pool…That’s not our policy.” If you’re worried that situational judgement tests won’t give you a chance to show your best self, then look out for schools like this when you’re searching for places to apply.

The best thing of all, though—especially if you’re currently a premed student—is that, ultimately, these new hoops to jump through aren’t changing the rules of the game. The AAMC’s report on their situational judgement test research shows that they’re still looking for the same competencies that you’ve been (hopefully!) thinking about already: “Service Orientation, Social Skills, Cultural Competence, Teamwork, Ethical Responsibility to Self and Others, Reliability and Dependability, Resilience and Adaptability, and Capacity for Improvement.” So if you’re a premed student right now, don’t change your plans! You should still be looking to develop these qualities through volunteering, work experience, and other extracurricular activities. Why not begin your preparation right now by making a list of times when you’ve displayed the eight skills the AAMC lists above? If you find you’re stuck for ideas, get out there and start working on building your skillset so you’ll know what kind of interpersonal qualities to draw on in any tests you take.

And rest assured, we will continue our search for a better, scientifically sound and fair admissions screening tool and until then, we will support each of you as you navigate your way through the current admissions practices. 

To your success,

Your friends at BeMo

BeMo Academic Consulting

References: 

• Lievins, F. Adjusting Medical School Admission: Assessing Interpersonal Skills Using Situational Judgement Tests. Medical Education. 47(2): 182-189.
• Patterson, F. et al., Evaluations of Situational Judgement Tests to Assess Non-academic Attributes in Selection. Medical Education. 46(9): 850-868.
• Patterson, F. et al., Situational Judgement Tests Represent a Measurement Method and Can Be Designed to Minimise Coaching Effects. Medical Education. 47(2): 220-1.
• Cullen, M.J., et al., Threats to the Operational Use of Situational Judgment Tests in the College Admission Process. International Journal of Selection and Assessment. 14(2): 142-155.
• Whetzel, D.L., Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis. Human Performance: 21(3). 291-309.