How the CASPer Test is Really Scored

Who are the CASPer evaluators?

As part of the first applicant cohort required to take the newly-introduced CASPer test, I didn't thoroughly review how the test was scored beforehand. Initially, I perceived CASPer as a test that required no special preparation—an opportunity to naturally showcase my personal qualities. I believed that since CASPer assessed traits like empathy and ethics rather than academic knowledge, there wasn't a need for specific preparation beyond being authentic to myself.

However, in hindsight, I realize that gaining a deeper understanding of how evaluators analyze and score CASPer responses could have better guided my approach. So here’s a quick introduction to who will be assessing your CASPer answers:

Each CASPer assessor is given an online orientation and training session on the logistical aspects of marking CASPer tests (e.g. how to use the secure login system, how to ensure they have the appropriate software for running the media clips, etc.) and the criteria they must apply to each response when grading it. This is the aspect of the test that is claimed to minimizes subjective judgment in favor of objective assessment criteria (notice that we use the term “minimize”, not “remove”). Tests with written answers marked by humans, even ones who are experts in this field, will never be 100% objective.

The assessors for the test are individuals from all walks of life. They include medical professionals, medical students and residents, members of the public, undergraduate students, or anyone provided with a registration code which allows them to go through the online training process and become a CASPer assessor.

The online training program gives an overview of the test and how it should be scored. Recall that, each assessor only sees one scenario for each applicant and they are not aware of the applicant’s personal identifiers. Thus, each applicant’s CASPer test will be seen by 12 distinct assessors, helping the claimed increase in the inter-observer reliability of the test. The assessors are given general information about the concepts important to each scenario and major ideas that should be discussed, but they are not provided with an answer key.

As the assessors analyze your responses, they will be applying the CASPer marking criteria to each answer, which is why effective CASPer test prep with professionals and knowing how long it takes to prepare for CASPer is key to succeeding. Application of the criteria will vary from question to question as, you may be given a scenario centered around informed consent, but then be asked a question on patient autonomy such as the following:

Example: 

“You are a surgical resident in an outpatient clinic seeing a patient who has been diagnosed with ovarian cancer that requires surgical intervention. You are discussing the details of the surgery, and explain that both open and laparscopic oophorectomy are possibilities in her case."

1. How would you ensure any consent this patient provides for surgery is informed?

After your discussion, the patient expresses that she does not wish to undergo surgery, stating that she would like to give chemotherapy a chance first.

2. How would you respond to this decision?”

First note that advanced technical knowledge of “open and laparscopic oophorectomy” is irrelevant and if you see a technical term in CASPer it is just meant to shake you up a bit and stress you. As you can see, this sample question encompasses numerous areas and deals with several ethical concepts. While the first question specifically asks about informed consent, the second then adds a dimension of patient autonomy and could even be argued to branch into Evidence-based Medicine.

From an assessors’ point of view, the more dimensions of the question you can demonstrate that you have recognized and considered, the better the grade you will receive. In this example, you would need to appreciate what important aspects of the patient encounter contribute to informed consent (e.g. proposed procedure, details, possible alternatives, risks and complications, consequences of doing nothing, etc.) and recognizing that, provided the patient has the capacity to make their own decisions about their healthcare, that making the decision to decline surgery, although potentially inadvisable within the context of their illness, is a patient’s inalienable right as part of their autonomy.

The grading of CASPer responses is done using a numerical Likert-style scale. The scale runs from 1 to 9 with 1 signifying a “unsatisfactory” response and 9 signifying a “superb” or superior one.

What if you miss one or two questions in a CASPer scenario in the typed response section of the test? 

Although all of your responses to the questions for the same scenario will be graded by the same assessor, the score that you receive is representative of your overall performance on that station. For instance, if during one of the scenarios you take a lot of time to provide a well-thought out and mature answer to only one of the questions and only have limited time to answer the remaining questions, you can still score well on that station, granted that the answer you provided to question one was strong, appropriate and professional. One high quality response that addresses the scenario in a comprehensive way can still score highly if you miss one or two of the questions, but I want to emphasize that you should strive to answer all the follow-up questions for each station. During my prep and even on the test I did miss some of the questions, but I made sure that I addressed them in my overall answer thus making sure I covered all my bases. 

Assessors are not required to comment on the score they give or provide any feedback justifying the score. However, they do have an option to flag a particular answer and/or indicate the possibility of technical glitches that may have led to a poor performance at a given station. (N.B: The CASPer system is claimed to be designed to detect technical problems and report it back to the testing center). This highlights your response when university admissions staff review your results. 

What type of a CASPer response constitutes a red-flag?

Here’s another tidbit of knowledge I gained as I learned how the CASPer test is scored:

An assessor will flag a typed or a video answer for the following reason:

"The response given indicates a dangerous, unethical, unprofessional or potentially harmful approach to the scenario that displays a clear deficiency in the candidate’s ability to appreciate the complex ethical issues raised."

In the event that a response is flagged, the assessor is then obligated to comment on why they flagged the answer. The other reason assessors must elaborate on their flagging of an answer is to justify their reasons for flagging it. This mechanic acts as something of a fail-safe for a candidate’s application process – the red-flagged response and the assessor’s comments will be reviewed by an admissions officer or committee. If it is felt that the assessor was excessively harsh in their judgment of the response, and provided the remainder of the candidate’s responses scored well, this red flag can be overlooked within the context of that candidate’s overall application.

However, multiple red flags on your test score are likely to result in rejection of your application. You may be asking yourself, “What happens if my test is marked by an assessor with excessively high standards?” I totally understand your panic! But this is unlikely because importantly each scenario is marked by a different assessor making it very unlikely that you ended up with 12 assessors that were all difficult graders. Furthermore, given the instruction CASPer assessors receive with regards to objective marking criteria, the marking system is set up to account for the possibility that some assessors have more critical approaches than others.

When marking CASPer, an assessor will usually be given a series of candidates’ responses to the same scenario and questions. The advantage in doing this is two-fold: the assessor can apply the same criteria to multiple applicants’ responses rather than having to shift focus between subject matter every time AND you have a different assessor marking each question rather than one continually harsh (or lenient) assessor for your whole test. This is deemed fairer and gives a more accurate representation of your interpersonal skills and ability to respond to the challenging scenarios CASPer presents.

Here's what you need to AVOID when taking CASPer!

How CASPer scores are released to you

This is probably the most burning question you have about how the CASPer test is scored! Well, there's the truth:

If you take the CASPer test now, you will be able to receive your quartile scores, but not the raw scores or the z-scores. This was not the case when I took CASPer - we got no scores! So what do quartile scores mean? Let’s find out.

When you take the CASPer test, you are evaluated in comparison to your peers who took the same test type on the same test date and time as you. This means that your score is relative to the performance of everyone else who was taking the same CASPer test form at the same time as you. This is exactly what quartile scores demonstrate. Every student’s score is sorted into one of the following quartiles:

• 1st quartile includes applicants whose score is in the 0-24 percentile in comparison to their peers
• 2nd quartile includes applicants whose score is in the 25-49 percentile in comparison to their peers
• 3rd quartile includes applicants whose score is in the 50-74 percentile in comparison to their peers
• 4th quartile includes applicants whose score is in the 75-100 percentile in comparison to their peers.

To put it more plainly, if you scored in the 4th quartile, it means that you scored higher than at least 75% of your peers who took the test at the same time as you. Correspondingly, if you scored in the 1st quartile, you scored lower than 75% of your peers; if you scored in the 2nd quartile, you scored higher than at least 25% of your peers but less than 50% of your peers; and if you scored in the 3rd quartile, you scored higher than at least 50% of your peers and less than 25% of your peers.

It’s important to remember that your quartile score represents how you performed in comparison to others. It does not represent the percentage of questions you got “wrong” or “right”. 

Most likely you will receive your quartile score one month after you complete the CASPer test, except applicants for US Graduate Medical Education programs, who will receive their quartile scores at the end of October.

And while you may be glad to hear that you do get some sort of score to see how you did on CASPer, this information should not affect your decision to apply to programs of your choice no matter what quartile score you receive. Remember that medical schools and most other professional programs have not revealed how they choose to use the CASPer score in the admissions process. Some medical schools may use them as simply additional pieces of information about you, others may review the scores to decide who gets the interview invites, while others may use it for final admissions decisions. You simply do not know, so apply to schools where your academic achievements and background fit the school’s profile regardless of what quartile score you get!

How CASPer scores are released to schools

While you receive your quartile scores, the schools receive z-scores. Just like a quartiles score, a z-score tells the schools where your score stands in comparison to all CASPer takers that took the test on the same day and time as you. The z-score shows how many “points” you are away from the mean score, i.e., where you are in the distribution of scores across all candidates. For example, if your z-score is 0, it means that your score is the same as the mean. If your score is +1 that means that you scored higher than the mean by one standard deviation point. A score of -1 would indicate that you scored lower than the mean by one standard deviation point. So instead of sorting you into quartiles that indicate how your score stands in percentile comparison to other quartiles, the z-score simply shows where you stand statistically in comparison to other test-takers.

Your z-score is released directly to the programs you indicated in your CASPer profile 2-3 weeks after you complete the test. Each school has its own way of using your score in the admissions process. Some programs may post this information on their official websites, so make sure to check them out. 

How can you prepare for your CASPer test?

As always, practice and expert feedback, just like any other skill set in life. 

FAQs

To your success,

Your friends at BeMo

BeMo Academic Consulting

Disclaimer: CASPer is a claimed trademark of McMaster & Altus. BeMo does not recommend, endorse nor affiliate with CASPer, Altus or McMaster and vice versa.

Want more free tips? Subscribe to our channels for more free and useful content!

TikTok

YouTube

Instagram

Apple Podcasts

Spotify

LinkedIn

Twitter