How the CASPer Test is Marked/Scored:
Possibly the most frequently asked question about CASPer is, “How is CASPer scored?”. While this isn’t a secret per se, it is not something medical schools will actively provide you with, although the publicly available research by CASPer creators provide many hints. Keep in mind that knowing how the scoring system works is not the same (and not even as important) as knowing what the assessors are looking for when scoring the questions. It also shouldn’t change your approach to the questions or how you construct your responses.
Each CASPer assessor is given an online orientation and training session on the logistical aspects of marking CASPer tests (e.g. how to use the secure login system, how to ensure they have the appropriate software for running the media clips, etc.) and the criteria they must apply to each response when grading it. This is the aspect of the test that is claimed to minimizes subjective judgment in favor of objective assessment criteria (notice that we use the term “minimize”, not “remove”). Tests with written answers marked by humans, even ones who are experts in this field, will never be 100% objective.
The assessors for the test are individuals from all walks of life. They include medical professionals, medical students and residents, members of the public, undergraduate students, or anyone provided with a registration code which allows them to go through the online training process and become a CASPer assessor.
The online training program gives an overview of the test and how it should be scored. Recall that, each assessor only sees one scenario for each applicant and they are not aware of the applicant’s personal identifiers. Thus, each applicant’s CASPer test will be seen by 12 distinct assessors, helping the claimed increase in the inter-observer reliability of the test. The assessors are given general information about the concepts important to each scenario and major ideas that should be discussed, but they are not provided with an answer key.
As the assessors analyzes your responses, they will be applying the CASPer marking criteria to each answer. Application of the criteria will vary from question to question as, you may be given a scenario centered around informed consent, but then be asked a question on patient autonomy such as the following:
“You are a surgical resident in an outpatient clinic seeing a patient who has been diagnosed with ovarian cancer that requires surgical intervention. You are discussing the details of the surgery, and explain that both open and laparscopic oophorectomy are possibilities in her case."
1. How would you ensure any consent this patient provides for surgery is informed?
After your discussion, the patient expresses that she does not wish to undergo surgery, stating that she would like to give chemotherapy a chance first.
2. How would you respond to this decision?”
First note that advance technical knowledge of “open and laparscopic oophorectomy” is irrelevant and if you see a technical term in CASPer it is just meant to shake you up a bit and stress you. As you can see, this sample question encompasses numerous areas and deals with several ethical concepts. While the first question specifically asks about informed consent, the second then adds a dimension of patient autonomy and could even be argued to branch into Evidence-based Medicine.
From an assessors’ point of view, the more dimensions of the question you can demonstrate that you have recognized and considered, the better the grade you will receive. In this example, you would need to appreciate what important aspects of the patient encounter contribute to informed consent (e.g. proposed procedure, details, possible alternatives, risks and complications, consequences of doing nothing, etc.) and recognizing that, provided the patient has the capacity to make their own decisions about their healthcare, that making the decision to decline surgery, although potentially inadvisable within the context of their illness, is a patient’s inalienable right as part of their autonomy.
The grading of CASPer responses is done using a numerical Likert-style scale. The scale runs from 1 to 9 with 1 signifying a “unsatisfactory” response and 9 signifying a “superb” or superior one.
What if you miss one or two questions in a scenario?
Although all of your three responses to the questions for the same scenario will be graded by the same assessor, the score that you receive is representative of your overall performance on that station. For instance, if during one of the scenarios you take a lot of time to provide a well-thought out and mature to only one of the three questions and only have limited time to answer the remaining two, you can still score high on that station, granted that the answer you provided to question one was strong, appropriate and professional.
Assessors are not required to comment on the score they give or provide any feedback justifying the score. However, they do have an option to flag a particular answer and/or indicate the possibility of technical glitches that may have led to a poor performance at a given station. (N.B: The CASPer system is claimed to be designed to detect technical problems and report it back to the testing center). This highlights the candidates’ response when university admissions staff review their results.
What type of a CASPer response constitutes a red-flag?
An assessor will flag an answer for the following reason:
"The response given indicates a dangerous, unethical, unprofessional or potentially harmful approach to the scenario that displays a clear deficiency in the candidate’s ability to appreciate the complex ethical issues raised."
In the event that a response is flagged, the assessor is then obligated to comment on why they flagged the answer. The other reason assessors must elaborate on their flagging of an answer is to justify their reasons for flagging it. This mechanic acts as something of a fail-safe for a candidate’s application process – the red-flagged response and the assessor’s comments will be reviewed by an admissions officer or committee. If it is felt that the assessor was excessively harsh in their judgment of the response, and provided the remainder of the candidate’s responses scored well, this red flag can be overlooked within the context of that candidate’s overall application.
However, multiple red flags on a candidate’s test score are likely to result in rejection of the application. You may be asking yourself, “What happens if my test is marked by an assessor with excessively high standards?” This is unlikely because importantly each scenario is marked by a different assessor making it very unlikely that you ended up with 12 assessors that were all difficult graders. Furthermore, given the instruction CASPer assessors receive with regards to objective marking criteria, the marking system is set up to account for the possibility that some assessors have more critical approaches than others.
When marking CASPer, an assessor will usually be given a series of candidates’ responses to the same scenario and questions. The advantage in doing this is two-fold: the assessor can apply the same criteria to multiple applicants’ responses rather than having to shift focus between subject matter every time AND the applicants have a different assessor marking each question rather than one continually harsh (or lenient) assessor for their whole test. This is fairer and gives a more accurate representation of the candidates’ interpersonal skills and ability to respond to the challenging scenarios CASPer presents.
How can you prepare for your CASPer test?
As always, practice + expert feedback, just like any other skill set in life.
To your success,