The for-profit company that administers the CASPer test, recently released a study on the effects of coaching on CASPer performance. In the following, we share our opinion about the study and note that their claim that third party prep does not improve performance directly contradicts our own findings that students’ CASPer practice scores increase significantly with expert feedback. A close examination of their study appears to suggest why our results differ and why their design may not be able to address the question they set out to answer. The major points from our foray into this study are outlined below. For clarity, note again that the following is our opinion and our interpretation of the published study by the administrators.
Disclaimer: CASPer stands for computer-based assessment for sampling personal characteristics and is claimed to be a trademark of McMaster & Altus. BeMo does not recommend, endorse nor affiliate with CASPer, Altus or McMaster and vice versa. BeMo only provides preparation services and practice tests. To take CASPer, contact Altus directly.
>>Want us to help you get accepted? Schedule a free strategy call here.<<
Point 1: Some may argue that the investigators appear to misunderstand the nature of third-party test prep – at least when it comes to BeMo’s CASPer prep.
The CASPer test administrators appear to believe that commercial test prep courses undermine the validity of CASPer because they teach students test-taking strategies rather than the concepts and skills that the test aims to evaluate (e.g., ethical decision-making, empathy).
One might argue, as the investigators themselves admit, their study was not designed to evaluate what kind of coaching students received since participants in the survey only indicated whether or not they used a third-party test prep course, without any information about the contents of the course.
Had the investigators included more detailed questions about the nature of the third-party test prep courses students used, they would discover that their assumptions on this matter do not hold. While we cannot speak for other test prep companies, at BeMo, we focus on teaching our students lifelong skills. We help students improve their interpersonal skills, ethical reasoning, and professionalism by teaching them to identify and navigate challenging, real-life scenarios. Through working with us, they are then able to ace situational judgment tests like CASPer because they have grown in their maturity and ability to make sound decisions.
Point 2: Some may argue that the investigators appear to misunderstand the source of bias in the admissions process.
The administrators appear to infer that situational judgment tests like CASPer are not biased against low socioeconomic status individuals, while third party test prep companies advantage those with more financial means.
While they do not produce any evidence in the form of research studies indicating such claims, it is important to note that other studies show the reverse. For example, studies showed that use of admissions prep is not correlated with wealth. We do not find this surprising because the cost of test prep is but a fraction of tuition costs and other living expenses. Applicants are aware of this as well as the competitiveness of admissions, so they are smart enough and motivated enough to invest in themselves. Every great performer in the history of man kind has taken advantage of experienced mentors. Think Einstein, Aristotle, Alexander the Great to Steve Jobs, Serena Williams and any other high achiever for that matter. Having a mentor is a must. In fact going to school itself is a form of seeking mentorship and schools charge tens of thousands of dollars.
On the other hands, studies have also shown that there is a positive correlation between scores on situational judgement tests and socioeconomic status. Similar findings have been found for gender bias. This may not be surprising since any form of human behavior is learned behavior, and the traits tested on SJTs are no different. Nobody is born with situational judgement test skills. Individuals from higher socioeconomic status may score higher on such tests because perhaps their upbringing allows them to learn the common socially acceptable behaviors that are assessed on such tests.
This is why our mission is to reduce the gap by making sure everyone knows how to answer such questions, regardless of gender, socioeconomic, cultural, or racial background.
Point 3: Importantly, some may argue further that the study design appears to invalidates the claims about coaching effects on CASPer.
The results of the administrator's study showed that most of the prep strategies had a significant positive impact on test scores but that participation in a third-party test prep course had a significant negative impact. We find it difficult to take this result seriously for the following reasons.
First, only 4% of the 16,642 respondents to their survey indicated that they used a third-party CASPer test preparation course. This is an incredibly low number that is likely inaccurate.
Second, since the administrators appear to denounce the use of third-party preparation courses for CASPer, one may argue that test-takers might be reluctant to indicate their use of such prep courses for fear that it may impact their score. In an ideal study, the participants would be selected at random by a neutral third party so that they are representative of the population of test-takers, and there would be no pressure to give a specific response.
Third, while the investigators note that participation in third-party test prep only accounts for 2% of the variance in scores, we would like to highlight that, according to the adjusted R2 value given in their own results, the amount of score variance accounted for by all factors combined in their study is a mere 4%. That means that 96% of the difference in CASPer score between applicants is accounted for by factors outside of their study, a point they don’t appear to discuss.
Most importantly, they make their claims based on final test scores only. However, the only way to assess the impact of coaching is to first establish a baseline score and compare that to scores received after coaching. Note that the administrator of CASPer do not have access to any students’ baseline scores before preparation and therefore they are unable to conduct such a study and make any claims about impact of coaching. In our own study we tracked the practice scores of randomly selected participants using BeMo’s CASPer prep program as they progressed from their first realistic CASPer practice test to their final realistic CASPer test. This allowed us to directly compare the practice score of each applicant before and after coaching. Our results indicate that there was a significant increase in applicants’ practice scores, which demonstrates that applicants can improve significantly with proper coaching on life skills and professionalism – at least in practice.
The bottom line is this: Take charge of your own future, and don’t hesitate to invest in preparation for CASPer and any other component of your application to ensure that you have the best chance of success. The form of preparation you receive does not have to be from BeMo but whatever you do, make sure you prepare well in advance.
Like our blog? Write for us! >>
Have a question? Ask our admissions experts below and we'll answer your questions!
Comments