The Health Sciences Reasoning Test in the Pharmacy Admissions Process

INTRODUCTION

The Accreditation Council for Pharmacy Education requires PharmD degree programs to use a variety of quantitative and qualitative measures in evaluating candidates for admission. While ACPE standards do not specifically state that critical-thinking skills should be measured in the admissions process, Standards 11, 13, and 15 state that critical-thinking skills should be taught and assessed throughout the PharmD program.¹ Additionally, critical thinking is one of the educational outcomes established by the Center for Advancement of Pharmaceutical Education of the American Association of Colleges of Pharmacy (AACP).² Finally, participants in the 2009 AACP Curricular Change Summit concluded that the most essential components that pharmacy graduates should possess upon graduation were critical-thinking and problem-solving skills.³

Prior to 2012, the PharmD program at the College of Pharmacy, Nursing, and Allied Sciences, at North Dakota State University offered admission into its program primarily based upon quantitative measures such as grade point average (GPA) and Pharmacy College Admissions Test (PCAT) scores, combined with the PCAT essay, the Defining Issues Test (DIT) (a standardized test assessing the use of critical thinking in moral decision problems), and a structured interview that evaluated 6 qualitative (and/or noncognitive) attributes of its candidates (ethics, relevant life and work experience, emotional maturity, commitment to patient care, leadership, and understanding of the pharmacy profession⁴). Other criteria (for example, the applicant’s state of residency) were also included in the admissions process but were omitted from this study because they were not explicitly under the candidate’s control.

In 2012, the DIT was eliminated from the college’s admissions process because (1) the North Dakota State University Pharmacy Admissions Committee determined that the DIT scores were difficult to interpret on a question-by-question basis; and (2) the sheet on which students recorded their answers was sufficiently complex that a number of students miscoded their answers. Also, students commonly misinterpreted the intent of the test as a “test of their personal ethics” rather than a test of critical thinking applied to ethical problems. The Pharmacy Admission Committee’s decision was supported by Lobb and colleagues’ findings that DIT results did not correlate with subsequent academic performance.⁵ Additionally, the committee felt that it could more accurately assess moral development in the admissions interview.

The Pharmacy Admissions Committee emphasized selection of candidates with well-developed critical-thinking skills as there is an established relationship between critical thinking and other measures of academic performance. For example, performance on the California Critical Thinking Skills Test is a significant predictor of pharmacy students’ classroom and experiential GPAs.⁶ Also, performance on the Health Sciences Reasoning Test (HSRT) and PCAT scores are significantly correlated.^7-9 A masters of physician assistant program concluded that preprofessional student HSRT scores were modestly associated with program success as measured by GPA.¹⁰

Based on these considerations, the North Dakota State University Pharmacy Admissions Committee replaced the DIT with the HSRT as part of the admissions process in 2012. The HSRT contains 33 multiple-choice questions that are built around health care scenarios, but those questions do not require prior healthcare experience or knowledge. There is only 1 correct answer for each question and each question receives equal weight in the total score. Domains assessed on the HSRT include: Inductive Reasoning Contexts of Uncertainty, Deductive Reasoning Precisely Defined Contexts, Analysis and Interpretation, Inference, and Evaluation and Explanation. Student HSRT scores were tabulated and proportionally adjusted to ensure that the HSRT received the same weight in the admissions decision as the DIT did in previous admission processes. The HSRT was selected because its validity is comparable to that of other critical-thinking assessment tools, and its subscale content requires application of classic reasoning skills to contexts more appropriate for healthcare professionals than does the content of other tools.^7,8 No other significant changes in the admissions processes occurred during the study.

The studies cited pose questions that have not been adequately addressed in the health education literature. While critical thinking tests may be psychometrically valid and conceptually important screening devices to use in the admissions process, scores may be correlated with other traditional measures of academic success, including student GPAs. Thus, the question arises as to whether the inclusion of critical thinking tests adds any unique information to the admissions process. Additionally, if there is unique information contained in critical-thinking test scores, to what extent does it affect who is accepted into the professional program and which students are declined admission? This study addressed both of these questions. The primary goals of this study were to determine (1) whether the inclusion of a critical-thinking test such as the HSRT added any unique information to the admission process and (2) how much inclusion of the HSRT score impacted the outcome of the admissions process.

METHODS

In 2012, the Pharmacy Admissions Committee invited 122 candidates to participate in the interview process and complete the HSRT. Prior to taking the HSRT, an Institutional Review Board-trained faculty member who was not part of the research team verbally explained the desire of the researchers to use the students’ HSRT results to evaluate the usefulness of HSRT scores in the admissions process and gave the student an informed consent form to read and sign. Although completing the HSRT was a required part of the admissions process, students were under no obligation to consent for their score or other admissions data to be included in the research study. Whether they were admitted to the college was in no way impacted by their participation in the study.

Upon completion of the HSRT, all data reported to North Dakota State University was entered into a Microsoft Excel spreadsheet. This information was then combined with other evaluation criteria used in the admissions process including the student’s GPA in core preprofessional pharmacy courses, PCAT scores, and interview scores. As explained earlier, additional criteria (eg, the applicant’s state of residency) were included in the admissions process but omitted from the study because they were not explicitly under the candidate’s control. Data of candidates who did not wish to participate in the study were removed from the database, and the personal information and HSRT scores data of candidates who consented to participate were de-identified. North Dakota State University Institutional Review Board approval was obtained prior to conducting the research.

Any competitive admissions process is, by definition, a ranking procedure. However, the North Dakota State University Pharmacy Admissions Committee made a decision several years ago to keep the exact process used for ranking candidates confidential. For the purposes of this study it is sufficient to state that the evaluation criteria were aggregated in a way that candidates were sorted from most qualified to least qualified, and the 85 candidates who ranked the highest were offered admission. To preserve the anonymity of the college’s actual process used to rank candidates and because any combination of aggregation processes can yield the same ordering, we focused instead on the candidates’ rankings. To account for the impact of the HSRT on the admission process, 2 sets of rankings were created. The first set ranked the candidates based on the actual process used by the Pharmacy Admissions Committee. The second set ranked candidates using the same process, but excluded the HSRT score as an evaluation criterion. In both cases, the top candidate was ranked as 1, the second-best candidate was ranked as 2, and so on. If the HSRT score did not add any unique information to the admissions process (the null hypothesis), then the ranking of candidates would remain unchanged whether or not HSRT scores were included or excluded from the formation of these rankings. If the null hypothesis was rejected, then inclusion of HSRT scores would lead to a fundamentally different ranking of candidates. To evaluate the null hypothesis, median sign and Wilcoxon 2-sample (matched pairs) nonparametric hypothesis tests were used.

To evaluate the degree of uniqueness of the HSRT score as an admission criterion, and to identify the degree to which it impacted the outcome of the admissions process for students, a two-step process was used. The relative impacts of the HSRT score can be evaluated at the mean by examining the rank-order (Spearman) correlations between the 2 ranking variables, as well as between each of the candidate rankings and specific admissions criteria, such as GPAs, PCAT scores, etc. To evaluate the impact of the HSRT score on specific groups of students, cross-tabulations were created to disaggregate students based on where they placed in the rankings. For example, a 2x2 cross-tabulation was constructed to identify the top 85 students when the HSRT score was included and the top 85 students when it was excluded. If the HSRT score impacted the outcome of the admissions process, we expected the same students to be ranked in the top 85 applicants (and the bottom 27 applicants) regardless of whether HSRT scores were included. By altering the ranking used as the cutoff point, it was possible to determine to what extent specific groups of students were affected by the inclusion of HSRT scores. We also studied the group of students who were “on the bubble” in the admissions process (where we defined the bubble as the last 10 students offered admission and the first 10 students not offered admission) vs those students who were not on the admissions “bubble.” In all cases, statistical significance was assessed at the 5 percent level, and all calculations were conducted using the SAS, version 9.3 (SAS, Inc., Cary, NC).

RESULTS

All 122 admissions candidates consented to participate in the study. Table 1 contains descriptive statistics for several key admission criteria and candidate ranking information. Panel A in Table 1 provides descriptive statistics for PCAT national percentile scores, PCAT essay scores, candidate GPAs in core preprofessional pharmacy courses, candidate interview scores, and candidate HSRT scores. As noted earlier, these scores are presented in their “raw” or untransformed formats to avoid disclosing the college’s pharmacy admission process.

Panel B in Table 1 provides information on the ranking of candidates with and without the inclusion of the HSRT score. The means for both ranking variables was approximately 61.5. The Wilcoxon signed rank test (p=0.62) and the median sign test (p=0.22) both failed to reject the null hypothesis of no difference in rankings across these 2 variables.

Spearman (nonparametric) correlations were used to assess the relationship between each of the ranking variables and admission criteria. The correlation between the 2 ranking variables is 0.98 and highly significant (p<0.001). Correlations between each of the ranking variables and the individual admission criteria are negative (because better performance led to a higher ranking, which is expressed as a smaller number) and significant (p<0.050 for each of these correlations). PCAT cumulative percentile scores exhibited the strongest correlations between rankings calculated with and without the HSRT score (-0.734 and -0.678, respectively). Core GPAs exhibit the second highest correlations, with rankings at -0.462 and -0.520, respectively, while the correlations between the PCAT essay and the rankings (with and without the HSRT score included) are -0.289 and -0.299, respectively. Even when the HSRT score was not included in the ranking process, student HSRT scores exhibited a significant correlation with candidate rankings (correlation = -0.195; p=0.031). While interview scores were significantly correlated with candidate rankings, the magnitude of the correlation was rather small, at -0.199 including the HSRT score and -0.227 excluding the HSRT score.

Spearman correlations also provided insights into the amount of information overlap between individual admission criteria. For example, the HSRT score was significantly and positively correlated with the PCAT cumulative percentile score (correlation = 0.483; p<0.001), but was not significantly correlated with candidate core GPAs, interview score, or PCAT essay scores. Student core course GPAs were not significantly correlated (at the 5% significance level) with any of the other admission criteria, while the interview score exhibited only one significant correlation, and that was with the PCAT essay (correlation = 0.210; p = 0.020).

Only 3 students moved into (or out of) the top 85 based on whether the HSRT score was included. That is, inclusion of the HSRT score in the candidate ranking process only impacted acceptance decisions for 6 students; 3 who were accepted and 3 who were not accepted because of the inclusion of the HSRT score in the evaluation process.

Candidates ranked between 76 and 95 effectively represented the last 10 students accepted into the professional program, and the first 10 applicants who were declined admission. Among these students, the inclusion or exclusion of the HSRT score in the evaluation process played a more prominent role. For candidates “on the bubble,” there were 8 applicants whose rankings moved them into this group if the HSRT score was used in the candidate ranking process. These same students would have been excluded from the admissions “bubble” (and replaced by a different group of 8 students) had the HSRT score not been used in the ranking process.

DISCUSSION

Inclusion of HSRT scores did not significantly affect admission decisions for the vast majority of applicants who applied to North Dakota State University’s professional pharmacy program in 2012 or for those students who were “on the bubble” and competing for the final 10 spots in the professional program. Thus, the value of the HSRT score as an admissions criterion appears to be limited.

Perhaps of even greater concern was the finding that HSRT scores were largely redundant of PCAT cumulative percentile scores. The correlation between the 2 scores was nearly 50%. This is troubling because the PCAT is explicitly designed to cover knowledge of important scientific principles that form a foundation for professional pharmacy programs. As noted earlier (and aside from critical thinking), the HSRT has no academic discipline-specific content, which implies that as much as half of a student’s success on the PCAT is not based on what they actually know about core science disciplines (chemistry, biology), but on their ability to think critically and interpret test questions (which is a very different set of competencies). This is supported (but not conclusively proven) by the weak correlation between PCAT cumulative percentile scores and candidate core GPAs, whose courses largely comprise the academic discipline-specific content covered by the PCAT examination. Thus, the difficulty lies not so much with the HSRT as with the interpretation of PCAT examination scores. The Pharmacy Admissions Committee continues to use both the HSRT and the PCAT examinations, but it has decided to reassess the relative weight given to scores on each of these examinations in the candidate ranking process to ensure that the final candidate rankings do not overemphasize critical-thinking skills at the expense of other desired applicant characteristics and competencies.

When the HSRT scores were removed, the remaining evaluation criteria were not highly correlated. This suggests that each of these criteria adds some unique content to the admissions evaluation process. Unfortunately, interview scores were not highly correlated with final evaluation rankings. Further study is needed to understand this trend. Possible explanations include (but are not limited to) that interviews received too little weight in the formation of candidate rankings compared to other criteria, or that students’ scores on their interviews were relatively consistent with their other rankings relative to their peers such that the impact of the interviews on candidates’ rankings was mitigated.

This study is subject to a number of limitations. The data were collected from only 1 academic year’s applicants from 1 pharmacy program. If this candidate pool differed from that of other years at North Dakota State University, or was fundamentally different from candidate pools at other colleges and schools of pharmacy, then our results may not be generalizable to other groups of candidates or to other institutions. Additionally, in order to maintain the privacy of the admissions process, the authors were unable to divulge the college’s exact aggregation process used to establish candidate rankings. Moreover, we focused on those criteria that were within candidates’ ability to control (grades, test scores, etc) rather than on factors outside of their control (eg, state of residency). If we had been able to analyze the full set of admissions criteria and procedures, additional inferences may have been found that are not contained in this study.

CONCLUSIONS

The HSRT can be an effective method to evaluate a candidate’s critical thinking ability as part of the admissions process for a PharmD program. However, the usefulness of the HSRT as an admissions screening tool is mitigated by information redundancies with other evaluation criteria. In particular, HSRT scores correlated significantly with cumulative PCAT scores.