APPE Evaluations are Positively Associated with MMI, Pre-pharmacy GPA and Pharmacy GPA

INTRODUCTION

The pharmacy profession continues to advance as pharmacists assume new roles in the delivery of patient care. To keep pace, pharmacy school admission processes should also evolve to be more holistic (ie, examine more than academic traits of applicants) to identify those who can potentially develop into a practice-ready professional.¹ Admissions processes that are not holistic have increased the potential for bias in their approach and should not be considered best practice.² Processes must include tools that holistically evaluate the applicant and assess their behavioral/social traits in addition to academic ability. In addition, the landscape facing pharmacy school admission committees has drastically changed during the last few years. There have been many influences driving this change, including evolutions in pharmacy practice in response to dynamic changes in the health care system, increases in the number of pharmacy schools and expansion in class sizes, changes in the accreditation standards for PharmD programs, and the continuously changing demographics of the pharmacy applicant pool.^3,4 It is also concerning when one considers the increase in the number of pharmacy schools and the recent trend of declining applications. The national applicant to seat ratio is approaching a low of 1:2.⁵ This indicates that institutions are competing to recruit highly qualified applicants. These influences have led admissions committees to adopt holistic processes when considering applicants.

As part of a holistic admissions process, every effort should be made to examine all aspects of an applicant to ensure the most qualified candidates capable of achieving academic success and contributing value to the profession are selected for admission. Nonetheless, predicting future success based upon admissions criteria is challenging. Thus, the Accreditation Council for Pharmacy Education (ACPE) requires pharmacy schools to have in place a monitoring program for identifying admitted students at risk of academic and behavioral concerns, including those that arise during APPEs.^3,6-9

There are many studies in the literature that show that pre-pharmacy GPA and PCAT scores are early identifiers of at-risk students and correlate with student success in the PharmD curriculum.^6,7,9-15 These factors can be easily assessed in the admissions process. In addition to these academic measures, researchers have correlated behavioral/social qualities (sometimes referred to as “soft skills” or non-cognitive qualities) including communication skills, empathy, ethical behavior and professionalism with outcomes in the PharmD curriculum.¹¹ These qualities are difficult to evaluate in standard or structured interviews, but can be assessed through the multiple-mini interview (MMI).^6,9,16-18 Recently, Heldenbrand and colleagues showed that the MMI used during the admissions process is another tool that can be used to predict academic difficulty (previously defined as course grades of D, F, WF, course/year repetition, summer school, programmatic withdrawal or dismissal) in the didactic PharmD curriculum and is an early identifier of academically at-risk students. This study showed that students scoring in the bottom 10^th percentile on the MMI were three times more likely to experience academic difficulty than those applicants with an MMI score above a 4.5 (minimum score of 1 and maximum score of 7).¹¹ These results emphasized the need for additional focus on the evaluation of applicants’ behavioral/social qualities through the MMI.¹⁹ Experiential education represents approximately one-third of the PharmD curriculum. Moreover, students demonstrate many of the behavioral/social qualities that are assessed in the MMI during the experiential component of the curriculum, particularly in their APPEs. Research in medical education demonstrates MMI results predict medical clerkship performance scores.¹⁹ However, there are no studies in pharmacy education that explore the relationship between MMI results and APPE performance. The UAMS College of Pharmacy implemented the MMI admission interview in the fall of 2008, this provides years of data in order to effectively evaluate if MMI performance is associated with performance in the APPE setting.

The primary objective of this study was to determine whether overall APPE scores were associated with admission MMI scores, and to establish whether performance on the MMI is independently associated with APPE scores. The secondary objective is to evaluate whether other admission and pharmacy school variables (demographic data, pre-pharmacy GPA, PCAT scores, pharmacy GPA) are associated with overall APPE scores.⁶

METHODS

This retrospective, multivariable analysis examined pre-pharmacy and pharmacy academic data from the UAMS College of Pharmacy during the admission years 2008-2011. The study protocol was determined to be exempt from human subject research by the UAMS Institutional Review Board.

Applicants who participated in the MMI process from 2008-2011, were enrolled in the College of Pharmacy, and successfully completed the APPE program were evaluated in this study. Successful completion of the APPE curriculum requires completing nine month-long APPE rotations. Like other components of the professional curriculum, the APPE program has a progression policy. More than 99% of UAMS students achieve successful APPE completion within an academic year (ie, 12 months); however, for a variety of reasons, some students experience difficulty, need remediation or must withdraw from a rotation. These individuals can achieve successful completion of the program if they ultimately pass all nine month-long APPE rotations within 24 months of starting their first APPE.

De-identified student admissions and academic data were obtained from the college registrar, associate dean for student affairs, and the associate dean for experiential education for the study period. Study identification numbers were assigned in a progressive manner for all years evaluated. Data obtained for evaluation included admissions data, pharmacy GPA, and overall APPE evaluation scores. Admission data included PCAT (percentile composite score), pre-pharmacy GPA (4.0 scale), MMI score, age, gender, rurality (rural, metropolitan, urban), state of residence (Arkansas resident, out- of-state resident), previous degree (none, bachelor of science [BS] /bachelor of arts [BA], graduate [master’s or doctorate]), admission year (2008-2011), and underrepresented minority status. If there was more than one PCAT score for a participant, the highest PCAT score was used. Pharmacy GPA and pre-pharmacy GPA were based upon completed coursework. Rurality was determined by dividing hometowns into rural, metropolitan, or urban using 2010 US Census data.²⁰

The admissions procedure has included the MMI process for many years as previously described.^6,10,11 This method includes four condensed encounters ranging from standard interview encounters with faculty to case-based scenarios with standardized interviewers.⁷ All encounters are scored with evaluation forms that include interviewer instructions, a 7-point Likert scale (1=unsatisfactory to 7=outstanding) assessment, and space for interviewers to comment. Scores collected for each applicant’s MMI included overall performance score on the encounter, the communication score, and whether the standardized interviewer had any concerns about the applicant. MMI scores are summarized for the admissions committee and presented as the mean overall score across all four encounters. Pre-pharmacy GPA, PCAT, and other data are presented to the admissions committee and used in the assessment of applicants during the admissions process.

The APPE component of the experiential education program at this institution consists of nine, 4-week experiences designed to assist the student in applying the knowledge they gained during their didactic coursework. Each student is required to complete five direct patient care experiences consisting of two acute care, one ambulatory care, one selective in acute or ambulatory care, and one advanced community pharmacy rotation. In addition, students are required to complete two indirect patient care experiences including health-system and community management experiences. Students are allowed two elective rotations. Preceptors for these experiences grade the students using a standardized electronic assessment rubric. Student performance is evaluated at the midpoint and at the end of the experience.

The APPE scores were generated electronically from an online experiential learning system using a standardized consortium APPE rubric used by the school (UAMS APPE Longitudinal Assessment Form. https://pharmcollege.uams.edu/wp-content/uploads/sites/6/2018/09/Appendix-2.-UAMS-APPE-Longitudinal-Assessment-Form-2012.pdf). The APPE rubric consists of 70 multiple choice items (22 patient care; 5 documentation; 9 medication distribution, processing, and dispensing; 4 resource management; 6 communication; 4 public health; 10 drug information and evidence-based medicine; 10 professionalism) using a 5-point Likert scale (1=does not meet competency to 5=exceeds competency expectation) for student performance assessment, with additional non-assessment options of 0=student did not engage in an available activity and N/A=competencies not applicable to this experience or rotation), and three free-form text areas to describe student strengths, competency areas that need improvement, and a plan of action for continued student competency development. Scores of zero and N/A were not included in this analysis. All preceptors were trained and oriented to the APPE rubric and the intended use of scoring scale according to new preceptor orientation and education policies and reinforced during regular site visits by the experiential office. This training is reinforced upon subsequent site visits and at preceptor development continuing education conferences.

The overall APPE score reported and used for this analysis is a composite score from all APPE domains, and each participant’s individual competencies for all completed rotations in the APPE (P4) year. All APPE scores for completed rotations were incorporated in calculating the mean APPE overall score. This included scores for failed rotations, repeated rotations, and for additional elective rotations students enrolled in, thus forgoing their off month. APPE domains include patient care, documentation, resource management, medication distribution, drug information/evidence-based medicine, public health, communication and professionalism. APPE score (mean overall performance) and mean APPE domain scores were collected.

Statistical Analysis Software (SAS Version 9.3; Cary, NC) was used to analyze a single dataset and generate all results reported in this study. Frequency and means of all study covariates and outcome measures were calculated. To identify useful categorizations of key study measures that are continuous (pre-pharmacy GPA, composite PCAT score, MMI, pharmacy GPA, and APPE score), frequency distributions of these measures were examined. Unadjusted correlations between study covariates and MMI were assessed using Pearson product-moment correlations. A multivariable analysis using ordinary least squares regression was performed to identify the influence of pre-pharmacy GPA, PCAT score, overall MMI score, age, gender, rurality, resident status, degree, underrepresented minority status, pharmacy GPA and mean APPE score (Ordinary Least Squares Regression Model of APPE Scores. https://pharmcollege.uams.edu/wp-content/uploads/sites/6/2018/09/Appendix-1.-APPE-MMI-Pearson-CORR.pdf). To address the possibility of a non-linear relationship between overall APPE scores, pre-pharmacy GPA and pharmacy GPA in this model both pre-pharmacy and pharmacy GPAs were divided into performance groups using the frequency distribution characteristics of each group. Pre-pharmacy GPA groups included a reference GPA of >3.5, GPA 3.0–3.5, and GPA <3.0 were compared to overall APPE scores individually. Pre-pharmacy GPA groups included a reference GPA of >3.5 and two comparison GPA groups of 3.0–3.5 and <3.0. Pharmacy GPA used the same categorizations but included one additional categorization: >2.6–3.0 (GPA of >3.5, GPA >3.0–3.5, GPA >2.6–3.0 and GPA <2.6).

RESULTS

There were 432 student pharmacists included for evaluation from the admission years 2008-2011. The mean age was 22.8 years upon admission to the program, mean pre-pharmacy GPA was 3.59, mean composite PCAT was 73^rd percentile, and the mean MMI score was 5.5 (Table 1). Pharmacy school didactic GPA over the P1 to P3 years (Table 2) was 3.14 and the mean overall APPE score was 4.6 (Table 2). Comparing the mean scores of the individual APPE domains, professionalism had the highest score (4.8) and drug information had the lowest (4.5).

Both pre-pharmacy GPA (r=0.20; p<.001) and pharmacy GPA (r=0.17; p<.001) positively correlated with overall APPE scores, whereas no correlations were seen between overall APPE scores and composite PCAT scores (r=0.078; p=.10) (Table 3). Small but significant positive correlations were observed between overall APPE scores and MMI scores (r=0.14; p=.003). Individual APPE domains also demonstrated significant positive correlations with MMI scores, including patient care (r=0.14; p=.003), documentation (r=0.110; p=.022), drug information/evidence-based medicine (r=0.12; p=.011), public health (r=0.16; p=.001) and communication (r=0.24; p=<.001).

The ordinary least squares regression seen in Table 4 demonstrated that pre-pharmacy GPA of less than 3.0 was negatively associated with overall APPE scores when compared to the reference pre-pharmacy GPA of >3.5 (β=-0.10; p=.029). Two pharmacy GPA groups showed negative correlations with APPE scores including pharmacy GPA groups >3.0–3.5 ((β=-0.05; p=.02) and 2.6–3.00 (β=-0.069; p=.004), however pharmacy GPAs lower than 2.6 did not reach significance (β=-0.056; p=.094). The MMI score showed significant positive association with overall APPE scores (β=0.035; p=.001). The R-squared for this model is 0.11.

DISCUSSION

Selecting the best applicants is paramount to ensure colleges and schools of pharmacy admit candidates with the cognitive ability and behavioral/social skills that are required for new pharmacy graduates. The MMI minimizes interviewer bias and is a component of a holistic admissions process.²¹ A study that explored relationships between admission variables, including a standard structured interview with faculty, performance scores from objective structured clinical examinations (OSCEs) students took as part of the curriculum, and final evaluation scores from APPEs indicated admission interviews were not predictive of APPE performance.⁸ By design, the MMI circumvents many of the limitations associated with assessing behavioral/social traits inherent in standard or structured interviews.⁸ It has been shown that poor MMI scores predicted academic difficulty in the didactic portion of the institution’s PharmD curriculum.¹¹ The results of the current analysis specifically demonstrate that MMI scores show positive associations with mean rotation scores in the APPE year, the largest component of the experiential curriculum.^10,11 This study, which is the first to demonstrate such an association in pharmacy education, provides insight into predicting student performance outside the classroom in the practice settings. The experiential learning portion of the PharmD program is intended to prepare student pharmacists for contemporary real world practice.²² In that context, the findings are somewhat expected. Given the patient care demands in pharmacy practice settings, it is intuitive that students who possess higher levels of empathy, critical thinking, ethics, collaboration and communication skills should excel in the experiential component of the PharmD curriculum. Therefore, these data suggest that using MMI in admissions processes can not only serve as a means to predict academic difficulty in the professional curriculum, but can also be used to predict a student’s potential to be successful in the APPE setting. Armed with this insight, strategies can be developed to assist students to improve their behavioral/social qualities before they reach the experiential components of the PharmD curriculum.

The strongest correlation with APPE scores and the MMI was in the communication domain. This section of the APPE rubric contains items that relate to verbal and non-verbal communication with both patients and other health care professionals. This is a key finding because with the requirement for interprofessional education, it is important to assess an applicant’s ability to communicate effectively with others. Communication is assessed at each MMI station, and at some stations it is the focus of the MMI encounter.³ Therefore, these data suggest that MMI is appropriately assessing communication skills and identifying applicants that demonstrate a strong ability to communicate with patients and other health care professionals during their APPEs.

Although the MMI score did not correlate with the two domains of the APPE scoring rubric, medication distribution and professionalism, it did correlate with multiple domains, which suggests that the current MMI topics are useful in identifying applicants who will be successful on their APPEs in multiple areas. Recently through an MMI continuous quality improvement (CQI) process, a critical thinking case was added to the list of topics evaluated in the MMI. Although critical thinking is essential to medication distribution, it is only one case in the MMI and may not provide enough data to correlate overall scores with APPEs. Since traits like empathy and ethical decision-making may not correlate well with the distributive functions in a pharmacy, it is not surprising that it did not show a correlation with that APPE rubric domain. In the future, this may improve the correlation between MMI scores and the medication distribution domain as well as other domains on the APPE rubric. The study authors do not believe that the MMI should necessarily correlate with the professionalism domain on the APPE rubric. Although professionalism is measured in the MMI, it is a malleable characteristic that can be developed throughout the curriculum and then assessed in the APPEs. Therefore, it is not surprising that it did not correlate with that APPE rubric domain.

Although pre-pharmacy and pharmacy GPA were also independent predictors of APPE performance, PCAT was not. This is important to note because in the admission process, schools often place great significance on the PCAT since it has been shown to be an independent predictor of success in the didactic curriculum.^{12,14,18,23-27} Although GPA and PCAT can both be used to predict success in the didactic curriculum, they do not measure non-academic skills. Likewise the MMI is not a surrogate for the applicant’s academic skills. However, perhaps both pre-pharmacy GPA and MMI used in combination could provide more reliable insight into the admission decisions and facilitate the selection of applicants who will be most successful in the professional program, including APPEs.

Despite the advantages to using MMIs, some pharmacy colleges may be reluctant to adopt the MMI in place of traditional interviews due to limited space or facilities (ie, clinical skills center), staff and faculty time associated with training, case generation, and MMI administration or the cost of MMI compared to a faculty interview. Corelli and colleagues reported a cost-effectiveness comparison study incorporating faculty time and facility costs that showed that the MMI process ($75.30) was almost twice as cost effective per candidate compared to traditional faculty interviews ($136.64).²⁸ MMIs are often done in a dedicated space at an institution, but several schools are thinking outside of the box by using faculty offices or occupied spaces that can be accessed temporarily on interview days. This shows the flexibility associated with the implementation of the MMI and the resources needed. Collaborating with schools that already use the MMI to share cases and being creative with the space and other resources needed have provided many institutions the ability to use the MMI even if they do not have dedicated physical space.

This study has some limitations. This is a single institution sample, which limits the generalizability of the results. The study authors recognize that admission and APPE outcomes likely vary depending on institution, instruments and raters. They also recognize that students develop throughout the curriculum. Thus in a broader context, this study shows that data used for admissions decisions (like MMI scores) may also provide feedback that programs can use to identify students who need additional focus on improving their behavioral/social skills and performance in the first three years of the professional program. Although small but significant associations between MMI and overall APPE scores were observed, the authors recognize that this model only explained about 11% of variability in APPE scores. Student experiences gained outside the didactic curriculum may contribute to how prepared they are to succeed on APPEs and this study did not capture measures that were co-curricular in nature.

Secondly, overall APPE scores were positively associated with MMI scores; however, the authors acknowledge that such scores are not individual educational outcomes, but rather the average of nine distinct experiences. Because the APPE measure is an average of nine individual APPE scores, the authors were not able to account for the clustering of preceptor or faculty APPE ratings which may have violated the independence assumption of the ordinary least squares regression. Violation of this assumption, in general, does not bias the estimates but can influence the standard errors and caution is warranted when interpreting the p-values of the models. The APPE programmatic and individual APPE rotation failure rates over the years evaluated in this study were too low to be individually assessed. Therefore, overall APPE scores were used as a surrogate for success in the experiential curriculum. Also, the validated MMI process only involves four cases (stations) and some have proposed to minimize biases in applicant assessment the number of cases be at least seven.²⁹ It is unknown if using more cases will improve the ability for MMI scores to predict experiential performance but the authors will explore this possibility with future graduation years since the school has expanded to a seven case MMI process starting with the 2016 admission cycle.

CONCLUSION

GPA (pre-pharmacy and pharmacy) and MMI showed positive associations with overall APPE scores in the PharmD curriculum. This study demonstrates that students having a combination of higher cognitive achievements in the classroom setting coupled with effective expression of behavioral/social (non-cognitive) skills are linked with higher preceptor-rated performances in the APPE year.