Development and Assessment of the Multiple Mini-Interview in a School of Pharmacy Admissions Model

INTRODUCTION

In addition to cognitive skills and academic ability, noncognitive attributes are vital to the success of health care providers. Noncognitive attributes, also referred to as professional attributes or skills, generally include nonacademic indicators such as empathy, collaboration, leadership, and integrity. A growing body of literature demonstrates the need for health care providers to possess these noncognitive skills, including the ability to work collaboratively on interdisciplinary health care teams and to communicate effectively.^1-7 The success of pharmacists during ongoing health care reform is likely to depend on the development of these skills as they take on expanded roles.

The evolving health care environment and growing importance of noncognitive skills in the workplace has prompted numerous calls for curricular reform in the health professions.^4-7 The University of North Carolina (UNC) at Chapel Hill Eshelman School of Pharmacy is in the midst of redesigning its PharmD curriculum to transform education, and ultimately advance health care.⁸ Desired outcomes of the new curriculum include: exemplary practitioners who provide high-quality, team-based, patient-centered care; leaders and innovators who recognize the health care needs of patients and society and who lead teams toward improvement and change for the betterment of patient care; and lifelong learners who continually strive for positive impact.⁸ The new curriculum places a renewed emphasis on noncognitive attributes that will help student pharmacists prepare for and participate in class, contribute to innovative problem solving, adapt to a changing health care system, and work as part of a health care team to solve complex problems.

As pharmacists take on more responsibility in the care of patients and curricula are redesigned to meet this evolving role, pharmacy educators must be prepared to design and implement strategies for identifying and developing these professional attributes in student pharmacists.^7,9 To assess the noncognitive attributes of prospective student pharmacists, colleges and schools have traditionally used the structured interview. This approach typically consists of 1-2 interviewers spending time with a single candidate, asking pre-established questions and engaging in discussion about the candidate’s experiences, opinions, and/or beliefs. However, research suggests that the subjectivity and bias associated with this approach can compromise the validity and reliability of interview results.^10,11 In addition, evidence of a correlation between structured interview scores and success as a student or practitioner is lacking.^12,13 To address the shortcomings of the structured interview, the multiple mini-interview (MMI) was developed and implemented as an admissions tool in Michael DeGroote Medical School at McMaster University.¹⁴ The logistics of the MMI are similar to those of an objective structured clinical examination (OSCE), which consists of approximately 5-10 stations. The candidate generally has 5-10 minutes at each station to respond to a situational prompt or case. Unlike structured interviews, the MMI allows the candidate to independently interact with multiple interviewers, which can reduce bias in assessing candidates.¹⁴

The MMI was developed to differentiate among prospective students based on noncognitive abilities and ideally better predict which applicants would be the most successful practitioners.¹⁴ Since its first description in the literature, the MMI has been replicated in medical schools and, more recently, in pharmacy schools, pharmacy and medical residency programs, and veterinary schools.^14-21 Across these settings, the MMI seems to be a useful admissions tool for measuring professional attributes. In a systematic review of studies exploring MMI use for student selection in health professions training, Pau et al concluded that candidates and interviewers found the process acceptable and fair.²² Additionally, MMI performance does not strongly correlate with previous academic performance and appears to successfully evaluate nonacademic traits as intended.²² The MMI may be predictive of success as a practitioner, as MMI performance predicted clerkship performance among medical students.¹²

Given the importance of noncognitive attributes in achieving the outcomes of the proposed new curriculum, the school identified the need to design and implement a new admissions model to effectively identify and select students who possess characteristics likely to aid their success in the new curriculum and beyond. For the 2013-2014 admissions cycle, the school implemented the MMI model as part of the PharmD admissions process. The purpose of this paper is to describe the development, implementation, and evaluation of the MMI as an admissions tool. This study is unique because it examines a PharmD program in the United States that uses the MMI to replace the previous interview process and not just as a pilot program.

METHODS

Multiple-Mini Interview Implementation

Planning for the new PharmD admissions model began in the spring of 2012. A faculty committee with student representation was formed to design the new admissions model and determine the best interview method as a part of the model. The committee realized the importance of noncognitive assessment in the admissions process and identified several methods to evaluate such attributes. After much research and discussion, the MMI was selected as the preferred interview technique based on literature supporting its use as a reliable assessment of noncognitive attributes with less bias than traditional interview methods.^23-25

In the fall of 2012, the committee researched companies that could assist with administration of MMIs and found the ProFitHR Candidate Assessment System, developed by faculty members at McMaster University.²⁶ Select committee members met with ProFitHR developers and faculty members from 2 other schools of pharmacy with experience using ProFitHR and the MMI to gather information about the process and tips for successful use of the system. This part of the process was invaluable to the planning, and the committee decided to use ProFitHR to assist with the MMI administration, specifically supplying validated scenarios for evaluation of noncognitive attributes.

The next step of the process was planning the implementation of the MMI in conjunction with the school’s 3 Candidates’ Days during the 2013-2014 academic year. The goal of Candidates’ Day was not only to interview candidates, but also for candidates to learn about the school and PharmD program, experience the culture of the school, and meet and interact with current students, faculty members, and staff. Each Candidates’ Day was scheduled to include a brief welcome and overview of the day’s events followed by 3 concurrent sessions: the MMI, a mock class, and an overview of the Office of Student Affairs’ programs and services designed to support enrolled students. After candidates completed the 3 concurrent morning sessions, they were scheduled to eat lunch with current students and then attend faculty and student panels in the afternoon, where they could ask specific questions about the PharmD program and student life.

Since the MMI method was drastically different from the structured interview used previously, efficient, effective, and complete training for the interviewers was necessary. Two months prior to the first Candidates’ Day, faculty members from all academic divisions within the school were solicited to serve as interviewers for all 3 Candidates’ Days. All academic divisions were represented and invested in the process. Off-campus faculty members also participated in the process. Twenty-eight faculty members from across all academic divisions and from off campus were stationed throughout the MMI circuits.

Interviewers were asked to view online training documents on their own time before attending a mandatory, 90-minute, live training session one month prior to the first Candidates’ Day. Using established MMI research and documents provided through ProFitHR, an offloaded training presentation was developed that included the history and development of the MMI, benefits of its use in the admissions process, an overview of the MMI process on Candidates’ Day, expectations for the interviewers, and sample MMI scenarios and probing questions. The in-person training highlighted important aspects of the MMI process and focused on the role and expectations of the interviewers on the day of the MMI. Trainees were asked to role-play 3 MMI scenarios, during which one participant acted as the student and the other participant acted as the interviewer. The trainees were timed and asked to go through the MMI process, as if it were an actual interview. Afterward, they were asked to reflect on their role playing experience, and the facilitator and other trainees offered feedback.

The MMI was designed and implemented according to available space and Candidates’ Day logistics. Based on research, the committee identified the need for 6 MMI stations evaluating noncognitive attributes vital for success in the PharmD program and pharmacy practice (integrity, adaptability, empathy, critical thinking, and 2 teamwork stations) and a seventh station asking candidates why they were interested in the school (why UNC). The seventh station was less structured so candidates could tell the interviewer more about themselves. In addition, stations 1 and 2 were designed to measure related teamwork constructs. Using scenarios provided by ProFitHR, members of the admissions committee selected and reviewed specific MMI scenarios that targeted each noncognitive attribute, which were further reviewed by the MMI interviewers prior to Candidates’ Day for face validity. Similar but not identical scenarios were selected for each Candidates’ Day, but the same 6 noncognitive attributes were evaluated at all Candidates’ Days.

Candidates’ Days were held on both the Chapel Hill campus and on the satellite campus in Asheville, North Carolina. At the Chapel Hill campus, the MMI consisted of three 1-hour circuits scheduled consecutively within a 3-hour block. In each of the 1-hour circuits, there were 4 concurrent MMI groups in 4 separate locations (Figures 1 and 2). Each MMI contained the 7 stations described above, with 1 faculty member assigned to each station. With this model, we were able to accommodate 84 candidates each Candidates’ Day using 28 faculty interviewers and 4 staff members to keep time at the 4 MMI locations. At the Asheville campus, the MMI consisted of 3 consecutive MMI circuits in a 3-hour block with only one MMI group in 1 location. Up to 21 candidates per Candidates’ Day were accommodated on the satellite campus using 7 interviewers and one staff member. Faculty office suites were used as MMI locations to allow the close proximity of interview stations. Each candidate had 2 minutes to read the scenario before entering the interview room and 6 minutes to talk with the interviewer.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Schematic of Multiple Mini-Interview (MMI) Circuits.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Illustration of Students Moving Through a Multiple Mini-Interview (MMI) Circuit.

RESULTS

As seen in Table 1, 253 candidates were assessed during Candidates’ Days with the MMI. One-hundred and sixty-two (63.5%) of the MMI participants were female, 235 (n=92.2%) possessed a bachelor’s degree or higher, 152 (60.1%) were white, and the mean age was 22.17 (3.41) years. Mean composite PCAT score of MMI participants was 87.47 (11.51) and mean uGPA was 3.54 (0.31).

Mean scores for each station were: station 1 (teamwork 1), 6.66 (1.74) (range 2-10); station 2 (teamwork 2), 6.70 (1.88) (range 1-10); station 3 (integrity), 7.17 (1.75) (range 1.75-10); station 4 (adaptability), 7.22 (1.67) (range 3-10); station 5 (empathy), 7.14 (1.63) (range 1.75-10); station 6 (critical thinking), 6.91 (1.99) (range 2-10); and Station 7 (why UNC), 7.09 (1.88) (range 2-10). When examining the relationship between MMI station scores and academic indicators (uGPA, composite PCAT, and all PCAT subscores), all correlations were negligible and nonsignificant (r_p≤0.2). In addition, age presented a negligible and nonsignificant relationship with each of the 7 stations (r_p≤0.2). An independent t test found no significant difference in average station scores based on gender. Kruskal-Wallace one-way analysis of variance indicated no significant difference in average station scores based on race. However, results indicated that students with less than a bachelor’s degree scored lower overall than students with a bachelor’s degree or postgraduate degree on the unstructured station (5.94 (2.34); 7.09 (1.97), p=0.06; 7.87 (2.26), p=0.02, respectively).

In a factor analysis of MMI scores, each station formed a single factor with loads ranging from 0.80 to 0.95 (Table 2). The factor analysis converged in 6 iterations and accounted for a total of 87.25% of the total variance. The correlations between stations, based on average scores, and Cronbach alpha (in parentheses) are shown in Table 3. Intercorrelations ranged from 0.16 to 0.62. Because the constructs measured in stations 1 and 2 were related, it is not surprising that these 2 stations presented a strong positive correlation. Cronbach alpha was greater than 0.90 for all stations (range 0.92 to 0.96), suggesting that the items used to compute scores for each station demonstrated high internal consistency.

Of the 253 candidates that participated in the MMI, 171 (67.6%) completed an online feedback survey about their experiences and perceptions of Candidates’ Day (Table 4). Only 19.9% of respondents indicated having participated in an MMI prior to the interview at the UNC Eshelman School of Pharmacy. On a 5-point Likert scale ranging from (1) strongly disagree to (5) strongly agree, 57.3% agreed or strongly agreed that “the interviewers got to know me through the questions I answered” (3.5 (1.0)), 73.7% agreed or strongly agreed that “the MMI allowed me to showcase communication, critical thinking, and opinion” (3.9 (0.9)), and 67.8% agreed or strongly agreed that “overall, I thought I did well in the MMI” (3.7 (0.8)). When considering the entire Candidates’ Day, which included the MMI, the majority of respondents (98.2%) agreed or strongly agreed that “I had positive interactions with current students, faculty members, interviewers, and staff” (4.6 (0.5)), and 98.9% agreed or strongly agreed that “Candidates’ Day was a positive experience” (4.5 (0.6)). Following Candidates’ Day, 97.8% of respondents indicated they were still interested in attending the UNC Eshelman School of Pharmacy. No significant differences were found between mean responses based on the date that the candidate attended Candidates’ Day and participated in the MMI.

Thirty-five faculty members (71.4% response rate) completed an online survey about the MMI training and implementation. Seventy-five percent of respondents reported participating in the traditional (structured) interview in the past, 82.6% of which agreed that the MMI was a better interview method to assess a candidate’s noncognitive attributes. Table 5 presents additional findings from the survey. Of note, 90.7% agreed or strongly agreed on a 5-point Likert scale that “after the interviewer training, I understood my role as an interviewer” (4.1 (0.9)) and 77.5% agreed or strongly agreed that “the time commitment required as an interviewer was reasonable” (3.8 (0.9)).

DISCUSSION

The analyses reported here describe the psychometric properties of the MMI as an admissions assessment tool and the candidate and faculty perceptions of the MMI and its implementation at the UNC Eshelman School of Pharmacy. This is one of the first studies in pharmacy education to examine the psychometric properties of the MMI used in a PharmD program admissions process. The findings of this study support the validity and reliability of the MMI and suggest that the MMI was well-received by candidates and interviewers.

In general, our findings suggest the MMI is able to distinguish between the attributes it was designed to assess. The factor analysis revealed 7 distinct factors with high factor loads and the majority of variance accounted for by the analysis, providing support for content specificity. With the exception of stations 1 and 2, which measured the related teamwork constructs, the weak to moderate correlations between stations suggest the stations were in fact measuring different constructs. High Cronbach alpha scores provide evidence of consistency among subscores within each station and further support the stability of station scores for each candidate. While we cannot ensure that each station measured the intended construct, review of each scenario by multiple stakeholders strengthened face validity for each scenario. These results also support other studies examining the psychometric properties of the MMI.²⁴

Weak correlations between academic indicators and MMI scores indicated the MMI measured something other than academic preparedness. This is consistent with the findings from previous studies in health sciences students, where MMI scores were not associated with academic parameters or aptitude tests.^22,23 This is important because the MMI was designed to assess noncognitive attributes and not academic ability. Additionally, no significant differences were found between MMI scores based on age, gender, or race, which suggests the process was not biased toward these factors. This finding is supported by the interviewer survey, which found that the majority of interviewers agreed that every candidate had equal opportunity to demonstrate the noncognitive attribute being assessed. Similar results have also been reported in studies of the MMI used in medical school admissions and medical residency interviews.^15,23,25

During the interviewer training, interviewers were encouraged to use the entire scoring scale when evaluating applicants within a circuit. As seen in this study, interviewers used the full 10 points of the scale, effectively enabling differentiation in MMI performance between the candidates. On average, the candidates agreed that “the MMI allowed me to showcase communication, critical thinking, and opinion,” and interviewers agreed that the MMI allowed them to effectively differentiate between applicants. With total MMI scores ranging from 7-70, this approach enabled the admissions committee to rank candidates based on their performance on the MMI with a large spread. However, moderate correlations between some stations suggest that future iterations of the MMI may benefit from further scenario refinement and additional interviewer training.

While the MMI provided valuable information for admissions decision making, this approach was also well received by candidates. One consideration when redesigning the interview process was the candidates’ perception of and experience during the interview. The MMI is a more rigorous process where the interviewer’s role is to challenge the student, which can be uncomfortable for both the candidate and the interviewer. Candidates’ Day was designed, however, so that candidates would have time to interact with faculty members, staff, and current students outside of the MMI and experience the culture of the school. The candidate survey indicated they accepted the process, felt interviewers got to know them, and were able to showcase their communication and critical-thinking skills. Similar results were found in studies by Razack et al²⁷ and Kumar et al¹⁷ in an evaluation of the perception and satisfaction of applicants and interviewers within medical schools.

Faculty members and staff interviewer investment in the process was vital for success since the program was relying on these individuals to volunteer their time to assist with the MMI. Survey results indicate that interviewers accepted the MMI and agreed it was a better tool for measuring noncognitive attributes compared to the previous interview method. They also felt the time requirement was reasonable. Interviewer training emphasized the importance of admitting the best students into the program, the rationale for the use of the MMI, and the reliance on the interviewers to help with the process. Most interviewers were faculty members who interacted with students in the program on a daily basis in various courses or pharmacy practice experiences, so they had a vested interest in helping to select the students who possessed the desired noncognitive attributes.

Identifying and measuring applicant attributes indicative of student pharmacist success is a complex undertaking.^28-30 Limitations associated with rater bias and context specificity, for example, can challenge the identification and measurement of these attributes at the point of admissions. While the findings of this study suggest the MMI provides valid and reliable information about noncognitive attributes, several limitations are worth noting. First, the single institution sample limits generalizability of results. As more pharmacy schools implement the MMI, these results should be considered within this growing body of literature. Second, this study did not examine the variability in MMI scores associated with interviewer bias, which should be examined in future studies with analyses like the Many-Faceted Rasch Model, which can be used to estimate variance associated with construct-irrelevant variance. In addition, the association between MMI scores for this cohort and their academic performance in the program remains unclear. Future research will evaluate the relationship between MMI scores and performance in the curriculum. Ongoing assessment of the MMI and its use as a tool for identifying qualified applicants will further inform refinements to this approach.

CONCLUSION

The MMI was implemented at the UNC Eshelman School of Pharmacy as a more valid and reliable strategy than traditional interview methods for measuring noncognitive attributes in prospective students. The results from this study suggest that the MMI can differentiate between such attributes and that it successfully measures something other than academic parameters. Additionally, candidates and faculty members perceived the process positively. Given the focus on noncognitive attributes during the admissions process and with the MMI being one part of this focus, we anticipate that admitted students will be better prepared for our new curriculum, pharmacy practice experiences, and the practice of pharmacy after graduation.