Pharmacy Student Perceptions of the Entrustable Professional Activities

INTRODUCTION

Entrustable professional activities (EPAs) are duties that a healthcare provider is expected to competently perform.¹ Collectively, the EPAs for a profession describe the essential work of the discipline or occupation. Entrustable professional activities are intended to guide the education and training of health professionals and offer a practical approach to assessing competence in real-world settings.² Entrustable professional activities have been described for medical school graduates and for internal medicine, pediatric medicine, family medicine, and pathology residencies.^3,4 In 2016, the American Association of Colleges of Pharmacy (AACP) published a list of 15 EPA statements for new pharmacy graduates which define a set of tasks and functions that all pharmacists should be able to perform.⁵ While many stakeholders provided input into the development of the EPAs published by AACP and experienced practitioner-educators have expressed their support of the 15 core EPAs, it is unknown whether pharmacy students consider the EPAs relevant to their future work or believe they are being sufficiently prepared to perform them.^4,5

Entrustable professional activities are now being used by colleges and schools as a tool to guide curricular development and assess each student’s progress towards independence.^6-8 Ultimately, the Doctor of Pharmacy (PharmD) curriculum should prepare graduates to perform the EPAs autonomously, requiring only indirect supervision at the time of licensure.⁵ Given that EPAs are a relatively new construct, soliciting input from pharmacy students on their perceptions can provide useful information regarding the clarity of the statements and may reveal gaps in their understanding about the roles and responsibilities of pharmacists.

This study involved capturing pharmacy students’ reactions to the 15 core EPA statements using a survey and then analyzing their responses.¹ Specifically, we attempted to answer the following questions: Do pharmacy students believe the EPAs are relevant to the practice of pharmacy? Are they an expectation in most practice settings? Do pharmacy students feel prepared to autonomously perform the EPAs as they progress toward graduation?

METHODS

A random sample of PharmD students from each of the four professional years at four institutions (Universities of Minnesota, Mississippi, Oklahoma, and St. Louis College of Pharmacy) were invited to participate in this study. A power analysis was conducted using G*Power, version 3.1.9.2 (http://www.gpower.hhu.de/) to determine an appropriate sample size given a proportional difference of 20% (ie, 90% vs 70% agreement), power of 0.80, and a Bonferroni-adjusted alpha of 0.00833. We set the alpha lower because we anticipated conducting multiple pairwise comparisons between program years. Based on these parameters, a total sample size of 400 would be needed. Using a multistage sample approach, we randomly selected students, stratified by professional year and school.^9,10 This was done to reduce overrepresentation of respondents from larger schools or from a particular class year. For example, students for the P3 or P4 year might be more inclined to respond to the survey instrument, causing their responses to be more heavily weighted in the overall analysis. By ensuring a relatively equal number of respondents in each stratum, we attempted to create a representative sample from each professional year and school. Therefore, the study was designed to include responses from 25 students from each professional year cohort at each school for a projected institution total of 100 students and an overall study total of 400 students. The study received approval (exempt status) from the institutional review boards at the four participating institutions.

A four-part survey instrument that included 56 questions was administered to student participants via the online survey tool, Qualtrics (https://www.qualtrics.com/ Provo, UT, and Seattle, WA). Part 1 of the survey instrument contained demographic questions regarding the participant’s age group, institution attended, year in the program, level of awareness of EPAs, and estimated hours of experience in pharmacy practice accumulated to date, including pharmacy technician and intern hours. Part 2 of the survey instrument asked student participants to indicate if the EPA statements were relevant to the practice of pharmacy and expected of pharmacists in multiple practice settings. This part of the survey instrument was adapted from the validated Quality of Entrustable Professional Activities (QUEPA) tool initially developed for internal medicine residents.¹³ The QUEPA survey questions were modified to reflect differences in pharmacy practice versus medical practice, account for the pharmacy specific EPA statements published by AACP, and to coincide with EPA survey questions previously administered to experienced practitioner-educators.⁴ An additional modification to the QUEPA tool was the change from scaled responses to a binary response of yes or no. Our version of the instrument was not validated after these modifications. Part 3 of the survey asked participants to read a patient case scenario to provide context and then to rate their current level of autonomy (level of entrustability) for each EPA. Eight community practice pharmacists (two from each study institution’s region) vetted the scenario to ensure that it was a complex and realistic case that a new pharmacy graduate would likely encounter. Last, in part 4 of the survey instrument, respondents were asked to select the three EPA statements they believed they were most prepared to perform and the three they believed they were least prepared to perform at this point in their education.

A draft survey instrument was piloted with 11 junior-level pre-pharmacy students (ie, undergraduate students accepted into the pharmacy program but not yet pharmacy students) enrolled at St. Louis College of Pharmacy. The purpose of the pilot test was to determine the time required for survey administration, the technical functionality of the survey instrument, and the clarity of the instructions and questions. The final survey instrument, instructions, and items can be viewed here (https://www.pharmacy.umn.edu/sites/pharmacy.umn.edu/files/appendix_survey.docx).

A randomized number list for each of the four program years was used to randomly select students to participate in the study at each institution. For the first round of data collection, 30 students from each professional year at each institution were invited to participate via an email that explained the purpose of the study, briefly defined EPAs, and provided a link to the survey instrument. Participants were invited in February 2018 (day 1) and reminded on day 8 and day 15. On day 15, study invitations were sent to additional students if the 25-participant goal for each professional year had not been met. The number of additional students invited equaled the shortfall for the professional year at each institution. For example, the goal of 25 minus the 20 P1 respondents equaled a shortfall of five P1 respondents (shortfall group 1), resulting in a total of 35 invitations sent to P1 students on day 15 (five new invitations and 30 reminders). On day 22, shortfall group 1 received an email reminder and a new shortfall group 2 was invited to participate. The initial 30 students who had been invited did not receive any further reminder emails. This pattern continued every seven days, ie, invited participants would receive reminder emails for only two weeks and then a new shortfall group would be added to replace the dropped shortfall group. All data collection was completed by April 2018.

Results were aggregated into descriptive summaries and compared based on demographic characteristics. Categorical variables were reported in percentages and evaluated using Pearson chi-square tests. Exact Pearson chi-square tests were used if expected cell counts were less than five. In part 2 of the survey instrument, participants were asked to evaluate EPA statements on two separate attributes (relevancy and expectation in multiple practice settings). Participants were said to have agreed with an item if they had responded yes to a yes/no question. Agreement does not refer to paired inter-rater agreement or reliability. To determine if differences in agreement exist among program years (P1-P4), a chi square test was performed on the 2x4 contingency table. The total number of statements a participant agreed with was reported with median (interquartile range, IQR), and differences in program years were evaluated using Kruskal-Wallis tests. Statistical significance was set at p<.05. All analyses were conducted using SAS software, version 9.4 (SAS Institute, Cary, NC).

RESULTS

The survey instrument was distributed to 1161 pharmacy students at the four participating institutions. Four hundred eighty-eight responses were received between February 20 and April 27, 2018 Sixty-five responses were excluded because the participant only completed part one of the survey instrument. The demographic information for the participants across the four institutions is detailed in Table 1. Statistical differences between age groups and pharmacy work experience reflect the variability in the structure of the four participating PharmD programs. Because of the recruitment design, there were similar numbers of students for each program year for each of the four participating institutions. As anticipated, the University of Minnesota had higher numbers of students familiar with EPA statements, as this was the only institution of the four that was using EPA statements at the time of the study (although not the AACP’s EPA statements) as their evaluation strategy for advanced pharmacy practice experience.

Of the 423 usable responses, 48 (11.3%) students stopped after completing part 2 (EPA relevancy/setting questions), two (0.4%) students stopped after completing part 3 (self-reported skill level), and 373 (88.2%) students completed the entire survey. The overall response rate was 36% of the 1161 invitations sent. Because of our distribution methods, each institution invited different numbers of students to participate in the survey. Response rates differed by institution: St. Louis College of Pharmacy, 25% of 436 invitations sent; University of Minnesota, 32% of 317; University of Mississippi, 46% of 216; and University of Oklahoma, 59% of 192, p=.001. However, the goal set after conducting a power analysis of obtaining 25 completed survey instruments per professional year at each of the four study institutions was met and, in some instances, exceeded.

Table 2 details the overall level of agreement for both relevancy and expectation in multiple practice setting attributes. Overall, there was high agreement that the EPAs were relevant to pharmacy practice. The percent agreement was >94% for 14 out of 15 EPA statements. The lowest level of agreement was 85% of students indicating that “Create a written plan for continuous professional development” was a relevant professional activity for pharmacists. The level of agreement across student program years were significantly different for two EPA statements: “Analyze information to determine the effects of medication therapy, identify medication-related problems, and prioritize health-related needs,” and “Minimize adverse drug events and medication errors.” The level of agreement for these two statements increased significantly among students from the P1 year to the P2. The median (IQR) number of statements with which students agreed was 15 (ie, 15 of 15) and did not differ by program year.

The overall level of agreement that the EPA statements represent activities that pharmacists are expected to perform in multiple practice settings varied, but there was 90% agreement or higher for nine of the 15 statements. The lowest overall percent agreement was 67% for the statement, “Create a written plan for continuous professional development.” No significant differences in the percent of agreement across program years were found for 13 of the statements. Significant differences were observed in two statements. Students in the P2, P3, and P4 years universally (100%) agreed that “Minimize adverse drug reactions and medication errors,” was an expected professional activity in multiple practice settings compared to 96% of P1 students. The P1 and P2 students were less likely than P3 and P4 students to agree that the EPA “Maximize the appropriate use of medications in a population” was an expected professional activity of pharmacists in multiple practice settings. The median number of statements that students agreed were expected in multiple practice settings was 14 (IQR=12-15) and was similar across program year.

Self-reported ability to perform each EPA statement without supervision increased across program year (Table 3). The median response among P1 students was 2, need direct supervision, on 13 of the 15 statements. The median response among P2 students was 3, need indirect supervision, for 10 of the 15 statements. The median response among P3 students was 3, need indirect supervision, on about half of the EPA statements, and 4, need periodic supervision, on the other half. Finally, the median response among P4 students was 4, need periodic supervision, on 13 of the statements. Within Table 3, the transition in the median self-perceived supervisory needs for P1 students to P4 students decreases as the level of self-perceived entrustability increases (higher level of entrustability), indicating greater confidence in performing each EPA statement.

Students were asked to select the three EPA statements they felt most prepared to fulfill at this point in their education (Table 4). Regardless of program year, the majority of students included the EPA statement “Collect information to identify a patient’s medication-related problems and health-related needs” among the three activities they felt most prepared to perform. Sixty-four percent of P1 students selected this EPA statement, with decreasing percentages across the other program years: P2, 62%; P3, 52%; and P4, 51%).

The second most often selected EPA statement was “Fulfill a medication order.” However, this EPA statement was most frequently selected by P1 students, with 68% saying they were most prepared to perform this activity. The frequency with which this statement was selected diminished across program years (P1=68%, P2=54%, P3=46%, and P4=45%). The third mostly frequently selected EPA statement differed by year in program. The EPA statements students indicated they were most prepared to perform were significantly different across program years. Overall, significant differences in students’ responses across years in the curriculum were found on eight of the 15 EPA statements.

Of the EPAs that students indicated they were least prepared to perform, a high percentage of students selected “Overseeing the pharmacy operations for an assigned work shift.” The percentage who indicated they were least prepared to perform this EPA steadily increased each year (from 31% for P1s to 57% for P4s). In addition, the percentage of students who indicated they were least prepared to “Create a written plan for continuous professional development” steadily increased from the P1 to the P4 year (24% for P1 students to 53% for P4 students). Conversely, while “Establish patient-centered goals and create a care plan,” was among the EPAs students felt least prepared to perform, the percentage of students indicating this steadily declined from the P1 (45%) to the P4 (24%) year.

DISCUSSION

This study provides evidence that among pharmacy students at four colleges of pharmacy, the AACP Core EPAs for New Pharmacy Graduates are perceived as relevant and applicable to multiple work settings. Overall, this study found consistently high agreement (≥85% for all 15 statements) among pharmacy students that the EPAs describe activities relevant to entry-level pharmacy practice. Pharmacy students’ agreement regarding the relevancy of the EPA statements was consistent across PharmD program years (P1-P4) and between the four institutions. The overall level of agreement was high (≥67% for all 15 EPA statements), but greater variability existed among pharmacy students’ responses regarding whether the EPA statements were expected activities for pharmacists in multiple practice settings. Interestingly, student agreement that the EPA statement of “Collaborate as a member of an interprofessional team” was expected across multiple practice settings declined as students progressed through the PharmD program years. Although this decline was not significant, it suggests that students may not be observing pharmacist preceptors routinely participate in interprofessional activities in all practice settings. However, this contradicts ratings among experienced practitioner-educators who ranked the interprofessional collaborator role as second highest in both importance and amount of time spent performing.⁴ The potential disconnect between experienced practitioner-educators with close ties to a college and the broader pharmacy practice community regarding interprofessional collaboration should be explored to ensure curricular emphasis and expectations are in alignment with IPPEs and APPEs.

The EPA statement “Create a written plan for continuous professional development” garnered the lowest percentage agreement as being both relevant and an expectation found across multiple settings. As pharmacy practice continues to evolve and change, colleges and schools need to help students gain understanding of the importance of continuing professional development (CPD) and intentionally integrate self-directed learning activities into PharmD curricula, as well as make explicit through preceptor-student discussion prompts how clinicians use CPD in their own careers. The results of this study regarding the importance and challenges related to preparing students for CPD align with input from other pharmacy stakeholders.^4,5

Student pharmacists self-rated their ability to perform the EPAs in an increasingly independent manner as they progressed through a PharmD program, moving from needing direct supervision in the P1 year to periodic supervision in the P4 year. This finding suggests that PharmD curricula, from the student’s perspective, are adequately preparing them to perform expected entry-level responsibilities and tasks. Perhaps somewhat expectedly, 57% of students felt that overseeing the pharmacy operations for an assigned work shift was among the responsibilities they were least prepared to perform. This is an important pharmacy practice activity that is difficult to authentically simulate in pharmacy curricula or otherwise provide students with experience. As Zgarrick argues, this unit of pharmacy practice requires the same purposeful curricular design as pharmacotherapy to prepare students for direct patient care roles.¹¹ Our results suggest students do not feel ready to carry out this practice management role. The percentage of students who identified this EPA among the roles they were least prepared to perform steadily increased with each year of the curriculum, suggesting students may become more aware of the complexity of managing pharmacy operations as they progress through the program.

Similarly, “Create a written plan for continuous professional development” was increasingly identified as a responsibility pharmacy students were least prepared to do as they progressed through the curriculum. This may have been because of students’ growing awareness of the importance and difficulty in maintaining their knowledge and skills in a rapidly evolving field. However, this EPA statement was also ranked lower on both relevancy and expectation in multiple settings, so feeling less prepared to perform this EPA may have simply reflected a perceived lack of importance. This differs from the EPA statement “Establish patient-centered goals and create a care plan,”. While among the top three EPAs students felt the least confident about, the percentage steadily declined from P1 to P4, potentially suggesting that as students acquired additional knowledge and experience, their confidence to perform this activity increased. While students’ belief in their ability to independently perform EPAs appears to increase over time, colleges and schools of pharmacy still need to develop assessments to document that students can independently perform the EPAs.¹

In 2017, senior medical students were surveyed to measure their self-perceived readiness for residency based on the Association of American Medical Colleges 13 EPA statements.¹² Similar to the findings from our study, alignment was found between student and other medical stakeholder perspectives. Rhodes and colleagues also found agreement between student learners at University of North Carolina Eshelman School of Pharmacy (UNC) and preceptors within early pharmacy practice education experiences using an institution-specific EPA list that predated the AACP EPA statements, but most mapped to the AACP EPA statements.⁷ While a direct comparison between projects is difficult because of different research questions, the P2 pharmacy student responses in Table 3 and the P2 pharmacy student self-ratings of entrustability in the UNC project show similar levels of student confidence at the midpoint of the UNC immersion experience, but much higher levels of confidence at the final evaluation point for UNC students. The UNC P2 pharmacy students rated their level of entrustability as a 4 (independent) on nine of the 14 UNC EPA statements by the end of the two-month immersion experience.⁷ The discrepancies could be related to the confidence that comes with being in the practice setting, an incomplete student understanding of the complexity associated with units of pharmacy practice, and differing descriptors for each level of supervision. The UNC P2 students, similar to the P2 students in this project, confirmed the relevancy of the EPA statements to pharmacy practice in multiple settings.

Obtaining pharmacy student input is valuable information needed to align expectations and to guide curricular priorities. This research adds to the accumulating evidence that the AACP list of EPAs for new pharmacy graduates are perceived as relevant to the practice of pharmacy and expected of pharmacists in multiple setting by multiple stakeholder groups.

The results of this study, while compelling, must be taken into context with potential limitations. Student participants were drawn from four colleges of pharmacy located in the Midwest and Southern regions of the United States. While most participants were within the common age range for pharmacy students, students at the four institutions varied in age, pharmacy work experience, and prior knowledge of EPAs, reflecting the differences in the regional program structures. While these four schools could be classified as geographically diverse (ie, not located in the same state), generalizability to other pharmacy schools may be limited. Pharmacy students at schools in states with different practice laws and regulations may hold different beliefs about the relevancy of the EPAs and the professional activities that pharmacists are expected to perform. Also, while we reached our target number of student participants in each program year at each institution, the overall response rate was less than 50%. Whether participation bias was present is unknown, ie, there is no way to know whether nonresponders would have felt differently than the responders did about the EPA statements. However, our power analysis and multistep sampling process yielded the needed sample size by institution and by program year, determined a priori, to find statistical differences among the responses received. This study represents a snapshot of pharmacy student beliefs, which likely will change over time. However, surveying students in each year of the PharmD program provides general insights into beliefs among students as they progress through the PharmD program. Finally, the QUEPA tool was developed and validated to assess the quality of EPA statements for internal medicine residency programs.¹³ The QUEPA tool was modified for the purpose of this study and has not been validated in settings beyond internal medicine residency programs.

CONCLUSION

Pharmacy students at four universities consistently agreed that the core EPA statements were relevant to pharmacy practice and that pharmacists were expected to perform these activities in multiple settings. The students self-rated their ability to perform the EPAs in an increasingly independent manner as they progressed through the PharmD curricula. Students believed they were most prepared to collect information to identify patient medication-related problems and health-related needs and to fulfill medication orders. Conversely, they felt least prepared to oversee pharmacy operations for a work shift, create a written plan for continuous professional development, and establish patient-centered goals and care plans.