Assessment of Practice Management Entrustable Professional Activities by Pharmacists in North Dakota

INTRODUCTION

In health care professional training programs, a performance gap can occur when a student transitions from didactic coursework to experiential education settings, and again when transitioning from educational settings to practice and/or residency training. In 2007, ten Cate and Scheele were among the first to explore translating individual academic competencies into clinical practice proficiencies or activities essential to the profession.¹ Each professional activity is comprised of individual competencies that students must master in order to perform the task independently in practice. Termed entrustable professional activities (EPAs), ten Cate described them as “professional practice units defined as unsupervised tasks or responsibilities entrusted to students during their work after having acquired the competency level necessary to carry them out.”²

The practical use of EPAs in the United States began in 2014 with the Association of American Medical Colleges (AAMC) publishing a list of 13 core EPAs that medical residents were expected to perform on the first day of residency without supervision, regardless of specialty.³ Residency training programs also use EPAs as a means to translate competency statements (ie, general abilities that every health professional should possess) into specific duties that supervisors can both detect and delegate to trainees in a practical setting.⁴ Entrustable professional activities have been formulated for postgraduate residency training in family medicine, internal medicine, and pediatrics, and are currently being incorporated into medical school curricula in other countries, including several European nations and New Zealand.^5,6

Similar to medical education, pharmacy is predicated upon a competency-based approach to curricular development. The 2016 Accreditation Council for Pharmacy Education (ACPE) Standards specifically use the term “competencies” to frame student outcomes.⁷ While competencies may be defined differently depending on the source, they frequently integrate multiple educational attributes (eg, knowledge, skills, values, and attitudes) to describe observable abilities of a particular health professional. Unfortunately, the operationalization of competency statements in both educational and practice settings has led to confusion, as the phrasing of specific competency statements lacks professional practice relevance.^8,9 Unlike competency statements, which are typically seen as abstract and academic to non-educators,² EPAs are intended to be intuitively understood by patients, practitioners, and policymakers. Thus, EPAs can serve as the bridge connecting competencies and clinical skills, and in doing so, help to eliminate some of the ambiguous language inherent in competency statements.

The American Association of Colleges of Pharmacy (AACP) has been instrumental in the development of EPAs for use in pharmacy education. AACP charged their Academic Affairs Committee to identify EPAs for pharmacy students as they transition from advanced pharmacy practice experiences (APPE) into practice and/or residency.⁷ According to AACP, “EPAs are units of professional practice or descriptors of work, defined as specific tasks or responsibilities that trainees are entrusted to perform without direct supervision once they have attained sufficient competence. EPAs are independently executable, observable, and measurable in their process and outcome.”⁷ Those EPAs that all new pharmacy graduates must be able to perform without supervision on the first day of practice were designated as core EPAs. The AACP identified 15 Core EPAs, which are organized by function among six domains: patient care provider, interprofessional team member, population health promoter, information master, practice manager, and self-developer.

AACP’s work to define EPAs relative to the pharmacy profession creates a common language to guide pharmacy education. Core EPAs for new pharmacists denote professional activities expected of all graduates independent of practice setting. After combining multiple experiences in the didactic and experiential realm, core EPAs can also be used to substantiate competency in an experiential setting rather than assume students’ competency at the time of graduation.¹⁰ The use of EPAs in the pharmacy curriculum can also serve as a baseline measure of competency where, with training and mentoring, pharmacy students become entrusted to perform specific professional activities.^10-12 Prior to pharmacy schools incorporating core EPAs into the curriculum and assessment practices (a costly and time-consuming endeavor), it would be helpful to understand how frequently licensed pharmacists actually perform specific core EPAs in their day-to-day pharmacy practice.

This study’s primary objective was to quantify the use of core EPAs in contemporary pharmacy practice. More specifically, a survey was designed based on the AACP core EPAs and example activities, and survey responses were used to measure the extent to which pharmacists practicing in community and hospital settings in North Dakota perform them. Because of the large number of core EPAs, this study focused only on EPAs and supporting tasks categorized within the practice manager domain.

METHODS

North Dakota is considered a rural state that had an estimated population of 756,927 as of 2015. More than 50% of its counties have population densities of six or fewer persons per square mile and are categorized by the US Health Resources and Services Administration as “frontier counties.” For the purposes of this study, rural pharmacists (nonmetropolitan) were defined as those working in North Dakota communities with populations under 24,999. Urban pharmacists (metropolitan) were defined as those working in North Dakota communities with populations of 25,000 or more.¹³ The latter group primarily comprised pharmacists employed in North Dakota’s four largest metropolitan areas: Fargo/West Fargo, Bismarck, Minot, and Grand Forks.

The survey was designed using the North Dakota Pharmaceutical Care Survey as a template, and employed standard survey research criteria.^13-15 The survey incorporated all of AACP’s 15 core EPAs, including those in the practice manager domain, ie, “oversee the pharmacy operations” and “fulfill a medication order,” along with example supporting tasks for each. The “oversee pharmacy operations” core EPA included nine supporting tasks, and the “fulfill a medication order” core EPA included eight tasks. Pharmacists were asked to respond to the question, “How many times in the past 30 days have you delivered the following services in your practice setting?” for each core EPA and each set of example support tasks. To encourage higher response rates, an integer response scale, capped at an upper limit, was used. After discussions with both academic and practicing pharmacists, we determined that a pharmacist who performed an EPA 10 or more times per month (whether it was performed 11 or 50 times per month) performed the EPA with “regularity.” As a result, the response options for each EPA were: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 or more times.

The survey contained demographic items, including gender, age, highest level of pharmacy education, pharmacy practice setting, respondent’s role, population of the city in which your pharmacy is located, whether this pharmacy serves as a preceptor for NDSU pharmacy students (and, if a preceptor, the typical number of NDSU precepted students per year). There was one open-ended item that stated, “Please add anything else you would like to tell us.”

The first survey draft was designed based on the training and experience of the research team. The instrument was refined by comparing survey items against other pharmacy surveys in the literature^13,14 and subsequently pilot testing it with five pharmacists. Numerous revisions were made to the survey based on these processes. The study protocol and procedures were reviewed and approved by the NDSU Institutional Review Board. Once approved, the research team obtained a list of pharmacists’ email and mailing addresses from the North Dakota Board of Pharmacy. After removing addresses outside the state of North Dakota from the list, the investigators were left with contact information for 990 pharmacists. As much of North Dakota’s population lives along the state’s eastern border, we assumed that many pharmacists in that area might work in neighboring Minnesota. Thus, the 990 names may have been an overestimate of the study population and therefore should not be interpreted as the total population size. The investigators used a modified total design method, which has been used successfully in mail and internet surveys to obtain a meaningful response rate.^16,17 The survey instrument was emailed to all pharmacists registered and living in North Dakota in September 2018. Typically, online response rates to surveys are lower than those for paper-based surveys.¹⁷ Realizing this, the investigators increased the number of email reminders to respondents, sent out during the 2nd, 4th, 6th, and 12th week of the study, mainly by the first author. The investigators also sent out a reminder from the senior associate dean during the 8th week and from the dean of school of pharmacy during the 10th week.

Qualtrics Survey Software (Provo, UT) was used to compile the data collected from the online survey. All the respondent data was stored in a secured database and access was limited to the author responsible for data analysis. Respondents were not individually identified by the researchers in the analysis or reports. As noted earlier, all survey items related to EPAs asked respondents to disclose the number of times over a 30-day window that a particular task or activity in a domain was performed. Thus, if most respondents performed a task infrequently, the response could be treated as an (approximately) ordinal variable, and responses would vary across the continuum of response options. For tasks that were performed frequently, responses could possibly cluster around the “10 or more times” response option. To address these possible trends, we report both means and standard deviations for each item, as well as the proportion of responses for each item that report conducting the activity 10 or more times (ie, perform the EPA regularly each month) versus the proportion of respondents that report conducting the activity less than 10 times.

When analyzing whether significant differences existed in the frequency of performing specific activities across different groups of pharmacy (ie, by gender, practice setting) two different methods of analysis were used. Both methods operate under the null hypothesis of no relationship (or mean/median differences) between the frequency with which a task was completed and a specific pharmacist demographic. The chi-square test of homogeneity was used to assess possible differences in the frequency of respondents who undertook an activity 10 or more times, versus those who self-reported that they undertook the activity less than 10 times per month. This is the appropriate a priori approach to analyzing responses when pharmacists perform them frequently, and there is a clustering of observations on the upper end of the response scale. Alternatively, in cases where pharmacists perform a task infrequently, responses may be (relatively evenly) distributed across the response scale, in which case the chi-square test does not make efficient use of the data. In such cases, the Kruskal-Wallis test (ie, a nonparametric analog of analysis of variance assessing whether two or more groups of respondents exhibit similar distributions of responses) was used to assess whether specific pharmacist characteristics led to (self-reported) higher or lower frequencies of undertaking a specific task within a given 30-day window.¹⁸ The results of both tests are reported and allow the reader to make a determination concerning which test is more appropriate given the distribution of responses for a given EPA. All tests were conducted using the SPSS Statistical Package, Version 24 (IBM, Armonk, NY), and use 5% significance levels.

RESULTS

Four hundred fifty-seven people responded to the survey, resulting in a response rate of 46%. Of those who responded, 355 (78%) were licensed pharmacists practicing in North Dakota. Thus, 102 individuals met the study’s inclusion criteria. Of these 102 individuals, 33 were not employed in the state, 27 were not employed in a patient care setting, seven were not working in a pharmacy related career, 28 were retired, and seven were not currently employed; With the ca

Respondents’ demographics are reported in Table 1. Of the 102 respondents, 72% were female and the average age was 43.5 years; 48% were less than 50 years of age. Most respondents had a Doctor of Pharmacy degree (52%) as their highest pharmacy degree, while another 33% had a Bachelor of Pharmacy degree. The most common practice setting was independent community pharmacy (38%), followed by hospital (32%) and chain community pharmacy (14%). Forty percent reported being a pharmacy manager, followed by staff pharmacist (30%) and clinical pharmacist (22%). Sixty-six percent of the pharmacist responders served communities of 25,000 or more (eg, urban) with the remaining 35% serving communities of 24,999 or fewer residents (eg, rural). A majority of the respondents (51%) reported serving as an NDSU preceptor.

Descriptive statistics for the practice manager EPA domain are summarized in Table 2. The first practice manager EPA, “oversee the pharmacy operations for an assigned work shift,” was performed an average of 7.8 (SD=3.9) times per month. Within this professional activity, the most commonly occurring EPA supporting task was, “supervise and coordinate the activities of pharmacy technicians and other support staff,” which was performed an average of 8.0 (SD=3.7) times per month. Overall, the second practice manager EPA, “fulfilling a medication order,” was performed most often at an average of 9.1 (SD=2.7) times per month. Within this core EPA, “identify and manage drug interactions,” was the most frequently performed supporting task (mean=9.0, SD=2.7, times per month,) followed by, “determine if a medication is contraindicated for a patient” (mean=8.8, SD=2.8, times per month). As expected, for most items with high means, a majority of respondents reported that they performed the task 10 or more times per month. For example, 80% of respondents reported that they, “oversee the pharmacy operations for an assigned work shift,” 10 or more times per month, while 90% of respondents reported that they were “fulfilling a medication order,” 10 or more times per month.

Differences attributable to practice setting and position are reported in Table 3. Significant differences were noted among practice settings for both, “oversee the pharmacy operations for an assigned work shift” (Kruskal-Wallis p=.02; chi-square, p=.01), and “fulfill a medication order” (Kruskal-Wallis p=.02; chi-square, p=.02) core EPAs (Panel A is position) with independent and chain community pharmacists reporting higher frequencies of performance. Significant differences between practice sites were also reported for nearly half of the corresponding supporting tasks for both core EPAs, with pharmacists at independent and chain community practice sites reporting a higher frequency of performance. In regards to EPA and pharmacy position (Panel B is pharmacy setting), there were significant differences in all but four supporting tasks: “fulfill a medication order” (Kruskal-Wallis, p=.06; chi-square, p=.06); “preparing sterile or non-sterile compounding” (Kruskal-Wallis, p=.11; chi-square, p=.08); “determining if a medication is contraindicated” (Kruskal-Wallis, p=.28; chi-square, p=.29); and “identifying and managing drug interactions” (Kruskal-Wallis, p=.17; chi-square, p=.19). Examining Panel B indicates that substantial variation exists in which role self-reports conducting that activity most (or least) frequently.

Table 4 summarizes the practice manager EPA domain and example supportive tasks by population of community served. In the first core EPA, “oversee pharmacy operations,” only two supportive tasks, “maintain the pharmacy inventory” (p=.01) and “assist in the management of the pharmacy budget” (p<.01), were significant as determined by the chi-square of homogeneity test. The Kruskal-Wallis test reported the same supporting tasks as significant, as well as 2 others: “assist in the evaluation of pharmacy technicians” (p=.03) and “interpret pharmacy productivity indicators using continuous quality improvement techniques” (p=.02). In all instances, pharmacists serving populations less than 5000 residents performed these supporting tasks most frequently. For the second EPA, “fulfill a medication order,” only one supportive task, “determine the patient copay or price for a prescription,” was significant as determined by the chi-square test (p=.05) with pharmacists serving populations less than 500 residents performing this task most frequently.

Table 5 reports the practice manager domain by preceptor status (ie, NDSU preceptor and non-preceptor). Preceptors were defined as those reporting they served as an NDSU preceptor during the past year. Four of the supportive tasks: assist in the evaluation of pharmacy technicians (Kruskal-Wallis, p=.01; chi-square, p=.03); identify pharmacy service problems and/or medication safety issues (Kruskal-Wallis, p=.01; chi-square, p=.01); assist in the management of the pharmacy budget (Kruskal-Wallis, p=.01; chi-square, p=.01); and interpret pharmacy productivity indicators using continuous quality improvement techniques (Kruskal-Wallis, p=.01; chi-square, p=.03), in the first EPA (oversee pharmacy operations) were significant. Pharmacist preceptors performed these supporting tasks more frequently than non-NDSU pharmacy preceptors. The second EPA (fulfill a medication order) was by itself significant (Kruskal-Wallis, p=.02; chi-square, p=.03) but no example supportive tasks within this activity were significant.

DISCUSSION

The development of core EPAs for new pharmacy graduates attempts to create a common language intuitively understood by patients and practitioners^7,10,11 to guide pharmacy education, professional development, and serve as a baseline performance measure.¹ This EPA model is being applied to pharmacy education to further structure the curriculum and APPEs to guide preceptors in assigning greater autonomy to students in their work.¹ Entrustable practice activities can also be incorporated into milestones, which are stages within the student’s journey toward achieving a competency, or to define matriculation standards as Pittenger and colleagues did for the University of Minnesota PharmD program.¹⁰ Entrustable practice activity instruments used as descriptors of observable behaviors have also been used to establish skill assessments in high-stakes performance assessments (ie, objective standardized clinical examinations or OSCEs).^9-12 To our knowledge, however, there is a lack of data to assess the relevancy of the core EPAs in the day-to-day practice of a licensed pharmacist in different practice settings. The objective of this study was to assess the frequency of performing core EPAs as reported by community and hospital pharmacists practicing in North Dakota. This manuscript focuses on reporting EPAs and example supporting tasks in the practice manager domain only.

Of the two practice manager EPAs, North Dakota pharmacists reported performing the task, fulfill a medication order, more frequently (mean=9.1, SD=2.7) than oversee the pharmacy operations (mean=7.8, SD 3.9), perhaps because the majority of study participants were in staff or clinical positions. Pharmacy managers were more likely than others to perform eight of the nine supporting tasks associated with overseeing pharmacy operations, including, “supervise and coordinate the activities of pharmacy technicians and other support staff.” This is somewhat surprising considering the close working relationship between staff pharmacists and pharmacy technicians. Conversely, North Dakota has a unique state law requiring that each community pharmacy have a North Dakota-licensed pharmacist owning 51% or more of the pharmacy, thereby restricting the number of chain stores in the state.¹⁹ Independent pharmacies are typically managed by the pharmacy owner, who also serves as the primary pharmacist, thereby offering a possible explanation for the high response for overseeing pharmacy operations and their supportive tasks.

Considering differences among practice setting for the core EPA, oversee the pharmacy operations, significant differences were seen with five supporting tasks: supervise and coordinate the activities of pharmacy technicians and other support staff, assist in training pharmacy technicians and other support staff, maintain the pharmacy inventory, assist in the management of a pharmacy budget, and interpret pharmacy quality and productivity indicators. In all cases, pharmacists in hospital pharmacy practice settings performed those tasks less frequently than independent and chain community pharmacists. Likewise, the frequency of hospital pharmacists performing the core EPA, fulfill a medication order, supporting tasks showing statistical differences: enter patient-specific information into an electronic system, determine patient copay or prescription price, obtain authorization for a non-preferred medication, and assist a patient to acquire medications through support programs was less than other practice sites.

In regard to population served, Kruskal-Wallis analysis found five supporting tasks associated with “oversee the pharmacy operations,” with significant differences: assist in the training of pharmacy technicians and support staff, assist in the evaluation of pharmacy technicians and support staff, maintain the pharmacy inventory, assist in the management of the pharmacy budget, and interpret pharmacy productivity indicators using continuous quality improvement techniques. In all cases, pharmacists serving populations under 5,000 residents performed those supporting tasks most frequently. Again, this may be related to a higher number of independently owned pharmacies operated by the pharmacist owner and manager. None of the supporting tasks for the second EPA, fulfill a medication order, were found to have statistical differences among population size.

In our study, the practice manager EPAs were also reported by preceptor status (NDSU preceptor and non-preceptor). Preceptors (51% of respondents) were defined as those reporting they were a NDSU preceptor in the past year. In the first EPA, “oversee pharmacy operations,” five supportive tasks: assist in the evaluation of pharmacy technicians and other support staff, identify pharmacy service problems and/or medication safety issues, assist in the management of the pharmacy budget, interpret pharmacy productivity indicators using continuous quality improvement techniques, and assist in the preparation for regulatory and inspection, were significant, with NDSU preceptors performing those tasks more frequently than non-NDSU preceptors. Only one supportive task, enter patient-specific information into an electronic record, in the second EPA, “fulfill a medication order,” showed a significant difference in preceptor status.

The study results identify what will be a very important issue for academic pharmacy over the next decade. The results of this study suggest that practicing pharmacists do not frequently perform certain EPAs. This creates both “push” and “pull” incentives between the academy and practice. If these EPAs are truly essential, then additional training or experience for practicing pharmacists must be provided to ensure that these activities are entrustable in practice. The results also provide useful guidance for deciding what should be entrustable (as a graduation requirement). On the other hand, if specific EPAs are not truly essential, ie, if they are tasks that very few people do or that all pharmacists do but infrequently, then the EPA in question should be revised or eliminated. Time and resources are better reallocated towards other more frequently used EPAs.

This study has several limitations. First, to be useful as an educational outcome, EPAs require a set of underlying competencies.^10,12 The core EPAs should also make competency-based education operational through measurable units of practice (workplace EPAs).^10,12 Although we used the published list of EPA domains, activities, and supportive tasks to develop the survey instrument, the study findings suggest that, while the EPA list is a good start, it lacks reliability and validity. More specifically, EPAs should prepare pharmacists to enter practice in a variety of modalities which may not be represented in the current population of practicing pharmacists in North Dakota. A second limitation is that the EPA activity domains and example supportive tasks have not been previously tested and may represent an educational assessment that is still in process. While EPAs have been proposed as competency specific and more relevant to pharmacy practice, our findings suggest that it merely represents a beginning rather than an end to a reliable and valid list of EPAs for pharmacy practice. Pharmacy academicians often propose how practice should be in an idealistic world rather than what it is in an actual pharmacy environment. Our findings tend to support this reality. Many of the entrustable professional activities and supportive examples were not highly reported by pharmacists in North Dakota. This suggests that pharmacy practice is not being appropriately measured by this instrument, as it is lacking in validity (construct and content). In other words, it does not measure what it was intended to measure. The inconsistent response rate among many items was concerning. Another concern is the lack of distinction between practice and competence. People can undertake tasks, and do so frequently, yet fail to perform the task competently.

A third limitation of this study was the survey’s response rate. Of the 990 names initially provided by the Board of Pharmacy, only 102 met the study’s exclusion criteria and provided a full set of survey responses. Since the response rate is 10.3% (as more than 100 individuals in the 990 did not meet the study’s inclusion criteria), it is well below the threshold expected to ensure that the survey accurately and precisely reflects the underlying population. The large number of items in the survey created a disincentive to fully complete the survey, which contributed to the low response rate. In any case, the low response rate necessitates that the reader consider this analysis as a pilot study. Future research is necessary to replicate our study in a context that achieves a much higher survey response rate.

A fourth limitation is that the response scale used in this survey may lead to truncated responses. If the EPAs and supportive examples are highly representative of pharmacy practice, a respondent likely performs those tasks on a daily basis. As a result, certain items in the survey exhibited substantial proportions of respondents who reported that they performed the activity 10 or more times in a 30-day period. Within the context of statistical analysis, we accounted for this possibility by using both chi-square tests (in the event that statistical truncation occurred) and Kruskal-Wallis tests (in the event that truncation did not occur). Several items in the survey exhibited different outcomes for both of these tests (ie, one test was insignificant while the other was significant), indicating that the truncation impacted the reliability of survey responses, and by extension, the findings of our manuscript. Future iterations of this survey that used a response scale with a different (and perhaps broader) set of possible responses would facilitate more reliable responses and improved statistical estimates than are contained in the current study.

A final limitation is that the findings represent only one state, thus, they cannot be generalized to other states. They represent a cross-section of time (fourth quarter 2017 and first quarter 2018) and practice in North Dakota; thus, the findings are not necessarily representative of pharmacy practice in other states. Because EPAs are considered by many academic leaders to be the latest phase of competency training and assessment for pharmacists, EPAs warrants further refinement and assessment in other states.

CONCLUSION

This study provides empirical (albeit partial) evidence suggesting that EPAs can be a useful means to assess outcomes in pharmacy education. However, our results are limited by the validity of the current AACP EPA list. Our study sets the stage for further assessment of EPAs to measure the impact of how this process relates to outcomes of care.