An Exploration of Preceptor-Provided Written Feedback on Entrustable Professional Activities During Early Practice Experiences

INTRODUCTION

The pharmacy profession finalized a set of entrustable professional activities (EPAs) in 2016, and the Accreditation Council for Pharmacy Education endorsed shortly after.^1,2 Entrustable professional activities are tasks or responsibilities that trainees are entrusted to perform unsupervised once they have attained sufficient competence. Preceptors evaluate a student’s performance of EPAs on an ongoing basis, and adjust their level of supervision of the student accordingly over time. Originating in medical education when the Association of American Medical Colleges defined core EPAs for medical students to enter residency training, EPAs can be used to assess student progression towards practice readiness. As independently executed, observable, and measurable skills, EPAs require the integration of baseline competencies from multiple domains of pharmacy practice.^3,4 A competency is an observable ability and behavior demonstrated by a health professional that marks a specific level of performance reflecting the achievement of milestones in knowledge, skills, values, and attitudes. Each EPA statement is accompanied by a description of the expected behaviors specific to each competency milestone.^1,2,5

Using EPAs to guide preceptors’ ongoing assessment of students may help to focus their teaching on areas where students need to improve and thereby build both the student’s confidence on the journey to mastery and the preceptor’s confidence on entrustment decisions.⁷ As student pharmacists receive feedback on their performance on EPAs during each subsequent practice experience, continued reinforcement of their skills and behaviors are refined until mastery of each EPA is achieved. Literature on EPAs for pharmacy is limited compared to that available for other health professions. In pharmacy education, literature to date has largely focused on the development of EPA statements, connecting EPA statements to programmatic and accreditation requirements, defining the level of entrustment necessary for practice readiness, validation of EPA assessment tools, and pharmacy preceptor and resident perceptions on EPAs for postgraduate training.^1,6-9 A detailed study by Rhodes and colleagues of the integration, validation, and quantitative assessment of EPAs during practice experiences has been previously published.⁶

According to The Cambridge Handbook of the Learning Sciences and a report by Ten Cate and medical colleagues, high-quality, individualized qualitative feedback provides important insight into learner abilities and real-world assessment of skills.^10,11 No published studies have evaluated the contents of written preceptor-provided feedback on EPAs in pharmacy education. Thus, this study was a logical next step to further understanding the student pharmacist’s learning journey as they seek to attain adequate entrustment on the path toward becoming competent pharmacy practitioners. Reviewing the feedback provided to pharmacy students by preceptors during early practice experiences presents an opportunity to further elucidate this learning process. Therefore, the purpose of this study was to explore and analyze open-ended feedback provided by preceptors to students regarding their performance on EPAs during their first pharmacy practice experience.

METHODS

After finishing their first professional (P1) year, student pharmacists at the UNC Eshelman School of Pharmacy completed a two-month practice experience between May and August 2018 in either a community or health-system setting.⁶ During this practice experience preceptors evaluated the student pharmacists using the validated UNC EPA framework at the end of the first (midpoint) and second (final) month.⁶ Preceptors were also asked to provide open-ended comments on student performance in three sections (EPA comments, professionalism comments, and overall performance comments). At a minimum, preceptors were required to provide students with feedback in the overall performance comment section during both evaluation timepoints. The required feedback prompt read, “Please provide comments that summarize your assessment of this student. The narrative may include a summary of strengths and/or weaknesses, suggestions for improvement, or evaluate attributes not captured in the evaluation items above. You must enter text into this box to submit the evaluation.” Prior to the evaluation, preceptors underwent training on how to provide quantitative assessment of student performance. Emphasis was also placed on the need for preceptors to provide some form of written feedback on student performance, although a specific section of the evaluation in which to do so was not prescribed.

Demographic data on the students was collected from the Office of Curricular and Student Affairs. A qualitative content analysis was manually completed in Excel on preceptor-provided comments extracted from the experiential programs management system and de-identified for all students. An initial codebook was developed by two researchers (LS and AB) using the established EPA framework. Analysis was broken down into two stages.

Stage 1 involved the pilot coding of a random sample of responses as present or not and as positive performance (achieved or did well performing EPA), negative/poor performance (opportunities identified for performance improvement), or neutral (ie, no assessment of performance documented) in nature based on preceptor comments regarding EPA performance. Examples of preceptor quotes that support the definitions of positive, negative, and neutral are provided in Table 1. Feedback describing the exposure or student’s comfort with completing a task was coded as neutral given a lack of evidence outlining the student’s ability to perform a particular EPA function. For example, if a preceptor documented that a student had exposure to taking medication histories but did not provide an assessment of the student’s ability to perform this EPA then the comment was coded as neutral. Two of the researchers then reviewed all piloted codes. Next, all of the researchers, including an experienced qualitative researcher, met to reach consensus on the codes and finalized the codebook. Responses were coded for all assessments present (ie, feedback that included positive and negative performance were coded as both positive and negative). Preceptor feedback often included reference to student-completed tasks that were difficult to associate with a specific EPA and therefore difficult to code. For example, vaccine check-ins, adherence calls, wellness visits, and medication therapy management were tasks frequently referenced by community preceptors. Vaccine check-ins could be coded as vaccine administration (EPA 14) or as monitoring/follow-up (EPA 8) depending on the actual activity performed by the learner; however, that was not specified. Therefore, these tasks were not coded and therefore not included in the study results in order to prevent overinterpretation of the data.

Stage 2 involved coding all preceptor responses as positive, negative, or neutral performance for all evaluations. Consensus was reached by two researchers regarding all codes. Data were then grouped to identify the focus of feedback (knowledge, skill, and behavior) provided by practice setting within the codebook by three researchers who reached consensus.

Throughout the study, the primary investigator conducted a dependability audit to track contacts made with the data. The audit was reviewed by research team members to ensure reliability and trustworthiness. This study was exempted for review by the Institutional Review Board at UNC.

RESULTS

Of the 153 student pharmacists who began the PharmD program in fall 2017 (class of 2021), the majority were female (n=103, 67%). Overall, the demographic data were representative of the internal student admission demographics within the school’s PharmD program. A total of 306 preceptor evaluations (midpoint and final) for the 153 student pharmacists were extracted from the school’s database. Of the 83 student pharmacists who completed a practice experience between May 1 and June 30, 2018, 41 (49.4%) completed the experience in a community pharmacy setting, while 42 (50.6%) completed an experience in a health-system setting. Of the 70 student pharmacists who completed a practice experience between July 1 and August 31, 2018, 39 (55.7%) completed the experience in a community setting, and 31 (44.3%) completed the experience in a health-system setting. Six health systems (15 hospitals) and 44 community pharmacies in North Carolina hosted student pharmacists for this early practice experience. A total of 60 unique hospital and 65 unique community preceptors were identified in the experiential programs management system and contributed to written feedback on student performance. Students were evenly spread across preceptors, with only a few preceptors precepting two learners simultaneously and/or in both time periods.

The frequency of positive, negative, or neutral EPA coding reported by preceptors in open-ended responses is presented in Table 2. Preceptors from both settings provided overwhelmingly positive feedback on all EPAs. Example quotes for corresponding positive and negative codes by practice setting are provided in Table 3. The feedback focus areas by practice setting and supporting examples are depicted in Figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Areas of Feedback Emphasis by Practice Setting

Documentation style varied noticeably between preceptors, regardless of practice setting. For example, one preceptor only documented that a student was “making good progress,” another only wrote “great student,” while a third preceptor outlined the opportunities provided at the site that aligned with each EPA (eg, the student “has been exposed to a variety of situations and patients”). Most preceptors provided feedback in the overall performance comment section. Preceptors in community settings did not report on students’ specific performance of clinical EPAs 1-11 as frequently as preceptors in health-system settings did. Preceptors from both community and health-system settings provided written feedback on EPAs 12-14, which focus on operational EPA performance. Health-system preceptors commented on student performance of EPAs more often than community preceptors (498 vs 315, respectively) and tended to provide more negative assessments of student performance compared to community preceptors (70 vs 21, respectively). Frequency of neutral comments was similar among preceptors at health-system and community settings (118 vs 100, respectively).

Responses from preceptors in health-system settings focused on knowledge performance and skill-based achievement. Positive comments coded to EPAs 1 (n=29), 2 (n=38), 3 (n=30), and 8 (n=28) emphasized the student pharmacist’s ability to integrate these EPAs in relation to patient care activities. Positive feedback coded for EPA 12 (n=47) discussed appropriate integration of operational performance with medication preparation, product verification, and distribution. Neutral feedback by preceptors in the health-system setting was most frequently coded to EPAs 2 (n=14) and 12 (n=23). This neutral feedback primarily focused on exposure to the skill or activity and the student’s confidence or willingness to contribute, but did not provide an assessment of the student’s actual performance on the task or skill. The majority of preceptor feedback that was coded negatively was for EPA 1 (n=17) and emphasized the need for students to focus on improving their independence and taking ownership of their education by researching challenging or unfamiliar concepts/disease states in advance, particularly by referencing appropriate guidelines and primary literature. Other negative preceptor feedback on other EPAs also commented on opportunities for students to take ownership of their learning and become more independent while improving efficiency (EPA 3, n=9 comments).

Feedback provided by preceptors in community settings primarily emphasized behavior and skill performance. Preceptor comments provided on EPAs 2 (n=25), 4 (n=21), and 7 (n=18) in community settings particularly emphasized a student’s strong positive performance related to adapting during patient interactions. Positive feedback coded for EPAs 12 (n=27 comments) and EPA 14 (n=36 comments) described preceptor comfort with the student’s increased independence in performing EPAs and rapid integration of EPAs into pharmacy operations. Neutral feedback provided by preceptors in the community setting was coded most frequently to EPAs 12 and 14. This was usually because preceptor comments only described student exposure to certain tasks (n=19) and functions (n=31). Negative feedback from preceptors in community settings was infrequent, with EPA 12 being the activity most often coded negatively (n=5).

DISCUSSION

This study explored and evaluated open-ended feedback on EPAs provided by preceptors to student pharmacists completing early pharmacy practice experiences. Literature from the learning sciences has reported that providing high-quality, individualized feedback is critical to the learning process.¹⁰ Optimal feedback is reliable, skill-specific, timely, and flexible, allowing for learner repetition and practice of the evaluated skills at separate points in time. Optimal feedback also provides correction, evaluation, and specific direction for progress, and is structured around common and consistent predetermined rubrics for tasks.¹⁰ Duijn and colleagues conducted a study in 2017 that evaluated medical students’ perceptions of meaningful feedback on EPAs.⁷ The researchers reported that student’s most valued feedback emphasized both their strengths and areas for improvement and was delivered in a safe environment by a credible, trustworthy supervisor who knew the student well.⁷ The findings of Duijn and colleagues add further weight to the importance of providing optimal feedback on EPAs. Consistent with the medical education literature, the findings of this study broadly show that written preceptor feedback was provided across all EPAs and comprehensively included positive, neutral, and negative feedback.^7-9 Further, while feedback varied in meeting the definitions for “optimal feedback” as recommended by scholars in the learning sciences, key opportunities identified for preceptor development include the provision of directional, goal-oriented, and skill-specific feedback. This study was a first step in highlighting a potential area of opportunity to enhance preceptor-provided written feedback on EPAs in pharmacy curricula.

The learning science literature does not broadly quantify the ideal distribution of positive, neutral, and negative feedback. However, the successful progression of the vast majority of student pharmacists in our pharmacy curriculum during this practice experience suggests that the high frequency of positive feedback observed in this study may be consistent with student ability to perform these EPAs. The high prevalence of positive feedback provided to student pharmacists may also be an indication that positive feedback feels better and is easier to provide. Because the evaluations assessed in this study are considered official written evaluations of student performance and/or preceptors may have wished to increase confidence within an early learner, this may have also influenced the overwhelming amount of positive feedback provided.

There are many possible reasons why the frequency of neutral and negative feedback may have occurred in this study. Factors such as time available (either to assess the student or document student performance), challenges in providing feedback for high performers or struggling students, and differences in feedback provided in writing vs verbal feedback provided during an informal discussion may also explain the neutral and negative feedback given. Further, there may be limited training available to preceptors on how to provide individualized feedback to student pharmacists and a progress plan toward achieving appropriate levels of entrustment. Roze des Ordons and colleagues studied medical resident preceptors who reported feeling uncertain regarding how to individualize feedback, navigate tensions between residents and preceptors over identified goals, coach residents with insight gaps, and mediate overly confident residents.⁸ Additionally, preceptors may have felt less certain on how to tailor the set level of EPA entrustment to their unique practice setting and specialty. In a study conducted by Moon and colleagues, preceptors reported being less certain of what the EPA entrustment level should be to consider an objective or skill achieved for residency.⁹ Lastly, workplace constraints such as differences in institutional regulations allowing for student pharmacists to perform various EPAs and/or preceptor comfort with the level of entrustment permitted for a learner may also explain the findings of this study, particularly the neutral feedback provided.

An emphasis on knowledge and relationship skills between pharmacy practice settings were consistent with reported literature and residency accreditation standards in each setting.^12-14 Preceptor-provided feedback on students’ knowledge, skills, and behavior suggests differences and similarities exist between practice settings in terms of their patient and business models. Greater documentation by health-system preceptors regarding students’ knowledge and skill performance in health-system settings may be the result of a difference in students’ experiences in these settings, ie, the level of care acuity, engagement in face-to-face collaboration with the treatment team, and time-sensitive nature of the medical care provided. Alternatively, the greater emphasis placed on behavior performance in the positive feedback provided by preceptors in community pharmacy settings may have been the result of students’ frequent interaction with the same patients and/or population, performance markers related to adherence, and the types of clinical services offered (eg, medication therapy management and motivational interviewing). This explanation is further supported by current literature indicating a strong association between pharmacist participation in the pharmacist-patient care relationship and the perceived quality of the pharmacist-patient relationship within the community setting.^15,16 While the EPA framework used in this study does not explicitly collect behavior performance, this is implied by the functions associated with each EPA and is assessed quantitatively as a component of the overall evaluation tool. These areas of emphasis suggest that appropriate, professional behavior and attitudes may be necessary for successful performance of EPAs.

The findings of this study provide valuable insight for pharmacy schools, especially those engaged in overseeing experiential training as well as institutions currently assessing or considering EPA feedback assessment. Our study findings emphasize the value of student pharmacists receiving written feedback on their EPA performance. This is consistent with medical and nursing literature indicating the need to evaluate students’ skills at separate timepoints throughout longitudinal experiences to ensure optimal progression.¹² This research also indicates the need to provide more opportunities for training and support for new and established preceptors regarding the provision of optimal feedback to students. Additionally, it is important for each institution to determine and communicate to preceptors the expected level of documented feedback on student EPA performance. Further, the neutral and negative feedback that preceptors gave students in this study may indicate a need to provide preceptor training regarding evidence-based methods for delivering feedback effectively to struggling students in addition to those who are performing well. Active, ongoing dialogue with current preceptors regarding development opportunities and beneficial training related to providing feedback to learners is also critical. It is equally key to provide guidance and support to learners on how to interpret feedback and identify appropriate next steps for progression to practice readiness. The need for this is further emphasized when the influence of student engagement and motivation during practice experiences is considered. Finally, professional development and resources that educate both preceptors and students on how to set appropriate expectations early on is also an essential consideration.

This study had some limitations worth noting. The EPA statements used for this analysis were developed specifically for use at our institution and differ slightly from the American Association of Colleges of Pharmacy Core Entrustable Professional Activities for Pharmacy Graduates.³ Our program has previously mapped the EPA framework used at our institution to the AACP Core EPAs for Pharmacy Graduates, suggesting that our EPA framework may be considered generalizable.⁶ Similarly, preceptors were not required to comment on every question with an open-ended response. However, the overall performance comment section was required, which may explain why the majority of responses were provided in this section. This may indicate that in some instances optimal feedback may have been provided to the learner but was not captured for analysis in the study if a preceptor chose not to document the feedback provided. Additionally, the feedback provided by preceptors may have been influenced by the learners’ early stage in training. Finally, the findings of this study may not be representative of the feedback on EPAs provided to learners on advanced practice experiences.

CONCLUSION

As EPAs continue to become imperative in documenting student pharmacist progression, subjective assessment of student performance longitudinally will play an important role in illuminating the student’s abilities and progression to practice readiness.^3,4,11 Results from this study suggest that subjective, preceptor-provided feedback can provide greater insight into the strengths and challenges of student pharmacists’ progression on practice experiences. Qualitative studies using structured interview or focus group methodology are needed to explore the collective implementation experiences of preceptors regarding EPA assessment in all real-world practice settings.