Implementation of Mock Acute Care Advance Pharmacy Practice Experience Simulations and an Assessment Rubric

INTRODUCTION

While pharmacy students are encouraged to cultivate clinical knowledge and patient skills throughout their didactic training, there is a limit to becoming proficient in performance-based competencies without the use of activities that closely mirror actual patient care.¹ Recognizing this challenge, schools of pharmacy are expanding use of mock or patient simulation activities in an attempt to enhance student application of knowledge, communication, and professionalism in structured “patient safe” settings.^2,3 Many schools are creating active-learning centers that recreate or mimic patient care delivery settings as well as specific clinical situations (eg, Clostridium difficile infection or critical care).^4-6 Despite widespread use of simulation activities for clinical specialties, there has been a lack of focus on scenarios relating to acute care, defined here as the practice of pharmacy within a hospital setting as it relates to an acute patient problem. This includes patient care activities that pharmacists will encounter frequently in an inpatient setting.

Effective assessment of integrated patient clinical skill performance via simulations is also important and requires a critical evaluation tool that objectively examines competency. Rubrics and measures in this regard are currently limited, and are primarily centered on student and faculty self-assessed surveys (indirect data), or are adapted from assessments developed for use in medical or nursing students.^4,7 In accordance with the Accreditation Council for Pharmacy Education’s updated standards for proficiency in student pharmacists to enter advanced pharmacy practice experiences (APPEs), educators now face a comprehensive list of essential skills to evaluate in student pharmacists.² The goal of the following study was to implement mock acute care APPE simulations (MACAS) throughout the second-year pharmacy (P2) curriculum, and validate an assessment tool to evaluate and track student performance in the simulations.

METHODS

An assessment rubric was created to measure student competency during each MACAS and to gauge student progression over time (Appendix 1). To provide a level of content validity, the rubric was developed by two board-certified clinical pharmacy faculty members with input from all of the college’s acute care preceptors. The initial rubric was subsequently pilot-tested on two postgraduate year one (PGY1) pharmacy residents. The residents evaluated a virtual MACAS patient and presented relevant clinical information as if they were presenting the case to an attending pharmacist or physician. The presentations were recorded and subsequently scored independently by 10 clinical pharmacy faculty members using the assessment rubric. As a measure of criterion validity, resident scores on the individual rubric items as well as their overall test scores were compared to identify any outliers (items or scorers). Minor changes were made to the rubric after initial pilot testing.

Prior to the implementation of the MACAS, the college had previously developed simulation sessions related to ambulatory and community setting patients, with no focus on the acute care setting. The MACAS were developed with input from various stakeholders, including acute care preceptors, PGY1 residents, and acute care faculty members. Three MACAS were implemented in the P2 spring semester didactic curriculum at Touro University California College of Pharmacy (TUCA-COP). The college is a 2+2 curricular-based school, with two primarily didactic years followed by two experiential years of training. Each of the three sessions spanned two consecutive Friday mornings, with half (n=52) of the class assigned to participate in each Friday session. Students were scheduled for a 90-minute time slot for each MACAS, which consisted of 60 minutes to do a work up on an acute care patient, followed by 15 minutes to present the patient case to a faculty preceptor, and 15 minutes allotted to the faculty preceptor to provide feedback to the student. Faculty preceptors were all TUCA-COP faculty members who had practice experience in acute care. The majority of the faculty preceptors also were involved in teaching the didactic curriculum but not necessarily all acute care topics.

During the 60-minute patient work-up, students used an electronic medical record to find pertinent patient information as it related to various patient disease states. For example, the first MACAS involved a patient admitted to the hospital for altered mental status secondary to a urinary tract infection. In addition, the patient had a past medical history of chronic kidney disease and various other minor chronic medical problems. Students were tasked with creating a prioritized list of medical problems. For each medical problem, students assessed laboratory values and current drug therapy to evaluate each medical problem and formulate a drug therapy plan and appropriate monitoring parameters. Students repeated this process for each of the patient’s medical problems. Each student individually performed these tasks using a formalized patient monitoring form created by a clinical pharmacy faculty member specializing in acute care.

After the student completed the patient work-up and/or the 60 minutes had expired, the student met one-on-one with a faculty preceptor. During the patient case presentation, the preceptor used the assessment rubric to assign a point value for the student’s performance on the MACAS. While students were not provided a copy of the actual assessment rubric prior to the evaluation, they were informed of the rubric’s major content areas and the relative weighting of each. After the student presentation, the preceptor provided formative feedback to the student. The feedback was focused on improving the student’s ability to use an electronic medical record to find pertinent patient information and formalize a succinct patient presentation that addressed all of the patient’s medical problems. This preceptor feedback was intended to improve student performance on subsequent MACAS and APPEs. While neither the presentation nor feedback sessions were recorded, students were provided with their rubrics along with any specific notes taken by the preceptor.

To determine inter-rater reliability of the rubric, a subset of faculty (raters) were chosen to be paired with another faculty rater in each of the three MACAS sessions. Faculty parings were assigned upon arrival and were dependent upon the number of “extra” faculty raters volunteering for each session. The faculty rater pairs independently evaluated a random subset of students and a measure of inter-rater reliability was calculated using Krippendorff's alpha.^8,9 The predictive validity of the MACAS assessment rubric was determined by calculating the Pearson correlation coefficient between the three sessions, the mean average of the three MACAS, and the student’s acute care APPE scores. Individual component scores of the APPE included communication, professionalism, and patient care. A total or composite acute care APPE score was also analyzed. The APPE scores were obtained from the students’ experiential training (ie, during their proceeding final two years at TUCA-COP). Students’ grades for APPEs at TUCA-COP are based on a point system, with each domain consisting of 100 points. A score of 70% (or above) in each domain is needed to pass the APPE.

Student MACAS rubric scores were compared across the three sessions (time) using an analysis of variance (ANOVA) model with a Bonferroni follow-up comparison test. Upon completion of the three sessions, an online seven-item survey was distributed to the students. The student survey was developed by the acute care faculty members and preceptors at TUCA-COP. The faculty members first created a rough draft of the survey instrument, which was then distributed to the other Touro faculty members for feedback. The survey instrument asked students to self-assess what their readiness for acute care APPEs had been prior to undergoing the MACAS. The survey instrument also queried them regarding the perceived benefit of the sessions using a series of questions rated on a four-point Likert-type scale. Results from the survey were recorded as medians, along with means and standard deviations. An a-priori p value of <.05 was considered significant. The study was reviewed by the Touro University California Institutional Review Board and determined to be exempt.

RESULTS

One hundred six P2 students (2017 graduation year cohort) completed the MACAS series in spring 2015. Two students subsequently withdrew or were held back from the program during their P2 academic year. In addition, three students missed one or more MACAS and their scores were not included in the final cohort of 101 students. Results of the student scores across the three sessions are shown in Table 1. Overall results were significantly higher over time (ANOVA, p<.05). Follow up comparisons found a significant increase between MACAS sessions one and two and sessions one and three. A significant difference was not found between sessions two and three.

Inter-rater reliability for MACAS scores ranged from 0.90 to 0.93 across the three sessions and averaged 0.91 overall, exceeding the accepted or customary cut off ranges.¹⁰ More details of the inter-rater reliability calculations can be found in Table 2. Correlation coefficients for the student MACAS scores and their subsequent acute care APPE scores are shown in Table 3. The correlations ranged from 0.04 to 0.80. Average MACAS and acute care APPE scores were significantly related, suggesting student MACAS performance might be a good predictor of future acute care APPE performance. However, student acute care APPE communication and professionalism scores were not significantly related to MACAS performance.

Results from the student survey (response rate=61.5%) are shown in Table 4. Students indicated a low degree of readiness for acute care APPEs prior to the sessions and reported improvement of skills following the MACAS intervention. Specific skill areas surveyed included ability to navigate electronic medical records, prioritize medical problems in acute care patients, and present patients to preceptors. Students also reported that feedback they received during the initial MACAS improved their performance on subsequent sessions and helped them gain a better understanding of the pharmacist’s role in an acute care setting. No changes were deemed necessary in the basic format of the MACAS or scoring rubric based on results of the student survey.

DISCUSSION

A series of mock acute care APPE simulations (MACAS) was successfully implemented during the P2 academic year at TUCA-COP, along with a rubric for use in assessing and tracking activities. While other studies describing pharmacy students’ performance in patient simulations exist, they tend to be critical care or code based (eg, cardiopulmonary arrest situation), accompanied with the use of a mannequin, elective in nature, focused on a single case, and/or set in an ambulatory care setting.^6,11 This is believed to be the first study examining at the use of a mock acute care APPE simulation experience that was required of all students. In addition, this study provides evidence of validation of a rubric and reports a link between student simulation performance and future performance in a required acute care APPE. While the mock patient simulations in this study were used in a 2+2 curricular program, they may be equally suitable for use in more traditional 3+1 programs.

The importance of using patient simulation to enhance pharmacy student learning has been highlighted in the literature.¹² However, there has been limited research focused on didactic activities relating to student readiness for acute care APPEs and the associated assessment tools used. A recent study by Smith and colleagues found significant improvement in students’ self-assessment scores before and after completing simulations.¹ However, Smith and colleagues’ research was based on outpatient pulmonary and rheumatology content, while our study focused on acute care scenarios. We found an overall positive student response to using simulations to enhance learning. Earlier work by Seybert and colleagues also found a similar positive student reaction to the use of mannequins as patients in a critical care type simulation, highlighting the potential benefit of patient simulation activities.^6,11

Similar to prior work by Bray and colleagues, we demonstrated high overall inter-rater reliability or consistency for an assessment rubric created for a student patient simulation activity.⁴ The inter-rater reliability analysis for our study used “pairs” of raters on a sub-sample of 10 students per MACAS or activity (three total), as opposed to four raters on what was reported to be 23 groups of three to four students by the prior researchers. Whether the patient simulation activity described by Bray and colleagues was repeated more than once during the student’s training in pharmacy school was not clear. In addition, the lack of an apparent individual or single student assessment score by Bray and colleagues would appear to limit the ability to link their assessment with future student APPE performance, somewhat hindering a proper validation of their work.

Several limitations of this study should be noted. First, the research was limited to a single class at a single college of pharmacy and the results may not necessarily be generalizable to future class cohorts, other schools, or other topic areas (eg, ambulatory care or community pharmacy). Second, although we found relatively high inter-rater reliability and good predictive ability for the MACAS rubric, good reliability and predictive ability do not guarantee the validity of an assessment. While MACAS appear to be a predictor of future APPE acute care performance related to patient care, student APPE communication and professionalism scores were not significantly related to the mock simulation performance. These skills are perhaps more universal, and we hypothesize that students may have had an opportunity to further refine these skills on their first (eg, institutional or community pharmacy) APPEs. Lastly, we do not know whether the assessment would be useful for tracking cohorts of P2 classes over time, especially as curriculum changes are made and/or student admission characteristics shift.

CONCLUSION

Pharmacy students will undoubtedly encounter many patients in an acute care setting during their experiential training. While these patients often have overlapping chronic problems, the reason for the encounter is primarily driven by an acute problem, such as an infection, complication after a procedure, or a cardiovascular event. These patients may or may not also have an intensive care component associated with their encounter. This is one of the first studies to date to describe the successful implementation of acute care patient simulation or mock activities into a pharmacy curriculum and provide a link to APPE performance. Given the findings, TUCA-COP has incorporated the use of MACAS into the curriculum indefinitely. Other schools seeking to enhance their students’ preparedness and performance for acute care APPEs should consider implementation of a patient simulation laboratory in the didactic curriculum that is focused on acute care.