Pharmacy Student Knowledge Retention After Completing Either a Simulated or Written Patient Case

INTRODUCTION

The Accreditation Council for Pharmacy Education standards allow for the structured use of simulation as part of pharmacy students’ required introductory pharmacy practice experiences.¹ Additionally, simulation is offered as a tool to facilitate interprofessional interaction. Simulations create a realistic setting in which standardized scenarios allow students to safely and repeatedly practice skills in a consistent environment.² The literature regarding the use of simulation mannequins in the doctor of pharmacy (PharmD) curriculum has focused on teaching interdisciplinary team skills and practicing advanced cardiovascular life support (ACLS).^3,4 Simulation technology also has been used to evaluate pharmacy students’ competency in performing physical assessments and providing therapeutic recommendations.^5-7 There has been minimal research investigating student retention of knowledge acquired through participation in mannequin-based simulations.

A mannequin-based simulation course was used to teach and evaluate ACLS protocols in second-year internal medicine medical residents.^8,9 The study showed improvement in residents’ knowledge and skills from baseline, and follow-up studies at 6 months and 14 months found no significant deterioration in residents’ ability to implement ACLS protocols. Nurse anesthetists’ knowledge and skill of ACLS protocols were evaluated before, immediately after, and 3 months after a brief ACLS course using a simulation mannequin.¹⁰ The study found significant immediate improvements in knowledge and skills; however, after 3 months, the nurse anesthetists’ scores reverted to pre-course levels.

There is no published research comparing outcomes from participation in simulation-based patient cases with participation in written patient cases in the PharmD curriculum. Prior to this study, there was no elective course at the University of Tennessee College of Pharmacy using simulation to allow students to practice clinical scenarios in an active-learning format. The Drug-Induced Disease elective course used team-based learning to teach identification and management of commonly encountered drug-induced diseases. This seemed the most appropriate course in which to conduct this study, as it would allow students to take active-learning a step farther than discussing clinical scenarios. The primary aim of this study was to determine whether completing a patient case using a high-fidelity mannequin results in greater student knowledge retention than completing a similar written patient case. The secondary objective was to determine whether students’ comfort level with completing the patient case was influenced by the type of learning format used.

DESIGN

This study was an IRB exempt, parallel-group, randomized controlled trial. All fourth-year PharmD students enrolled in the month-long Drug-Induced Disease elective in February 2011 were included in the study. Teams of 2 to 3 students were randomly assigned to 1 of 2 teaching methods: written case or simulated patient case using a high-fidelity mannequin. The case that the students were asked to complete involved a patient who had taken an overdose of a narcotic and acetaminophen (Appendix 1). Prior to study participation, students were asked to read 2 articles on the use of acetylcysteine and naloxone to prepare them to complete the patient case. Students were excluded from the study if they were absent on the day on which the exercise or study case took place.

The Drug-Induced Disease course is a month-long elective course offered to fourth-year PharmD students. This course provides students with an organ-system approach to recognizing and treating adverse drug reactions. This patient case was incorporated into the course to give students the opportunity to treat an acute drug misadventure. Because of the team-based learning format used in the course, students were accustomed to being given a required reading assignment prior to the class session and taking a quiz at the beginning of each class.

Sim-Man 3G and corresponding software (Laerdal Medical, Stavanger, Norway) were used in the simulated case. A hemodynamic monitor was displayed in the simulation environment and controlled by a trained operator. The student teams who participated in the simulated case stood in the simulation area, which was arranged to look like a room in an emergency department. The simulation mannequin and hemodynamic monitor were programmed to exhibit the symptoms and vital signs of a patient experiencing narcotic overdose (eg, pinpoint pupils, hypoxia). The students were asked questions by a simulated physician and nurse (portrayed by pharmacy residents and faculty members) about drug selection, dosing recommendations, and drug monitoring. Additionally, students were asked to prepare medications for administration to the patient. As medications were administered, the mannequin responded accordingly. All simulations were modified slightly to allow each of the students in the team to participate in medication preparation, patient assessment, and drug information.

After the student teams completed the simulation, they were debriefed by a pharmacy faculty member who discussed all points covered on the multiple-choice test, but did not provide answers to specific questions. A debriefing tool was used to ensure that information was consistently addressed with the simulation group as it would be with the group completing the written case. Any questions or issues that arose during the simulation that would not usually occur between a physician and pharmacist in a similar real-life situation were addressed in the debriefing session.

The student teams that completed the written patient case were given a packet that contained the case and a series of questions. The questions required students to use subjective and objective data to assess the patient, formulate a problem list, create a patient-specific therapeutic monitoring plan, and provide instructions for medication preparation.

Students in both groups were provided a Lexi-Comp Drug Information Handbook and a copy of the Rumack-Matthew nomogram for managing acetaminophen overdose.^11,12 Students were also allowed to use personal drug information resources. After student teams in the written-case group completed the exercise, a pharmacy faculty member reviewed the case; however, as with the simulation group, he did not offer answers to specific questions on the test.

EVALUATION AND ASSESSMENT

To measure knowledge retention, students were given a 15-question multiple-choice test at 3 different intervals: before the case (pretest), immediately after the case (posttest), and 25 days later (retention test). The same 15 questions were used in each test. In an attempt to minimize pattern recognition, the questions were reordered on each test. Students were not told how they performed on the tests or what the correct answers were until all phases of the study were complete.

To evaluate student perceptions of the experience, subjects completed a 5-statement survey instrument. The survey tool was administered at the same 3 time points as the multiple-choice test. Responses were based on a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree), with a maximum score of 25. The pretest survey instrument contained 4 additional questions to determine whether the students had any previous experiences that might influence their performance during the simulation or on the written-case exercise.

A mixed ANOVA was used to test for significant main effects across time and between learning interventions in both the multiple-choice tests and the comfort survey. Mauchly’s test of sphericity was used to assess the statistical assumption of sphericity for the analysis. The Bonferroni correction was used to explain any significant main effects. Fisher exact tests were used to evaluate the students’ responses to the questions asked at baseline relating to previous experiences. All analyses were conducted using SPSS, Version 19 (SPSS Inc., Chicago, IL), and significance was assumed at a p < 0.05 level. The assumption of sphericity was not violated for the knowledge retention analysis according to Mauchly's Test of Sphericity.

Twenty-six students were enrolled in the study. There were no significant differences in students’ previous experiences between groups at baseline (Table 1). Also, there was not a significant difference between the groups with regard to knowledge retention (p = 0.85). There was a significant difference in knowledge retention across time between the 3 testing points (p < 0.001). Therefore, post hoc analyses were conducted and significant differences were found between pretest and posttest (p < 0.001), and between pretest and the retention test (p = 0.001). There was not a significant difference between posttest and the retention test (p = 0.99).Marginal means for the interaction appear in Table 2.

The composite data obtained from the comfort survey indicated a violation of the assumption of sphericity (p < 0.001), and there was not a significant difference between the groups (p = 0.52). There was a significant difference across time with regard to self-reported comfort over the 3 trials (p = < 0.001). Therefore, post hoc analyses were conducted and found a significant difference between comfort levels at baseline and posttest (p < 0.001), but there was no difference in comfort levels between the posttest and retention test. The means for self-reported comfort are presented in Table 3.

Resources used to conduct this study included the simulation laboratory space, Sim-Man 3G (Laerdal Medical, Stavanger, Norway) and corresponding software, 2 faculty members and 3 pharmacy residents in the simulation laboratory (one of whom was a trained simulation operator), 1 faculty member in the classroom, and extended class time to allow multiple groups of students to participate in the simulation.

DISCUSSION

To our knowledge, there are no previous studies comparing PharmD students’ knowledge retention after participating in a paper-based patient case versus after participating in a simulated patient case. As seen in the nursing literature, students’ test scores improved immediately after participating in either type of patient case.¹⁰ A decrease in test scores in the collective group of students from posttest to retention test was seen, but unlike what was seen in other studies, the decrease was not significant and students’ scores remained significantly higher compared to pretest scores. When comparing test scores between the written and simulation groups, test scores of students in the written-case group decreased slightly from posttest to retention test, while test scores in the simulation group remained the same. The difference in scores, however, was not significant.

There were some interesting findings concerning students’ comfort levels. Similar to the results with the multiple-choice test, an immediate increase in students’ comfort level with the material was seen after participating in the patient case; however, the change in scores between the posttest and the retention test was unexpected. The comfort levels of students in the written-case group increased when tested 25 days after completion, while the comfort levels of students in the simulated-case group decreased. One possible explanation is that students in the written-case group felt more comfortable because of the traditional learning format and/or the classroom setting. Another possibility is that by participating in a simulated patient case, students acquired practical, hands-on experience and therefore could more accurately self-assess their knowledge and comfort than could students in the written-case group. If their self-assessments were accurate, then the self-assessments of comfort level by students in the written-case group is unsettling because their knowledge, as reflected by scores on the retention test, declined, yet their comfort level did not reflect this.

The researchers observed some differences between the groups that were not reflected in the findings from the multiple-choice test or comfort survey. First, students in the simulation group were more enthusiastic and engaged in the activity than students in the written-case group. Students who participated in the simulation were also more interactive in the debriefing session that followed than were students who discussed the written patient case after completing it.

Although not a comparison between the 2 groups, the instructors also observed that the students in the simulation group used poor medication preparation technique. Use of appropriate aseptic technique was assessed by 1 of the questions on the multiple-choice test which asked the students to appropriately sequence the steps necessary to prepare a parenteral medication. Although most students answered this question correctly, the students in the simulation group rarely demonstrated appropriate aseptic technique. Providing the opportunity for students to practice and receive feedback on hands-on pharmacy practice skills, like intravenous admixture, is an important benefit of simulation. Using simulation to identify areas for skills development was described in a review discussing the value of simulation to improve pharmacy practice.¹³

This study was limited by several factors. First, the study enrolled fourth-year pharmacy students in their last semester of their PharmD training. At the time of this study, this class of students had completed a full year of clinical practice experiences. Therefore, students had gained clinical experience that may have influenced their performance. Second, the study was small, only enrolling 26 students, and therefore, likely underpowered. Another limitation is that the same test was administered 3 different times. Steps were taken to minimize pattern recognition, but students likely remembered some of the test questions from previous attempts. Finally, the 25-day period between posttest and retention test was shorter than that in previous studies and may not be an adequate interval to observe changes in knowledge retention.

Although advantages and disadvantages of incorporating simulation into healthcare education have been described,¹⁴ the optimal use of simulation in pharmacy education continues to be evaluated. The considerable amount of resources required to complete this exercise and the fact that posttest scores, although significantly improved from pretest scores in the simulation group, did not improve significantly more than those in the written-case group must be considered. However, students’ excitement while participating in the simulation may be an inciting factor to continue this method of teaching.

More simulation exercises will be incorporated into future offerings of the Drug-Induced Disease elective, as it appears to be as effective as a written-case teaching format and encourages student interest and participation in the activity. Future studies should enroll more students earlier in the PharmD curriculum and allow for more time between administration of the posttest and retention test. Additionally, as described by a review of assessment practices for simulation-based activities,¹⁵ careful consideration should be given to using reliable assessment tools aimed at measuring not only satisfaction and knowledge but also encompassing clinical performance and critical thinking.

CONCLUSION

We found no difference in knowledge retention after a 25-day period between students who participated in a simulated patient case and those who completed a written patient case. We did see that students’ participation in a patient case (written or simulated) led to significant improvements in test scores within a short period of time. There was an insignificant decrease in test scores of students in the written-case group after 25 days, whereas the scores of students in the simulation-case group remained constant. The role of simulation technology in the PharmD curriculum is still being elucidated. Given our students’ enthusiasm for learning in the simulation environment, further research into the most appropriate use for simulation in the PharmD curriculum is warranted.