Student Self-Assessment and Faculty Assessment of Performance in an Interprofessional Error Disclosure Simulation Training Program

INTRODUCTION

Every year approximately 250,000 people in the US die as a result of medical error, making it the third leading cause of death in the country.^1,2 It has been advocated that training in medical error disclosure is needed to create a culture of safety and build trust among health professionals and patients.^3,4 While training in medical error disclosure has been undertaken through a variety of teaching methods, interprofessional education (IPE), and by single health professions (physician-in-training programs provide numerous examples of medical error disclosure training),^5-17 training in disclosing medical errors in dental medicine, nursing and pharmacy education remains limited.^18-22

Although dental medicine, nursing and pharmacy programs present limited examples of single discipline error disclosure training, there is a growing number of IPE error disclosure trainings that include these professions as well as a variety of other health professions.^23-26 Error disclosure as a framework for IPE is an ideal combination, as patient safety is a theme across health professions. The availability of IPE error disclosure training toolkits, curriculum, and training programs has aided health profession programs in implementing IPE error disclosure.²⁷ As the number of IPE error disclosure experiences begin to increase, there is a need for enhanced assessment of these experiences.

Assessment strategies should consider a combination of two objectives, error disclosure and interprofessional team performance. Oakuyama and colleagues reviewed tools that are used to assess patient safety competencies of health care professions and discovered that a tool to assess the disclosure of adverse events according to Miller’s four competency levels: “knows, knows how, shows how, and does currently” does not exist.²⁸ Although the optimal assessment tool does not exist, physician-in-training error disclosure examples in the literature provide a variety of assessment tools and strategies with several examples reporting improvements in knowledge and self-efficacy.^6,8,9,15 The majority of examples demonstrated attainment of skills via use of Objective Structured Clinical Exams (OSCEs).^{5,6,11-13,15,29} Documentation of skills attainment for disclosing a medical error was assessed in the above references primarily by a checklist and was most commonly assessed by faculty, peers, and standardized patients.^{5,6,11-13,15,29} Two examples, specific to physician-in-training error disclosure, went beyond observer assessment of skills and incorporated video-based self-assessment to enhance the trainees’ metacognitive skills.^7,15

Although there are many assessment tools and strategies to assess error disclosure, most of the examples in the literature are specific to a single discipline. Two abstracts and the University of Washington error disclosure curriculum describe pharmacists’ involvement in learning error disclosure as an IPE opportunity.^25-27 The two pharmacy abstracts reported measuring knowledge, skills, and attitudes.^25,26 In the abstract by Nappi and colleagues, skills were assessed by a checklist similar to OSCE skill assessment utilized in physician-in-training programs and similar to those provided in the University of Washington error disclosure toolkit.^25,27 Metacognitive skills were cultivated through peer assessment and guided debriefing was provided in Nappi and colleagues’ report. Error disclosure debriefing, a strategy to enhance metacognition, is included as a component of the University of Washington error disclosure curriculum and toolkit.^25,27 The debriefing described in IPE error disclosure fosters some metacognitive skills, but could be enhanced through self-assessment of error disclosure simulation by video recording and utilizing a skills checklist for both team and individual performance.

Assessing interprofessional team performance is the second objective when error disclosure and IPE are combined. Assessment tools for interprofessional team performance specific to error disclosure are lacking. Most descriptions of assessing team performance during error disclosure refer to Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS), which is not specific to error disclosure.^26,27,30 Debriefing after error disclosure to discuss team performance is part of the University of Washington error disclosure curriculum and toolkit, but utilization of video-based self-assessment to enhance metacognitive skills for IPE error disclosure has not been described.²⁷ In addition to limited tools for assessing IPE team performance specific to error disclosure, a tool to assess development of the Interprofessional Education Collaborative (IPEC) core competencies specific to interprofessional error disclosure has not been described in the literature.^31,32

In 2015, Southern Illinois University Edwardsville (SIUe) instituted an IPE error disclosure simulation training program involving an interprofessional team of students from dental medicine, nursing, and pharmacy schools. Reflections from the initial experience allowed coordinators to identify areas of improvement for error disclosure assessment tools and strategies, as well as the need to assess achievement of IPEC core competencies. Therefore, the primary purpose of this project was to conduct a prospective evaluation to determine the effectiveness of an error disclosure assessment tool in combination with review of video recordings to enhance student learning and metacognitive skills while also assessing the IPEC core competencies. Data from the simulation training generated four research questions that were statistically evaluated:

• 1) Are there differences between the pre- and post-video student self-assessment scores for the four categories of planning, communication, process and team dynamics?

• 2) Do professions and different portrayal of affect by the standardized family members have an effect on the self-assessment scores pre- and post-video?

• 3) Are there differences in self-assessment scores for pre- and post-video depending on the family members’ affect?

• 4) Are there differences between pre- and post-video student self-assessment scores versus faculty real-time assessment for communication, process and team dynamics?

DESIGN

A student summary guide utilized in the inaugural 2015 IPE error disclosure simulation training, which consisted of criteria that measure key steps in the error disclosure process, along with video recordings were used to develop the student self-assessment and faculty assessment instruments.³³ The student self-assessment instrument (Appendix 1) contained a yes/no checklist that was categorized into planning, communication, process, and team dynamics, as well as questions adapted from previous validated tools using a Likert scale to assess achievement of the IPEC core competencies (Appendix 2). This checklist was chosen because this format is widely used in performance assessments like with other OSCEs.^{5,6,11-13,15,29} Two additional questions to obtain qualitative perception data on areas where students performed well and areas for improvement were also included. For coding purposes for SPSS (IBM SPSS Inc, Chicago, IL), yes responses were coded as one and no responses as zero. A similar instrument was also created for faculty to make real-time assessment of the simulation training, focusing only on communication, process and team dynamics (Appendix 3). The study was granted exempt status by the Institutional Review Board at the Southern Illinois University Edwardsville (SIUe).

There were 48 interprofessional teams each composed of four to five students from the schools of dental medicine, nursing and pharmacy. These teams had not previously worked together nor were there any training on teamwork. Students were asked to review materials on error disclosure and a video demonstrating error disclosure prior to attending the simulation event. Each team participated in the simulation training and also observed two other teams disclose the same medical error, but with variations in the affect of the standardized patient’s family member. Each scenario was presented in three different affects portrayed in a consistent order: relieved (1^st), angry (2^nd) and distrustful/sad (3^rd). Six actors were trained by one pharmacy faculty on the case and served as the case patient’s family member. Each simulation was recorded using EMS Simulation IQ (Education Management Solutions, Exton, PA), which allowed immediate access to the recordings.

Student self-assessment and faculty real-time assessment of the error disclosure were conducted on the day of the simulation training. Students individually completed the self-assessment immediately after their team disclosed the medical error (pre-video self-assessment). They were not given copies of the self-assessment prior to conducting the simulation. They were provided a summary guide on steps in the error disclosure process. After all three teams finished their error disclosure and following self-assessment, students then viewed a video recording of their error disclosure simulation and completed the same self-assessment of their performance a second time (post-video self-assessment). The student self-assessment were all completed prior to the faculty debriefing. During the post-video self-assessment, students were also asked for their perceptions regarding achievement of the IPEC core competencies. A unique identifier was used to match the pre- and post-video self-assessment data. Students were required to indicate their team number, case scenario (l^st, 2^nd, or 3^rd affect), and profession. Students had the option to exclude their data from the research analysis.

Faculty, on the other hand, performed real-time assessment of the communication, process and team dynamics for each team during the error disclosure simulation training. Faculty were trained on using the assessment instrument prior to the event by using video records of error disclosure from the 2015 simulation. Generally, there was one faculty from each profession that completed the real-time faculty assessment for each team.

The primary outcome of the evaluation was the student self-assessment scores in four categories (planning, communication, process and team dynamics) and faculty real-time assessment scores in three categories (communication, process and team dynamics). Scores were computed by adding all the yes responses and reversely coded if it is a negative statement, (ie, “We overpromised patient outcomes when discussing with the family member.”). Paired t-test was used to find significant differences in student pre- and post-video self-assessment scores. For comparisons between the student self-assessment scores across different professions (dental, nursing, pharmacy) and family affects (relieved, angry, distrustful/sad), a two-way multivariate analysis of variance (MANOVA) to test for multivariate effects of the four self-assessment categories was used. After detecting a significant multivariate effect, an individual analysis of variance (ANOVA) on the four self-assessment categories and post-hoc analyses using Tukey’s HSD test were performed. For comparisons between student self-assessment scores and faculty real-time assessment on the three major categories, the team average of self-assessment scores and its corresponding average faculty real-time assessment score were computed. Consequently, the 48 pairs of average scores between student and faculty real-time assessments were compared using paired t-test.

The perception data for achievement of IPEC core competencies, which were assessed using a seven point Likert scale, were averaged within each competency category. The Cronbach’s alpha was calculated to determine correlations between each question in each competency category.

EVALUATION AND ASSESSMENT

A total of 233 students from the SIUe schools of dental medicine (50), nursing (109), and pharmacy (74) completed the error disclosure simulation training. Two hundred twenty six student data were included in the analyses.

A significant difference was detected only in communication (p<.01) between pre-video scores (Mean=3.8, SD=.7) and post-video scores (Mean=3.7, SD=.8) for the student self-assessment scores. (Question #1) The self-assessment scores in other categories did not differ significantly between pre- and post-video (Table 1).

A significant multivariate effect was detected with the family affects but not among the different professions. (Question #2) Further analysis showed that family affects significantly affected only two out of four self-assessment categories for both pre- and post-video scores: planning (p<.01 for both) and communication (p=.01 for both). (Question #3) Post-hoc analyses indicated that both pre- and post-video self-assessment scores in the planning category are significantly lower for the angry family affect as compared to the distrustful/sad family affect (p<.01 for both). Meanwhile, both pre- and post-video self-assessment scores in communication are significantly lower for the relieved family affect in comparison to the distrustful/sad family affect (p<.01 for both) (Table 2).

Student self-assessment scores and faculty real-time assessment scores were significantly different within the process and team dynamics categories for both pre- and post-video, with student scores higher than faculty (p<.01 for both). (Question #4) However, student self-assessment scores and faculty real-time assessment scores within the communication category were only significantly different for the pre-video scores (p<.01), again with student scores higher than faculty scores (Table 3). Table 4 provides the descriptive statistics and paired t-test for individual questions under the four categories and where comparisons could be made between student scores and faculty scores respectively.

The means of the perception data on achievement of IPEC core competencies were 6.5 for roles/responsibilities, 6.8 for values/ethics, 6.7 for teams and teamwork, and 6.7 for interprofessional communication on a seven-point Likert scale. Cronbach’s alpha values for each category were > .8, which suggests relatively high internal consistency within each category (.89 for roles/responsibilities, .95 for values/ethics, .84 for teams and teamwork, .93 for interprofessional communication).

The most common student responses to what they did well included that all team members disclosed their specific role in the error; students provided full disclosure of the error; students expressed empathy toward the family member; students did not shift blame onto other members of the team; and disclosed steps to prevent future error. There was agreement among the faculty for what the students did well, including that students expressed empathy; that all team members disclosed their specific role in the error; that all participated in the disclosure; and that students provided full disclosure of the error. Generally, areas for improvement included that students felt they should have introduced all members of the health care team to the family member; should have anticipated family member’s questions and reaction and more equal distribution of participation from team members. There was agreement among faculty regarding that all members of the team should have been introduced; more equal distribution of participation from team members; should have anticipated family member’s questions and reaction; and should not have overpromised outcomes.

DISCUSSION

This article described using student self-assessment and video recordings along with faculty assessment to enhance student learning and metacognitive skills in an interprofessional error disclosure simulation training. This evaluation also addresses another area lacking in the literature by assessing both IPE error disclosure skills and IPEC core competencies simultaneously.

The relatively large interprofessional sample size provided sufficient power to determine significant differences in pre- and post-video assessments.^23-24,28 Incorporating a multitude of assessments in the training program design, including two student self-assessments (pre- and post-video) and interprofessional faculty real-time assessments, enhanced student learning and metacognitive skills related to error disclosure and team performance during an interprofessional activity. Individual student review of video recordings of his/her error disclosure simulation also enhanced cognitive skills to regulate learning. Self-assessment including use of video review are useful metacognitive techniques to enhance learning described by Schraw and Moshman.³⁴ Self-assessment and video review help students realize how well they can perform error disclosure and what needs to be done to enhance their performance. The inclusion of faculty real-time assessment was critical for enhancing the quality of the debriefing and utility of simulation as a learning process.

While the assessment instruments utilized in the simulation were face validated (ie, was determined to represent the proper steps in error disclosure), lack of further validation could be seen as a limitation. The purpose of the assessment instruments was to enhance student learning and metacognition, rather than being used as a high-stake indicator of performance. The instruments were not intended as a summative assessment for grading purposes. They were more formative in nature to provide feedback that could be used during the debriefing process. Another limitation of the simulation training is that although students were instructed to complete the pre-simulation preparation, they were not held accountable nor required to do it. Future offerings will address ways to enhance student preparation prior to the simulation training. In addition, the short interaction in a simulation may not allow accurate assessment of team dynamics. In real practice, issues with team dynamics are often seen with longer periods of time where conflicts and benefits can be more fully appreciated. While focusing on student learning and metacognition related to error disclosure performance, this evaluation did not measure knowledge or attitudes. The evaluation instead focused on Miller’s “knows how” competency level.²⁸ In contrast to Sukalich’s study, self-efficacy was not evaluated using a Likert scale, nor was there completion of a second simulation after reviewing video recordings. Further enhancements of learning could be achieved by incorporating a second simulation experience.

Self-assessments were generally positive for all categories as students perceived their performance favorably. Changes in the post-video scores indicate that the use of video recordings enhanced students’ metacognitive skills related to the ability to identify areas for improvement, especially in the communication category. However, as the changes are small and only the communication category is statistically significant, the overall practical significance of these differences is questionable. The lack of difference in the planning category may possibly be explained by the planning phase of the simulation not being recorded. The lack of difference in the team dynamics category may be explained by the short period of time students interacted during the simulation which prevented any major team conflicts from arising.

The lack of significant differences in student self-assessment scores among the three professions in both pre- and post-video may reflect a similar professional training culture across the three programs. Student pre- and post-video self-assessment mean scores in all three family affects were generally positive for all categories as students perceived their performance favorably. Improvements in pre-video student self-assessment scores within the communication category, as the simulation experiences progressed from relieved, angry and finally to distrustful/sad family member affects, may indicate that peer observation of previous simulations enhanced student performance. Students participating in the distrustful/sad simulation, the third simulation, had the benefit of watching the previous simulations with relieved and angry affects. This allowed students to identify areas that went well or areas for improvement and incorporate those observations into their simulation. The pre- and post-video self-assessment scores being the lowest with the angry affect for the planning category may indicate that students did not anticipate or were not prepared to encounter a family member with this type of affect. In real practice, it is much more challenging to deal with an angry patient or client than one who is relieved or sad. Future offerings will address the need to prepare students to better plan for how a family member might respond to disclosure of medical errors.

Comparison of student pre- and post-video self-assessment scores with faculty real-time assessment scores revealed differences in almost all categories (the exception is post-video communication). Overall, faculty tended to assess students more critically, which is consistent with other reports.³⁵ The use of video recordings to enhance student self-assessments resulted in students reporting mean scores more closely aligned to the faculty. This suggests that video recordings enhanced recognition of areas for improvement and is a useful tool to enhance metacognitive skills.

Looking more specifically at individual assessment items under each category for student pre- and post-video self-assessments, no differences were noted in scores for most items under communication. The lack of differences in pre- and post-video self-assessment scores may indicate the difficult nature of assessing emotional components using a yes/no checklist versus a Likert scale format. Assessing empathy, building rapport, or showing respect would likely be better evaluated using a Likert scale.¹⁵

The student qualitative data reaffirm the quantitative data and reveal that students enhanced their metacognitive skills in identifying areas they did well and areas for improvement. The use of pre- and post-video assessments identified what went well and areas for improvement were more extensive than what was observed during the process used in the first error disclosure training program simulation in 2015.³⁴

In addition, the data reiterates that error disclosure is a good vector for IPE. Students identified that error disclosure helped them develop roles and responsibilities and teamwork, which are IPEC core competencies. We believe that this IPE experience in the final didactic year of the pharmacy curriculum may serve as a pinnacle activity that further enhances the IPEC core competencies.^36-38 The more robust assessment helps us to benchmark our school’s achievement of the IPEC competencies and provides data supporting learning of error disclosure skills. Using the IPEC competencies as a framework for assessment of student performance and learning also serves as a benchmark for our program outside the school. This assessment report of using developed instruments for IPE error disclosure and IPEC competencies compares favorably to a recent report by Ragucci and colleagues where student perception of confidence and proficiency in disclosing medical errors was described.³⁹ However, Ragucci and colleagues did not assess team performance nor the IPEC competencies.

SUMMARY

The use of self-assessment instruments and video recordings enhanced student metacognitive skills for assessing performance in interprofessional error disclosure, specifically in the area of communication. As metacognitive skills are enhanced, this suggests that enhanced learning about error disclosure skills may have occurred. The specific family member affect experience made a difference in student self-assessments. Using videos resulted in self-assessments more closely aligned with faculty assessments in the category of communication. The quantitative data is also supported by the qualitative data in terms of areas teams did well and areas for improvement. Students also appeared to have a positive perception of achievement of the IPEC core competencies after completing the error disclosure simulation training.

The tools and process used can easily be transferred to other programs for addressing IPE error disclosure and IPEC competencies. Overall, the assessment process addresses best practices in assessment (the Plan, Do, Check, and Act model) including: 1) measurable performance goals (Plan); 2) informed planning with internal and external stakeholders (Plan); 3) scientific methods and processes for obtaining data (Do); 4) outcomes are directly linked to desired learning outcomes (Do and Check); 5) use and reflecting on data to enhance improvement (Check); 6) benchmarking outcomes within and outside the School (Act); and 7) communication of findings to internal and external stakeholders. (Act).⁴⁰