A Tool to Assess Student Performance in a Clostridium difficile Infection Simulation Scenario

INTRODUCTION

Assessment activities surrounding human patient simulation have focused on enhancing student pharmacists’ exposure and confidence related to situations that occur in clinical practice. The majority of simulation activities incorporate formative feedback while providing student pharmacists the opportunity to practice skills and integrate various aspects of knowledge, communication, professionalism, and clinical application.¹ Educators commonly use survey data from learners to evaluate simulation-based activities.² Educators are also using pre- and post-knowledge testing more frequently.^3-6

Effective assessment of clinical performance and critical thinking with simulation requires pharmacy educators to move beyond satisfaction and self-efficacy surveys and knowledge assessments to the development of reliable, valid tools to assess student performance during the simulation. This effort is becoming even more relevant as doctor of pharmacy (PharmD) curricula becomes more competency based. This transition is partly in response to the Accreditation Council for Pharmacy Education’s (ACPE) recently approved Appendix D, a list of “must have” abilities indicating a student pharmacist’s readiness to enter advanced pharmacy practice experiences (APPEs). The pre-APPE domains and abilities outlined in Appendix D require evidence of student pharmacist achievement of specific performance-based competencies prior to entering APPEs or clerkships.⁷ Furthering the necessity for sound assessment practices to determine competency is that ACPE allows simulation-based activities to count for up to 20% (approximately 60 hours) of total introductory pharmacy practice experience.⁸

Basic components during the development of valid assessment tools include determination of content and construct validity and inter-rater reliability. Validity is a tool’s accuracy in measuring what it was intended to measure. An accepted procedure is to use clinical experts to establish both content and construct validity. For content validity, the experts review the criteria in the tool and provide feedback regarding the appropriateness of the items and the key measurements required to evaluate each item. For construct validity, the experts assess how well an action represented the concept being evaluated.² Finally, the experts ensure that all important items are included on the tool and provide feedback regarding the relevance and importance of the items.

Once the tool is developed, the next step is to establish its reliability. Reliability refers to the tool’s ability to provide scores that are substantially free of measurement error. Of the many types of reliability that can be determined, inter-rater reliability is critical to establish when assessment of student learning will be conducted by many different graders from various clinical backgrounds. Inter-rater reliability is typically assessed when multiple raters independently evaluate the same performance using the same tool.

Faculty members at the Washington State University College of Pharmacy, initiated simulation-based activities within the PharmD curriculum during 2007. One of the initial simulation scenarios implemented was an infectious disease case related to Clostridium difficile infection (CDI). We selected this content area because of its importance to the healthcare system. Clostridium difficile is a widespread disease within healthcare systems and is the leading cause of nosocomial diarrhea.⁹

As part of the patient care team, pharmacists must/should understand the appropriate management of CDI. Pharmacists can play a critical role in identifying unwarranted agents that promote colonic stasis. Health care providers should avoid use of agents that reduce gastric motility in patients with CDI because of the potential risk of severe complications such as toxic megacolon.⁹ Pharmacists can assist with treatment optimization, thereby reducing the infectious spread of CDI to other patients. A multidisciplinary CDI reduction team led by a pharmacist was associated with a reduction in the rate of CDI within the Huntsville Hospital System.¹⁰ As healthcare professionals with direct patient contact, pharmacy personnel must actively use proper infection control procedures to prevent the nosocomial transmission of this pathogen.

We developed a simulation evaluation tool, including proper infection control procedures and therapy optimization, to address the role of the pharmacist in appropriate treatment of CDI. Despite the expansion of simulation-based teaching methodologies, few studies have been published in the pharmacy literature regarding the validation of simulation assessment tools. Thus, the purpose of this study was to validate the evaluation tool with a specific focus on establishing content validity and inter-rater reliability.

METHODS

The WSU Institutional Review Board granted “exempt” status to this study as normal education practice conducted in an established educational setting. We completed the tool validation process over a 2-year period, beginning in fall 2010, with content validation, tool creation, and preliminary testing in spring 2011, followed by tool revision and further testing in spring 2012. The tool developers identified the following 6 content domains which they believed were important to include on an assessment tool: isolation precautions, assessment of home medications and current medication orders, appropriate monitoring parameters, evaluation of diagnostic results, accurate treatment of CDI, and appropriate supportive care. We used a deductive process to develop individual items for each content domain. A 6-person clinical expert panel that included 3 infectious diseases pharmacists, 1 infectious diseases physician, 1 internal medicine pharmacist, and 1 postgraduate year 1 (PGY1) pharmacy resident reviewed the 6 content domains and their associated items.

The experts evaluated the items in the 6 content domains, rating each as critically important, important, not very important, or not important at all. Additionally, they suggested revisions to existing items and identified missing items. The expert reviewers also ranked items of pharmacist-specific patient care in order of importance relative to each other. Based on the results of the expert review, we developed the first 18-item draft of the checklist style tool, including the assignment of point distribution on the tool.

We used the initial draft tool to evaluate the performance of 24 student pharmacist groups during the spring 2011 simulation. Each simulation session was video recorded and saved in digital format to a secure university server. Four raters independently assessed student performance from the video recordings. We analyzed 4 independent raters’ summed scores across the items to determine the internal consistency and reliability of the entire evaluation tool, as well as for individual items. The tool validation team reviewed the results from the 2011 spring semester analysis and revised the tool. The revised tool (Appendix 1) with 19 check-list items, was then used to evaluate the performance of 31 student groups during 2012 spring semester. The video-recordings and the independent video-recording rater processes were consistent with the methods used in 2011. Team members again reviewed the overall tool and each item and compared it to the previous tool to determine if its performance had improved. Special focus was given to the items regarding accurate therapy to determine if those had changed from the previous tool.

The course faculty members created an education module to introduce second-year student pharmacists to a hospitalized patient with CDI. The educational module was delivered in an applied patient care laboratory which included a 1-hour tutorial followed by a 2-hour laboratory component. The student pharmacists were required to read Clinical Practice Guidelines for Clostridium difficile Infections in Adults prior to tutorial and laboratory. In addition, prior to the laboratory session, course faculty members provided the students with the patient’s admission note from the physician. ¹¹ Learning objectives for the CDI simulation included that after completion of the module, the student would be able to do the following: (1) list risk factors for CDI, (2) recommend appropriate CDI therapy, (3) indicate measures used for supportive care and identify medications that should be avoided or used with caution, and (4) establish appropriate monitoring parameters to evaluate the therapeutic improvement or treatment failure of a patient. The tutorial included a brief orientation to the CDI case, a review of the patient chart, and a review of a simulation confidentiality statement.

Student pharmacists were divided into teams of 3 to 4 students. The simulation began with a brief interaction with the facilitator who provided a short, scripted introduction describing the simulated patient case setting. The student pharmacists began the simulation by conducting a chart review at a mock nursing station. They then entered the patient room and performed a patient interview. Immediately following the patient interview, the facilitator entered the patient room where 1 of the student pharmacists on the team initiated a short, informal patient case presentation. The facilitator encouraged other team members to participate by adding relevant information to the presentation. After the students completed the simulation, the facilitator conducted a short debriefing and discussion with the student pharmacists in a separate conference room. During the debriefing, the facilitator evaluated the student pharmacists’ performance and provided formative feedback.

The simulations were conducted over 2 days. The same basic patient case was used for all simulations but the patient medical record and outcome were changed between days to limit the impact of discussion between student pharmacists who had completed the simulation and those who had not. In one case the patient’s condition deteriorated and he did not respond to the initial therapy of metronidazole. For the second case, the patient improved with the initial therapy. Each scenario provided opportunity for the student pharmacists to obtain pertinent information from the patient, practice appropriate infection control procedures, modify drug therapy regarding duration and dosing, and discontinue contraindicated therapy.

The statistical analyses were consistent for both the 2011 and 2012 tool versions. Summed scores across all the items from 4 independent raters were analyzed using Cronbach alpha to determine internal consistency reliability of the entire evaluation tool. We assumed the overall instrument to be internally consistent at a Cronbach alpha value >0.80.^12,13

The summed scores were derived from dichotomous ratings per item by each rater (1 = aspect observed, 0 = aspect not observed). Cronbach alpha across all raters for all groups was calculated as well as for groups with the “deteriorating” patient and those with the “improving” patient. Observed percent agreement values amongst the 4 raters per item across all scenarios were calculated. Gwet’s first-order agreement coefficient, AC₁, was used to estimate chance-adjusted agreement among the 4 raters for each item on the evaluation.^14,15 AC₁ uses a chance-agreement probability that is calibrated to the propensity of random ratings estimated from observed ratings. Unlike other measures of chance-adjusted agreement, such as Cohen’s Kappa, AC₁ is not distorted by high (or low) prevalence of the trait being rated; thus, it represents an unbiased estimate of ‘true’ inter-rater agreement.^16,17 Confidence intervals for AC₁ were derived with unconditional variance estimates in order to characterize precision of the agreement coefficients to the general population of raters. Items with AC₁ values below 0.60 were scrutinized for revision. All calculations were performed with SAS software.¹⁸

RESULTS

The experts rated accurate treatment of CDI and assessment of home medications and current medications as critically important. They rated all other domains as important. The experts ranked the most relevant pharmacist-specific patient-care item as accurate treatment of CDI followed by, in order of decreasing importance, evaluation of home and acute care medication orders, patient monitoring, evaluation of diagnostic tools, isolation precautions, allergy status, and estimation of renal function.

DISCUSSION

We conducted the study to determine the validity and reliability of an evaluation tool designed for the purpose of assessing student pharmacists’ skills during a simulation scenario involving a patient with CDI. The final 2012 version of the simulation assessment tool was sound with respect to content validity, demonstrated acceptable levels of inter-rater reliability per item, and exhibited sound internal consistency, whether the scenario involved a patient who was improving or deteriorating.

Potential weaknesses of the study were identified by the team. Prior to using the 2011 and 2012 CDI evaluation tools, graders did not participate in norming sessions. By discussing how the tool would be used, the raters may have been able to identify a clear vision of the intent for each item, potentially increasing inter-rater reliability. Evidence of this was discovered with the “correct route” item which demonstrated a low AC₁ score. Student pharmacists tended to leave the route out, particularly if they were modifying the drug dose but retaining the original route of administration. The route of administration was often “implied.” Second, the evaluators then inconsistently gave credit (or not) for route of administration used. Some raters indicated that if the students did not explicitly state the route used, they did not give credit for it, while others gave credit for the students when they implied which route was used. This emphasizes the importance of including norming sessions during tool validation processes.

During 2011, the audio was not recorded while the students were reviewing the patient charts, and the video capture of the patient charts was sometimes of a low quality. Video raters reported difficulty seeing what portion of the chart was being reviewed and did not have the ability to listen to discussions between student pharmacists while evaluating the patient chart. During 2012, both audio and video of the chart review were available for the raters to review during the evaluation.

Traditional assessment of student learning relies heavily on the ability of students to demonstrate achievement of knowledge-based content through the use of written examinations. However, performance on a written examination does not ensure competency in the provision of patient-specific care. An example relevant to this scenario is that student pharmacists may pass a written assessment illustrating they understand the importance of following isolation precautions when encountering infectious material. However, whether they can incorporate this knowledge into practice is unknown. Performance-based assessment allows evaluation of this competency. Assessment of performance “requires students to display their learning…actively practice their skills and synthesize their knowledge.”²⁰ In the area of advanced cardiac life support (ACLS), checklists can be used to make valid judgments of minimum competencies.^21,22 Also, reliable scores can be obtained through the use of an ACLS checklist during a high stakes performance assessment, called a Megacode, used during ACLS certification.²³ To this end, the need for valid, reliable assessment tools to evaluate skills-based performance of student pharmacists during simulation cannot be overstated, particularly as pharmacy education programs are called to provide evidence of competency/mastery of pre-APPE domains. Future studies based on these results are needed to target the best approach for using evaluation tools to set standards of proficiency in pharmacy practice.

Additionally, there is a need to continue to define and implement authentic assessments as a best practice in pharmacy education. While there is not a single definition, authentic assessment is defined by Frey and colleagues as an assessment that is a “mirroring of real-world tasks or expectations.”²⁴ For healthcare workers our “real-world” is the provision of patient care. Developing reliable assessment tools for simulated patient cases may be the first step to authentic assessment. Reliable assessment tools must have a defensible cut point, consistent between raters, and include content needed to treat a patient.²⁵

CONCLUSION

Using the evaluation tool described for assessing a Clostridium difficile infection simulation scenario, student pharmacists were graded consistently among multiple graders. Additionally student pharmacists were evaluated on CDI content deemed important and relevant by experts in the field. Establishing a valid, reliable assessment tool to evaluate student performance during a simulation scenario lays the foundation for simulation to be used more commonly as summative evaluation to ensure that student pharmacists meet the pre-APPE core domains and abilities.