Abstract
Objective. To evaluate a telehealth simulation rubric for a pharmacy skills lab course using interrater reliability (IRR) to compare different evaluator types.
Methods. A rubric was developed and used to score student pharmacists’ competency in a telehealth simulation for a pharmacy skills lab course. Intraclass correlation coefficients (ICCs) were calculated to determine the IRR.
Results. Fifty-five recorded telehealth simulation sessions were scored three times by a pool of evaluators using the rubric. The overall ICC for the rubric was 0.631 across different types of evaluators, indicating good IRR. Differences in ICCs were observed when they were calculated for type of evaluator and rubric subcomponents.
Conclusion. The rubric demonstrated good IRR across multiple types of evaluators when used in a skills lab course. Additional evaluation of the rubric and continuous quality improvement is warranted to ensure that assessment becomes more consistent with continued use, regardless of the evaluator type.
INTRODUCTION
During the COVID-19 pandemic, health care providers increased their use of telehealth services to provide continued health care access amidst stay-at-home orders and social distancing protocols. Telehealth services involve the use of electronic information and telecommunications technologies to support and promote long-distance care, monitoring, and education.1 Telehealth visits increased by 50% in March 2020 compared to the same period in 2019, and at that time more than one in eight Americans had engaged in a video consult with a health care provider.2 Furthermore, temporary changes to federal law and regulations implemented early in the pandemic increased access and created opportunities for pharmacists to receive reimbursements for telehealth services3; in the pharmacy setting, these services are called telepharmacy to refer to pharmacy care provided to patients through telecommunications.4 Telehealth services provided by pharmacists include medication therapy management, chronic disease state management, remote order entry, medication dispensing, patient education, transitions of care, and others.5 Early in the pandemic, pharmacists and other health professionals had to quickly transition to delivering telehealth services to ensure continuity of care.
The global pandemic also highlighted opportunities for pharmacy educators to evaluate how they teach timely topics such as telepharmacy and telehealth, which is especially relevant because the increased use of telehealth services accompanied by regulatory changes will likely lead to continued use of these services after the pandemic.6 Therefore, pharmacy students will need training on various aspects important to the delivery of telehealth services.7 The literature published on telehealth and pharmacy students describes, through simulation activities and student perceptions, that a virtual environment affects students’ understanding, confidence, communication, and abilities in interprofessional collaboration.8-12 While the literature has also described students’ performance scores on virtual skills-based assessments compared to in-person skills-based assessments, a method for evaluating the competency of skills performed in a telehealth activity that can be used by different types of evaluators is not currently available.12-14
Various competencies and skills seem to be particularly relevant to telehealth settings. For example, telehealth etiquette refers to unique behaviors outside of typical professional actions that are used to conduct an effective telehealth visit.15 The components of telehealth etiquette that lead to increased patient satisfaction and successful interactions involve both verbal and nonverbal communication skills.16,17 Some considerations to help improve verbal communication are the timing and pace of a telehealth visit as well as the use of motivational interviewing.18 Regarding nonverbal techniques, making eye contact is more challenging to accomplish via telehealth, but looking at the camera rather than at the patient on the screen can help the provider to give the appearance of making direct eye contact. Further, in a virtual setting, health professionals need to develop the patient-professional relationship differently and more deliberately than an in-person encounter,19 such that health professionals may need to pay more attention to personal appearance, environmental factors, privacy, and problem-solving skills during a telehealth encounter than they would for an in-person visit.15 Regarding personal appearance, during a virtual visit patients may be more affected by the health professional’s clothing color, pattern choices, or jewelry selection.19 For environmental considerations, health professionals could try minimizing background noise, ensuring proper lighting, and determining what is viewable behind them. Regarding privacy and security, these issues may be of concern for patients if they cannot see that the virtual environment is secure. To minimize these concerns, health professionals could communicate that the area is secure, introduce all individuals present, and use the camera to show the patient around the room.12 Lastly, health professions should possess problem-solving skills to overcome potential technological barriers and to manage any unforeseen factors that may impact the effectiveness of a telehealth session. Given the skills and competencies health professionals must develop to offer successful telehealth sessions, the primary goal of the following study was to evaluate a rubric for assessing a telehealth simulation using interrater reliability (IRR).
METHODS
The Doctor of Pharmacy curriculum at the University of Toledo includes a six-semester pharmacy skills development (PSD) laboratory course. In the fall of 2020, 67 third-year pharmacy students were enrolled in the fifth PSD course of the series (PSD-5). A one-hour lecture was given to students as an introduction to the topic of telehealth. The lecture topics included telehealth definitions, recent regulatory changes, reimbursement, privacy and confidentiality, audio and visual technologies, and communication best practices/etiquette. At that time, the lecture component of the course was delivered remotely.
Following the lecture, students participated in a telehealth simulation as an in-laboratory graded activity where they met with a mock patient using Blackboard Collaborate (Blackboard Inc). The virtual platform had audio and video capabilities, and all participants were expected to have their microphones and cameras on during the session. The laboratory was conducted remotely, and participants were primarily connecting from home. Students were scheduled in 20-minute time blocks to participate in the simulation. Sessions were recorded following normal course practice in the case of grading discrepancies. The purpose of the session was to collect a medication history from a patient recently discharged from hospital to home and to reconcile discrepancies identified in real time. To assist with preparation, prior to the session students were provided a patient scenario and medication list, medication history interview handout, and documentation template.
The role of the patient was fulfilled by volunteer faculty (n = 2), residents (n = 6), and one advanced pharmacy practice experience (APPE) student completing an elective academia rotation. To standardize the interactions and patient responses, volunteers playing the mock patients were given a mock patient scenario that included background, patient information, and current medication knowledge from the patient’s perspective. Those playing the mock patients also served as the evaluators. Student performance was scored using a competency and communication rubric from the course series that was modified to include statements to assess telehealth etiquette skills. The coordinator of the course determined which telehealth etiquette skills would be included in the rubric based on practice experience and a review of the literature as outlined above.
The rubric used for the study contained 13 assessment statements for competency and six for telehealth etiquette (Figure 1). Each statement was evaluated on three dimensions, namely “satisfactory,” “unsatisfactory,” or “not performed.” The rubric did not describe criteria for any of the dimensions beyond the assessment statements. Students were provided the rubric one week prior to participating in the simulation. A brief overview of the activity scenario and the rubric was provided to evaluators before the session, but no specific training or guidance was provided on how to interpret the rubric. Upon finishing the activity, the completed rubric containing quantitative and qualitative feedback was provided to the student.
Excerpt from the Telehealth Simulation Assessment Statements. The rubric used for the study contained 13 assessment statements for competency and six for telehealth etiquette. Each statement was evaluated on three dimensions, namely “satisfactory,” “unsatisfactory,” or “not performed.”
Upon semester completion, two additional faculty evaluators reviewed the recorded sessions and evaluated the students using the rubric to gather retrospective data on interrater reliability (IRR). The intended purpose of using IRR for this study was to assess reliability of the mean ratings from multiple coders. Of 67 students in the class, 55 recordings were able to be reviewed a total of three times each. Seven sessions were evaluated by three faculty members, 36 sessions were evaluated by a resident and two faculty, and 12 sessions were evaluated by the APPE student and two faculty. There were four missing recordings and eight incomplete recordings that were excluded from the sample. The intraclass correlation coefficient (ICC) was calculated to determine the IRR of the rubric overall and its subcomponents of competency and telehealth etiquette. A p<.05 was considered significant. A one-way random effect model and average measures mean (ICC [1,3]) was used to calculate the ICC when the evaluators for each student were not the same three evaluators each time. A two-way mixed effect model and average measures mean (ICC [3,3]) was used to calculate ICC when the same three evaluators reviewed a recording. Higher ICC values indicated stronger IRR, which was suggestive of stronger consistency among evaluators. ICC scores less than 0.4 indicated poor IRR, scores of 0.4-0.59 indicated fair IRR, scores of 0.6-0.74 indicated good IRR, and scores of 0.75-1 indicated excellent IRR.20 Statistical analysis was performed using SPSS Statistics, version 27 (IBM Corp). This study was reviewed by the University of Toledo Social, Behavioral, and Educational Institutional Review Board and was deemed exempted.
RESULTS
The ICC results and their corresponding IRR outcomes are presented in Table 1. When including all evaluators in the pool, the ICC was 0.63 (p<.001; 95% CI, 0.423-0.772), indicating good IRR for the rubric overall. When the sessions were evaluated only by faculty members as the three evaluators, the calculated ICCs showed the strongest consistency among evaluator types. Comparatively, when the evaluator pool contained residents or a student, the IRR diminished.
Intraclass Correlation Coefficient Results for Interrater Reliability
For the 13 competency statements of the rubric, ICCs were calculated and assessed for each evaluator type. For all evaluators together, the ICC for the competency statements of the rubric was 0.69 (p<.001; 95% CI 0.517-0.809), indicating good IRR. When the rubric was evaluated by faculty, the calculated ICC resulted in excellent IRR. Again, when the evaluator group included residents, IRR diminished for the competency component of the rubric.
Comparatively, for the rubric’s six telehealth etiquette statements, the ICC for all evaluators was 0.47 (p=.003; 95% CI, 0.169-0.672), indicating fair IRR. For the faculty evaluators, the ICC for the rubric’s telehealth etiquette statements suggested fair IRR, but it was not significant. For the resident-faculty evaluator group, the ICC for the rubric’s telehealth etiquette statements indicated fair IRR. When a student was included as an evaluator, the ICC for the telehealth etiquette statements suggested poor IRR but was not significant.
DISCUSSION
In a recent commentary, Frenzel and Porter clearly described the relevance and need to prepare pharmacy students for delivering services remotely.7 While descriptions of telehealth simulation activities have been published, a method to evaluate telehealth etiquette skills that can be used by varying evaluator types is not available.8,10,12,14 Candelario and colleagues described a telemedicine simulation where pharmacy students worked in a group to identify medication-related problems (MRPs) incorporated into a discharge follow-up encounter. Standardized patients completed a patient perception scale checklist to assess the students’ communication skills. Findings showed that less than half of the student teams performed communication skills for this activity accurately and completely.8 Although methods to assess student performance in a telehealth simulation were discussed, this study focused more on transitions of care concepts and did not mention whether telehealth etiquette skills were assessed for the activity. In another study, Gustin and colleagues described a training program developed to teach health care students telehealth etiquette skills. While this study used a telehealth etiquette checklist to help facilitate discussion, reflection, and feedback on techniques, it was not used as an assessment tool.12
In the current study, the IRR of the rubric’s telehealth etiquette statements was not consistent across evaluator types, which indicates that improvement efforts should be focused on this area. In addition to potentially revising the rubric statements, the overall clarity of the rubric could be improved by adding a description of expectations for the different dimensions. Telehealth may also have been a new concept to the evaluators and something they may not have had enough personal experience with. While not recorded in the study, IRR outcomes may also have been affected by the professional backgrounds of the evaluators, since evaluators may have had different types of experiences to draw upon when completing the rubric. Another factor impacting IRR was that no formal training was given to evaluators before participating in the activity to help them interpret the telehealth statements. To help standardize scoring and improve consistency among evaluators, training will include instruction on how to evaluate student performance, discussions of the rubric, and opportunities to assess a sample session using the rubric.
The current study demonstrated good IRR across different evaluator types, which also supports the need for continued development of the assessment tool. The IRR of the rubric was highest when faculty members used the tool to evaluate students, but this may not be a realistic recommendation to make, considering faculty workloads and responsibilities. Allowing residents and students to assist in the simulation and evaluation process is also valuable to their development as potential educators. Of note, the student-faculty evaluator group had a higher ICC for the rubric compared to the resident-faculty evaluator group, but this was likely because the student evaluator assisted with developing some rubric statements as part of a rotation project. When reviewing the IRR results of the competency assessment statements in the rubric, fair to good IRR was generally seen across all groups except for the faculty-only group, which showed excellent IRR. This is likely because the faculty evaluators have used the competency and communication portion of the rubric in other skills lab courses. The competency statements had already been standardized for the course series and were originally developed with assistance from the faculty involved in this research. Using external faculty for further evaluation of the rubric will be important for expanding IRR results.
Some limitations of the study should be considered. First, these results capture data from the first use of the rubric among a limited pool of evaluators for one cohort of students. Continued use of the rubric with additional evaluators may improve reliability of the rubric over time. Second, the original evaluator of each session was potentially multitasking, acting as both the mock patient and evaluating the student in real time. The first evaluator completed the rubric after each session and did not have the recording to view, whereas the subsequent evaluators watched the recordings; this may have introduced some bias. Lastly, validation of this rubric is needed before it is used in a high-stakes assessment rather than a formative assessment.
CONCLUSION
No validated rubric is available to assess students in telehealth etiquette skills, and additional research is needed before the rubric described in this study can be universally implemented in pharmacy education. The rubric applied in this study demonstrated good IRR overall among various evaluator types when used in a skills lab course. Evaluating and improving the quality of the rubric’s telehealth etiquette statements are necessary to aid in improving the feedback provided to students on their performance. An evaluator training session on interpreting and using the rubric is also planned to help improve reliability with future assessments.
ACKNOWLEDGEMENTS
The authors would like to acknowledge Dr. Tian Chen for providing statistics consulting, and Dr. Mariann Churchwell for her guidance in this research.
- Received June 17, 2021.
- Accepted October 8, 2021.
- © 2022 American Association of Colleges of Pharmacy
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.