Skip to main content

Main menu

  • Articles
    • Current
    • Early Release
    • Archive
    • Rufus A. Lyman Award
    • Theme Issues
    • Special Collections
  • Authors
    • Author Instructions
    • Submission Process
    • Submit a Manuscript
    • Call for Papers - Intersectionality of Pharmacists’ Professional and Personal Identity
  • Reviewers
    • Reviewer Instructions
    • Call for Mentees
    • Reviewer Recognition
    • Frequently Asked Questions (FAQ)
  • About
    • About AJPE
    • Editorial Team
    • Editorial Board
    • History
  • More
    • Meet the Editors
    • Webinars
    • Contact AJPE
  • Other Publications

User menu

Search

  • Advanced search
American Journal of Pharmaceutical Education
  • Other Publications
American Journal of Pharmaceutical Education

Advanced Search

  • Articles
    • Current
    • Early Release
    • Archive
    • Rufus A. Lyman Award
    • Theme Issues
    • Special Collections
  • Authors
    • Author Instructions
    • Submission Process
    • Submit a Manuscript
    • Call for Papers - Intersectionality of Pharmacists’ Professional and Personal Identity
  • Reviewers
    • Reviewer Instructions
    • Call for Mentees
    • Reviewer Recognition
    • Frequently Asked Questions (FAQ)
  • About
    • About AJPE
    • Editorial Team
    • Editorial Board
    • History
  • More
    • Meet the Editors
    • Webinars
    • Contact AJPE
  • Follow AJPE on Twitter
  • LinkedIn
Brief ReportBRIEF

Evaluation of a Rubric to Assess Pharmacy Student Performance in a Telehealth Simulation Exercise

Michelle N. Schroeder and Aaron J. Lengel
American Journal of Pharmaceutical Education November 2022, 86 (8) ajpe8778; DOI: https://doi.org/10.5688/ajpe8778
Michelle N. Schroeder
University of Toledo, College of Pharmacy & Pharmaceutical Sciences, Toledo, Ohio
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aaron J. Lengel
University of Toledo, College of Pharmacy & Pharmaceutical Sciences, Toledo, Ohio
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • PDF
Loading

Abstract

Objective. To evaluate a telehealth simulation rubric for a pharmacy skills lab course using interrater reliability (IRR) to compare different evaluator types.

Methods. A rubric was developed and used to score student pharmacists’ competency in a telehealth simulation for a pharmacy skills lab course. Intraclass correlation coefficients (ICCs) were calculated to determine the IRR.

Results. Fifty-five recorded telehealth simulation sessions were scored three times by a pool of evaluators using the rubric. The overall ICC for the rubric was 0.631 across different types of evaluators, indicating good IRR. Differences in ICCs were observed when they were calculated for type of evaluator and rubric subcomponents.

Conclusion. The rubric demonstrated good IRR across multiple types of evaluators when used in a skills lab course. Additional evaluation of the rubric and continuous quality improvement is warranted to ensure that assessment becomes more consistent with continued use, regardless of the evaluator type.

Keywords:
  • telehealth
  • telepharmacy
  • tele-education
  • etiquette
  • rubric

INTRODUCTION

During the COVID-19 pandemic, health care providers increased their use of telehealth services to provide continued health care access amidst stay-at-home orders and social distancing protocols. Telehealth services involve the use of electronic information and telecommunications technologies to support and promote long-distance care, monitoring, and education.1 Telehealth visits increased by 50% in March 2020 compared to the same period in 2019, and at that time more than one in eight Americans had engaged in a video consult with a health care provider.2 Furthermore, temporary changes to federal law and regulations implemented early in the pandemic increased access and created opportunities for pharmacists to receive reimbursements for telehealth services3; in the pharmacy setting, these services are called telepharmacy to refer to pharmacy care provided to patients through telecommunications.4 Telehealth services provided by pharmacists include medication therapy management, chronic disease state management, remote order entry, medication dispensing, patient education, transitions of care, and others.5 Early in the pandemic, pharmacists and other health professionals had to quickly transition to delivering telehealth services to ensure continuity of care.

The global pandemic also highlighted opportunities for pharmacy educators to evaluate how they teach timely topics such as telepharmacy and telehealth, which is especially relevant because the increased use of telehealth services accompanied by regulatory changes will likely lead to continued use of these services after the pandemic.6 Therefore, pharmacy students will need training on various aspects important to the delivery of telehealth services.7 The literature published on telehealth and pharmacy students describes, through simulation activities and student perceptions, that a virtual environment affects students’ understanding, confidence, communication, and abilities in interprofessional collaboration.8-12 While the literature has also described students’ performance scores on virtual skills-based assessments compared to in-person skills-based assessments, a method for evaluating the competency of skills performed in a telehealth activity that can be used by different types of evaluators is not currently available.12-14

Various competencies and skills seem to be particularly relevant to telehealth settings. For example, telehealth etiquette refers to unique behaviors outside of typical professional actions that are used to conduct an effective telehealth visit.15 The components of telehealth etiquette that lead to increased patient satisfaction and successful interactions involve both verbal and nonverbal communication skills.16,17 Some considerations to help improve verbal communication are the timing and pace of a telehealth visit as well as the use of motivational interviewing.18 Regarding nonverbal techniques, making eye contact is more challenging to accomplish via telehealth, but looking at the camera rather than at the patient on the screen can help the provider to give the appearance of making direct eye contact. Further, in a virtual setting, health professionals need to develop the patient-professional relationship differently and more deliberately than an in-person encounter,19 such that health professionals may need to pay more attention to personal appearance, environmental factors, privacy, and problem-solving skills during a telehealth encounter than they would for an in-person visit.15 Regarding personal appearance, during a virtual visit patients may be more affected by the health professional’s clothing color, pattern choices, or jewelry selection.19 For environmental considerations, health professionals could try minimizing background noise, ensuring proper lighting, and determining what is viewable behind them. Regarding privacy and security, these issues may be of concern for patients if they cannot see that the virtual environment is secure. To minimize these concerns, health professionals could communicate that the area is secure, introduce all individuals present, and use the camera to show the patient around the room.12 Lastly, health professions should possess problem-solving skills to overcome potential technological barriers and to manage any unforeseen factors that may impact the effectiveness of a telehealth session. Given the skills and competencies health professionals must develop to offer successful telehealth sessions, the primary goal of the following study was to evaluate a rubric for assessing a telehealth simulation using interrater reliability (IRR).

METHODS

The Doctor of Pharmacy curriculum at the University of Toledo includes a six-semester pharmacy skills development (PSD) laboratory course. In the fall of 2020, 67 third-year pharmacy students were enrolled in the fifth PSD course of the series (PSD-5). A one-hour lecture was given to students as an introduction to the topic of telehealth. The lecture topics included telehealth definitions, recent regulatory changes, reimbursement, privacy and confidentiality, audio and visual technologies, and communication best practices/etiquette. At that time, the lecture component of the course was delivered remotely.

Following the lecture, students participated in a telehealth simulation as an in-laboratory graded activity where they met with a mock patient using Blackboard Collaborate (Blackboard Inc). The virtual platform had audio and video capabilities, and all participants were expected to have their microphones and cameras on during the session. The laboratory was conducted remotely, and participants were primarily connecting from home. Students were scheduled in 20-minute time blocks to participate in the simulation. Sessions were recorded following normal course practice in the case of grading discrepancies. The purpose of the session was to collect a medication history from a patient recently discharged from hospital to home and to reconcile discrepancies identified in real time. To assist with preparation, prior to the session students were provided a patient scenario and medication list, medication history interview handout, and documentation template.

The role of the patient was fulfilled by volunteer faculty (n = 2), residents (n = 6), and one advanced pharmacy practice experience (APPE) student completing an elective academia rotation. To standardize the interactions and patient responses, volunteers playing the mock patients were given a mock patient scenario that included background, patient information, and current medication knowledge from the patient’s perspective. Those playing the mock patients also served as the evaluators. Student performance was scored using a competency and communication rubric from the course series that was modified to include statements to assess telehealth etiquette skills. The coordinator of the course determined which telehealth etiquette skills would be included in the rubric based on practice experience and a review of the literature as outlined above.

The rubric used for the study contained 13 assessment statements for competency and six for telehealth etiquette (Figure 1). Each statement was evaluated on three dimensions, namely “satisfactory,” “unsatisfactory,” or “not performed.” The rubric did not describe criteria for any of the dimensions beyond the assessment statements. Students were provided the rubric one week prior to participating in the simulation. A brief overview of the activity scenario and the rubric was provided to evaluators before the session, but no specific training or guidance was provided on how to interpret the rubric. Upon finishing the activity, the completed rubric containing quantitative and qualitative feedback was provided to the student.

Figure 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 1.

Excerpt from the Telehealth Simulation Assessment Statements. The rubric used for the study contained 13 assessment statements for competency and six for telehealth etiquette. Each statement was evaluated on three dimensions, namely “satisfactory,” “unsatisfactory,” or “not performed.”

Upon semester completion, two additional faculty evaluators reviewed the recorded sessions and evaluated the students using the rubric to gather retrospective data on interrater reliability (IRR). The intended purpose of using IRR for this study was to assess reliability of the mean ratings from multiple coders. Of 67 students in the class, 55 recordings were able to be reviewed a total of three times each. Seven sessions were evaluated by three faculty members, 36 sessions were evaluated by a resident and two faculty, and 12 sessions were evaluated by the APPE student and two faculty. There were four missing recordings and eight incomplete recordings that were excluded from the sample. The intraclass correlation coefficient (ICC) was calculated to determine the IRR of the rubric overall and its subcomponents of competency and telehealth etiquette. A p<.05 was considered significant. A one-way random effect model and average measures mean (ICC [1,3]) was used to calculate the ICC when the evaluators for each student were not the same three evaluators each time. A two-way mixed effect model and average measures mean (ICC [3,3]) was used to calculate ICC when the same three evaluators reviewed a recording. Higher ICC values indicated stronger IRR, which was suggestive of stronger consistency among evaluators. ICC scores less than 0.4 indicated poor IRR, scores of 0.4-0.59 indicated fair IRR, scores of 0.6-0.74 indicated good IRR, and scores of 0.75-1 indicated excellent IRR.20 Statistical analysis was performed using SPSS Statistics, version 27 (IBM Corp). This study was reviewed by the University of Toledo Social, Behavioral, and Educational Institutional Review Board and was deemed exempted.

RESULTS

The ICC results and their corresponding IRR outcomes are presented in Table 1. When including all evaluators in the pool, the ICC was 0.63 (p<.001; 95% CI, 0.423-0.772), indicating good IRR for the rubric overall. When the sessions were evaluated only by faculty members as the three evaluators, the calculated ICCs showed the strongest consistency among evaluator types. Comparatively, when the evaluator pool contained residents or a student, the IRR diminished.

View this table:
  • View inline
  • View popup
  • Download powerpoint
Table 1.

Intraclass Correlation Coefficient Results for Interrater Reliability

For the 13 competency statements of the rubric, ICCs were calculated and assessed for each evaluator type. For all evaluators together, the ICC for the competency statements of the rubric was 0.69 (p<.001; 95% CI 0.517-0.809), indicating good IRR. When the rubric was evaluated by faculty, the calculated ICC resulted in excellent IRR. Again, when the evaluator group included residents, IRR diminished for the competency component of the rubric.

Comparatively, for the rubric’s six telehealth etiquette statements, the ICC for all evaluators was 0.47 (p=.003; 95% CI, 0.169-0.672), indicating fair IRR. For the faculty evaluators, the ICC for the rubric’s telehealth etiquette statements suggested fair IRR, but it was not significant. For the resident-faculty evaluator group, the ICC for the rubric’s telehealth etiquette statements indicated fair IRR. When a student was included as an evaluator, the ICC for the telehealth etiquette statements suggested poor IRR but was not significant.

DISCUSSION

In a recent commentary, Frenzel and Porter clearly described the relevance and need to prepare pharmacy students for delivering services remotely.7 While descriptions of telehealth simulation activities have been published, a method to evaluate telehealth etiquette skills that can be used by varying evaluator types is not available.8,10,12,14 Candelario and colleagues described a telemedicine simulation where pharmacy students worked in a group to identify medication-related problems (MRPs) incorporated into a discharge follow-up encounter. Standardized patients completed a patient perception scale checklist to assess the students’ communication skills. Findings showed that less than half of the student teams performed communication skills for this activity accurately and completely.8 Although methods to assess student performance in a telehealth simulation were discussed, this study focused more on transitions of care concepts and did not mention whether telehealth etiquette skills were assessed for the activity. In another study, Gustin and colleagues described a training program developed to teach health care students telehealth etiquette skills. While this study used a telehealth etiquette checklist to help facilitate discussion, reflection, and feedback on techniques, it was not used as an assessment tool.12

In the current study, the IRR of the rubric’s telehealth etiquette statements was not consistent across evaluator types, which indicates that improvement efforts should be focused on this area. In addition to potentially revising the rubric statements, the overall clarity of the rubric could be improved by adding a description of expectations for the different dimensions. Telehealth may also have been a new concept to the evaluators and something they may not have had enough personal experience with. While not recorded in the study, IRR outcomes may also have been affected by the professional backgrounds of the evaluators, since evaluators may have had different types of experiences to draw upon when completing the rubric. Another factor impacting IRR was that no formal training was given to evaluators before participating in the activity to help them interpret the telehealth statements. To help standardize scoring and improve consistency among evaluators, training will include instruction on how to evaluate student performance, discussions of the rubric, and opportunities to assess a sample session using the rubric.

The current study demonstrated good IRR across different evaluator types, which also supports the need for continued development of the assessment tool. The IRR of the rubric was highest when faculty members used the tool to evaluate students, but this may not be a realistic recommendation to make, considering faculty workloads and responsibilities. Allowing residents and students to assist in the simulation and evaluation process is also valuable to their development as potential educators. Of note, the student-faculty evaluator group had a higher ICC for the rubric compared to the resident-faculty evaluator group, but this was likely because the student evaluator assisted with developing some rubric statements as part of a rotation project. When reviewing the IRR results of the competency assessment statements in the rubric, fair to good IRR was generally seen across all groups except for the faculty-only group, which showed excellent IRR. This is likely because the faculty evaluators have used the competency and communication portion of the rubric in other skills lab courses. The competency statements had already been standardized for the course series and were originally developed with assistance from the faculty involved in this research. Using external faculty for further evaluation of the rubric will be important for expanding IRR results.

Some limitations of the study should be considered. First, these results capture data from the first use of the rubric among a limited pool of evaluators for one cohort of students. Continued use of the rubric with additional evaluators may improve reliability of the rubric over time. Second, the original evaluator of each session was potentially multitasking, acting as both the mock patient and evaluating the student in real time. The first evaluator completed the rubric after each session and did not have the recording to view, whereas the subsequent evaluators watched the recordings; this may have introduced some bias. Lastly, validation of this rubric is needed before it is used in a high-stakes assessment rather than a formative assessment.

CONCLUSION

No validated rubric is available to assess students in telehealth etiquette skills, and additional research is needed before the rubric described in this study can be universally implemented in pharmacy education. The rubric applied in this study demonstrated good IRR overall among various evaluator types when used in a skills lab course. Evaluating and improving the quality of the rubric’s telehealth etiquette statements are necessary to aid in improving the feedback provided to students on their performance. An evaluator training session on interpreting and using the rubric is also planned to help improve reliability with future assessments.

ACKNOWLEDGEMENTS

The authors would like to acknowledge Dr. Tian Chen for providing statistics consulting, and Dr. Mariann Churchwell for her guidance in this research.

  • Received June 17, 2021.
  • Accepted October 8, 2021.
  • © 2022 American Association of Colleges of Pharmacy

REFERENCES

  1. 1.↵
    U.S. Department of Health & Human Services. What is telehealth? https://www.hhs.gov/hipaa/for-professionals/faq/3015/what-is-telehealth/index.html. Accessed October 18, 2022.
  2. 2.↵
    1. Boner L
    . Telehealth basics for pharmacists during COVID-19 and beyond. Pharmacy Today. 2020;26(6):28-29. https://www.pharmacytoday.org/article/S1042-0991(20)30517-X/pdf. Accessed October 18, 2022.
    OpenUrl
  3. 3.↵
    American Society of Health-System Pharmacists. Issue brief: COVID-19 and telehealth changes. https://www.ashp.org/advocacy-and-issues/key-issues/other-issues/additional-advocacy-efforts/issue-brief-covid-19. Accessed October 18, 2022.
  4. 4.↵
    1. Alexander E,
    2. Butler CD,
    3. Darr A,
    4. et al.
    ASHP statement on telepharmacy. Am J Health Syst Pharm. 2017;74(9):e236-e241.
    OpenUrlFREE Full Text
  5. 5.↵
    American Pharmacists Association. What is telehealth? https://www.pharmacist.com/Practice/Practice-Resources/Telehealth. Accessed October 18, 2022.
  6. 6.↵
    1. Koonin LM,
    2. Hoots B,
    3. Tsang CA,
    4. et al.
    Trends in the use of telehealth during the emergence of the COVID-19 pandemic – United States, January-March 2020. MMWR Morb Mortal Wkly Rep. 2020;69(43):1595-1599.
    OpenUrlCrossRefPubMed
  7. 7.↵
    1. Frenzel J,
    2. Porter A
    . The need to educate pharmacy students in telepharmacy and telehealth. Am J Pharm Educ. 2021;85(5):Article 8566.
    OpenUrl
  8. 8.↵
    1. Candelario D,
    2. Cunningham K,
    3. Solano LA,
    4. Pabst A,
    5. Srivastava S
    . Description of a transitions of care and telemedicine simulation lab activity. Curr Pharm Teach Learn. 2019;11(11):1184-1198.
    OpenUrl
  9. 9.↵
    1. Almodovar AS,
    2. Chang HK,
    3. Matsunami M,
    4. Coleman A,
    5. Nahata MC
    . Confidence in skills applied to patient care among PharmD students in telehealth medication management programs versus other settings. Curr Pharm Teach Learn. 2018;10(5):558-565.
    OpenUrl
  10. 10.↵
    1. Begley K,
    2. O’Brien K,
    3. Packard K,
    4. et al.
    Impact of interprofessional telehealth case activities on students’ perceptions of their collaborative care abilities. Am J Pharm Educ. 2019;83(4):Article 6880.
    OpenUrl
  11. 11.↵
    1. Beal JL,
    2. Weber ZA,
    3. Isaacs AN,
    4. et al.
    Pharmacy student perceptions and preferences of in-person versus video recorded evaluations in skills-based courses. Am J Pharm Educ. 2020;84(11):Article 7976.
    OpenUrl
  12. 12.↵
    1. Gustin TS,
    2. Kott K,
    3. Rutledge C
    . Telehealth etiquette training: a guideline for preparing interprofessional teams for successful encounters. Nurse Educ. 2020:45(2);88-92.
    OpenUrl
  13. 13.↵
    1. Skoy ET,
    2. Eukel HN,
    3. Frenzel JE,
    4. Schmitz TM
    . Performance and perceptions: evaluation of pharmacy students’ consultation via telepharmacy. J Pharm Technol. 2015;31(4):155-160.
    OpenUrlCrossRef
  14. 14.↵
    1. VanLangen KM,
    2. Sahr MJ,
    3. Salvati LA,
    4. Meny LM,
    5. Bright DR,
    6. Sohn M
    . Viability of virtual skills-based assessments focused on communication. Am J Pharm Educ. 2021;85(7): Article 8378.
    OpenUrl
  15. 15.↵
    1. Haney T,
    2. Kott K,
    3. Fowler C
    . Telehealth etiquette in home healthcare: the key to a successful visit. Home Healthc Now. 2015;33(5):254-259.
    OpenUrl
  16. 16.↵
    1. Bulik RJ
    . Human factors in primary care telemedicine encounters. J Telemed Telecare. 2008;14(4):169-172.
    OpenUrlCrossRefPubMed
  17. 17.↵
    American Telemedicine Association. A concise guide for telemedicine practitioners: human factors quick guide - eye contact. https://www.telemedecine-360.com/wp-content/uploads/2019/02/2016-ATA-Eye-Contact-Quick-Guide.final_.pdf. Accessed October 18, 2022.
  18. 18.↵
    1. Rienits H,
    2. Teuss G,
    3. Bonney A
    . Teaching telehealth consulting skills. Clin Teach. 2016;13(2):119-123.
    OpenUrl
  19. 19.↵
    1. Iafolla T
    . 10 telemedicine etiquette tips to deliver professional care. https://blog.evisit.com/10-telemedicine-etiquette-tips-deliver-professional-care. Accessed October 18, 2022.
  20. 20.↵
    1. Cicchetti DV
    . Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284-290.
    OpenUrlCrossRefPubMed
PreviousNext
Back to top

In this issue

American Journal of Pharmaceutical Education
Vol. 86, Issue 8
1 Nov 2022
  • Table of Contents
  • Index by author
Print
Download PDF
Email Article

Thank you for your interest in spreading the word on American Journal of Pharmaceutical Education.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Evaluation of a Rubric to Assess Pharmacy Student Performance in a Telehealth Simulation Exercise
(Your Name) has sent you a message from American Journal of Pharmaceutical Education
(Your Name) thought you would like to see the American Journal of Pharmaceutical Education web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
2 + 4 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Evaluation of a Rubric to Assess Pharmacy Student Performance in a Telehealth Simulation Exercise
Michelle N. Schroeder, Aaron J. Lengel
American Journal of Pharmaceutical Education Nov 2022, 86 (8) ajpe8778; DOI: 10.5688/ajpe8778

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Evaluation of a Rubric to Assess Pharmacy Student Performance in a Telehealth Simulation Exercise
Michelle N. Schroeder, Aaron J. Lengel
American Journal of Pharmaceutical Education Nov 2022, 86 (8) ajpe8778; DOI: 10.5688/ajpe8778
del.icio.us logo Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • INTRODUCTION
    • METHODS
    • RESULTS
    • DISCUSSION
    • CONCLUSION
    • ACKNOWLEDGEMENTS
    • REFERENCES
  • Figures & Data
  • Info & Metrics
  • PDF

Similar AJPE Articles

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Predictors of Academic Success in a Nontraditional Doctor of Pharmacy Degree Program at a Historically Black College and University
  • Student Performance on Graded Versus Ungraded Readiness Assurance Tests in a Team-Based Learning Elective
  • Evaluating Pharmacy Faculty’s Awareness of Teaching and Learning Myths and Misconceptions
Show more BRIEF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Keywords

  • telehealth
  • telepharmacy
  • tele-education
  • etiquette
  • rubric

Home

  • AACP
  • AJPE

Articles

  • Current Issue
  • Early Release
  • Archive

Instructions

  • Author Instructions
  • Submission Process
  • Submit a Manuscript
  • Reviewer Instructions

About

  • AJPE
  • Editorial Team
  • Editorial Board
  • History
  • Contact

© 2023 American Journal of Pharmaceutical Education

Powered by HighWire