Comparison of Pharmacy Students' Performance in a Laboratory Course Delivered Live Versus by Virtual Facilitation

INTRODUCTION

An increasing number of pharmacy schools are employing distance education, requiring continued evaluation of student performance to validate these models. The use of various course designs, including flipped classrooms, continue to be a focus of educators who seek to improve student engagement and learning in the classroom and fit within the distance model.^1-5 Effective learning environments that focus on achieving multiple outcomes require the use of advanced technology and a good instructional design.⁶

Technology used to deliver course content in pharmacy education has been investigated and discussed, including web conferencing and interactive video teleconferencing, by schools and colleges of pharmacy.^7-9 Additionally, academic performance and student opinions have been evaluated to determine whether these experiences are equivalent to those gained in the traditional face-to-face learning model. Synchronous delivery of course content between two pharmacy schools using video teleconferencing resulted in similar academic performance by both groups.¹⁰ However, it is unclear whether virtual teaching methods would yield the same outcomes when used in a laboratory course setting. No difference in student performance outcomes was reported among students who completed pharmacotherapy courses at a distant site using videoteleconferencing.¹¹ Positive student perceptions of the flipped classroom model have been reported in the literature; however, opinions about virtual classrooms have not been reported as extensively.^1,5,12,13 Students enrolled in a virtually facilitated laboratory course concluded no difference in virtual or in-person facilitator communication.¹⁴ Additionally, the use of Zoom, version 4 (Zoom Video Communications, Inc., San Jose, CA), an internet-based video and chat conference room, for facilitation provided a unique learning environment and did not hinder students perception of learning.¹⁴

Student demographic information can serve as a baseline for comparison when assessing student performance in a course. In one study, students’ grade point averages (GPAs) prior to enrollment was correlated with student performance in pharmacotherapy courses taught face to face and in distance sites.¹¹ Lenz and colleagues reported higher scores on performance-based assessments and equivalency of the curricular experience for pharmacy students receiving their education via a distance pathway.¹⁵ While distance education has been shown to be effective, challenges such as the quality of instruction, student engagement, and technology used remain for the students and instructors involved.^{1,10,11,14-16} Course evaluations that are completed at the conclusion of the semester are often used as a measurement of student satisfaction and how well a course is taught.¹⁶

Traditional laboratory course design includes students attending class within a laboratory space while an instructor is physically present for activity facilitation. The Bernard J. Dunn School of Pharmacy at Shenandoah University has two campuses approximately 80 miles apart with a shared curriculum. Video teleconferencing is used to deliver lecture content synchronously between campuses. Traditionally, a course coordinator has been located on each campus to facilitate laboratory courses. In fall 2016, the outpatient pharmacy practice laboratory course administered in the first professional year (P1) consisted of an hour-long traditional lecture followed by a separate laboratory session with live instructor facilitation on both campuses. In fall 2017, an alternative teaching method and course design was needed to facilitate laboratory activities and deliver course content to both campuses because of the scheduled absence of a course instructor. A hybrid course format using the web-based Zoom platform for laboratory session facilitation was planned, piloted, and delivered successfully.¹³

The purpose of this study was to compare pharmacy students’ performance in the laboratory course taught in a traditional laboratory classroom with that taught by virtual (ie, remote) facilitation. Course and instructor evaluations were compared between groups to assess students’ perceptions.

METHODS

This analysis of academic performance was conducted over two years. The data included were derived from a one-credit hour outpatient pharmacy practice laboratory course delivered to first-year pharmacy students during the fall semesters of 2016 and 2017. In 2016, the course was delivered in a traditional laboratory setting to a classroom of 69 students. A year later, the same course was virtually facilitated via Zoom to 91 students. The subject matter and skills taught were the same in each year. Different lecturing methodology was used to deliver the pre-laboratory content for the two classes and was directly related to the activities and assignments to be completed during the laboratory session.

The methods of teaching for each course are summarized and compared in Table 1. In fall 2016, pre-laboratory content was taught using a traditional classroom lecture prior to the designated laboratory session. The instructor was present on site for laboratory sessions which provided face to face interactions between students and the instructor. The same ABL exercises and assignments were performed as in 2017. In fall 2017, the hybrid course design included a pre-laboratory in-person class followed by a virtually facilitated laboratory session. Prior to attending a mandatory pre-laboratory class, students were told to complete required readings and review the content for the self-paced flipped lecture that was posted to the learning management system. Weekly top 200 quizzes and active-learning activities were completed during this time to reinforce the flipped lecture content. Materials included PowerPoint slide presentations, an Excel document with a list of the Top 200 medications, links to assigned readings, and Word documents with other supplemental activity information. This in-person class session preceded the virtually facilitated laboratory session that was conducted the following day.¹³

The virtually facilitated laboratory sessions were delivered from a remote office space equipped with a web camera, laptop computer, and additional desktop monitor. Zoom, an online video/audio conferencing platform, was used to facilitate communications between the instructor and the distance learners from both campuses in real time. The instructor was able to toggle between screens to display either a shared computer screen or the video/audio conference call, which displayed an image of all students in attendance. Each student had their school-issued laptop computer which was equipped with an internal web camera that allowed them to see the instructors shared screen, video image, and/or instant message thread within the Zoom chat room. In an effort to increase student engagement within the virtual environment, students were able to interact with both the instructor and other classmates via the video/audio conference and were encouraged to communicate during Zoom sessions either verbally by queuing their microphone or through the meeting room chat feature. Students could chat with other students in the group and send private messages to the instructor. Additionally, students were permitted to use Google GSuite chat (Google, Mountain View, CA) or email if they needed to share screenshots with the instructor that supported their questions or comments. Students were permitted to work with their peers or individually on laboratory assignments. Laboratory assignments consisted of activity-based learning exercises including prescription order data entry, errors and omissions quizzes, patient case-based questions, and peer review assignments.¹³

In order to establish baseline knowledge and work experience, demographic data were pulled from student records and a one-question survey was administered. The survey was given to all students during the introduction sessions for the course to determine which students had prior experience working in an outpatient pharmacy setting. In 2016 the survey consisted of one question that prompted a description of prior work experience as a pharmacy technician in a community pharmacy. In 2017 the survey question prompted students to answer yes or no to previous experience as a pharmacy technician. Any description of prior work experience outlined in the 2016 survey was considered comparable to a yes when computing the totals for analysis. Students incoming grade point average and pre-pharmacy admission grade point average (PCAT) was obtained from student records for each class as additional demographic data for baseline comparison.

To determine whether the change in course design and laboratory facilitation would affect student performance on summative assessments, a comparison of student performance between the 2016 and 2017 fall semester classes was conducted using scores from written examinations and objective structured clinical examinations (OSCEs). The OSCE scenarios were the same activity-based exercises of transcribing a prescription (graded according to a rubric), entering the prescription, generating a prescription label (graded according to a rubric), and answering case-based questions relating to the medication and patient.

Examination questions were categorized to determine whether there was a change in the level of difficulty between groups based on overall student performance. The categories were established using standard guidelines for the difficulty index that are outlined by ExamSoft, version 2.1 (ExamSoft, Dallas, TX). In both the 2016 and 2017 class, a midterm and final examination were administered as well as a midterm and final OSCE. Written examination questions with the exact same question stem and answer choices that were used on both 2016 and 2017 examinations were identified and matched. Scoring for each matched question was compared between the 2016 and 2017 class to determine whether there was a difference in performance between the two classes. Item analysis data generated from ExamSoft assessment reports was recorded for each question and also used to evaluate the difficulty of the question itself and the percentage of the upper 25% and lower 25% of students who answered the question correctly. The difficulty of questions was evaluated using ExamSoft item analysis and categorized as either difficult (<0.6), moderately difficult (>0.6 to <0.8), or less difficult (>0.8). The evaluation of overall question performance was analyzed for significance using unpaired t tests based on the assumption of unequal variances in SPSS, version 26.0 (IBM, Armonk, NY). Scores on the midterm and final OSCE were averaged and compared between the 2016 and 2017 class and analyzed using unpaired t tests based on the assumption of unequal variances in SPSS.

Course evaluation scores were compared between the two groups to determine whether there was a difference in students’ satisfaction. Course and instructor evaluations were administered to each class using an online course evaluation software, IOTA 360 (IOTA 360, LLC, Richmond, VA) at the conclusion of the course. The questions rated students’ satisfaction of the course and instructor using a five-point Likert scale ranging from 5=strongly agree and 1=strongly disagree. The evaluations and final course grades were analyzed for overall significance using independent sample t tests based on the assumption of the unequal variances in SPSS, version 26.0.

RESULTS

Demographic data of the students from each class were obtained from student records in order to provide a baseline analysis. The students who took the course in 2016 (N=69) had a higher mean PCAT score (44.0, SD=0.40), and GPA, (3.1, SD=24.18). More students in the 2017 class (54 of 77 respondents, 70.1%) had experience working as a pharmacy technician prior to starting the course than in the 2016 class (22 of 56 respondents, 39.3%).

There was no significant difference between final course grades of the two groups. Students who completed the course in 2016 had an average final course grade of 90.1% compared to an average final course grade of 89.6% for students who took the course in 2017 (p=.05). The response rate for the course and instructor evaluation was 40% (28 of 69 students) for the 2016 traditional live course and 45% (41 of 91 students) for the remotely facilitated course in 2017. The mean evaluation score for the 2017 students was lower (4.7 out of 5) than for the traditional live students (4.8 out of 5) in 2016 (p=.58), but this was not significant. Results of the analysis of 19 individual course and instructor evaluation questions divided by course year and campus are presented in Table 2. Students ranked all evaluation questions at 4.3 or higher on a five-point Likert scale and no significant difference was noted for individual question responses between the 2016 and 2017 classes of students.

Results of the analysis of upper and lower performers on 19 individual assessment questions that were tested in the 2016 and 2017 midterm and final written examinations are presented in Table 3. Ten questions were categorized as less difficult questions, seven questions as moderately difficult, and two questions as most difficult for both groups. There were differences noted in student performance between the 2016 and 2017 classes on those questions categorized as moderately difficult. Students performed better on some questions in 2016 versus 2017 and vice versa for this category of difficulty (Table 3). In the analysis of upper and lower performers on the individual mid-term and final written examinations, students in the virtual course (2017) had a higher overall score by 3.2% than students in the live course; however, this finding was not significant (p=.59). Average midterm and final objective structured clinical examination (OSCE) grades for each year were evaluated. Students in 2016 had higher average midterm OSCE scores (95.4% in 2016 vs 94.7% in 2017) and final OSCE scores (88.2% in 2016 vs 84.7% in 2017) compared to students in 2017, but these differences were not significant (p>.99). In addition, there was no significant difference in final course grades between the two groups of students (90.1% in 2016 vs 89.6% in 2017, p=.05).

DISCUSSION

This paper reports a study that compared the academic performance and opinions of pharmacy students who completed a required outpatient pharmacy practice laboratory course in a traditional classroom setting with students who completed the course online via Zoom. This study was conducted to determine whether students performed similarly regardless of course delivery mode. Student performance on individual assessment questions and midterm and final OSCEs as well as their opinions of the course were evaluated. The flipped classroom and active-learning exercises used to enhance critical-thinking skills and problem-solving were applied to support learning in a virtual environment. The live and virtual courses were taught in successive years (2016 and 2017). By delivering the course to the entire group of students each year using the same method, it This eliminated student perceptions of unequal instructor interaction and inferiority compared to students in the live course.

The results from the analysis of upper and lower performers on the individual midterm and final written examinations, midterm and final OSCE scores, and final course grades indicate that the use of a virtual learning environment did not negatively affect student performance in this course. The authors wondered whether students with previous pharmacy experience would perform better than those without any experience in a virtually facilitated environment. The evaluation of question difficulty did not appear to show major differences between the groups, although no statistical analysis was performed. Students in the 2017 group had a higher mean incoming PCAT score and more students who had previous pharmacy experience, but their performance on individual questions was not always better than that of students in the 2016 group. Student performance on OSCEs was better among students in the 2016 live group, which may be attributed to students being present in the classroom with the instructor. However, this is difficult to conclude based on this small study design. Other studies have reported that students do not perform differently in an online environment vs a traditional classroom when the content being taught remains the same.^10,11,18 In our study, the content that was taught was the same for both groups, but how the lecture material was delivered and how the laboratory sessions were facilitated was different. It appears that this difference did not prevent students in the virtually facilitated laboratory course from performing the same as students from the traditional course. Faculty who are planning to teach in a virtually facilitated environment should consider evaluating student performance and opinions to ensure that the teaching model is suitable for student learning and engagement. Laboratory courses that require students to demonstrate their ability to perform or complete specific tasks can be facilitated virtually without a significant impact on student performance. This virtual environment may also provide an opportunity for students to work more independently and allow schools of pharmacy to use faculty more efficiently by removing the need to have faculty present in the physical classroom space.

The mean evaluation score for the virtual course was lower than that for the live course, which may have been a result of the unequal response rates between the virtual (40%) and live (45%) groups. While student satisfaction with asynchronous distance education or online courses have been shown to be generally less favored over traditional courses, there is not much in the literature reporting on student satisfaction with synchronous virtual laboratory facilitation. Many students are less satisfied with online distance courses because of the lack of synchronous interaction and/or quality engagement with the faculty member or fellow students, but most ultimately conclude that distance learning is comparable to learning in traditional classroom settings.^16-18 Facilitation of a course using Zoom offers the opportunity for the faculty and students to engage in real time, which may result in satisfaction comparable to that with a traditional course design. More evaluations using this virtual laboratory course design may add to the current body of literature to determine if student satisfaction of asynchronous distance courses compares to satisfaction of traditional courses.

In addition to the noted differences in evaluation response rates between the 2016 and 2017 classes, this study had several other limitations. Statistical analyses were not conducted for student demographic data, but there were marked differences in PCAT scores, admission GPA, and community pharmacy experience between groups. Whether this difference was because of uneven class sizes between 2016 and 2017 remains unclear. Grade point average has been correlated to student performance in other studies that have evaluated video teleconferencing models.¹⁰ Therefore, the higher scores, GPA, and experience of students in the 2017 class may have skewed data with regard to differences in examination/OSCE scores and overall grades between the virtual and live groups. However, in this study, higher incoming GPA, PCAT scores, and previous community pharmacy experience did not indicate that a correlation that students may perform better in a virtual environment. Finally, the increased usage of Zoom to deliver course content during the 2020 pandemic may prevent applying the data from this study to the current academic landscape.

Finally, there was no report on quantitative data or perceived learning impact in regard to technological complications in the virtual group. Regardless of these limitations, the findings in this study still offer valuable evidence that a virtual course can be facilitated and executed in place of a traditional course without a significant impact on student performance.

CONCLUSION

An outpatient pharmacy practice laboratory course was redesigned to facilitate the delivery of course content to two campuses by one instructor. Students completing this course via virtual facilitation performed well and documented their positive experience in course and instructor evaluations. Major assessments in the form of OSCEs and written examinations showed comparable performance with both the traditionally taught students and students taught through virtual facilitation.

To ensure that a virtually facilitated laboratory course does not affect student performance on course assessments or overall grades, future studies may need a more robust design that includes testing of teaching formats to determine pedagogical equivalence. Student performance on OSCEs and written examination questions did not change significantly with the change to the virtually facilitated laboratory course, and this new course design was well received by the students and had a positive impact.