Evaluating the Effects of Flexible Learning about Aseptic Compounding on First-year Students in a Pharmacy Skills Laboratory

INTRODUCTION

When students begin a course of study, they bring an array of prior experience, learning preferences, learning needs, and personal characteristics. Educators can help address these differences by incorporating varying degrees of flexibility into a learning environment.¹ Providing learners with choices is a key element of the concept of flexible learning.^1-3 These choices can relate to how, when, where, and what is learned. When these kinds of choices are included, learners are empowered to make more of their own decisions related to learning.¹ Studies have shown that students feel instructional materials in a flexible format offer more control and support course outcomes.^4,5

The use of audio-visual resources can be used to create flexible learning experiences. Web-based, on-demand audio and video formats allow students to control the pace at which they consume content, as well as determine when and where they view it.⁴ Abdous et al suggest that recorded podcasts are useful for review, provide more flexible access to learning materials than traditional formats, and can be used to compliment other course resources.⁶ McKinney and Page found that more than 60% of students in a nursing class appreciated being able to access lectures from home, and 89% reported that the recordings aided their understanding of pathophysiology.⁷ Similarly, in Bolliger et al’s study, the use of podcasts led graduate and undergraduate students to feel confident they understood what they were supposed to learn from the instruction in an online course.⁸

Although learner satisfaction with online video is a prevalent theme in the literature, reporting on direct measures of learning (course performance) has been less consistent. Traphaganet al reported that, for students who had access to lecture videos, more video viewing was linked with higher performance.⁹ Similarly, pharmacotherapy students who were given access to a series of renal-pharmacotherapy videos to be watched prior to in-class activities performed significantly better on the renal examination section than students from the prior year who had no access to the videos or in-class activities.¹⁰ However, several studies conducted in health professions learning environments, including a physical therapy course, nursing course, and undergraduate medical course, found no significant difference in performance, as measured by examination scores, between students who had access to on-demand, audio-visual resources and those who only received live instruction.^11-13 Additionally, a study of undergraduate geography students found no significant differences in examination scores prior to and after adoption of 6 podcasts.⁴

Studies considering the relationship between technology and learner self-efficacy also have relevance to the present study. Research suggests that self-efficacy, an individual’s perception of his or her ability to produce a particular outcome, is a powerful predictor of academic performance.^14,15 Bandura argued that self-efficacy influences cognitive processes and learner success.¹⁶ Several studies looked specifically at the effects of technology on learner self-efficacy. Isiksal and Askar suggested a link between the use of geometry software and students’ mathematics self-efficacy when compared with a group who received only traditional instruction. Cauble and Thurston found interactive multimedia lessons provided multiple benefits to students such as influencing their confidence level and sense of competence.^17,18 Additionally, providing learners with the ability to manipulate elements within a multimedia learning tool has a significant impact on learner self-efficacy.¹⁹ bosse et al found training involving the use of instructional video, specifically improved the self-efficacy of problem-based learning (PBL) tutors in terms of techniques for dealing with difficult situations in PBL environments.²⁰

A literature search revealed a knowledge gap regarding the use of on-demand recordings to increase flexibility in a pharmacy skills laboratory. The primary objective of this project was to determine the effect of flexible learning through supplemental online videos about aseptic compounding techniques on student performance in such a laboratory. The secondary objective was to determine the effects of the same videos on student self-confidence. Investigators hypothesized that students who accessed videos would have improved final assessment scores as well as increased self-confidence.

DESIGN

The doctor of pharmacy (PharmD) program at the University of Georgia College of Pharmacy requires 5 semesters of pharmacy practice skills laboratory. Two semesters are offered in the first (P1) year, two in the second (P2) year, and one in the third (P3) year. Class sizes average 145 students. Four laboratory sections of are available each week and each section is offered 8 times over 2 weeks to reduce the number of learners to approximately 20 per section. Aseptic compounding is introduced in the first semester of the P1 year. All students enrolled in skills laboratory courses are also enrolled in a parallel online course via the course management system, Desire 2 Learn (D2L, Kitchener, Ontario, Canada). Assignments, handouts, and assessments are available to learners through this technology.

During the summer of 2013, investigators, the course coordinator, and college’s instructional designer discussed the development, implementation, and design (Figure 1) for the research project. Researchers agreed that the P1 class was an ideal group on whom to study the effects of flexible learning about aseptic compounding. The rationale included the low likelihood that any P1 student had received previous aseptic compounding training and that collection of true baseline self-confidence data were realistic. Researchers agreed that 5 short videos (about 5 minutes in length) focused on selecting the diluent, gowning and garbing, sterilizing the hood, preparing the product, and checking the final product were ideal.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Sterile Compounding Exercises and Assessments for First-year Pharmacy Students.

Investigators mapped out a filming plan on paper and discussed how to optimize it. Video recording required about 4 hours and was completed on one day. The instructional designer videoed the coordinator as the coordinator completed each activity. An additional 10 hours were required for postproduction editing. All videos were uploaded to a media server and corresponding hyperlinks, and descriptions for each video were placed on the homepage of a distinct “research course” in D2L. The tracking and statistics features were activated in the research course to monitor the number of visits and duration of visits by students to each video. Other features (eg, chat, discussion board, e-mail, news) were disabled in the research course.

The coordinator introduced the research project and obtained consent from students during the first week of the course. He explained that the course would be identical to previous years, but that supplemental videos about aseptic compounding would be made available during the semester to assess the effects of flexible learning. The subsequent weeks are outlined in Figure 1. This project was approved by the University of Georgia Institutional Review Board.

The primary outcome was assessed at the conclusion of the semester. Investigators hypothesized that students who clicked video links would obtain better performance scores on the final assessment than their colleagues who did not view the videos. Researchers attempted to correlate the number and duration of video clicks with final assessment scores by using the D2L statistics data captured from the research course. Two faculty members, a skills laboratory faculty member (grader 1) and the college’s director of assessment (grader 2) each observed and assessed about 74 randomly assigned students during week 14. Both worked with the coordinator in advance to practice using the evaluation rubric. The coordinator attempted to maximize inter-rater reliability for the 2 assessors by meeting with them during the summer of 2013 (approximately 45 minutes) and just prior to the final student assessments (approximately 20 minutes). During the review processes, the coordinator explained and demonstrated the most common student mistakes observed in previous years. He compounded several products while both evaluators observed and graded his performance. The group discussed why assessment scores differed between graders at the completion of each mock assessment.

At week 14, groups of 4 students were scheduled to enter the compounding area every 15 minutes and were given the same instructions by the coordinator. Each student was given an IV label for Gentamicin 60mg in 100mL of 0.9% NS and calculators were permitted. Each student was expected to select the correct medication and diluent from the storage shelves, correctly interpret the intravenous product label, properly sterilize the horizontal laminar flow hood, and use aseptic technique to accurately compound a gentamicin drip. Time did not permit assessments of hand-washing, garbing and gowning, or the pharmacist final check; students compounded in plain clothes and the use of gloves was optional. Faculty members observed 2 student compounders simultaneously and used the same rubric introduced to students at weeks 7 or 8 to evaluate performance. Students received a score of 0-100% on the final assessment which counted 20% toward their course grade.

The secondary measure of student self-confidence was evaluated using online assessments (baseline, midpoint, final) created on Qualtrics (Qualtrics, Provo, Utah). Students used a 5-point Likert scale (1=strongly disagree to 5=strongly agree) to evaluate each statement. Investigators sought to establish a correlation between the number and duration of views of videos with self-confidence scores. The baseline assessment was administered prior to any formal instruction during weeks 5 or 6. The midpoint assessment was administered at the conclusion of laboratory activities during weeks 9 or 11. At this stage, students had observed correct techniques demonstrated by the coordinator, had been given formative feedback using the grading rubric, and had compounded 2 large volume preparations. The final self-confidence assessment was completed after students had compounded 3 sterile products and was administered at the conclusion of weeks 12 or 13.

Final assessment scores were compared using a t test to evaluate potential differences between faculty graders. Two-way analysis of variance (ANOVA) was used to control for the difference between graders and to determine the relationship between number and duration of video views and final assessment scores. An ordinal logistic regression (OLR) model was used to determine the relationship between the number and duration of views and student self-confidence. Statistical analyses were completed using SPSSv21 (IBM, Armonk, NY). All p values were 2-sided and considered significant if less than 0.05 for all results.

EVALUATION AND ASSESSMENT

One hundred forty-three of 144 students (99%) agreed to participate in the research project. A majority of participants (85%) indicated that they had no hospital experience. Only 1 student indicated more than 5 years experience compounding IV preparations (Table 1). No attempt was made to obtain data regarding aseptic compounding experience in nonhospital settings.

Sixty-nine percent of participants clicked on at least one of the videos. Sixty-three percent of participants viewed video 4, preparing the product, and 46% viewed video 5, checking the final product (Table 2). The videos most related to content on the grading rubric and final assessment were videos 1, 3, and 4.

The mean final examination score was 86.1 (SD 1.4, range 0-100). One student received a zero on the final assessment; this student’s score was not included in the final comparisons between groups. The mean scores for those who completed the final assessment were 89.4 (8.76) for grader 1 and 84.01 (14.9) for grader 2. A 2-sample test revealed the difference between graders was significant (p<0.01). This difference was controlled during the analysis of the primary and secondary outcome measures (discussed below).

A postdata collection experiment revealed that data captured by the D2L research course did not accurately report number and duration of video views. Students could click on a video link in the research course and leave the resulting window open for extended periods without actually watching the video (viewing times ranged from 5-200 minutes for some videos). Thus, duration measured how long the video webpage was open and not how long the video actually played. Additionally, D2L counted the number of times students clicked the video link instead of counting the number of times students actually played the video. Ultimately, researchers decided that duration should be removed from consideration and that the number of clicks should be collapsed into an indicator variable. To this end, indicator variable I_x was created to distinguish between students who clicked on the link for video x at least once (I_x=1) and students who did not (I_x=0). Because students had access to 5 videos, there were 5 such indicator variables created. A sixth indicator variable, I_t, was created to distinguish between students who clicked on at least one of the 5 video links (I_t=1) and students who did not click on any of the video links (I_t=0). Frequency distributions for these 6 indicator variables are shown in Table 2.

To determine whether students who clicked on the video links had better final assessment scores than students who did not access the videos, investigators used a 2-way ANOVA model. Controlling for the effect of grader, researchers ran separate 2-way ANOVA models with each of the 6 indicator variables described above. Results are only shown in Table 3 for videos 1, 3, and 4 as content on those videos was most related to the assessment. Although students who clicked specific video links typically outperformed those who did not click video links, the ANOVA results indicated the difference was only significant for video 1 (p=0.02). The students who clicked on at least one of the 5 video links also typically outperformed those who did not, but the difference between the 2 groups was not significant (p=0.12). Researchers removed data for 9 students with hospital experience and reanalyzed the data, but again found no significant differences.

When investigators considered scores for each grader separately, they observed no statistical difference between those evaluated by grader 1 who clicked the link for video 1 and those evaluated by grader 1 who did not click the video 1 link (p=0.59). In contrast, those evaluated by grader 2 who clicked the link for video 1 had a significant improvement in final scores over those evaluated by grader 2 who did not click the link for video 1 (p=0.01). A similar pattern was observed for all of the other videos, but the result was only significant for video 1. Investigators could not explain these differences.

Baseline and final mean self-confidence scores were significantly different (p<0.001) for all 7 questions. Mean scores improved for all items (Table 4). The OLR model was used to determine whether or not clicking on a video link at least once had an impact on final self-confidence scores. To accomplish this, each of the 7 self-confidence questions had to first be linked to exactly one of the 5 video segments. Controlling for baseline self-confidence scores, the OLR results indicated that clicking on a particular video link at least once had no significant impact on a student’s final self-confidence score. This was true for all 5 video links (Table 5).

DISCUSSION

Published literature indicates that flexible learning can be used in lecture courses to improve students’ performance on assessments and to bolster their self-confidence. Investigators hypothesized that this premise would also hold true in the skills laboratory environment. When data were analyzed, investigators were surprised that those who took advantage of flexible learning opportunities through online videos seemed to have no significant advantage in measures of performance or self-confidence over those who did not.

Although great care was taken to plan and design the study, investigators realized during data analysis that the number of times a video link was clicked as well as the duration of the view captured from the statistics tracking features in the D2L research course were not accurate. This has important implications and should serve as a warning for faculty members who use statistics from D2L to evaluate student engagement and performance in an online course.

Mean scores for grader 1 were significantly higher and the standard deviation was much smaller for grader 1 than for grader 2. Grader 1, a pharmacist, had previous experience using the grading rubric and assessing compounding, whereas grader 2 was not a pharmacist and did not have such experience. As mentioned previously, the coordinator attempted to overcome these differences in experience through multiple exercises where both assessors graded mock compounding exercises and scores were compared. Although statisticians controlled for the differences that emerged during data analyses, these differences prompted questions. Researchers could not determine whether students assessed by grader 2 were truly poorer performers than those assessed by grader 1. Researchers could only speculate about whether similar results would have been observed if both assessors had been pharmacists or nonpharmacists.

Faculty assessors also differed with their approach to assessment. Grader 1 observed student pairs from approximately 3-5 feet away, while grader 2 stood between student pairs as they compounded. A 1977 study by Geen and Gange revealed the presence of an audience is associated with poor performance on difficult tasks relative to the performance on the same tasks performed in isolation.²¹ In 1968 and 1972, Cottrell further argued that audience elicited evaluation apprehension in the subject, which effected performance.^22,23

Access to flexible learning through videos may not have significantly improved scores because they were already commendable (average 86%). Moreover, the experiment may not have been sufficiently powered to detect significant differences between the final scores of viewers versus nonviewers on final assessment scores. Published literature supports beneficial effects of flexible learning on student self-confidence, but this effect was not observed in this experiment. This group of P1 pharmacy students was homogeneous with respect to previous training and experiences. Researchers were pleased to see that self-confidence scores improved significantly from baseline for all statements. Only 66 (46%) of the students clicked the video link to learn about the pharmacist final check, and this was purposefully not taught during this laboratory. Rather, this video was included to see if students might access it more or less often than the other content. Researchers were unable to determine if the improved scores were a result of inflated student self-confidence or the availability of flexible learning opportunities.

SUMMARY

Aseptic compounding is one of the more anxiety-provoking and difficult skills for P1 pharmacy students to master because most have limited previous experience. Proper skills are modeled for students during the first semester of the P1 year at the University of Georgia, and all students have the opportunity to compound 4 intravenous preparations. Laboratory activities are sufficient practice for most students to perform well on a final assessment. Although short videos that review concepts can provide flexible learning opportunities for students, participants in this study did not take advantage of the opportunities. Those who reviewed videos had no significant advantage in final assessment or self-confidence scores over their classmates who did not view the videos.