Effectiveness of Providing Video Podcasts to Pharmacy Students in a Self-Study Pharmaceutical Calculations Module

INTRODUCTION

Proficiency in conducting pharmaceutical calculations is an essential attribute for all pharmacists. Instruction in pharmaceutical calculations is a required element of the Doctor of Pharmacy (PharmD) curricula in all schools of pharmacy as mandated by the Accreditation Council for Pharmacy Education.¹ Student pharmacists need to achieve competency in performing calculations used to dispense and compound medications. No standardized methodology for calculations instruction exists. Because of this, pharmacy schools employ various instructional methods to teach pharmaceutical calculations.²

A 2007 survey found that most US pharmacy schools use live lectures and/or discussions to teach pharmaceutical calculations.² Growing evidence shows that using video instruction or video podcasts has beneficial effects on students’ performance compared to traditional methods of instruction in general education and mathematics-based courses.^3,4 As such, some pharmacy schools have started using video modules or condensed videos within pharmaceutical calculations courses in flipped classroom models. Students watch prerecorded video modules and complete practice problems prior to class.^5,6 While student perceptions of video podcasts are positive in these models, the evidence is conflicting as to whether student performance actually improves from watching video podcasts.^5,6 No studies have explored the effects of video podcasts on student performance in a self-study pharmaceutical calculations course or module.

The University of California San Diego Skaggs School of Pharmacy and Pharmaceutical Sciences (SSPPS) historically offered Pharmaceutical Calculations as a self-study module within a three quarter first-year Pharmacy Practice self-care course. Pharmacy students used a required textbook to review concepts and practice review problems.⁷ The calculations material tested from the textbook was divided into two, one-hour quizzes per quarter, scheduled evenly throughout the quarters, for a total of six quizzes during the year. Each paper-based quiz, covering material from two textbook chapters, contained 10 multiple-choice questions and required students to show their work for each question. Students were allowed 50 minutes to complete each quiz. A score of at least 90% on each quiz was required to pass. Students who did not pass were required to attend a quiz review session and complete and pass a make-up quiz of similar content the following week. To help students prepare for each quiz, the calculations facilitator provided review slides one week prior to each quiz. Students were encouraged to make an appointment by email (no set office hours) with the coordinator if they needed additional assistance; however, students rarely requested this option.

To better assist students with learning to perform calculations, the course facilitator decided to create video podcasts in hopes that the more personal, multimedia format would better engage students in preparing for each quiz. Based on those quiz topics on which students performed the worst, as well as on student feedback, the calculations facilitator created six video podcasts, each of which provided a step-by-step demonstration of how to solve textbook practice problems. The primary objective of this study was to assess whether the addition of the video podcasts made a difference in students’ performance on the calculations quizzes, including on passing rates, compared to students’ performance on quizzes the previous year, which did not employ the video podcast on each type of question reviewed in the video podcast. The secondary objective was to determine whether students perceived the video podcasts as valuable in learning pharmaceutical calculations.

METHODS

This descriptive study assessed all first-year (P1) pharmacy students (n=69) enrolled in a four-year PharmD curriculum, all of whom took the required Pharmacy Practice course series, and therefore had enrolled in the pharmaceutical calculations module. The mean entering grade point average for all P1 students at SSPPS, including both study cohorts, was 3.67 (SD 0.16 and 0.17 for 2017-2018 and 2018-2019 classes, respectively). The P1 students were not enrolled in any other courses that were self-taught or that regularly used video podcasts outside of this module.

Six videos were created demonstrating how to solve calculations problems using the course textbook, from where quiz content was taken. To identify the types of questions on each quiz on which students performed the worst, the calculations facilitator reviewed students’ quiz performance during the 2017-2018 academic year. Based on these findings, the facilitator created a video podcast demonstrating how to solve similar problems from the textbook for the most difficult question on each of the six quizzes. The Explain Everything app (explaineverything.com), an iPad Pro, and an Apple Pencil were used to create the video podcast.⁸ For each video, the calculations facilitator first provided an overview of the concept followed by a demonstration of the steps necessary to solve the problem. Videos were brief in duration, each lasting two to four minutes. During the 2018-2019 academic year, each video on how to solve a problem was posted online on a YouTube channel one week prior to the quiz in which the problem would appear.⁹ This gave all P1 students time to watch the video in preparation for the quiz. No other changes were made to the calculations module in the 2018-2019 academic year with respect to the course, facilitator, quiz content, instruction, or grading, nor were major changes made to the Pharmacy Practice course. To minimize sharing of quiz content between classes of students, quiz material was not allowed to be removed from the classroom during the quiz or reviews, and students were required to use nonscientific calculators provided by the school. Although the exact values used in questions on each quiz were modified annually, the items remained the same.

The primary outcomes assessed were student improvement in quiz performance and the passing rate for each individual quiz problem that had a corresponding video podcast as well as overall quiz performance. Quiz scores from the 2018-2019 academic year (video podcasts provided) were compared to quiz scores from 2017-2018 (no video podcasts) as a control. The secondary outcome was students’ attitudes toward the video podcasts as a learning modality. To assess this, students were surveyed to obtain qualitative feedback at the end of the academic year via a brief 11-item online survey conducted using Qualtrics. The survey was administered during the spring quarter after the sixth and final quiz but prior to the end of the course. An email containing a link to the survey and asking students to complete it was sent out to the class. Students were given class time to complete the survey; however, if they did not feel comfortable doing so, they could complete the survey on their own time. No incentive was offered for completing the survey nor was it mandatory to do so. The survey asked whether students watched the videos, and if they had watched them, if they felt they were worthwhile to watch, helped them learn the content, and improved their ability to conduct pharmaceutical calculations. The final survey item asked students to provide suggestions for improving the videos.

Differences in numbers of students with correct vs incorrect answers on the quizzes pre- and post-implementation of video podcasts for both individual items and overall grade were compared using Fisher exact and chi-square tests between the 2018-2019 and 2017-2018 academic years, with p<.05 considered significant in STATA, version 16 (StataCorp LLC, College Station, TX). Survey responses were summarized with descriptive statistics. The study was reviewed and deemed exempt by the University of California San Diego Human Research Protections Program.

RESULTS

Initial analyses determined whether the quiz questions selected for the video podcasts were indeed the most difficult ones. Student performance on these specific six quiz questions in 2017-2018 (no video podcasts) was significantly worse (ie, fewer students were able to correctly solve these six questions) compared to the rest of the quiz questions that year (p<.001). Student performance after the implementation of the video podcasts improved. The percentage of correct answers on these six quiz problems in 2018-2019 (with video podcasts) was significantly higher than the percentage of correct answers in 2017-2018 (p<.001). Student scores significantly improved for three of the six specific quiz questions that had corresponding videos in 2018-2019, and trended toward improvement for one question (Table 1). Performance on two of the questions that had corresponding videos was not different between the two years. In the 2018-2019 year, these six quiz questions were still the most challenging topics for students to correctly solve. Student performance on these six questions was still significantly worse than performance on all of the remaining quiz questions (p=.032). Overall, student performance significantly improved in 2018-2019 compared to the year before (p<.001) with respect to percent of correct answers for all quiz questions. When assessing overall first-attempt passing rates for all quizzes, there was a significant improvement in the 2018-2019 cohort (p=.001, Table 1). There was also a significant improvement seen in first-attempt passing rates on quiz 6 (p=.03). Finally, when removing the six quiz questions associated with video podcasts, student performance on the first four quizzes was not different between the groups, but improved on quizzes 5 and 6 in the 2018-2019 cohort (p<.05).

Ninety-nine percent (68/69) of students responded to the survey regarding attitudes about the video podcasts. Sixty-two (91%) students reported watching at least one of the video podcasts. About three-quarters of the students watched the first four videos (75%, 72%, 76%, and 74% for videos one through four, respectively). Almost two thirds of the students watched the final two videos (63% for both). Students’ responses to the remaining survey items are displayed in Table 2. Students strongly agreed with various positive statements regarding the effectiveness of the videos, with median scores of 5 out of 5 (5=strongly agree) on all questions, indicating that the videos were useful, helped teach the content, and improved math skills. Student-reported reasons for not watching the videos included: additional help was unnecessary (n=4), did not have enough time (n=2), forgot (n=1), and was unaware that a video was available (n=1). Finally, common student suggestions for improvement (provided in response to a free-text item) included: create more video podcasts (n=22), provide more difficult examples (n=8), and create longer video podcasts (n=3).

DISCUSSION

This study was designed to determine whether there was a difference in quiz performance and first-time pass rates when video podcasts were used to supplement a self-study pharmaceutical calculations module in the study group (2018-2019 cohort) compared to a control group (2017-2018 cohort) that did not have access to videos. The initial goal was to identify the six most difficult calculations questions that students struggled with and to create short video podcasts to aid in student learning and quiz performance. Results demonstrated that, of the 60 quiz questions, the six most difficult were correctly chosen to create video podcasts about as each cohort of students performed the poorest on the six selected questions.

Compared to the control group, overall student performance on the six selected questions significantly improved in the 2018-2019 cohort, which had access to the video podcasts. With respect to performance on each individual quiz question that had a corresponding video podcast, all scores demonstrated improvement with the exception of the question from quiz two. All students in both years who answered incorrectly solved the problem using the same incorrect method, ie, they failed to recognize an abbreviation, which was key to solving that problem. Fewer students viewed the second video, which also may have contributed to the lower performance. Analysis of student performance on quiz questions for which a video podcast was not created further suggests that overall improvement on the first four quizzes was due to the addition of the video podcasts. With the aid of the video podcasts, overall student performance improved for all questions grouped together, and for overall first-attempt pass rates on the quiz.

Feedback was positive, with a majority of students who responded to the survey strongly agreeing that the videos were helpful as a teaching method. Most watched at least one video at some point; however, the viewing rate decreased towards the end of the academic year. Despite that the quizzes were progressively more complicated through the year, student performance on quizzes 5 and 6 showed significant improvement compared to the scores of students in the control group (2017-2018 cohort). Because SSPPS requires that students complete a comprehensive examination at the end of the third year, prior to their final clinical year, future studies may determine whether the 2018-2019 study cohort performs better on calculation questions in their third year than did the 2017-2018 control group.

The literature evaluating instructional video podcasts for use in pharmaceutical calculations education is sparse. Flood and colleagues found online video modules on pharmacy calculations to be helpful for students preparing for and receiving feedback on online pharmaceutical calculations assessments; however, the program was geared for pharmacy students in the final year of the PharmD curriculum.¹⁰ While this method may be a valuable refresher prior to graduation, pharmacy students practice as intern pharmacists and in introductory pharmacy practice experiences during the first through third years of the programs, so assistance in correctly performing pharmaceutical calculations earlier in the curriculum is beneficial. Two studies assessed the outcomes of pharmaceutical calculations taught using a flipped classroom model in which students watched prerecorded video modules and completed problems prior to class.^5,6 In-class time was used to teach and reinforce how to perform pharmacy calculations. Both studies used comparatively longer (40-60 minute and 7-15 minute) videos compared to our much shorter videos (2-4 minutes in length). Cotta and colleagues found that longer videos with a calculations quiz given at the end of class yielded higher examination scores and improved student understanding.⁵ In contrast, Gloudeman and colleagues reported that the addition of videos did not improve student performance but students felt their understanding of the material improved. The lack of a performance effect may have been because of the low (1%) contribution to the course grade given for calculations quizzes, with corresponding low effort invested by students in this area.⁶ While the flipped classroom models were similar to our self-study calculations course, we did not dedicate time in class to reviewing calculations content as our module is entirely self-taught. Further, our curriculum differs in that 10% of the Pharmacy Practice course grade is determined by student performance on pharmaceutical calculations quizzes, which may result in students emphasizing this coursework to a greater degree. These findings are in closer alignment with Cotta and colleagues, where the pharmaceutical calculations component accounts for 12% of the overall course grade.⁵ With time constraints and substantial curricular content, having a self-taught pharmacy calculations module/course enhanced by short video podcasts may help other schools of pharmacy achieve improved performance measures without needing additional in-classroom time.

There are some limitations to this study that are inherent to its design. One such limitation was using the same questions each year; however, the exact values used in the questions were modified. Using slightly modified questions ensured that course objectives were met and students were required to show their work for credit on each question. Another limitation was that the survey used lacked an item that specifically asked students whether the quizzes aided in their understanding of the material. However, we believed that improved student performance on several items on the quizzes indicated improved understanding. Finally, while our student class size was small compared to some pharmacy schools, our survey response rate was very high. The outcomes of this study are generalizable to other pharmacy schools in that creating a podcast requires the same effort regardless of class size.

CONCLUSION

The addition of video podcasts improved pharmacy students’ performance on pharmaceutical calculation quizzes in a self-taught calculations module within a first-year pharmacy practice course. Students felt the video podcasts were helpful in improving their skills and facilitated learning the material. Schools of pharmacy may consider using this additional learning modality to supplement learning of pharmaceutical calculations, particularly in self-taught modules and courses.