Design and Evaluation of Video Podcasts for Providing Online Feedback on Formative Pharmaceutical Calculations Assessments

INTRODUCTION

Proficiency in completing pharmaceutical calculations is a core competency for health professionals as errors can compromise patient safety.^1,2 This brief evaluated the potential of video podcasts as a novel method of providing feedback on a formative assessment of pharmaceutical calculations to final year pharmacy students enrolled in the National Pharmacy Internship Programme (NPIP) in the Republic of Ireland – a workplace-based program with academic content primarily delivered and formatively assessed online. Delivery of educational material online requires reconsideration of traditional methods of assessment and feedback, and video podcasts are an option relevant to calculations.³ Video podcasts based on the principles of Mayer’s cognitive theory of multimedia learning⁴ were designed as a novel method to provide feedback on a formative calculation assessment. We evaluated this approach and compared it to standard written solutions.

Podcasts are media (audio) files that are distributed via the Internet for use on personal computers and portable media devices. The term originally referred to audio files played on the iPod (Apple Inc., Cupertino, CA) portable media device.⁵ Further development has led to the use of video podcasts (sometimes referred to as vodcasts), which combine audio and visual outputs. These have been used extensively in educational settings and have shown to have a positive impact on the student learning experience, and the application of video podcasts as a method of providing feedback has been considered an area of potential future benefit to learners.^6-8 In several studies, “worked example” video podcasts are described as a beneficial way of presenting complex material.^9,10 “Worked example” video podcasts are multimedia clips used to explain problems, articulate reasoning and assist students in learning how to solve specific problems. The benefit of worked examples has been reported in disciplines such as mathematics, science and engineering, although none have been reported specifically relating to pharmaceutical calculations.^11,12 This study provides an overview of the steps taken in design and evaluation of the feedback.

METHODS

NPIP students (n=162) were provided with a formative 20-item multiple-choice question (MCQ) assessment delivered via the virtual learning environment (VLE) in 2012/2013. Two kinds of feedback were developed and made available to all students. MCQs were separated into two groups of 10 questions, one group for video podcast feedback and the other for written solutions. Each group had questions of comparable difficulty level as determined using the item analysis and a difficulty ranking score ranging between 1 (0% to 20% achieving the correct answer) and 5 (81% to100% achieving the correct answer).

For the podcast group, 10 individual video podcasts were developed using Jing software (TechSmith, Okemos, MI), which enables simultaneous screencast and audio recordings of up to 5 minutes. A Bamboo digital stylus pen and tablet (Wacom, Kazo, Saitama Prefecture, Japan) were used to digitize and record handwritten solutions to each calculation – facilitating optimal synchronization between audio and visual materials in line with multimedia learning principles.⁴ Each video podcast took 30-45 minutes to develop and ranged in length from 2 to 5 minutes with a mean of 3 minutes. The written solution feedback was prepared using PowerPoint (Microsoft, Redmond, WA), here each step in the calculation was typed and there was no audio explanation. Both kinds of feedback were made available at the same time.

Evaluation was performed for both methods using a repeated measures design. Here, all students completed the evaluation of both types of feedback rather than allocating the students to groups receiving one feedback type only. A mixed methods survey was developed to evaluate the video podcasts. The combination of Likert rating scale questions and open-ended questions was used to strengthen the evaluation, and facilitate triangulation and comparison with other studies.⁸ This is especially important given the difficulty in achieving desirable response rates for surveys.¹³ The Likert rating scale questions are shown in Table 1. The open-ended questions asked respondents to describe which features of the video podcast feedback they found useful and provide suggestions for improvement. The survey was prepared and reviewed for face validity by the authors and one NPIP student. As per institutional requirements, all questions were optional and no incentive was offered. Ethical approval was granted by Lancaster University and permission to conduct the evaluation was obtained from the program director of the NPIP. All registered students were invited to take part in the evaluation and given 10 days to complete the survey, with a reminder after 8 days. Survey responses were collated and tabulated, and qualitative content analysis was used to analyze the open-ended responses.

RESULTS

Fifty-three students (33% of those invited) responded to part or all of the survey. Forty-four students gave details of their demographics; 73% (n=32) were female and 27% (n=12) were male. Respondents’ age was also surveyed; 66% (n=29) were 20-25 years old, 20% (n=9) were 25-30 years old, and 14% (n=6) were over 30 years old. In this evaluation, positive cognitive attitudes were reported toward video podcasts and written solutions. The survey questions and responses are detailed in Table 1. The majority of participants agreed or strongly agreed that video podcasts were easy to understand (91%; n=31), clearly explained (91%; n=31), and a useful form of feedback (88%; n=30), and participants had a comparable view of written answers (Table 1). The themes and frequencies arising from the qualitative content analysis, and illustrative quotes are presented in Table 2. Qualitative analysis of the open-ended responses triangulated these positive findings, with respondents indicating that they felt positively about particular features of video podcasts, including clear explanations, stepwise approach to explanation and the audio/visual features (Table 2). Respondents also suggested areas for improvement such as expanding their application and technical aspects.

DISCUSSION

As would have been predicted based on similar studies in other fields, respondents reported that they liked the feedback video podcasts and found them useful.⁸ The benefits reported in this study were consistent with those reported previously, namely: a clear explanation and a step-by-step approach,⁹ and the synchronization of delivery of audio and visual information.¹¹ Additionally, respondents requested more examples using video podcasts and identified technical aspects (eg, to improve sound quality) that may improve the audio and/or visual aspects of the podcasts, again a finding common to other research.⁸

Video podcasts take longer to produce than typed solutions, so we wanted to understand if there were distinct benefits that make video podcasts a worthwhile endeavor for faculty members. To this end, we simultaneously evaluated traditional feedback. The addition of this comparison group showed that a comparable number of respondents felt similarly about written solutions, which was a relatively unexpected finding. The positive review of both methods by the same cohort indicates that there may be particular advantages to both, and this is supported by free-text comments that clearly indicate that there are features particular to video podcasts that were considered useful, that are not afforded by written solutions. While there are several possible reasons, one possibility is the relatively high proficiency of final year students who can likely determine their errors from a written solution. It has been previously suggested that those with relatively low prior knowledge derive most benefit from material presented in this manner.⁴ We aim to explore this empirically through a future study using video podcasts for feedback on pharmaceutical calculations in an undergraduate student cohort.

Our findings provide further support for use of video podcasts as a method of feedback provision based on self-reported cognitive and affective attitudes, and we report qualitative data that will help guide future instructional design relating to online feedback in our institution. Although a low response rate, and the self-report nature of the study are limitations of this evaluation, the preliminary findings are strengthened by triangulation with the qualitative data. Aside from determining face validity for the survey, no reliability or other validity testing was performed, and we acknowledge this as a potential limitation of the study. We acknowledge that positive findings require further study to comprehensively understand the relative benefit of each feedback type and have taken care not to overstate our findings.¹⁴

CONCLUSION

Video podcasts are a beneficial form of online feedback on pharmaceutical calculations. The data suggest that feedback in video podcast form may be a useful addition to traditional feedback. Data collected may be of interest for those involved in instructional design for online learners or those teaching pharmaceutical calculations. Our illustration of the use of principles of learning sciences research on video podcasts, and cognitive load theory in the design of online learning material for health professions education is timely, and currently highlighted as a topic of importance in the wider health professions education literature.¹⁵