Advanced Screencasting With Embedded Assessments in Pathophysiology and Therapeutics Course Modules

INTRODUCTION

The Accreditation Council for Pharmacy Education has promoted active learning to enhance pharmacy students’ development of critical-thinking and problem-solving skills.¹ In a national survey of US colleges and schools of pharmacy, 87% of respondents reported using active-learning strategies in their classroom activities.² As active-learning strategies have been encouraged, the incorporation of technology to enhance student learning has gained widespread use in pharmacy education.³ In fact, students have reported using technologies such as electronic course materials, digital lecture recordings, and handheld devices more often than traditional course textbooks.⁴ A hybrid course uses a combination of face-to-face and online instruction to increase time spent on active-learning activities. This teaching strategy has been well received by students, with increased student preparation for in-class discussion and improved test performance.^5-10

Screencasting is a type of e-lecture that incorporates digital recording of computer screen actions with dubbed audio narration. For example, an instructor can record voice-over narration for Powerpoint slides to create an e-lecture that can be posted in a video format for students to view.¹¹ We define advanced screencasting as the incorporation of digital recording, narration, interactivity, and metrics into an e-lecture format. E-lectures allow students to stop, restart, replay, and skip sections, allowing them to progress at their own pace according to their individual learning needs as they watch concepts, ideas, and calculations evolve in a stepwise fashion. Assessment questions are embedded within e-lectures to provide students immediate feedback on their understanding of the topic. The instructor can monitor students’ time spent viewing the e-lectures and their achievement on embedded assessment questions to gain valuable preclass assessment data.

Pathophysiology and therapeutics courses require integration of newly learned information regarding complex disease states and corresponding pharmacotherapy. Additionally, many pathophysiology and therapeutics courses incorporate a problem-based learning (PBL) model whereby students apply newly learned material to a patient case in a clinical scenario to further enhance their understanding of the topic.^12,13 The tasks required in a pathophysiology and therapeutics course are complex, and a traditional lecture learning model may not suit every ability level and learning style.

The primary objective of this study was to compare students’ test performance before and after the creation of pathophysiology and therapeutics course modules using advanced screencasting with embedded assessments. Secondary analysis included performance on test questions stratified by Bloom’s taxonomy, student satisfaction with the use of e-lecture technology, and the length of time students spent completing the e-lectures. We hypothesized that implementing a hybrid learning model that used advanced screencasting e-lectures with embedded opportunities for self-assessment would improve examination scores and facilitate students’ progression to higher cognitive domains of Bloom’s taxonomy, such as application and synthesis.

DESIGN

This study was approved by the Institutional Review Board at St. John Fisher College and was conducted at Wegmans School of Pharmacy (WSOP). Informed consent was not required, but a section was added to the course syllabus outlining the addition of e-lectures and surveys to the course to evaluate students’ preference for this technology as part of this study. Two modules in the Pathophysiology and Therapeutics IV course in 2013, viral hepatitis and clinical pharmacokinetics of aminoglycosides, were selected for this study. At WSOP, clinical pharmacokinetic content is integrated within the pathophysiology and therapeutics curriculum. Pathophysiology and therapeutics IV was the final course in a 4-semester sequence and was taught in the spring semester of the third year of the doctor of pharmacy program. Content in this section of the Pathophysiology and Therapeutics course series predominately encompassed infectious diseases. There were 74 students in the class in 2012 and 76 students in 2013. The 5-credit-hour course was offered in conjunction with a 1-credit-hour PBL laboratory where students worked in groups on a clinical case(s), which they turned in for a grade. A typical weekly schedule included 3 hours of lecture on Monday, 2 hours of lecture on Thursday, and 3 hours of laboratory time on Friday to work on a PBL group case that corresponded to the topic taught in either the Monday or Thursday lecture for that week. In 2012, prior to the lectures, students were given a reading assignment designed to take 1 to 2 hours to complete. To encourage completion of the preclass readings, an unannounced quiz was given during the next class session. The same schedule was continued in 2013; however, the preclass reading assignments were replaced with preclass e-lectures for viral hepatitis and clinical pharmacokinetics of aminoglycosides modules. Also, assessment questions were embedded within the e-lectures instead of being given as an in-class quiz. Table 1 shows the time allotted to the viral hepatitis and clinical pharmacokinetics of aminoglycosides modules in 2012 and 2013. The e-lectures were developed using advanced screencasting with embedded assessment through use of Adobe Captivate 6 software for Windows (Adobe Systems Inc, San Jose, CA) and Explain Everything for iPad (MorrisCooke, Wroclaw, Poland). The instructor felt that static Powerpoint slides were a hindrance to student learning of these topics and sought a more dynamic method to walk students through the evolution of concepts, ideas, and calculations. E-lectures were made interactive with the use of tablet and stylus technology so that the instructor could write, draw, and manipulate figures, pictures, equations, and calculations. This asynchronous, self-paced, digital recording allowed students to see the instructor’s method and follow the instructor’s thought-process before applying it to an embedded assessment question in the e-lecture. These assessment questions were graded to encourage student participation but given a low point value so as not to inflate overall grades in the course. Assessment questions were embedded within the e-lecture using the question pool feature in Adobe Captivate. For each given concept or question to be assessed, 5 different permutations of the assessment were generated in the question pool. When students reached the embedded assessment question in the e-lecture, 1 of the 5 questions from the pool would be generated at random for them to complete before progressing. After question completion, a pop-up box would immediately alert students as to whether they answered the question correctly. Students could navigate backwards in the e-lecture at any time before or after the question to review material to facilitate their understanding of the information tested in the assessment questions. The question pools were developed to reduce students’ exchange of answers and encourage them to complete the assessment portion of the preclass assignment independently. By using question pools, the preclass assessment data was more likely to accurately reflect students’ understanding of the material, which afforded the instructor greater reliability in using the data to make decisions about how to tailor in-class lecture time. The e-lectures were posted via Blackboard (Blackboard, Inc, Washington, DC) as a Sharable Content Object Reference Model (SCORM) package.

Viral hepatitis and clinical pharmacokinetics of aminoglycosides were chosen as the topics for this study because of students’ dissatisfaction with the brisk pace and the complexity of the material for these topics in the past. Student learning objectives for the viral hepatitis and clinical pharmacokinetics of aminoglycosides course modules are outlined in Table 2. Three e-lectures were developed for the viral hepatitis module. They focused on serology interpretation, with a brief review of epidemiology and transmission of hepatitis A, hepatitis B, and hepatitis C viruses. Three e-lectures also were developed for the clinical pharmacokinetic module. They reviewed clinical pharmacokinetic concepts such as first-order elimination and 2-compartment models; empiric dosing of aminoglycosides using population-based equations; and patient-specific, aminoglycoside dose-adjustment calculations based on serum levels. E-lecture material predominately covered application- and synthesis-level concepts. As such, embedded assessment questions within the e-lecture were also predominately application-level or synthesis-level questions. This was in contrast to the typical educational theory of a hybrid-learning model where introductory material is covered before class and advancement to application and synthesis activities occurs in class.^14,15 From previous experience, the instructor noticed that it was difficult to review higher-level concepts in a class with 70 to 75 students where some students caught on quickly and others required more time to learn a concept. As a result, the instructor chose to focus on predominately knowledge-level material in class. Examples included the spectrum of coverage and indications for aminoglycoside use and risk factors for acquiring viral hepatitis. The instructor’s impression was that most students in the class could learn knowledge-level material at a similar pace but application- and synthesis-level concepts would be most effectively taught in a self-paced environment with the use of e-lectures.

EVALUATION AND ASSESSMENT

Student performance on examination questions after implementation of the hybrid modules using advanced screencasting with embedded assessments in 2013 was compared to student performance on examinations from 2012. All examination questions were categorized into 3 categories based on Bloom’s taxonomy: knowledge, application, and synthesis in accordance with WSOP’s assessment mapping practices.

To determine if a preference for this learning platform emerged, students were asked to complete 1 survey instrument for the viral hepatitis and 1 for the clinical pharmacokinetics of aminoglycosides modules. The survey instruments consisted of items rated using a 5-point Likert scale (1=strongly disagree to 5=strongly agree) and a comments field to elaborate on strengths and weaknesses of the method. The survey instruments were completed anonymously at the end of the corresponding examination. The length of time the students spent viewing the e-lectures was reported in Blackboard and collected.

Two-sample t tests were performed to compare average scores on test questions between the 2012 and 2013 examinations. Student survey scores were compared to a score of 3 (based on a 5-point Likert scale) using 1-sample t tests for each survey question. A score of 3 indicated neutral feelings towards the e-lectures and was therefore used for comparison to determine if there was a significant preference for the hybrid design. Content analysis was performed to assess themes in the comment section of the student survey instrument. Descriptive statistics were used to describe the length of time the students spent viewing the e-lectures. Statistical analysis was conducted using Microsoft Excel.

After implementation of the viral hepatitis hybrid module in 2013, the corresponding examination included 11 multiple-choice questions, 9 of which were application- or synthesis-level questions. In 2012, prior to hybridization, the corresponding examination included 10 multiple-choice questions, 5 of which were classified as knowledge and 5 were application or synthesis. The average score on all viral hepatitis examination questions was 83% in 2013 and 75% in 2012 (p<0.001). When comparing only questions that remained the same or were very similar from year to year, the difference in scores was still significant (Table 3). The average scores on application- and synthesis-level questions were significantly higher in 2013.

Seventy-one students (93%) returned a completed survey instrument on the viral hepatitis module. Students’ perceptions of the preclass e-lectures were favorable (Table 4). The most common theme on the open-ended survey question was related to technical difficulties encountered with the e-lectures (Table 5).

All 76 students (100%) completed the preclass e-lecture assignment for the viral hepatitis module. The total duration of the e-lectures in the viral hepatitis module was 70 minutes; however, the amount of time students spent viewing the e-lectures and completing the embedded assessment questions varied (Table 6).

In 2013, postimplementation of the hybrid clinical pharmacokinetics of aminoglycosides module, the corresponding examination included 9 items: 5 multiple-choice questions and a written, 4-step calculation question. There were 2 knowledge questions, 2 application questions, and 5 synthesis questions. In 2012, the types of questions remained similar, with 5 multiple-choice questions and a long-answer question broken into 4 components. Bloom’s taxonomy classification was also similar, with 1 knowledge question, 4 application questions, and 4 synthesis questions on the 2012 examination. The average score on the examination questions was 91% in 2013 and 87% in 2012 (p<0.001). When comparing only those questions that remained similar between the 2012 and 2013 examinations, the difference remained significant (Table 3). When broken down into Bloom’s classifications, there was no significant difference in performance on knowledge-based questions from 2012 to 2013. Scores on application-based questions went down between the same period, but performance on synthesis-level questions improved significantly (Table 3).

Seventy-five students (99%) returned a completed survey instrument on the clinical pharmacokinetics module. Results of the 5-point Likert scale questions showed that students’ perceptions of the preclass e-lecture were favorable (Table 4). The most common theme on the open-ended survey question was that students liked that the e-lectures were self-paced and allowed for multiple views (Table 5).

Table 6 summarizes the results of the length of time students spent viewing the e-lecture and answering the embedded questions. The total duration of the e-lectures was 76 minutes, with additional time needed to complete the embedded assessment questions. On average, it took more than double the duration of the e-lecture to complete the preclass assignment; however, the total time taken varied greatly among students.

DISCUSSION

We hypothesized that the use of a hybrid-learning model using advanced screencasting with embedded assessments would increase the examination scores of P3 students, particularly on application- and synthesis-level questions. The hybrid-learning model improved overall question performance and performance on similar questions for both modules from 2012 to 2013. The performance on application and synthesis questions for the viral hepatitis module and synthesis questions for the clinical pharmacokinetics of aminoglycosides module improved significantly. This novel hybrid-learning model using advanced screencasting with embedded assessments allowed students to spend more time learning higher-level application- and synthesis-level concepts at their own pace than would normally be afforded in a traditional face-to-face lecture.

We also hypothesized that the Clinical Pharmacokinetics of Aminoglycoside module would benefit most from hybridization using e-lecture technology because of the calculus-based pharmacokinetic calculations it contained and the known success of other online platforms that target mathematics, such as the Khan Academy.¹⁶ Although overall scores improved, scores on knowledge- and application-level questions in the clinical pharmacokinetics of aminoglycosides module did not improve with the addition of the hybrid-learning model using e-lecture technology. This may be because of the use of the group PBL activity in 2012 and 2013. Students were essentially given a hybrid-learning environment in 2012 where they were introduced to the material in class and then given clinical cases to work on as a group in a clinical laboratory setting to turn in for grading and feedback. This highlights an important point; technology may not be needed as long as students are provided with an arena to apply the material they have learned in class and given feedback on that application activity. However, our model may be more efficient for courses that cannot accommodate a laboratory or recitation credit hour and should be considered for use in courses with in-class time constraints.

The inclusion of a viral hepatitis module using advanced screencasting and embedded assessment improved overall examination scores and scores at every level of Bloom’s taxonomy. Additionally, the instructor shifted the proportion of examination questions towards more advanced-level questions in 2013 compared to 2012. Despite testing with arguably more difficult questions in 2013, test scores still improved, most likely because of the use of advanced screencasting with embedded assessments.

There was variability in the average length of time that students spent watching the e-lectures. This variability was most likely attributed to differences in the speed at which students learned new concepts, particularly those taught at the application and synthesis levels. The e-lectures allowed students who learned more slowly to move through the material at their own pace instead of being forced to move at the pace of the instructor lecturing in the classroom setting. This was supported by the results of the open-ended survey questions. The theme repeated most by students was how beneficial it was for the e-lectures to be self-paced and allow for multiple views.

Students also showed a preference for e-lectures as preclass assignments over traditional reading assignments, which is consistent with the Millennial generation’s heavy reliance on technology to support their academic endeavors.³ This study used e-lectures that ranged from 11 minutes to 45 minutes in length. Based on the themes from survey instrument data, students wanted to continue the use of e-lectures, but the e-lectures should be shorter in length. This is consistent with the smaller proportion of information that Millennials digest when searching for information on platforms like YouTube and Facebook.¹⁷ E-lectures may need to be tailored to be a larger series of short clips like those used by these technologies rather than a more traditional length of 30 to 40 minutes.

The greatest barrier encountered for the instructor was the technical difficulties in learning how to use the software effectively. It took a week to create the first series of e-lectures and incorporate them into the learning management system. None of the authors, who were entirely responsible for e-lecture creation and integration onto the learning management system, had any formal technology training. If an instructional technologist familiar with these technologies had been available, this time could have been dramatically reduced. The second series of e-lectures was created within a day after the instructor had learned the intricacies of the technology. One of the themes that emerged from the student survey data was technical difficulty with the e-lectures. Although we had previewed the e-lectures and made sure they played correctly, some students experienced frames freezing or problems with the embedded questions. For those students who experienced these difficulties, the instructor made time for them to play the e-lecture and answer the embedded assessment questions during office hours, which took another 2 to 3 hours of the instructor’s time.

This study had several strengths. To our knowledge, the use of advanced screencasting with embedded assessments was a novel method to use in conjunction with a hybrid-learning model in pharmacy education. Additionally, the educational method used in this study proved successful despite being somewhat unconventional. We chose to cover higher-level application and synthesis concepts in the preclass portion to allow for self-pacing with these more difficult concepts rather than cover introductory, knowledge-based material. This model of learning also gave the instructor more flexibility. By using question pools to increase reliability, the instructor was able to rely on the preclass assessment data to benchmark students’ understanding of several difficult concepts prior to class. This allowed for shortening of coverage of some material and increased coverage of other material based on the students’ needs.

There were some inherent limitations in this study. A historical control group from 2012 was used for comparison, which did not allow for matching baseline characteristics between groups. Also, the scope of this intervention was small, encompassing only 2 modules in a larger course. The applicability of these methods on a larger scale remains to be tested. Another weakness was the difference in the use of a group PBL activity between the Clinical Pharmacokinetics of Aminoglycosides module and the Viral Hepatitis module, creating some non-uniformity between the modules. Because of scheduling constraints, the Viral Hepatitis module did not include a group PBL activity in 2012 or 2013. Additionally, a different instructor taught the Clinical Pharmacokinetics of Aminoglycosides module in 2012 than 2013. The instructor who taught this module in 2013 observed the lecture in 2012 and kept overall content similar. The same instructor taught the Viral Hepatitis module in 2012 and 2013. Finally, the time students spent viewing the e-lectures could have been inflated. The time collected could have included time that the students were not physically sitting in front of the computer viewing the e-lectures because the lecture window would remain active as long as they were signed onto Blackboard.

SUMMARY

The addition of e-lectures using advanced screencasting with embedded asssessments to create 2 pathophysiology and therapeutics modules improved overall test performance for P3 students, particularly on higher-level Bloom’s taxonomy questions. Additionally, e-lectures allowed students to learn at varying paces and students preferred e-lectures over traditional preclass reading assignments.