Evaluation of a Flipped Drug Literature Evaluation Course

INTRODUCTION

Active-learning pedagogy in the classroom is highlighted in the 2016 Accreditation Council for Pharmacy Education (ACPE) Standards.¹ Specifically, the standards state the curriculum should actively engage learners, promote student responsibility of self-directed learning, and foster collaborative learning. The flipped model is one type of active-learning pedagogy that meets the 2016 ACPE Standards.

The term “flipped” was introduced in 2006, although the strategies behind the flipped classroom have been used for years.² Flipping a classroom essentially involves simple transfer of knowledge out of scheduled class time. Material can be delivered in a variety of ways such as videos, podcasts, readings, or websites and must be completed by the student before class. To ensure completion of materials, an assessment can be administered before or at the beginning of class. Homework is transformed into an in-class activity that allows the instructor to assess learning in a formative manner. Activities are typically completed in groups. The instructor interacts with students on an individual level, employs more active-learning techniques such as think-pair-share and team-based learning methodologies, and models expert thinking related to the subject material. The flipped model employs a constructivist approach to learning.³ In this approach, students learn by experiencing and are responsible for building their knowledge. Students build knowledge prior to class and apply what they have learned during class. Such an approach aligns strongly with principles of life-long learning. The flipped classroom may also fit better with millennial learners because of their preference for active learning.⁴

There are a number of reasons that make flipping a literature evaluation course an attractive approach. The flipped classroom employs methods that could reduce anxiety and fear typically encountered in courses that involve statistics. Fear and anxiety related to statistics courses can be broken down into six main themes: interpretation of statistics, test anxiety, computation anxiety, worth of statistics, fear of the instructor, and fear of asking for help.⁵ Test, interpretation, and computation anxiety are related to performing statistical tests and interpreting results. Fear of the instructor and asking for help can result in students not obtaining the help they need. “Worth of statistics” is related to students thinking that the material in the course is not applicable to the real world. The flipped classroom addresses these concerns by showing how the material is applicable to real world practice through activities, allowing for closer interactions with the instructor and giving students opportunities to practice skills prior to examinations. An additional benefit of the flipped classroom is that students are able to access material at any time outside of class. Lastly, the flipped model fits all recommendations from the Guidelines for Assessment and Instruction in Statistics Education (GAISE) from the American Statistical Association.⁶ These recommendations emphasize using active learning, real data, technology and assessments, improving conceptual understanding, and developing critical-thinking skills.

Studies in academic pharmacy literature show students perceive the flipped classroom model as a way of enhancing learning, increasing understanding of foundational content, increasing participation, increasing class attendance, and improving assessment scores.^7,8 Studies evaluating the flipped classroom have been conducted in a wide variety of health care education programs including radiology, nutrition, nursing, public health, and medicine.^7-13 Identified studies report mostly positive results, although some studies do not report quantitative outcomes.

We decided to flip the classroom after observing student inattention during lectures, difficulty completing homework assignments, and anxiety about the course. Therefore, the purpose of this study was to evaluate the effect of flipping a drug literature evaluation course on examination performance and student perceptions in first-year pharmacy students.

DESIGN

Literature Evaluation is a required 2-credit hour course for students enrolled in the pharmacy program at Wayne State University (WSU) Eugene Applebaum College of Pharmacy and Health Sciences. The purpose of the course is to enhance students’ ability to evaluate drug information with a strong focus on medical literature evaluation. The course is strategically placed in the second semester of the curriculum to allow for reinforcement in other courses. The course is sequenced after a pharmacy calculations course that includes descriptive statistics. Course content is mapped to programmatic ability-based outcomes.

Planning for the flipped classroom started one year in advance of the flipped course. A thorough review of the literature as well as consultation with faculty members at other colleges and schools of pharmacy informed the planning process. Activities were chosen intentionally to meet course outcomes. Activities employed evidence-based, active-learning strategies.

The course was coordinated by the primary author in both the 2013 and 2014 years. There were no changes in content, with small changes in sequence. The course content is displayed in Table 1. The flipped classroom model included two major changes to the course. The first change was movement of content from in-class lectures to videos to be viewed prior to class delivered via YouTube (Google, Mountain View, CA). The second major change was the movement of homework to in-class activities. Homework and activities are listed in Table 1. In-class activities were designed for 12 groups of approximately eight students. Groups were assigned to ensure equal distribution of student grade point average (GPA) across all groups, and groups remained the same throughout the semester. To accommodate the change in course structure, the location of the class was moved. The course was held in a lecture hall in 2013, but located in an active-learning classroom in 2014. The locations are within a half-mile of each other. The required text in both years was Drug Information: A Guide for Pharmacists.¹⁴ Reading suggestions were provided in both 2013 and 2014 but not required.

The course grade in 2013 was composed of examinations (55%), homework (20%), journal club (15%), quizzes (7.5%), and participation (2.5%). The course grade in 2014 was composed of examinations (45%), team-based activities (32.5%), quizzes (15%), and peer evaluations (7.5%). Quiz weight was increased from 7.5% to 15% in 2014 to emphasize the importance of preparation prior to class. Attendance was mandatory for journal club days in 2013 and all days in 2014. Excused absences required students to complete an extra journal article evaluation.

Examination format did not change between the two years. Students received two articles to read and critically evaluate using a journal club evaluation form one week prior to examinations. Articles were carefully selected to highlight concepts taught during the semester. Half of the examination questions pertained to these articles. The other half of the questions tested content covered in lectures or videos. Examination questions were written at the application, analysis, and evaluation levels of Bloom’s Taxonomy.^15,16

Quizzes from both years focused on content from the course previously delivered through lectures (2013) or videos (2014). Quizzes were taken individually by students both years and were open book with a time limit of 30 minutes for a maximum of 10 questions. Quiz questions were written at the knowledge and comprehension level of Bloom’s Taxonomy. Quizzes were administered electronically in class in 2013 and out of class in 2014 through Blackboard (Blackboard, Inc. Washington, DC). Quizzes were moved out of class to save time for more in-class activities and were allowed to be taken 24 hours prior to class. Quiz question and answer order was randomized both years.

Videos were developed with Camtasia (TechSmith, Okemos, MI) using narrated screen capture. In preparation for recording, faculty members prepared a script and practiced their presentations. They used Powerpoint or Word to display information during recording. Internet sites were occasionally used to present information; for example, a PubMed.gov search was demonstrated. Videos were embedded into the Blackboard course website and were provided as video links through course announcements.

Faculty members also prepared the in-class activity. Activities reinforced the journal club evaluation form, evaluation of statistical tests, PubMed searches, and the use of various drug information resources. Activities frequently utilized the Immediate Feedback Assessment Technique tool (Epstein Educational Enterprises, Cincinnati, OH). This tool is a scratch-off card that resembles a lottery ticket. Students scratch off the answer they believe is correct and then view a symbol indicating a correct answer or a lack of a symbol indicating the choice is incorrect. They can proceed through the answer choices until they identify the correct answer. The study was approved by the WSU Institutional Review Board.

EVALUATION AND ASSESSMENT

This project employed a quasi-experimental design to evaluate flipping a literature evaluation course. The primary assessment measure was comparison of examination scores between the 2013 and 2014 years. Secondary comparative assessments included evaluation of quiz scores and overall class Student Evaluation of Teaching Scores (a university instrument). Secondary assessment measures for the year the class was flipped included evaluation of student attitudes towards the flipped classroom and evaluation of factors that were associated with greater percentage of preclass videos viewed. Descriptive statistics were used for analyzing postcourse survey attitudinal data. Reliability of questions organized by constructs were analyzed using Cronbach’s alpha. Continuous primary and secondary outcomes were analyzed using the Student’s t-test. Multiple linear regression was performed to adjust for factors previously shown in the literature to affect examination scores.

Ninety-nine and 94 students completed the course in 2013 and 2014, respectively. These students had similar first-semester GPAs (3.06 vs 3.07, p=0.85), but PCAT composite scores differed (73.9 vs 80.8, p<0.001) between the 2013 and 2014 years, respectively. Average examination scores increased from 2013 to 2014 (75.6% to 86.1%, p<0.001). Linear regression controlling for GPA and PCAT composite scores further demonstrated that average examination scores were significantly increased (9.5%, 95% CI 7.4-11.5%, p<0.001) in the flipped classroom. Grade point average and PCAT composite scores were significant predictors of average examination scores (p<0.001, Table 2). Linear regression modeling explained 51.4% of the variance in the average examination scores. Examination 1 scores increased from 2013 to 2014 from 77% to 86.6% (p<0.001) and examination 2 scores increased from 74.1% to 85.6% (p<0.001). Quiz scores also increased from 2013 to 2014 from 66.9% to 96.2% (p<0.001), although the quizzes were moved outside of class.

Eighty-two of 94 (87.2%) students completed the postcourse survey in 2014, which was designed to test six constructs: importance of the course, video quality, flipped classroom quality, activity quality, classroom setting, and group dynamics (Table 3). Three questions were written to assess each construct. For each series of three questions, two were written in a positive direction and one question was written in a negative manner. The survey was developed by the authors and validated by a focus group of students. The survey was administered electronically through E-Value (Advanced Informatics, Minneapolis, MN) a week after the end of the course. Participation in the survey was anonymous, optional, and had no bearing on course grades.

After conducting initial reliability analysis using Cronbach alpha, negatively written questions were excluded from analysis because of a large effect of internal consistency. This occurred because seven students filled out all boxes on the survey as a specific category (ie, all strongly disagree). Reliability using Cronbach alpha after negative questions were excluded is listed for each construct: importance of course (0.94), video quality (0.72), flipped quality (0.78), activity quality (0.56), classroom quality (0.71), and group quality (0.73). Seventy-eight of 82 students provided feedback on the open-ended question at the end of the survey. This open-ended question was analyzed using conventional content analysis methodology.¹⁷ Themes identified on the open-ended question indicated that the majority of students preferred the flipped classroom to lecture (35 vs 16), eight students thought mini-lectures should be added, seven students disliked the walk to the active-learning classroom (∼1/2 mile from the pharmacy school), five students preferred videos made by particular faculty members, three students thought peer evaluations should be anonymous, two students thought the in-class activities could be improved, two students thought the course could be moved later in the curriculum, two students thought the videos were too long, and one student thought there was too much work involved with the course.

Comparison of Student Evaluation of Teaching scores focused on three questions: the overall course quality, the amount learned in the course, and the quality of the instructors teaching. Each item can be scored 1 to 5, which corresponds with a response indicating poor, fair, good, very good, or excellent. Student average class evaluation scores did not change significantly between the 2013 and 2014 years (8.5 vs 8.8, p>0.1; maximum score 15).

Fifty-eight videos were created for the 2014 course. Deidentified YouTube analytical tools provided aggregate data describing the time spent watching each video. The average video length was 13 minutes and the average view duration was seven minutes. Each video had an average of 93.6 views. The average percentage of videos viewed (defined as time spent viewing the video divided by the length of the video) was 61.9%. If students repeated watching portions of the video, this increased their average percent viewed by YouTube definition. Video view percentage was significantly correlated (r=0.67, p<0.01) with the length of the video (Figure 1). Average view percentage was not affected by the use of Powerpoint or Word to deliver information (62.1 vs 61.3%, p=0.86).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Correlation between Video Length and Percent Video Watched.

DISCUSSION

This is the first study describing the implementation of the flipped classroom model in a first-year drug literature evaluation course at a college of pharmacy. We saw improvements in both examination and quiz scores using the flipped classroom model. We hypothesize that the increase in examination scores reflects a better match between the level of application materials and examination questions. We hypothesize that quiz scores increased for a number of reasons: shorter videos allowed students to maintain focus, confusing portions of the video could be revisited, and quizzes were taken outside of class. We attempted to prevent sharing of quiz answers by instructing students to take the quizzes individually, randomizing question order and choices, limiting quiz time to 30 minutes, and limiting the quiz availability within 24 hours of the class period. Unfortunately, high quiz scores prevented identification of confusing concepts from the YouTube videos. We plan to address this in the future by having a “muddiest point” session during the first five minutes of class.

Students generally recognized the importance of the course and agreed that the in-class activities and working in groups improved their learning. Students also thought the setting of the classroom was conducive to learning. Approximately 60% of students preferred the flipped classroom model to the traditional lecture model. This finding corroborates comments received on the open-ended question at the end of the survey. In response to student requests for “mini-lectures,” we will incorporate a 5-minute overview of “take home points” at the end of class. This will highlight key points from the activities and videos and address a variety of learning preferences. Of note, mini-lectures (1-2 minutes) were employed in 2014 during activities to clarify points of confusion. The number of mini-lectures varied based on the activity. The postcourse survey also suggested some videos could be shortened. This finding was reinforced by YouTube data, which indicated that students watched a higher percentage of shorter videos. Future videos will not exceed 10 minutes.

Our findings are similar to those of a statistics course in which the flipped model was employed.¹⁸ This statistics course served a variety of health care and nonhealth care majors and enrolled approximately 25 students. The course also saw an increase in examination scores by 6.73%. The authors were unsure if these results could be replicated in a large classroom; however, we showed that they could be replicated. In contrast, a flipped classroom veterinarian skill course that had a research skills component saw a decrease in examination scores from 21.4 to 20.3.¹⁹ However, only 25% of the veterinarian skills class was dedicated to research skills, with the remaining dedicated to financial planning, communication, and career planning. This is likely an insufficient allotment of time for research skills and may not have been enough time to positively change examination scores.

Studies in pharmacy students demonstrate a range of impact on examination scores. A flipped classroom renal pharmacy module course reported a 3.9% increase in scores on the final examination.⁸ Scores also increased from 80 to 82.5% on a pharmaceutics course examination.⁷ A 3-class pilot study in a cardiac arrhythmia course saw significant improvements in examination scores in pharmacology (56.8 to 89.6%) and therapeutics (73.7 to 89.2%), but not basic science (84.1 to 88.3%).²⁰ Finally, one study of a flipped classroom in a graduate classroom course covering cardiovascular, renal, and respiratory physiology observed a 12% increase in examination scores.²¹ The level of questions was not highlighted in these reports, but we hypothesize if examinations are written at an application level, an improvement in performance may be seen. Our study adds to the evidence in the pharmacy literature supporting the flipped classroom and describes the first experience in a drug literature evaluation course. Unfortunately, few studies have evaluated video quality within the flipped classroom model. Although only mentioned subjectively, one study in a nursing PhD statistics course found students preferred 10-15 minute videos compared to longer videos.⁹ Our study supports this suggestion as we found the length of the video had a strong correlation to the percentage of video students viewed.

Improved performance on examinations in a flipped classroom model does not necessarily mean students are more satisfied with the course. A quasi-experimental study of nursing students in an adult therapeutics course demonstrated higher grades, but less satisfaction in a flipped classroom group compared to a lecture or a lecture plus video group.¹⁰ Another study of a flipped classroom in a self-care course at a school of pharmacy saw in improvement in the distribution of course grades but a decrease in student satisfaction.²² We did not see student evaluations change between the two years, and this was the first time the flipped model was introduced at our college. The lack of change may have occurred because the benefits of the flipped classroom were explained in depth to students at the start of the course. Alternatively, the lack of change in student satisfaction may reflect the inherent lack of interest students have in the topic (as opposed to a therapeutics course).

Prior studies have expressed concerns regarding implementation of the flipped classroom and costs associated with the requirement for additional teaching resources (faculty members in the classroom, preparation time, and technology).²³ We did not employ teaching assistants but instead used a combination of voluntary postgraduate pharmacy residents and adjunct faculty members to help facilitate classroom activities. This may be a more cost effective strategy in a resource-limited environment. A significant amount of faculty time and effort was invested in preparing videos and activities during 2014, which included training on use of the technology. Many articles regarding the flipped classroom mention a large increase in time spent the first year of implementation. We did not quantify time spent, but subjectively found a large increase in time spent on the course during 2014 in contrast to 2013. The time investment should decrease in future course offerings because videos may be edited and reused. Of note, it is important to have good Internet connections for in-class activities to go smoothly.

For those planning on using videos as a delivery mechanism for preclass content, we believe that allowing students to access videos on YouTube increased the frequency of viewing. Students could access lectures on mobile devices, which provided flexibility for student viewing. Video length was inversely correlated with percent of videos viewed. Although videos were used in this course, there are a variety of opportunities for knowledge acquisition outside of the classroom to scaffold in-class activities. Class activities were designed to make students apply their knowledge at higher levels of Bloom’s Taxonomy while using scientific journal articles. Observing how students were performing during activities allowed us to reinforce confusing points and guide students by modeling expert thinking.

Additional improvements will be made to this course in future years. While we felt group assignment by GPA worked well, prior experience with statistics will be added. This will distribute knowledge resources more equally between groups. Adjunct faculty and residents training will be provided in a more systematic manner. We will provide a guide for both facilitating and providing mini-lectures in the classroom based on student behaviors and points of confusion. Reflection activities will assess if mini-lectures are successful at addressing points of confusion. Lastly, we plan to have a minimum of two facilitators in the room in the next offering so we can respond to student questions promptly.

Our study has some limitations. Although we controlled for GPA and PCAT composite score, changes in examination scores may have been influenced by other unmeasured differences between the two classes. Furthermore, these results are reflective of one course with a limited sample size, which may decrease the external validity of our findings. Quiz score improvement was largely attributed to moving the quizzes outside of class, and we hypothesize the change in scores will be hard to replicate. Also, our level of agreement as measured by Cronbach alpha could have been higher for some constructs on the postsurvey. Including more questions per construct may have given better reliability data for the postcourse survey. Also, we did not systematically evaluate feedback from adjunct faculty members or pharmacy residents who participated in the course. Future studies may want to systematically evaluate feedback from these stakeholders. Lastly, flipping the entire class involved a significant time investment along with a number of changes made to the course structure. This makes our structure harder to replicate, although we comprehensively described the effort that went into designing the course.

SUMMARY

Intentionally flipping a classroom was successful in a drug literature evaluation course as measured by grades and student perceptions. We believe the flipped model is an excellent fit for drug literature content and courses that want to incorporate more active learning. Furthermore, our study mirrors findings in the pharmacy literature of the positive impact of the flipped classroom. Faculty members should plan early if an entire course is to be flipped and recognize that the first year will require increased preparation time. We believe the outcomes are worth the increased time investment, as the flipped classroom can prepare students for life-long learning.