Evaluation of a Flipped Examination Model Implemented in a Final-Year Undergraduate Pharmacotherapeutics Course

INTRODUCTION

Pharmacy educators advocate that in-depth learning and knowledge retention require the employment of novel pedagogic approaches that are more interactive and learner-centered than traditional approaches.^1-3 “Flipped classrooms” are a teaching innovation that have been increasingly embraced.^4-6 In “flipped learning,” the traditional teaching model is rotated, and classroom learners have already engaged with the study material prior to class via online lectures and prework activities. Thus, students’ time in class can be spent engaging in meaningful discussions, problem solving, and collaborating with other students.⁴ Another educational trend is the “flipped exam” or “flipped assessment,” where students construct their own well-reasoned examinations or assessments using educational tools provided to them. The purpose of flipped examination is to determine not only what students know, but to encourage them to think in novel ways that go far beyond just rehearsing memorized material. In a flipped examination, having students develop test questions serves as a learning experience, but having them actually study for and take the test they and their peers created can lead to a more profound and deeper understanding of the subject material.⁵

Research on flipped instructional models suggest that flipped examinations could enrich and enhance the learning experience. Flipped assessments provide students with opportunities to construct, articulate, and defend logical responses to complex questions or problems; provide immediate feedback to the students and instructor regarding knowledge and understanding of subject content, and offer the opportunity to use this feedback to improve future performance.^6,7 Flipped examinations also are considered more student-centered compared with traditional examinations. Some research suggests that traditional examinations place an emphasis on the numeric aspects of learning, which fosters a fear of failure, increases anxiety, undercuts interest, reduces intrinsic motivation, heightens competitiveness, and depresses creativity.^6-9 In contrast, the flipped examination model addresses these limitations and offers an occasion to determine how well students perform when they are given course resources and an opportunity to collaborate with and learn from their peers. Further, in a constructivist paradigm, the flipped examination allows students to actually focus on the process of learning rather than on the correct answer.^6-10 Educational intervention trials indicate that students performed better when the flipped examination model was used. For example, in a trial of this technique among postbaccalaureate students at Wayne State University School of Medicine, the students’ performance in their advanced cardiopulmonary physiology course was significantly better when flipped examinations were used versus traditional examinations.⁶ Various studies assessing the impact of flipped educational and assessment models in science and pharmacy undergraduate courses worldwide have found similar results.^4,7,10-13

Despite growing interest in the flipped classroom approach and the promising results obtained in terms of students’ performance and satisfaction in the pharmacy curriculum,^4,11-13 this assessment model has not been tested in pharmacy education in Australia. However, in 2016 the Pharmaceutical Society of Australia, which is a national professional pharmacy organization providing extensive educational programs and professional development activities for Australian pharmacists working in various pharmacy sectors, incorporated the flipped classroom approach as an interactive educational tool in some of its pharmacy certificates and diplomas.¹⁴ An Australian study published in 2015 elicited undergraduate nursing students' responses to a flipped classroom approach in pharmacology lectures and evaluated the impact of this approach on their understanding of drug science and its application to clinical practice.¹⁵ The authors concluded that the flipped classroom model provided students with a greater understanding of the subject and enhanced their critical-thinking skills.¹⁵ This study, however, was one of very few studies addressing the flipped instructional approach for health care professional students in Australia.^16,17

The University of Sydney is one of Australia’s largest providers of undergraduate and postgraduate pharmacy training. A trial of the flipped classroom model being conducted at The University of Sydney at the time of this writing was funded by an Educational Innovation grant from the Faculty of Science, However, this model was proposed only for struggling students who were at risk of failing. The aim of our study, which also was supported by an Educational Innovation grant, was to test pharmacy students’ performance and evaluate their experience in a flipped examination versus a traditional examination for an advanced clinical course in the final-year pharmacy undergraduate program at The University of Sydney. The researchers hypothesized that students would perform better than students in previous years, and as active designers of their assessment, they would engage with the subject material at a deeper scholastic level.

METHODS

Pharmacotherapeutics is a compulsory six-credit unit of study offered to fourth-year bachelor of pharmacy (BPharm) students during the first semester of their final year. In Australia, the BPharm degree is a four-year degree. Graduates may then register for an internship year, the completion of which renders them eligible for pharmacy registration. The fourth-year pharmacotherapeutics course focuses on developing students’ understanding of the use of medicines and related appropriate health measures in special patient populations, including pediatrics, geriatrics, pregnant/breastfeeding women, patients living with a disability, and patients with cultural and spiritual beliefs or practices that affect medication use. The course draws upon concepts in clinical pharmacy, pharmacokinetics, and clinical practice to help students explore information sources for drug use and integrate knowledge of clinical indices, laboratory data, medication use history, and demographic information to optimize drug therapy in response to the needs of individual patients. This unit of study aims at providing pharmacy students with hands-on practice in the provision of patient-specific medicine use education and exploring key issues concerning the maintenance of vigilance for medicines use specific to certain population groups. The Pharmacotherapeutics course was taught via lectures followed by three-hour small-group workshops (referred to as laboratories or “labs” in other countries).

For the purpose of the workshops, the entire class was divided into 12 smaller groups of 18-20 students each, and the groups were labeled Group 1 through Group 12. Thus, every Pharmacotherapeutics workshop was repeated 12 times each week with different facilitators overseeing different groups of students at different times. The students usually were allowed to choose which of the 12 groups they wanted to be in based on their preference of facilitator and peers. There were seven workshops conducted for this unit of study.

In the workshops, students were further divided among five stations. The students were allowed to choose their station as long as each group consisted of three or four students. Group members for each station were fixed for the entire semester. For every workshop, each station was allocated a task to be completed prior to attending the workshop (ie, prework). The prework task consisted of a PowerPoint template with case conundrums, clinical queries, and discussion points outlined for students to prepare and complete. These tasks were supported by a set of references and research articles. For an umbrella topic such as pediatrics, each of the five stations in the group was assigned a topic entirely different from those of the other four stations. Some examples of niche topics within the pediatrics workshop included common childhood ailments (eczema, diaper rash, coughs/cold, asthma, diarrhea/constipation, etc.), drug dosing (pharmacokinetics and pharmacodynamics), drug formulations, adverse drug reactions and medication safety, and infant and child nutrition. During the workshop, each station presented the work their group had completed. The facilitator reviewed and assessed the content of the presentations and suggested improvements and corrections if needed. The assessment for this unit of study included submission of a clinical portfolio (20%), workshop participation and presentations (35%), completion of online modules (15%), and completion of a traditional final examination (30%).

For the purpose of this study, in 2016, faculty members proposed that a mid-semester flipped examination consisting of student-generated multiple-choice examination questions be used in place of the traditional final examination. The midsemester flipped examination was scheduled during week 8 of the semester, ie, after the completion of five workshops (two on pediatrics, two on geriatrics, and one on pregnancy and lactation). To conduct this, during the first workshop of the semester, all students attended a three-hour training session on how to write good multiple-choice questions (MCQs). During the workshop, the characteristics of good quality MCQs were described and examples were provided. All students practiced writing MCQs in groups and were given feedback about the quality of their generated questions. For the remaining workshops, in addition to completing the prework, each station within each of the 12 weekly workshops held was asked to devise one MCQ, along with the correct answer and the rationale behind the proposed question. At the end of each station’s presentation, the MCQ was presented to the class and reviewed by the facilitator. Students who submitted a poorly-written question received feedback for improvement and were asked to revise the question and re-submit. All students in each workshop were told to make sure they understood the answers to the provided questions. If they did not, the facilitator and the presenting station members would discuss the question and go through the thought process for arriving at the correct answer. At the end of each workshop, the revised MCQs, the correct answers, and the thought process behind the correct answers were submitted by station members from each of the 12 groups to the course coordinator. After carefully reviewing all the questions in terms of meaningful stems, uniform options (no all/none of the above responses were allowed), logical reasoning, clinical accuracy, and aesthetics (grammar and spelling), the course coordinator posted the finalized questions on the online course management system as the Student-Created Exam Bank. Posting the questions online was undertaken using Respondus, version 4.0 (Respondus, Inc., Redmond, WA), the university’s e-learning platform.

During the first week of the semester, a research assistant presented all final-year pharmacy students with information on this study. A study questionnaire entitled “Evaluating a ‘Do It Yourself (DIY) Exam’ in a Final-Year Study Unit” was used to assess the students’ learning approaches. The questionnaire consisted of two main sections: a demographic section comprising questions related to gender, age, country of birth, work status, and history of participation in a DIY examination/assessment; and a section addressing student perceptions of this novel teaching and learning strategy. The second section included the 20 items from the Biggs revised 2-factor Study Process Questionnaire (R-SPQ-2F), which measures students’ deep vs surface levels of engagement in the learning process.^9,18 As prior research into the reliability and validity of the Biggs’ R-SPQ-2F was fairly extensive, we considered this validated instrument a reasonable tool to investigate students’ approaches to learning.^18-21 In order to evaluate the impact of the flipped examination model and assess any possible changes in the students’ attitudes and perceptions of the learning process, the same questionnaire was administered before and after the flipped examination was administered. The pre-examination questionnaire was administered to the students by a research assistant at the beginning of the first workshop and the post-examination questionnaire was administered during the last workshop of the semester. Students were reminded that their participation in the study was voluntary, confidential, and independent of course participation and assessment grades. Students then were asked to complete the questionnaire individually. Submission of a completed questionnaire was considered to be the student’s consent to participate in the study. Data from the completed questionnaires was collated and analyzed using SPSS Statistics, version 24.0 (IBM, Armonk, NY, United States). Student perception scores (total study process perception scores) from the 20-item questionnaire section were calculated and the average score was presented as mean (SD). The Biggs’ R-SPQ-2F consists of two scales, the Deep Approach (DA) scale (10 items) and the Surface Approach (SA) scale (10 items). Scale items were rated using a 5-point Likert-type scale, ranging from never or only rarely (1 point) to always or almost always (5 points). According to the scoring system provided by Biggs, total DA and SA scores were calculated as the sum of all DA scale items and SA scale items, respectively.¹⁸ The paired samples t test was used to compare students’ total study process perception scores and total DA and SA scores before and after the flipped examination.

To further gauge students’ feedback and insights into the flipped examination model, focus groups of 10 students each were conducted with students who volunteered to participate. While the flipped examination was embedded in the unit of study, participation in the focus group discussions and completion of the study questionnaire were voluntary activities. The focus groups were facilitated by the project research assistant during the last week of the semester, and neither the course coordinator nor the workshop facilitators attended any of the focus group sessions to allow students to share their feedback to ensure that students felt free to state their opinions and observations.

The focus group discussions were guided by a semi-structured topic guide (Table 3). The topic guide was developed based on proposed a-priori themes derived from the aims and objectives of the study and the available literature on the flipped classroom model. Major proposed a-priori themes included improved examination grades, reduced examination-related stress with the flipped examination model, potential active involvement of the students in the course material to develop good-quality MCQs, and a possible decreased motivation to study non-examinable topics. The semi-structured topic guide addressed the students’ experiences with writing their own assessment questions as compared to a traditional MCQ examination written by a course instructor, the impact of the flipped examination on their learning experience in this course, and their suggestions for improving the flipped model for the purpose of this course. All focus group sessions were digitally recorded using a handheld digital voice recorder and then transcribed verbatim by an independent professional transcriber. Transcribed data were systemically coded and analyzed using NVivo 11 software (QSR international Pty Ltd, Melbourne, Victoria, Australia). Thematic content analysis was conducted in line with the framework approach, which is a qualitative analytical approach commonly used in medical and health care research. This approach was used because it builds upon proposed a-priori themes while allowing data to be managed systematically and explored thoroughly to produce transparent results that enhance the rigor of the analytical process and the credibility of the findings.²² The framework approach involves familiarization of the emerging themes and key ideas; identifying a thematic framework by drawing on the a-priori issues and questions derived from the aims and objectives of the study, issues raised by the participants, and subjects that recur in the data; indexing by annotating the transcripts with numerical codes; charting by rearranging the data according to the appropriate part of the thematic framework to which they relate; and mapping and interpretation of the dataset to find associations, conceptualize themes, and provide explanations for the findings.²³ Two independent researchers validated the themes by comparing randomly selected sessions.

As an exercise of interest, the mean total examination grade for the flipped examination cohort was compared with the mean course grade for the previous year’s class. The course content was relatively the same in 2015 despite that the cohort of students and the examination questions were different for each year. Feedback and student responses on the compulsory survey administered by the university to all students were sought. The survey was administered online at the end of the course, as per university procedures. The study was approved by the Human Research Ethics Committee at The University of Sydney.

RESULTS

Two hundred thirty-three final-year pharmacy students were enrolled in the Pharmacotherapeutics course during Semester 1 of 2016. The pre-examination questionnaire was completed by 219 (98%) students, and the post-examination questionnaire was completed by 216 (97%) students. Table 1 describes the general demographic characteristics of the participants. Before taking the flipped examination, the students’ mean total study process perception score was 2.8 (0.4). After the flipped examination, the mean total study process perception score was 3.1 (0.5). The total scores are presented and compared in Table 2. Changes in mean total study process scores at both the deep and surface levels of learning were significant (p<.05).

Data saturation was obtained after three focus group sessions had been conducted. In all, 30 students participated in the focus groups. The duration of the focus group discussions ranged from 40 to 60 minutes, depending on the issues raised by the participating students. Analysis of the focus group transcripts revealed three major themes and 11 subthemes. The generated themes and subthemes are presented in Tables 4 and 5 along with corresponding participant quotes to illustrate each theme’s content.

The first emerging theme was the benefits of the flipped examination learning experience. Five subthemes emerged under this theme: actively engaging in the learning material, acquired skills in writing and evaluating examination questions, relaxing examination setting, improved assessment marks, and greater appreciation of academic lecturers’ roles. Most of the participants found the overall educational experience utilizing the flipped examination model enjoyable and satisfying. The students reported that the flipped examination model facilitated a less-stressful learning environment and improved their learning satisfaction, knowledge, and skills beyond the scope of the topics covered in the intended course. All the students reported that engaging thoroughly in the various topics covered in the workshop presentations was indispensable to overcoming the challenge of writing good examination questions. They all highlighted that their learning was self-directed and more consistent as compared to other traditional learning experiences. Although initially expected to be easy, students found the process of designing examination questions to be challenging and highly competitive. In an attempt to propose a well-written MCQ, the students reported investing greater effort and spending a longer amount of time analyzing the subject material as the course progressed. Some students reported that, besides their desire to get a high grade on the flipped examination, they wanted to challenge themselves and their peers to complete the examination questions within a short timeframe. Additionally, most of the students expressed their gratitude to all faculty lecturers for investing a significant amount of time and effort to develop well-written examinations to incorporate intended learning objectives.

The second emerging theme comprised the drawbacks of the flipped examination learning experience. Three subthemes were developed under this theme: decreased motivation to learn from non-examinable material, learning-related trust issues, and perceived inconsistencies in the content and quality of the MCQs. While the overall flipped examination experience was pleasant, certain drawbacks were highlighted by some participants. For instance, the majority of the students reported being motivated to understand and answer the questions provided in the Student-Created Exam Bank and to relate them to the corresponding workshops. As the examination questions were developed from the workshop presentations, motivation was very limited in terms of grasping a broader understanding of the subjects covered in the workshop lectures. Trust was another issue raised by the students. As the five stations covered different activities within each umbrella topic during the workshops, some students were hesitant to trust their peers from other stations to develop MCQs that were relevant to significant learning aspects. While all the students attended the same lecture on how to write sound MCQ examination questions, some students perceived inconsistencies in the quality and content of some questions with respect to the corresponding presentations. They related the latter to having different viewpoints on what is important from a learning perspective, as well as to different facilitators reviewing the proposed questions.

The third theme that we identified related to the suggested improvements for the flipped examination experience. Three subthemes emerged under this theme: the addition of a “non-flipped” examination element, timing of the flipped examination, and format of the flipped examination. Most of the students highlighted that the introduction of a flipped examination to a major course in the pharmacy curriculum was innovative and interesting. At the same time, most students perceived some limitations to flipped examinations that should be addressed to improve overall learning outcomes. For example, the majority of students indicated the significance of adding a traditional final examination for this course or having a single examination that encompassed both the flipped and traditional approaches, where some questions were written by the students and other questions were written by the lecturers or course coordinator. Students also suggested that the examination should cover both the workshop presentations and the lecture notes. Students anticipated that the addition of a non-flipped element could assist them in grasping a better understanding of every aspect of the course and could facilitate their retention and future application of the learned topics. A few students also pointed out that if only a flipped examination were to be used, then providing the questions without the correct answers in the examination bank might encourage even better learning and understanding as students would have to read more and analyze the information thoroughly to find the right answers. All the students indicated that having the examination at the end of the semester as opposed to midsemester would improve the learning outcomes for the topics that were covered after the examination was administered. Another suggestion made by several students was to add short-answer questions to the examination as they believed it would be harder to remember the answers to short-answer questions without actually reviewing course study notes.

Of the 200 questions from for all five workshops that were posted in the examination bank, 30 were randomly selected for inclusion on the flipped midsemester examination. Despite the expectation that 300 questions would be generated, there appeared to be duplication of some concepts in questions generated by the 12 groups. Also, some questions were poorly written despite the facilitator assisting students with their revisions. The mean total Pharmacotherapy grade on the flipped examination administered in 2016 was 87% (range, 80%-91%) while the mean total grade on the traditional examination administered in 2015 was 80% (range=64%-91%). The mean total flipped examination grade in 2016 was 99% (range=73%-100%), while the mean total traditional examination grade in 2015 was 76% (range=52%-94%).

Analyzing feedback from the university-mandated survey revealed greater student satisfaction with the learning process and the skills acquired compared to feedback received the previous year. As to whether the initiative to involve students in designing their examination questions had a positive impact on their learning experience, almost 77% of the students agreed, indicating that the flipped examination model may have achieved the proposed pedagogic outcomes. Students’ feedback on the survey regarding the best aspects of this course was highly favorable in terms of using the flipped examination approach. Suggestions to improve the learning experience in Pharmacotherapeutics were similar to the suggestions obtained from the thematic analysis of the focus group transcripts and included the addition of a traditional final examination or the addition of a set of questions written by the coordinator/lecturers to the flipped examination in order to encourage further learning.

DISCUSSION

This study was the first Australian study to assess final-year pharmacy students’ performance and evaluate their learning experiences after undertaking a “flipped exam” for an advanced clinical pharmacy course.

Notably, both deep and surface-approach learning scores significantly improved. In surface-approach learning, students tend to view tasks as being imposed, for which they have to develop coping strategies for assessment purposes only rather than for thorough understanding.²⁴ On the other hand, in deep-approach learning, students seek to comprehend ideas in a way that allows them to relate and integrate knowledge from other parts of their study and thereby conceptualize frameworks from which they can derive solutions to novel problems.²⁴ The flipped examination model used in our study served as a robust, student-centered learning tool that provided participants with an opportunity to actively construct their knowledge and enrich their learning experience by facilitating engagement and interest while encouraging rational thinking. Similar results have been obtained at Wayne State University School of Medicine, where the implementation of a flipped examination created an environment that assisted students in determining the essential concepts to be learned; promoted deep and meaningful learning; improved communication skills, critical thinking, and the exchange of ideas; and resulted in better performance scores on the examination.⁶ While research on the impact of flipped examinations is still scarce, research into the impact of flipped classrooms on students’ learning also suggests educational gains beyond surface gains and improvements in grades. Flipped classrooms result in pedagogic gains that entail independent learning strategies, deep and active learning, profound knowledge advantage and information retention, and valued opportunities for peer-peer and peer-instructor interactions.^10,25-27

Another finding in the present study was that the process of developing tactful examination questions improved students’ perceived writing skills and rendered the learning experience more constructive. Both the students and the workshop facilitators in our study reported an improvement in the quality of the MCQs generated as the semester progressed. Similarly, in a study at Cardiff University School of Medicine, a student-led approach to develop MCQs and create an online database of questions also was perceived to be highly valuable for learning. The students at Cardiff University described their experience as “enjoyable” and reported that the writing and review procedure of the questions enhanced their insight into question construction, potentially ameliorating their enthusiasm to learn and improving their overall examination performance.²⁸ In support of this, in another study evaluating the potential for medical students to learn about palliative care through the process of writing examination questions, all students considered the exercise to be constructive to their learning. Most students appreciated the experience and felt more confident in taking the examination.²⁹ From a pedagogic viewpoint, involving students in the actual writing, review, and presentation of examination questions is expedient. Developing and undertaking the flipped examination in our study provided the students with a binary opportunity to self-direct and deeply consolidate their knowledge acquisition and retention.

One of the biggest advantages of the flipped examination experience was creating a less stressful examination environment where students were actually excited and enthusiastic about undertaking an assessment. Another advantage was the high grades that students obtained on the examination, given that the examination questions along with the answers were provided prior to the examination. It is well-acknowledged in the literature that experiencing test anxiety dramatically impairs students’ cognitive performance during examinations.^30,31 The sources of academic stress most commonly perceived by medical students include fear of examinations, concerns about examination results and grades, and fear of failing the course at the end of the semester or year.^32-34 However, the flipped examination model in our study not only reduced pharmacy students’ test anxiety, it also addressed academic and test competence issues, which are substantial components of academic performance.³¹ For instance, academic competence, which refers to a students' ability to manage their study load,³¹ was undertaken by allowing the students to employ their understanding of the workshop presentations to design one question that they perceived as essential and relevant to learning. Similarly, by providing the students with the opportunity to study and review all the generated questions present on the examination bank, their test competence, which refers to a student's ability to manage and cope with the amount of study material for examinations, was addressed.³¹

Despite the high grades they obtained on the flipped examination, the majority of the students believed that the course lacked a significant assessment component. The suggestions of covering more material in the flipped examination and/or adding a non-flipped examination element were highly rewarding and unexpected. These suggestions reflect a deep level of motivation, maturity, learning appreciation, and competitiveness among final-year pharmacy students who are on the verge of beginning their professional career. In fact, pharmacy students generally exhibit a preference for the mastery-approach goal orientation and demonstrate substantial intrinsic motivational characteristics.^35,36 Pharmacy students are metacognitively aware of their motivation to learn and self-regulate their learning, and, as such, they highly value their learning behaviors.³⁶

While the flipped examination model in our study was associated with potential educational and performance-related benefits, overcoming the perceived drawbacks is warranted, and the design of a novel flipped examination model requires careful pedagogic attention. For instance, the addition of a traditional final examination in our case would ensure that the learning outcomes for the various topics covered through the lectures and workshops of this course have been sought. A traditional examination also would provide high-performing students with an opportunity to excel and be rewarded accordingly, an opportunity that clearly was unexploited through the flipped examination. Additionally, while MCQs are widely used in examinations for undergraduate and postgraduate health care and medical students, the addition of short-answer questions possibly could enrich the learning experience and test students’ nascent abilities rather than their ability to pass examinations.³⁷ Because data on the design, format, and potential impact of flipped examinations in pharmacy education is generally scarce, further research is recommended before our conclusions are generalized. The popularity of flipped classrooms seems to be increasing in higher education, and more promising results are expected in the future, particularly in terms of refining knowledge retention, encouraging personal accountability, and improving student learning.³⁸ In the meantime, numerous flipped classroom designs exist, which makes it difficult to understand all the aspects of this approach. Most approaches are providing students with an opportunity to gain exposure to content prior to class, motivating them to prepare for class, and providing clear connections between in-class and out-of-class activities.³⁸ While flipped classroom teaching methods may offer several advantages, the perspectives of both faculty members and students also should be accounted for prior to the implementation of such a paradigm change. Assessment methods used in courses that have adopted the flipped classroom approach need careful consideration as well to ensure that, as much as possible, grades accurately reflect the students’ knowledge, skills, and attitudes.³⁸

As with any other research, our study was not free from limitations. The study compared pre- and post-examination study process scores only using a validated instrument (Biggs’ R-SPQ-2F). Different results may have been obtained if we had used a different survey instrument. Another possible limitation was test-retest bias in that the students knew that the same questionnaire would be administered after the examination. Also, the study lacked a control arm and the comparison between 2015 and 2016 course grades was not conclusive given that the two cohorts completed different examinations. For the qualitative analysis, other analytical approaches, such as grounded theory, may have generated different thematic categories than the themes generated by the approach that we used. Also, this study did not address the students’ retention of knowledge acquired during the course. This should be taken into consideration in future studies addressing the development and application of flipped assessments as different results may have been obtained had traditional examinations been used.

CONCLUSION

Pharmacy education is rapidly progressing to accommodate the diversities and challenges of the growing pharmacy profession. This study highlights the role of the flipped examination as an innovative instructional approach that if pedagogically designed and implemented, potentially brings about significant improvements in student satisfaction, student performance, instructor-student and student-student communication, and most importantly, thorough learning and retention of acquired knowledge.