Student Engagement with a Flipped Classroom Teaching Design Affects Pharmacology Examination Performance in a Manner Dependent on Question Type

INTRODUCTION

Today’s students face intense cognitive challenges. Learning outcomes that increasingly address higher levels within Bloom’s cognitive domain require students to develop skills and attitudes needed to solve problems, predict outcomes, and deal with novel clinical scenarios.¹ Teaching methods must therefore be adapted to effectively prepare students.

Carefully designed instructional approaches can help students meet various aspects of the problem-solving process. These include: defining and understanding a problem, researching for information, delineating new terms and analyzing relationships between variables, hypothesizing or planning solutions to the problem, and checking that the proposed solution is consistent with all elements of the scenario. The flipped classroom approach offers a teaching design that can assist students to do many of these tasks. The basic premise of the approach to a flipped or inverted classroom is to require students to study content prior to class so that class time can be devoted to active learning activities.^2,3 Pre-learning and related feedback can help with defining terms (developing the language of the new area), and understanding the problem. In-class active learning approaches in which students tackle problems and apply, analyze and synthesize have been proven to improve student performance.^4-6 Specific approaches that ask students to produce diagram representations of their understanding such as concept mapping can assist with comprehension of the inter-relationships between variables.⁷ The flipped classroom approach proposes that students engage with pre-class activities that prepare them for in-class active learning. Thus, there is a proposed synergistic relationship between student preparation for class and in-class active learning⁸ that could be seen in assessments.

Within the flipped classroom environment, there are many different approaches to in-class activity design, depending on the nature of the intended learning outcomes. Scenario-based learning is a method that is based on the principle of situated cognition,⁹ which states that “knowledge is situated, being in part a product of the activity, context, and culture in which it is developed and used.” Scenario-based learning can help students learn problem-solving strategies by exposing them to the requisite variety and complexity of novel scenarios and problems that they might face as employed graduates. In this environment, students can learn to propose new hypotheses to solve novel scenarios and to evaluate outcomes to test whether their proposed solution fits all elements of the scenario.

The theoretical framework for our work combines instructional approaches that invoke Brown’s situated cognition⁹ as the specific arm of a social constructivist approach pioneered by Vygotsky.¹⁰ Ausubel and Novak’s meaningful learning theory offers an effective and behavioral viewpoint as it considers students’ attitudes to and motivation for learning.¹¹ The important requirements for deep meaningful learning proposed by Ausubel and Novak are connection of new information to what a learner already knows (prior knowledge); relevance of new concepts; and active integration of new concepts into a learner’s cognitive structure (learner’s choice). This theory stresses the active role of the learner in knowledge construction.

We aimed to investigate the relationship between student engagement with the key elements of a flipped classroom approach (preparation and attendance), their attitudes to learning, including strategy development, and their performance on two types of examination questions (knowledge recall and providing rational predictions when faced with novel scenarios). We predicted that students would develop strategies to solve novel problems if they prepared for and attended classes because the instructional approach would help them define terms, understand the problem and inter-relationships between variables, and develop skills in producing and testing hypotheses to solve the problem. More specifically, we predicted that students who both prepared for and attended classes would gain the greatest advantage in examination performance on questions requiring them to solve novel problems, compared to students who only prepared or only attended or did neither. To our knowledge, this question of synergy between preparation and attendance has not been thoroughly addressed. In examining the relationship between learner disposition and performance, we predicted that motivation, goal orientation, self-efficacy, and use of strategy would all positively associate with examination performance, and that self-efficacy would increase as a result of increased perception of problem-solving skill development.

METHODS

This study was approved by the Monash University Human Research Ethics Committee. There were 147students enrolled in one unit (course) of study within the bachelor of pharmacy at Monash University who consented to taking part in the study; 67 students undertook the unit in 2014 and 80 students in 2015. The unit (referred to using its university handbook code PAC2412) provides an introduction to disease state management. PAC2412 provides a foundation of basic knowledge required for the diagnosis and management of patients with cardiovascular diseases.

The teaching approach within the bachelor of pharmacy involves an active learning approach, as reported by White and colleagues in 2016, including some preparation for classes and active learning during classes. In 2013, prior to the intervention reported here, students engaged with active learning activities that were aimed at assisting students to understand cardiovascular pharmacology, including clicker questions at understanding and analyzing levels of Bloom’s taxonomy, and concept mapping. The intervention reported here involved a change to a sequential, strategic use of pre-class videos, concept mapping, practice with novel scenarios and problems and related metacognitive exercises as described below.

Students were briefed on the teaching design prior to the first class and instructed to watch four 20-minute pre-recorded videos. The videos featured information about cardiovascular disease, specifically about blood pressure regulation, high blood pressure, and systolic and diastolic heart failure. As part of this activity, students were assigned several tasks, including a concept map of the relationships that determine blood pressure. They were instructed to complete these tasks prior to class in order to link their preparation to active learning in class.

Students were asked to peer-assess and finalize their concept maps, indicate on these maps the specific factors responsible for increased blood pressure in hypertensive patients, and show the interaction between antihypertensive drugs and causative factors.

Students who attended class were each given four scenarios to tackle during the final three classes. These scenarios were also made available to students who missed class via online materials within the Learning Management System (Moodle). Students were presented with a problem-solving schema at the completion of the first scenario, which was similar to patterning Bodner’s problem-solving characteristics.^12,13 Students were advised to solve the problem in stages (sequence, source information, define relationships between key variables, compare known to unknown to generate hypotheses, and retest against scenario details). An example is shown in Appendix 2. Each problem had a short stem, and included a number of prompts that were predicted to require the student to use a problem-solving approach rather than simply look up a correct answer.

Students were asked to present answers/solutions to the novel scenario. The instructor read and gave feedback to any student who requested such. In the following class, students were presented with a range of solutions chosen from the submitted student answers, and asked to rate them. In the final class of the series, students were asked to discuss the strategies they used to solve such scenarios, and common approaches were presented to the whole class.

The relationship between three groups of variables was examined in this correlational study, and the primary outcome measure for this study was student performance on examination questions involving novel scenarios. First, individual student engagement in the flipped classroom was determined by measuring frequency of student access of pre-recorded videos prior to class and student attendance in class. Second, student dispositions to learning were determined using a survey instrument, and open-ended questions regarding solving novel problems were posed and the data analyzed. Third, student performance on two types of examination question was determined; performance on questions requiring knowledge of cardiovascular disease and performance on questions requiring students to solve novel scenarios (novel case studies). Student performance on knowledge and novel scenario examination questions was compared across the categories “prepared, attended,” “not prepared, attended,” “prepared, not attended,” and “not prepared, not attended” (see section on Binning Analysis). To understand the strategies that students used in solving novel problems, a structured analysis of survey response data was conducted. Finally, student final grades in four units (courses) taken the previous semester were used to analyze past student capability, and performance in examinations for the following year were used to analyze subsequent capability.

Data was collected for all 147 participants. The learning management system, Moodle, was used to record whether each student had interacted with each of the four preparation videos prior to the class, and the percentage preparedness was recorded from the number of videos accessed by each student. An audience response system was used to record participation in each of the four classes, and the percentage attendance was recorded for each student.

Students were surveyed to determine their disposition toward learning. Items in the survey were based on the widely used “Motivated Strategies for Learning” questionnaire developed by Pintrich, De Groot,¹⁴ and customized for the context. There were 20 items plus two open-text follow-up questions in the survey (Appendix 1). Students were instructed to respond to the items on a five point Likert scale from 1=never to 5=always.

The 20 items in the survey covered four key constructs (scales) related to dispositions about learning. These consisted of goal orientation (eg, I set myself specific goals to complete all the requirements in this section of PAC2412); self-efficacy (I was aware of my ability to complete all the requirements of this section of PAC2412); motivation (It was very important for me to do well in PAC2412); and learning strategies (I was very clear about my learning strategies in the study of this section of PAC2412). The follow-up question asked students to describe any strategies they used when tackling scenarios such as those they’ve encountered in cardiovascular pharmacology, and to indicate the extent to which the teaching design influenced how they approached the solution of novel scenarios in this subject on a five-point Likert scale ranging from 1=to a very small extent; 5=to a very large extent.

Scale reliability analyses were conducted on the student attitude to learning survey scales, with survey items organized and analysed to determine Cronbach’s alpha values for the four scales using Statistical Package for Social Sciences (SPSS) software (IBM, Chicago, IL). Scale reliability analysis indicated that two of the survey items should be removed to increase the scale reliability score; items GO5 and STR6 in the survey. Scale reliabilities for the scales were then determined to be self-efficacy: 0.71, goal orientation: 0.68, motivation: 0.78, and strategy: 0.75.

Performances on two types of examination question were recorded for each participant. In each case, students were provided with information about a patient and first asked to explain the pathophysiology underlying the patient’s condition (knowledge/comprehension question). Students were encouraged to predict the outcome of a secondary change to the scenario, which was unlike any scenario tackled in class or online (novel scenario prediction and reasoning). The examination questions used are shown in Appendix 2. Examinations were marked by a teaching associate who was not involved in the study, using marking criteria developed by one of this study’s authors.

Student grades were obtained for four units (courses) in the semester prior to and for four units after the one in which the study was conducted and mean scores for all eight units were used as an indicator of past and subsequent student academic performance independent of the current study. Analysis of examination questions on the prior units and subsequent units indicated that a range of assessments were used in these examinations; this range did not include open-ended problems or novel scenarios in these examinations.

All quantitative data analysis other than Cronbach’s alpha determination was conducted using GraphPad software (La Jolla, CA). Data are presented as mean (standard deviation (SD)). Independent t-tests were used to examine differences between two groups and a two-way ANOVA followed by Tukey’s test was used for data shown in Figure 1. Multiple comparisons increase the probability of false positive outcomes. Where inferential statistical methods were used, they were accounted for within the method (two-way ANOVA). For other comparisons, we have not applied a correction to the alpha value, as these were exploratory comparisons and avoidance of type II error was considered important.¹⁵ Absolute p values are reported for all comparisons.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Performance on examination questions by engagement category. Values shown are mean (SD) percentage mark for each type of examination question. *Indicates a significant difference compared to all other binned categories of preparation and attendance within that assessment type (two-way ANOVA, p<.01).

Correlational analysis was performed to determine relationships between each of the three groups of variables, namely student engagement with flipped classroom activities; student responses on each of the four categories on the student attitudes to learning survey; and student performance on knowledge and novel scenarios. Pearson’s correlation coefficient was calculated for each comparison and significant correlations were identified with α set at 0.05.

Student engagement with flipped classroom activities was further analyzed to determine whether the combination of preparation and attendance produced a synergistic benefit (ie, to test that a combination of preparation and attendance was related to examination performance compared to the combined individual benefit of preparation and attendance). Binning analysis, in which students were assigned prepared or not prepared status as described above, can be problematic in that it results in a “loss of information” and the binning can be seen as arbitrary. However in this case, binning was considered justified as the bins were easy to determine prior to the study, and the correlational analysis was unable to answer the question as to whether preparation and attendance had more than additional benefit because the majority of students were at one end of the distribution for both variables and, thus, dominated the analysis (ie, more than half the students attended most classes and prepared for classes). In order to uncover differences between those students who both prepared and attended, and those who attended but did not prepare or prepared but did not attend, each student was categorized as “prepared” or “not prepared” and “attended” or ”not attended.” Students were categorized as “prepared” if they accessed 50% or more of pre-recorded lectures and “not prepared” if they accessed less than 50% of recordings. Students were categorized as “attended” if they were recorded as present at 50% or more of the classes, and “not attended” if they were recorded as present at less than 50% of the classes. A two-way ANOVA was used to determine differences between the four resultant groups in performance on examination questions.

The following open-ended questions were posed to students in a survey at the end of the study:

Question 1. “Describe any strategies used when tackling scenarios such as those you encountered in cardiovascular pharmacology.”

Question 2. “Indicate the extent to which the teaching design and strategies in this subject influenced how you approached the solution of novel scenarios in this subject.”

Ninety-two and 59 responses were collected for questions 1 and 2, respectively. The length of the responses ranged from a single word to 2-3 sentence paragraphs, resulting in a total of 3,100 words. A list of themes (for student learning and instructional design strategies) was composed from these responses. A combination of deductive and inductive analysis was used as this study was also of an exploratory nature designed to achieve a descriptive interpretation of patterns within the data instead of illustrating the pre-existing theory with excerpts from the data.

The analysis followed the guidelines of Braun and Clarke’s study.¹⁶ Step 1 (data familiarization) involved the lead researcher reading and checking the responses. In Step 2 (initial coding), the researchers examined the responses using qualitative analysis software (NVivo 11, QSR International, Melbourne, Australia). Sentences, or small sections of text, were assigned to descriptive nodes. New nodes (compared to the initial ones identified by the lead researcher) were identified at this stage. Step 3 (searching for themes) involved the sorting of nodes into higher-level themes. Following steps 1-3, the researchers reviewed the themes to confirm if they were supported by the data (Step 4, reviewing themes). During the review, themes were removed, merged, or divided. They were also renamed, where necessary, using words from educational research and respondents.

The final nodes for question 1 were pre-determined: sourcing of required information, defining key variables and their relationship, solving the problem in stages, breaking the problem into component parts, hypothesizing; emergent: vocalizing. The final nodes for question 2 were all pre-determined: analysis of novel scenarios, concept mapping, peer discussion, viewing pre-class videos.

To ensure rigour, the researchers re-coded the data using the themes agreed upon. At this stage, full sentences were assigned to the nodes. If a sentence contained more than one conceptual idea, it was coded onto multiple nodes. No new nodes or themes were identified in the final round, suggesting that saturation was achieved. The final themes were discussed and refined to reach consensual validation. To test for the inter-rater reliability, Cohen’s Kappa coefficient was used as a statistical measure. The mean Kappa values for inter-rater reliability for question one was .70 (SD 0.2) and for question two was .71 (SD 0.2).

RESULTS

Ninety of the 147 participants (61%) were female, and the median age was 20 years. Mean attendance across the 147 participants was 68% (SD 32%) of the four classes, while mean participation was 69% (SD 36%) of the four pre-recorded videos. The average mark on the examination scores for the questions assessing student knowledge was 72.8% (SD 25%). The average mark on the examination scores for the questions assessing student scenario prediction and reasoning was 70.8% (SD 24%).

Data shown in Table 1 are Pearson’s correlation coefficients with p values shown where correlation is significant at the .05 level or below (2-tailed). Preparation for classes showed no correlation with performance on knowledge questions and correlated weakly with performance on scenario prediction questions. Attendance was weakly correlated with performance on knowledge questions and strongly correlated with performance on scenario prediction questions. The correlation between attendance and performance on novel scenario questions was consistent across the two cohorts, while the correlation between preparation and performance on novel scenario questions was significant for the 2014 cohort.

Table 2 shows the proportion of students in the 2014 and 2015 cohorts that were categorized as “prepared” or “not prepared” and “attended” or “not attended.” The mean number of recordings watched for “not prepared” students was .8 out of 4, or 20% of recordings, compared to a mean of 2.9 out of 4, or 72% of recordings for “prepared” students. The mean number of classes for students who had “not attended” was .1 out of 4 classes, compared to a mean of 3.4 out of 4 for students who “attended.”

Performance on examination questions was compared between the four engagement status groups, “prepared, attended”; “prepared, not attended”; “not prepared, attended” and “not prepared, not attended” (Figure 1). For the questions addressing knowledge, “not prepared, not attended” students performed significantly worse than the other three groups (two way ANOVA, p<.0001, n=147). For novel scenario questions, students who were both prepared and attended performed significantly better than all other groups; the mean for prepared and attended students was 80% (SD=15), which was 20 marks, or 1.3 SD higher than the next best performing group, “not prepared, attended” students (p<.0001, Figure 1). There were no significant differences between the groups in examination performance in prior units or subsequent units.

Of the 147 participants, 122 completed the 22 question survey (Appendix 1). Table 3 shows the descriptive statistics for the Student Attitudes to Learning survey.

Student responses on two of the four constructs in the dispositions to learning survey correlated significantly with students’ performance on novel scenario questions (Table 4). Motivation scale responses correlated positively with performance on novel scenario and knowledge questions for the pooled data. Strategy scale responses correlated positively with performance on novel scenario questions.

Student responses on all four constructs (scales) in the dispositions to learning survey correlated significantly with preparation for classes for the pooled data set (Table 5). A positive relationship was also observed for responses on self-efficacy and motivation scales and attendance for the pooled data set. Preparation for PAC2412 (2014) was strongly correlated with student goal orientation, learning motivation, learning strategy and notions of self-efficacy. There was a positive correlation between attendance in PAC2412 (2014) with learning motivation and self-efficacy, as well as a positive correlation between preparation for PAC2412 (2015) and motivation and self-efficacy scales.

In open-ended question 1, Participants were asked to describe any strategies they used when tackling scenarios that they encountered in cardiovascular pharmacology. There were 88 respondents to this question.

Five pre-determined themes were used to evaluate whether the student strategies related to the problem-solving schema presented to the students during the first scenario analysis. There were 39 comments that were coded as related to sourcing of required information, 27 comments that were related to defining key variables and their relationship, 13 that related to solving the problem in stages, 22 that related to breaking the problem into component parts, four that related to hypothesizing. An emergent theme identified by the four coders referred to the practice of verbalizing. Examples of student comments are provided in Table 6.

There were 114 respondents to open-ended question 2, “Please indicate the extent to which the teaching design and strategies in this subject influenced how you approached the solution of novel scenarios in this subject.”

Forty-one responses were assessed by four independent coders as referring to specific elements of the teaching design. There were 18 responses that were coded as referring to analysis of novel scenarios, six references to concept maps, seven references to peer discussions, 10 references to pre-recorded videos (Table 6). Students’ comments indicated that concept mapping helped explain the links between variables. Seven students commented that concept maps were useful in helping in the creative process of finding a solution to a novel problem. Five students did not understand what was meant by the term teaching design and strategies: “Teaching design and strategies I'm a bit confused as to what they refer to. But all the practice questions clicker questions, mind maps, pre recordings were brilliant!” Sixty-one students (54%) responded that the teaching design influenced their approach to a large or very large extent, while nine students chose “to a small extent” or “to a very small extent.”

DISCUSSION

This study demonstrated the relationships between student engagement with a flipped classroom teaching design, student disposition to learning, and student ability to solve novel cardiovascular scenarios in examinations. Student attendance in classes and preparation for classes both correlated positively with success in solving novel scenarios in exams, and students who both prepared and attended did far better at solving novel scenarios than students who attended or prepared, or those who neither prepared nor attended. A disposition to learning survey indicated that those students who scored highly on the motivation and strategy development scales of the survey performed better on novel cardiovascular scenarios in examinations.

Our hypothesis regarding the benefit of preparation and attendance at classes was supported by the data. The categorization of students as “prepared” or not and “attended” or not allowed us to reveal a synergistic benefit from the combination of preparation for class and attendance. There is a significant benefit from the combination of preparation and attendance. However, that benefit was only apparent on the questions asking students to deal with novel scenarios in the examination. This is the first time, to our knowledge, that the benefits of engagement in both components of the flipped classroom approach (ie, prior preparation and active engagement in class) have been shown to be synergistic, and also the first time that the benefit has been shown to be dependent on the type of question. Our findings are not in agreement with those of Jensen and colleagues,¹⁷ who suggest that the advantages of a flipped classroom may be solely due to the benefit of active learning, although we did find that attendance did correlate more strongly with exam performance than preparation. Our findings are consistent with those of Gross and colleagues, who found that the benefits of the flipped class are due in part to improved student interaction with course materials.¹⁸ Attendance at active learning classes was strongly correlated with examination performance on novel questions, as expected given the extensive body of literature supporting active learning approaches.^4-6,19,20

The 1.3 SD difference between prepared and attended students and all other groups is a significant outcome, close to approaching the 2 standard deviation (2 sigma) effect that is possible only with resource and time intensive instructional methods such as one-to-one tutorial instruction or problem-based learning.^21,22 It is possible that the differences between the groups relate to factors intrinsic to the students themselves – that prepared and engaged students are “better” students. However, the large difference in examination performance seen in the unit in question was not observed in prior units or subsequent units. Therefore, it is reasonable to contend that the specific intervention that was accessed by the prepared and engaged students (the strategic, sequential flipped classroom design) was responsible at least in part for the superior examination performance by these students.

Extra time-on-task, or more specifically “engaged time,” is an important determinant of learning outcomes.^23,24 Increased “engaged time” may have occurred in our study for the students who prepared for and attended classes, and may have contributed to the superior performance of the prepared and attending students. However, increased engaged time would have been expected to result in improved performance on questions addressing knowledge as well. However, in our study, performance on questions addressing knowledge was similar for students who prepared but did not attend compared to students who prepared and attended. This suggests that the particular teaching design in this subject had more impact on developing higher order skills, such as advanced problem-solving skills, than on the accumulation of factual knowledge.

While student-reported strategies for solving novel problems were often operational (listen to recordings, use of concept maps), they did include, in many cases, reference to one or more of the problem-solving approaches that were presented to them. A recent meta-analysis of studies relating to promoting critical thinking (CT) concluded that “the opportunity for dialogue, the exposure of students to authentic or situated problems and examples, and mentoring had positive effects on CT skills.”²⁵ Each of these approaches was used in the current study and was referred to by students in their qualitative responses.

Students commented that their engagement with concepts via the pre-class recordings and the concept mapping in class helped them understand the concepts (“get your head around how things link together”). They also said that practice how to solve novel problems was beneficial, both in preparing them for similar examination questions (and future clinical practice) and in helping them reinforce concepts and correct misconceptions. A significant proportion of students referred to solving the problems in stages, which is an important step in developing a personal problem-solving schema.

Concept mapping ¹¹ is a technique that involves students representing the relationships between variables. It has been shown in many studies to improve performance on related assessments. In pharmacy education, it has been shown to benefit higher-level learning.⁷ Students commented that concept mapping helped them produce their own strong conceptual framework ²⁶ and helped them explain the links between variables. A number of students also reported that concept maps were also useful in helping in the creative process of finding a solution to a novel problem, in agreement with the findings of González and colleagues.²⁷ A theme emerged in the student comments related to the benefits of developing a conceptual model prior to the scenario-based learning.

Peer learning was not uniformly recognized by students as a key element in the development of strategies to solve novel problems, based on the survey question item related to the impact of peers on student-learning strategies (STR6: mean response 2.8 out of 5). However, seven responses to question one did refer to peer discussions as being useful, consistent with other studies that showed the benefit of peer instruction on the ability to solve novel problems. ²⁸

Student dispositions toward their learning strategies correlated with performance on novel scenario questions. Students with higher scores on the motivation and strategy scales of the survey performed better on the novel scenario examination questions; albeit the correlations were modest. Interestingly, goal orientation and self-efficacy survey scale scores did not correlate significantly with performance on either type of examination question. Goal orientation might be expected to be important for the implementation of schema required for solving novel problems, and thus it was surprising that this construct (using our survey instrument) did not reveal a relationship with performance.

Students who attended classes were expected to reveal greater self-efficacy at the end of the semester, as Bandura’s theory indicates that past successful experiences increase self-efficacy, and increased self-efficacy is predicted to enhance performance.²⁹ Thus the practice and reflection on the scenarios in classes was predicted to have resulted in increased perceptions of self-efficacy. This did appear to be the case based on the correlational data, in that preparation and attendance did correlate with self-efficacy scores on that scale in the survey. However, this higher self-efficacy seen in the survey analysis did not translate to performance on examination questions, as self-efficacy score did not significantly correlate with performance on knowledge (p=.07) or novel scenario (p=.16) questions. This was unexpected, as we anticipated that “the beliefs they (students) hold about their capabilities powerfully influence the ways in which they will behave.”³⁰ Given the nature of the correlational data, a larger sample size would be required to address this issue definitively.

This was not a randomized, controlled study; instead it looked at interactions between the naturally occurring student engagement with the teaching design, the attitudes of students to their learning, and their performance on different types of examination question. Controlled studies in the field involve relatively small sections of a course or one or two classes.¹⁹ These controlled studies provide the benefit of causal relationships between variables in the study; however, they do have limitations. Controlled studies, by their very nature, constrain some variables that would ordinarily affect outcomes and thus render the learning environment within the study unnatural. Also, where elements of the intervention are well known to result in enhanced learning as is the case for active learning,⁴ it is ethically problematic to employ negative controls for such elements.

Participants in this study behaved as they would in any other teaching context. There is value in studying student behavior in field settings that are not affected by the study design. Although we acknowledge that the issue of causation cannot be solved by correlational studies alone, findings of randomized controlled studies cannot necessarily be generalized to an uncontrolled educational environment; the combination of hypothesis driven and correlational research is more powerful than either alone. Causality ought to be seen in much broader terms that include behavioral dimensions other than achievement on conventional cognitive measures.³¹

This study has implications for others who are thinking about adopting a flipped classroom approach to teaching. Firstly, our data suggest that preparation is an important element of the flipped classroom model, and that efforts to ensure that students prepare for classes will result in improved performance on assessments. Some key steps for instructors to increase the proportion of students that prepare for classes include providing clear expectations to students, keeping the preparation tasks focused, and explicitly linking preparation activities to in-class active learning.

Good practice in implementing a flipped classroom approach can help ensure that the majority of students prepare for and attend classes,^32,33 as seen in this study. Some students in our study did not engage with the teaching activities; however, just under 30% did not prepare and the same proportion (although not the same students) did not attend. This study underlies the importance of approaches to ensure that those remaining students who are disengaged can gain the benefits seen in summative assessments shown in the study, and this is an area that requires future investigation. The timely use of analytics to identify and communicate with those students who are not engaging with the designed teaching strategies may be a fruitful approach.³⁴

In summary, this study found that a flipped classroom teaching design that was strategic (using specific instructional strategies) and sequential (using information sourcing prior to discovering relationship between variables, then practising tackling novel scenarios and reflecting) produced improved examination performance on questions that involved novel scenarios by students who had engaged fully with this design.