Students’ Attitudes, Academic Performance and Preferences for Content Delivery in a Very Large Self-Care Course Redesign

INTRODUCTION

The volume of knowledge in pharmacy is rapidly changing and expanding, and practitioners are asked to evaluate complex patient situations using foundational knowledge and decision-making skills. Education literature states that students’ actively engaging with content and assuming larger responsibility for learning are keys to knowledge retention.¹ The rationale for moving toward a more learner-centered teaching approach instead of passive instruction varies among curricula. Many proponents of such pedagogy often cite the need to engage students, develop problem-solving and critical-thinking skills, and enhance application of course content as reasons for implementation.^2,3

Team-based learning (TBL) is a pedagogical approach that enhances learning by immersing students in an active-learning situation where TBL activities are facilitated by the instructor.⁴ Student preparation is assessed via individual and team-based quizzes at the start of each class prior to engaging in team-based activities that focus on application of newly acquired knowledge. The TBL classroom experience serves to enhance students’ understanding and retention of content via activities that promote collegial communication and collaboration.² Use of TBL framework allows for successful implementation of this format in a variety of college level courses, especially health care programs.³ Within the health disciplines, much of the literature describes use of TBL in schools of medicine, nursing, and pharmacy demonstrating encouraging outcomes including higher levels of student engagement,^5,6 enhanced test performance,⁷ and effective application of clinical knowledge.⁸

Colleges and schools of pharmacy in the United States have begun to adopt TBL principles. This is partly in response to the Accreditation Council for Pharmacy Education (ACPE) Standards calling for active learning in pharmacy curricula.⁹ Letassy and colleagues reported on successful TBL integration within the PharmD curriculum in an effort to increase critical-thinking skills and application of information in an endocrine module.¹⁰ The effectiveness of TBL was assessed based on performance on unit examinations and student evaluations. When compared to grades earned using a lecture-based format, unit examination grades after TBL integration improved from 81% to 86%. Student evaluations revealed that TBL was positively received—a result similar to other reports of TBL adoption.^11-17 Ferreri and O’Connor specifically evaluated how TBL emphasized the importance of teamwork, enhanced students’ confidence with patient assessment, and evaluated how this approach increased relevance to practice.¹⁸ Diaz and colleagues developed a survey to characterize the use of TBL in US pharmacy schools.¹⁹ Sixty-nine out of 100 faculty members responded to the survey, representing 43 schools of pharmacy. More than 90% of participants reported having experience with TBL, with 74% reporting 1-4 years of experience. The class size for all responding schools was less than 250 students.

Team-based learning is a flipped classroom format, where the course content is delivered online and utilized later during the class time for concept application and mastery through active student engagement.²⁰ A growing body of literature on flipped classroom in medical, nursing, and pharmacy education reports that flipped courses improves attendance^21,22 and increases grades^21-27 and performance in follow-up courses.²⁸ Conversely, some reports discuss decreased student satisfaction,²⁵ significantly decreased examination scores resulting from a lack of accountability for out-of-class preparation,²⁹ and student satisfaction being influenced by the quality of content and activities.²⁷ Some negative aspects of TBL in the literature include increased preparation time by students and faculty members as well as student dissatisfaction (especially among introverted individuals and those who prefer the traditional faculty role of lecturing and storytelling).^25,27,30,31

Based on the literature review, faculty members teaching this course hypothesized that a change to a flipped team-based learning classroom would allow them to dedicate efforts towards discussing patient cases and relevant materials and answering student questions. This, in turn, would increase students’ participation, enthusiasm, and interest in the course content. Faculty members hoped the redesign would improve students’ ability to process self-care information, communicate with others, enhance performance on assessment activities, and better prepare them for future practice. Until this study, limited data was available about the successful integration of flipped TBL format within “very large” classes (>300 students). Since previous studies suggested that some students were dissatisfied with the idea of flipped TBL classroom, faculty members wanted to determine if students who prefer the traditional lecture format would perform differently in this course. Our analysis aimed to present the findings of a student survey about their attitudes toward change in the course format, homework completion, their academic performance, and incoming grade point average (GPA) ranges based on preference for content delivery.

DESIGN

The majority of the faculty members teaching in the Over-the-Counter Drugs/Self-Care (OTC, or nonprescription) course have clinical practice sites and train fourth-year (PY4) students. They identified students’ lack of knowledge retention as a problem. Moreover, the faculty team perceived students’ inability to apply content and communicate with others as curricular deficits and reasons for innovative redesign of the course. Another reason for re-evaluation of content delivery was expanding class size. After reviewing the literature on large classroom instruction and successes in converting lecture to small group format,^18,32-34 we felt the need for consistency in topic delivery would be served well by dividing the very large class into two sections. The restructured self-care course remained a required 3-credit course in the third-year (PY3) year of the curriculum and intended to prepare students to handle routine questions in clinical practice experiences and future practice. The new content included introduction to TBL, patient counseling, gastrointestinal products, skin products, legal and regulatory issues, vitamins and minerals, home kits, cough/cold/allergy products, insomnia, ophthalmic/otic/oral products, women’s health products, nutritional replacements, complementary/alternative medicine, natural products, pain and fever, and weight loss products. The objectives of the course included: (1) implementing effective patient counseling strategies and utilizing appropriate techniques for selecting nonprescription drugs; (2) discussing common regulations pertaining to the use of nonprescription drugs; (3) recognizing specific active and inactive ingredients in nonprescription products; (4) comparing and contrasting properties to make correct therapeutic decisions; and (5) incorporating safety information into patient counseling.

Previously the team-taught (10 faculty members) course was offered in a traditional lecture style with 1-2 faculty-directed cases and other elements of active learning incorporated into lectures. To accommodate a growing class size (305 students), provide consistent delivery of topics, and attempt to increase knowledge retention, the course content was changed to a flipped format with TBL sessions delivered in class. Previously, all students met for 150-minute lectures in a large stadium style lecture hall. In the new format, to promote team work, a large one level function hall was set up with small rectangular tables facing each other to accommodate small teams participating in live application exercises for 75 minutes. To accomplish this, we divided 305 students into two sections and randomly assigned them to teams of four or five students. In preparation for the in-class session, students were expected to review prerecorded materials (approximately 75-100 minutes in length), which were recorded with iSpring (iSpring Solutions, Alexandria, VA)³⁵ and posted online along with handouts via Blackboard (Blackboard, Washington, DC) learning management system. All lectures were divided into smaller recordings of 10-20 minutes in length to accommodate the possibility of students having short attention spans, as recommended by the literature.^36,37 Other posted preclass materials depended on faculty members; for example, some instructors included readings assignments and study guides instead of prerecorded lectures.

Students were invited to enroll in a closed Facebook group for the course where they were able to post questions for instructors and course coordinators. Students were encouraged to answer each other’s questions whenever possible. Faculty members contributed to the discussions throughout the week and, at the start of each in-class session, summarized the questions posted on Facebook. Students were then quizzed on their preparedness with the use of an iRAT (individual readiness assurance test) containing 5-10 multiple-choice questions. Following each iRAT, teams completed the same quiz as a tRAT (team readiness assurance test). Students utilized IF-AT (immediate feedback assessment technique) scratch cards³⁸ during the tRATs. Teams were awarded credit for providing the correct response as follows: 100% if the correct response was scratched off from the first attempt, 50% from the second attempt, 25% from the third attempt, and 0% from the fourth attempt. The remainder of the class time was devoted to team-based case study application exercises. The goals of these exercises were to improve team-based communication and promote critical-thinking skills on therapeutic self-care topics. On average, each topic had 3-5 exercises. Students reported their consensus to the class either using multiple-choice ABCD card responses, or individual teams were called to report their findings. Students were not graded on their performance during application exercises.

Throughout the semester, students participated in three 60-minute examinations (each was 20% of total grade), 11 iRATs (15% of total grade), and nine tRATs (20% of total grade). On examination days, students only completed iRATs on the topic of the day and participated in case discussions. Because these sessions were shorter in duration, tRATs were omitted on examination days. The information on iRATs on examination days was included on the following examination. Students evaluated performance of team members twice during the course of semester (the evaluations counted for 5% of total course grade). The peer evaluations were based on the following criteria: preparation (30%), contribution (30%), facilitation (20%), and flexibility (20%). During the first year of course redesign, student peer evaluation percent grades and written responses were collected via Google Forms (Google, Mountain View, CA), processed, and grades were posted via Blackboard. Written comments were shared via e-mail with those who received grades of less than 85%. The remainder of students was able to obtain written responses per request.

At the end of the semester, we evaluated students’ perception of the redesigned course with a survey. The survey instrument was created by faculty members over a number of meetings, tested by postgraduate year-one residents, fellows, and PY4 students, and later revised. The university’s institutional review board approved the research and the survey instrument. An e-mail was sent to students enrolled in the course with an informed consent request and a link to the survey. Clicking on the link implied student consent. The 42-question survey was delivered via the Qualtrics online survey platform (Qualtrics Labs, Inc., Provo, UT).³⁹ Five-response Likert scales were used for the majority of questions with responses ranging from extremely useful to extremely not useful, strongly agree to strongly disagree, and always to never. We included one open-ended question with the option of a free-text response and used a summative content analysis approach to analyze keywords and then interpret responses as positive or negative.⁴⁰ Participation in the survey was voluntary but students received bonus points for completing it. Information about students’ performance during iRATs/tRATs, examinations and final grade, and GPA ranges of performance prior to the start of the course were evaluated (1=3.50 to 4.00; 2=3.00 to 3.49; 3=2.70 to 2.99, 2.7 being the minimum required GPA for all students in the professional phase of our program). Grades and GPA ranges were used only for students that consented to participate in the project.

The survey instrument consisted of questions about students’ preference for content delivery format, amount of routinely completed homework, attributes related to ability to learn, and frequency of technical problems. Additionally, perceptions on the benefit of home preparation on in-class activities, the impact of in-class activities on course interest, ability to communicate, and preparation for clerkships/practice were assessed. Feedback on readiness assurance testing and team interaction/evaluation process was also gathered. The Qualtrics online survey platform tabulated the responses and provided descriptive statistics. Multiple regression analyses were used to determine if significant relationships existed between quantitative measures of student performance (final grade, examinations, iRATs, tRATs) and students’ preference for delivery format, controlling for homework completed, GPA, gender, and age. Chi-square and Marascuilo’s post hoc analyses were used to assess for significant correlations between student responses to select survey items and their content delivery preference. All analyses were performed using either NCSS 2007⁴¹ or StatsToDo (Kaysville, UT).⁴² Results were considered significant if the observed level of significance was p<0.05.

EVALUATION AND ASSESSMENT

Following the last class of the semester, 286 students (94% response rate) completed the survey. Sixty-three percent of participants were female, and 76% were between 22 and 23 years of age, which reflects the demographics of the school of pharmacy student body. In the survey, students were provided with examples of course types of and, based on those, expressed content delivery preferences as follows: traditional lecture-based (traditional, 19%), flipped/team-based learning (flipped TBL, 30%), combination of traditional lecture/based and flipped/team based (mixed, 48%), and no preference (3%). Regarding homework completion prior to in-class sessions, 62% of students reported completing all of it, while 28% reported completing most, and 10%, some. Homework completion prior to in-class sessions was comparable among different content delivery preferences. The in-class sessions were delivered on Monday mornings. Based on a Blackboard activity report, the majority of students viewed content on Saturdays, Sundays, and Mondays.

Students’ performance (final grade) in the course was as follows: A (49%), B (48%), C (3%), and D (0%), which was similar to previous years. Final grades, as well as performance on iRATs, tRATs, and examinations, were comparable among content delivery preferences (p>0.05). Students who completed all or most homework and those who had higher GPAs before entering the course performed better (p<0.05, Table 1).

For the purpose of statistical analyses of student feedback, the group with “no preference” for content delivery (N=8) was excluded because this number was too small in comparison to the other groups and could skew analyses. Students with a preference for flipped TBL and mixed format had higher appreciation for all elements of the course than students with preference for a traditional format (p<0.05). (Table 2) All students, regardless of format preference, had about the same amount of technical problems with audio/video streaming (p>0.05). Students with preference for a traditional format experienced more obstacles than students with other preferences when reviewing homework (p<0.05, Table 3). Students with preference for flipped TBL or mixed format had more positive views than students with preference for a traditional format about how homework and in-class activities affected their learning and knowledge retention (p<0.05, Tables 4 and 5). Most students felt that in-class activities helped enhance their ability to communicate with others.

Students reported the following elements provided significant learning opportunity: prerecorded lectures (79%), tRATs (75%), application exercises (66%), and iRATs (55%). The following parts of the course were described as enjoyable: tRATs (69%), application exercises (47%), prerecorded lectures (26%), and iRATs (11%). Sixty-six percent of students liked the iRAT format, while 94% appreciated the ability to discuss tRAT choices with the team members, and 74% felt that RATs and activities prepared them for examinations. Students were also comfortable with the random formation of teams (85%), liked being in the same team for the entire semester (92%), got to know team members well (83%), and felt peer evaluations concentrated on important points (70%) and were fair (77%). Students with preference for a traditional format provided less positive feedback on attributes listed above except for appreciation for discussing the choices of tRATs with team members.

Qualitative analysis of the 244 free-text comments from the entire class was conducted using a summative approach.⁴⁰ Keywords derived from the course components were identified and counted, and then each comment was characterized as positive or negative. One hundred twenty-two comments (50%) were positive, 102 (42%) were negative. The two most common keywords were prerecorded lectures (26%) and iRAT or tRAT (20%). Positive comments included appreciation of the ability to watch the recordings on one’s own time, RATs helping with learning the material, and enjoyment of working in teams. Negative comments included having technical difficulty with the prerecorded materials, having too many RATs, disliking Monday morning classes, and having to purchase a required textbook. Students with preference for a traditional format for content delivery provided 41 comments, 15 (37%) of which were positive and similar in content to the class overall. Negative comments among this group echoed the dislike for Monday morning class and technical difficulties with prerecorded lectures, yet differed in stating prerecorded materials were too long, disliking iRATs, and feeling they lost connection with faculty members.

DISCUSSION

Typically nonprescription drugs/self-care education incorporates all four domains from the Center for the Advancement of Pharmacy Education (CAPE) Outcomes 2013 and 2016 ACPE Standards 1-4 and 25.^9,43 Restructuring the lecture-based course to a TBL format helped faculty members enhance foundational knowledge, approach to practice and care, and personal and professional development domains/standards. The new format allowed faculty members to concentrate on knowledge application, problem-based learning, and development of critical-thinking skills, as well as increasing self-awareness and professional development through team work. The innovation also allowed increasing the number of formative assessments.

The large number of students in our study (286) has not been captured in the literature. Similar to other large classes that converted from lecture delivery to small group format,^18,33,34 this course was well received based on students’ feedback and performance. The majority reported to have an increase in course interest, retention of knowledge, andpreparation for summative assessments and future practice experiences. What differentiated our experience was the successful transition in a very large class of 305 students. Our students enjoyed working in groups in class (especially on tRATs) and recognized the significance of team vs individual performance, consistent with previously reported literature.^14,16 In their comments, students echoed findings from previous literature about enjoying the new format, learning the material, and working with teams.^14,16 Some negative comments were related to the timing of the lecture on Monday mornings and the need for a required textbook rather than content delivery format. Students reported feeling they spent more time preparing for class, which is indicative of increased engagement but is often underappreciated by busy pharmacy students. The adjustment to a new format and understanding the benefits of active learning to their professional development took time, though we discussed this with students at the start of the course. It may be difficult for some students to change the way they approach learning if most of their courses rely on lecture-based format.^2,44

An important finding was that a subset of our students missed immediate responses and connection to instructors as a result of pre-class preparation and had a distracting surrounding environment that was not conducive for lecture viewing. Unfortunately, at this point we cannot describe a “typical” student who prefers a traditional lecture-based delivery—statistically, there were no differences in terms of age, gender, incoming GPA ranges, or final performance among different content delivery preferences, but there was a clear difference in perceptions of course redesign and usefulness of various elements. We contemplated whether students with preference for a traditional format might have busier lives (ie, children, outside employment) and would, therefore, be interested in completing more work in class. Exploring reasons for preference would be potentially helpful follow-up research. Additional areas of research that would benefit faculty members who choose to deliver their courses in this format include long-term retention in different pedagogical formats, ways of enhancing student satisfaction during out-of-class preparation, improving teamwork and team creation, and factors affecting student performance.

In the future, we plan to spend more time discussing with students the importance of creating an environment appropriate for studying. In addition to contacting faculty members electronically or in person, attending their office hours, or posting questions via Facebook group, we are currently creating more effective summaries for delivery at the end of in-class sessions to prompt live questions. We are also reevaluating our team formation and team evaluation strategies, as well as ways to enhance student participation during in-class application activities. We initially felt that student peer evaluations of teamwork would help keep students engaged during in-class application activities. However, we felt that some students did not take application activities as seriously as graded components of the course. Thus, we are considering introducing a participation grade for this part of the course to enhance student engagement and increase the value of at home preparation. We will continue to search for ways to entice more students to complete their homework prior to coming to class. Exploration of additional in-class activities and exercises should be given a priority even in courses that do not use team-based learning format.

After the first time the course was offered in a redesigned format, only 30% of students had a strong preference for learning via flipped team-based learning format. To be fully accepted by all students, it will take a culture change (probably through delivery of multiple courses in the same format). Such change will require a concerted effort on the part of faculty members and administration to develop new content and assignments.

Four areas of attention for future course offerings that would help it appeal to more students with diverse learning styles and content delivery preferences include encouraging students to create environments conducive to learning while reviewing online materials, highlighting multiple ways of accessing faculty members during the course, enhancing logistics of teamwork (formation and evaluation), and incorporating participation grade for application exercises. Further research on connection among learning styles, personality characteristics, and performance in flipped TBL classrooms will help with unanswered questions. We strongly believe that the success of curricular modification lies in faculty collaboration, acceptance of the overall pedagogical concept, and good planning. We were fortunate to have all three elements to accomplish this significant course revision.

Limitations of this research include that it was performed at one school of pharmacy and the results cannot be extrapolated to other schools. Students received bonus points for completing the survey.

SUMMARY

A flipped TBL format was positively received by our PY3 students, especially those who identified a preference for flipped TBL or mixed content delivery style. Those who appreciated the elements of the restructured course had fewer challenges and performance better academically. A minority of students with a preference for traditional teaching format appeared not to enjoy the flipped TBL format to the same extent. Even though these students had unique opinions about the course, their objective performance did not differ from others. Currently, the course described is the only large required course in the school of pharmacy that has adopted the flipped team-based learning model. As faculty members are becoming comfortable delivering content in this format and engaging students during in-class activities, they are also beginning to share their experiences and enthusiasm with colleagues internally, planting a seed for the future curricular changes.