Abstract
Objective. To compare outcomes (grades, resources, and perceptions) from a weekly in-person seminar capstone course (pre-revision group) to an intensive hybrid course design that included a two-day, in-person conference (10- and 25-minute student presentations) and asynchronous seminar skills sessions (post-revision group).
Methods. Students’ scores on seminar presentation rubrics were compared before and after the course revision. Between the groups, we compared resources, such as number of faculty and hours of involvement, and student time away from advanced pharmacy practice experiences (APPEs). We also assessed student and faculty satisfaction and perception. Comparisons between groups were made using statistical tests, and descriptive statistics were used to summarize student performance and survey responses.
Results. The study included 370 students, 205 in the pre-revision group and 165 in the post-revision group. No significant difference was found in mean overall scores for the 25-minute presentation between groups; however, the post-revision group had significantly lower subscores for objectives and slides and significantly higher subscores for critical analysis. The survey was completed by 82% of faculty and 43% of students from the class of 2018. Most students (80%) found all of the asynchronous sessions helpful, and 70.6% preferred the intensive hybrid course format. Compared to the weekly format, all faculty reported student presentations were similar or better in quality and workload was similar or decreased with the intensive hybrid format.
Conclusion. Changing the senior seminar capstone course to an intensive hybrid design reduced faculty workload and decreased student time away from APPEs while maintaining similar presentation grades and quality.
INTRODUCTION
Capstone courses, such as senior seminar courses, are one way to assess students’ higher-level application of critical thinking, problem-solving, educating, communication, and professionalism skills. This type of course, designated to have students use their drug information skills to research a clinical controversy and formally present their recommendations to peers and faculty, has been previously described in the literature.1-9 However, capstone courses can significantly vary in expectations, training, delivery method, and time commitment, especially when they occur during students’ advanced pharmacy practice experience (APPE) year. While the concept of a capstone course may not be new for programs, the literature has not reported the outcomes of delivering a seminar capstone course over a nontraditional time frame, such as via an intensive hybrid course design.
Use of intensive courses have increased, but the evidence comparing these formats to traditional semester formats is lacking.10 The lack of evidence may be due to the term intensive, which could mean having a reduced time, having an accelerated time, or rigorous; the term and definition are crucial.10-12 For example, one study found that an intensive Pharmacotherapy Scholars Program for fourth-year pharmacy students resulted in higher residency placement rates and significant improvements to knowledge and clinical skills; however, this intensive course was a yearlong program that focused on rigor.11 By contrast, intensive or accelerated teaching is defined as courses that are offered in less time or fewer contact hours than normal, and studies have shown that compared to traditional formats, such courses have no difference in outcomes.12,13 In three graduate health science programs, Harwood and colleagues found no significant differences on student performance (final assignments or course grades) and student satisfaction (course evaluation ratings) in an intensive (seven-week) course compared to a traditional semester-long course (15 weeks).10 While this study was conducted with students in health science programs, it did not evaluate Doctor of Pharmacy (PharmD) students specifically.
A senior seminar capstone is an appealing course to adapt into an intensive/reduced contact time format, since it occurs during the APPE year, when programs want to maximize the amount of time spent at an experiential site. Factors such as proximity of APPE rotation sites to campus, faculty availability, and classroom space are impacted during a traditional weekly class structure. While an intensive course structure may impact these factors to a lesser degree, it is important to ensure that the format does not negatively impact students’ learning outcomes. The current study was designed to evaluate the feasibility of transitioning a traditional weekly seminar capstone course into an intensive course primarily involving self-study plus two in-person sessions; the impact of this transition on student performance was also investigated. This study is unique because in addition to assessing the impact of an intensive hybrid course design with two synchronous face-to-face sessions on learning outcomes, it also evaluated the use of a hybrid format that included asynchronous videos. The primary objective of the study was to compare seminar presentation grades before and after the course revision. A second objective was to compare the resources needed before and after the course revision, including faculty and student time, number of faculty, and workload. The third objective was to capture perceptions of the students and faculty participating in the first year of the intensive hybrid course design.
METHODS
Since 2001, the University of Oklahoma College of Pharmacy has required a two-credit-hour senior seminar capstone course for all PharmD students during their APPE. Historically, the course met once weekly for two hours during designated months in the fall and spring semesters (ie, 19 two-hour sessions). During these weekly meetings, students completed five sessions of seminar preparation workshops (ie, presentation skills, writing objectives and assessment questions, creating presentation slides and handouts, refreshing statistical principles, and referencing and citing literature) according to continuing education/continuing professional development guidelines. Students then formally applied these skills and analyzed primary literature to present a clinical controversy to a small peer group and a full-time college of pharmacy faculty member. Each student delivered a 10-minute presentation during one of five weekly sessions in the fall semester and an expanded 25-minute presentation during one of six weekly sessions in the spring semester. Students also participated in two professional skills workshops (eg, curriculum vitae/resume review, interview skills, dining etiquette) in the course, where they received formative feedback about career skills, as well as one session of pharmacy law review.5
Due to a programmatic change at the college that resulted in a distant campus closure and a reduction in faculty in spring 2017, a major change in the course structure was required. In the 2017-2018 academic year, the weekly class sessions were changed to asynchronous self-study during June and July of the fourth professional year using eight videos, which were class recordings from the live delivery in the previous year and online quizzes. This was followed by two intensive days of face-to-face student presentation sessions, with a four-hour session in the fall (August) and an eight-hour session in the spring (January). Students presented to their assigned group of 10-14 students and a full-time college of pharmacy faculty member. Despite these changes, the course retained the same credit hours, content, goals, and activities of the previous weekly course design. To pass the course, students were required to complete all activities, attend all the presentations in their assigned group, provide feedback to their peers, and achieve a grade of 70% on the spring semester 25-minute presentation. Overall, this structural change provided an opportunity to compare the outcomes of this change in course delivery.
A presentation rubric was developed and used for evaluating the 10-minute and 25-minute seminar presentations. The 10-minute presentation evaluation rubric was weighted more heavily on the formatting of the handout and slides, development of objectives, and timing of presentation, whereas the 25-minute presentation rubric included the areas listed above but placed more emphasis on analyzing, critiquing, and applying primary literature. The rubrics have been modified since inception of the course in 2001 but have been consistent in content since 2010. All graders were trained on the rubrics, and interrater reliability was established. A total of 100 points are available on the spring semester presentation rubric, and specific subsections have a designated number of maximum points (Appendix 1). For this study, the overall scores, subsection scores, and percentage of students with a spring presentation grade <70% were compared between the pre– and post-revision groups. The specific subsections selected for comparison were objectives, handout, slides, assessment questions, and critical analysis, because they were heavily emphasized in the seminar preparation sessions.
This study included the academic years of 2015-2017 for the weekly seminar course format (pre–revision group) and 2017-2019 for the hybrid asynchronous and two-day conference format (post-revision group). All students (N=370) enrolled in the course during these time periods were included. The 2019-2020 academic year was not included in the post–revision group because of the COVID-19 pandemic, which required the entire course to be delivered virtually. To determine faculty workload, the number of faculty and hours spent in the classroom grading presentations were calculated for the pre- and post-revision periods. In addition, in an effort to determine time away from APPE activities, the hours students spent in the classroom were also calculated for the pre- and post-revision periods. Travel time from the APPE site to the college of pharmacy was not included in this calculation.
Two locally developed electronic surveys were distributed to the students and faculty involved in the 2017-2018 senior seminar course at the end of the spring semester (ie, the first post-revision year). The student survey (39 questions) assessed the students’ perception of the helpfulness of the seminar preparation videos and quizzes, satisfaction with course design, and preference for course format. The faculty survey (eight questions) assessed their perceptions of presentation quality, course workload, and course format satisfaction. Eighty students from the class of 2018 and 11 faculty graders for the spring presentations (excluding the two course coordinators) were invited to complete the respective survey and were emailed one reminder after two weeks.
Descriptive statistics including mean and standard deviation as well as frequency (percentage) were used to summarize the students’ performance results and their responses to the survey instrument. Overall score and subscale scores were converted to a percentage (0%-100%). Distribution of these variables was reviewed using quantile-quantile (QQ) plots to determine whether data were approximately normal. Differences in scores between pre-revision and post-revision groups were assessed using independent t tests. The percentage of students who failed the presentation was assessed using a chi-square test. The alpha for all statistical tests was set at 0.05. For all statistical analyses, SAS version 9.4 for Windows (SAS Institute Inc) was used. The study received approval from the University of Oklahoma Institutional Review Board.
RESULTS
The pre-revision group included 205 students (n=106 and n=99 students in the class of 2016 and 2017, respectively), and the post-revision group included 165 students (n=80 and n=85 students in the class of 2018 and 2019, respectively). Overall presentation scores were available for all 370 students. There was no difference in the mean overall presentation grade before and after the course revision, 85.5% versus 89.6% (p=.26). Subscores for specific sections of the rubric were not available for all students in the pre-revision group because evaluations from the distant campus were not available in the electronic database; subscores were available for 72 (67.9%) and 70 (70.7%) students in the classes of 2016 and 2017, respectively. However, subscores were available for 100% of the post-revision group. Significant differences in subscores were noted for the objectives, slides, and critical analysis, with the post-revision group scoring lower on objectives and slides and higher on critical analysis. Table 1 includes a breakdown of the overall scores and subsection scores based on graduation year. No differences were noted in the percentage of students that scored <70% and had to remediate the seminar presentation (p=.6). In the pre-revision group, 7.5% and 5.9% had to remediate in the classes of 2016 and 2017, respectively, compared to 2.5% and 7.0% in the post-revision group for the classes of 2018 and 2019, respectively.
Overall Grade and Subsection Grades for 25-Minute Presentation
Student and faculty time commitment differed between the two time periods. For the weekly course design, students had significantly more in-class time, with 38 hours of mandatory attendance (ie, 19 two-hour sessions) compared to 16 hours in the hybrid asynchronous and two-day conference format (p=.003). Although the total number of full-time faculty involved in evaluating presentations was similar (20 in pre-revision group vs 18 in post-revision group), the total time commitment for in-class grading of presentations decreased significantly for the 10-minute presentation session, with 72 hours in the pre-revision group and 36 hours in the post-revision group, p=.001. The total amount of in-class grading time faculty devoted to the 25-minute presentation decreased from 101 to 83 hours, although this was not significant.
A survey response was received from 34 (42.5%) of the 2018 graduates. Figure 1 includes the students’ ratings of helpfulness of the asynchronous seminar preparation videos and corresponding quizzes. Overall, most students rated the preparation videos as helpful. Although most students rated the quizzes as having an “average helpfulness” for reinforcing information learned in the associated video, the students did not perceive them to be as helpful as the videos. When asked about overall satisfaction with the course after the change in the fall semester, the majority (94%) were satisfied, with seven students (20.6%) reporting they were “extremely satisfied,” 11 (32.4%) were “above average satisfied,” and 14 were “average satisfied.” Responses were similar for the course in the spring semester, with 10 students (29.4%) reporting they were “extremely satisfied,” nine (26.5%) were “above average satisfied,” and 13 (38.2%) were “average satisfied.” When asked whether they would have rather attended a weekly seminar course, most students (79.4%) answered no and indicated they would prefer to continue the two-day format.
Student Ratings of Helpfulness of Preparation Videos, Quizzes, and Grading Rubric (n=34).
Nine of the 11 (82%) faculty who graded the spring presentations completed the faculty survey. Two faculty were not involved with grading in the previous course structure. Of those that were involved in the previous course structure, four faculty respondents (57.1%) believed that the two-day course format had a lower workload, and three respondents (42.9%) believed that the workload was no different than in previous years. No faculty respondents reported an increased workload from previous years. Regarding the quality of presentations, responding faculty rated student presentations as being better (14.3%) or similar (85.7%) to previous years, and no respondent considered presentations to be of poorer quality. When asked about preference of format for the seminar course, eight faculty respondents (88.9%) selected the two-day format, and one (11.1%) selected a monthly format. Overall, 22.2% of faculty respondents reported they were “extremely satisfied” with the post-revision fall semester seminar course, 44.4% were “above average satisfied,” and 11.1% were “average satisfied.” Satisfaction for the spring semester was similar, with 22.2% reporting “extremely satisfied” and 66.7% reporting “above average satisfied.”
DISCUSSION
While PharmD programs in the United States vary in curricular design and semester length, a common traditional academic year format is two semesters that are 16 weeks each.14 Intensive courses that are accelerated in time can occur within a traditional four-year PharmD program. While research about the use of intensive courses in traditional pharmacy programs is lacking, studies in health science programs with graduate students have found no significant differences on student performance or satisfaction.10 Our current study with pharmacy students in a professional program using a hybrid approach found similar results. The new intensive hybrid design did not affect the overall course performance (presentation grades); however, there were two significant differences noted in the subscores for objectives and slides. The differences in scores for objectives and slides may have resulted from the pre-revision group having more hands-on time working with faculty on writing objectives and formatting slides. Of note, one additional difference was found: Higher critical analysis scores were found in the post-revision group compared to the pre-revision group. This result may have been due to prior year curricular changes that updated the spring literature evaluation course given to third-year pharmacy students; it was less likely due to the change in delivery of the senior capstone course, as literature evaluation skills are not directly taught in this course. It is possible that adding an additional year of students to either group may have altered these results. Furthermore, it is important to consider that these differences may not be educationally significant, as the mean subsection scores were similar in grade range. There were also no differences in the number of students that failed the presentation (ie, presentation score <70%) in the pre- and post-revision groups. Therefore, for the primary objective, transitioning to an intensive hybrid design for a senior seminar course is feasible, and the structure does not negatively impact students’ overall performance. It is important to note that this is a core PharmD course, but it is primarily performance based. A core course with more didactic instruction may not find similar results related to the transition or performance outcomes. However, this result may be appealing to programs that are hesitant to use an intensive design because of concerns about learning and performance outcomes.
For the second study objective, one reason to shift to an intensive course design is to achieve resource savings related to student and faculty time, and this study demonstrated this outcome. The intensive hybrid design reduced faculty evaluation/grading time for the fall and spring sessions by almost 30% compared to the traditional weekly class design. The faculty hours contributed for in-class grading of the 10-minute presentation had the most significant reduction, with a 50% decrease in hours. This result was due to the ability to run sessions in more rooms simultaneously with fewer breaks between presenters, leading to a more efficient use of time. For the 25-minute presentation, there was only an 18% decrease in the faculty hours contributed to in-class grading of presentations; however, the course coordinators used more faculty to grade, but the grading was limited to either a morning or afternoon session based on availability. This allowed more faculty members to be involved because there was greater flexibility in scheduling. Student time away from APPE sites to attend in-classroom sessions was also reduced from 38 to 16 hours in the post-revision group. These hours did not include travel time from the APPE site to the college of pharmacy. However, the pre-revision group also likely missed more time away from the APPE site because they had to leave early to account for the drive time to the college for 16 different class sessions. Less time away from the rotation site would likely be more appealing to APPE preceptors since it would produce fewer disruptions to students’ APPE schedule. Having students absent from the site for two full days during two specific months may be easier for preceptors to manage than students’ weekly absences for two hours plus travel time each week for the semester.
Finally, for objective three, the intensive hybrid course design for the fall and spring was well received by most students and faculty. This result is important because it revealed that students and faculty perceived that an intensive hybrid course design maintained course outcomes, expectations, and performance. This result is limited by the low student survey response rate. This low response may be due to a lack of incentive to complete the survey, since it was administered after the course was completed. Similarly, over 75% of faculty members who participated in both formats preferred the intensive hybrid course design over weekly meetings. Faculty perceived the workload to be similar or less than in previous years and felt the quality of presentations remained the same.
There are two additional limitations of this study that should be noted. First, this study evaluated a single pharmacy program; student outcomes and acceptance by faculty and students may differ if this capstone course is conducted at another college of pharmacy. However, the data collection occurred over several years with many students and faculty, which should increase the generalizability to other programs. Second, a locally developed rubric was used to evaluate the presentations and has not been validated for use outside of our program. However, interrater reliability was established when this rubric was first used in the course. It is possible that different study results would be found if a different scoring rubric is used.
The intensive hybrid course design provided an ideal structure for virtual learning, which was made necessary during the COVID-19 pandemic. The only major change required was a transition from in-person to online video conferencing (ie, Zoom) on the two meeting days. This allowed for quick changes based on frequently changing epidemiologic data and provides an additional group to compare and assess for future research.
CONCLUSION
Changing the format of a senior capstone seminar course from a weekly meeting to an intensive hybrid design led to a reduction in faculty workload and student time away from APPE rotations, while maintaining a similar quality of presentations. A significant drop was found in the post–revision group's subscores for writing presentation objectives and slide development, so opportunities for hands-on practice of these skills may be needed. Most students and faculty were satisfied with the course design and preferred to continue the new format.
Appendix
Spring Semester 25-Minute Presentation Rubric
- Received October 6, 2021.
- Accepted February 4, 2022.
- © 2023 American Association of Colleges of Pharmacy
REFERENCES
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.