Use of an Online Spaced-Education Game to Study Top 200 Drugs in a Skills Laboratory Course

INTRODUCTION

The 2016 Accreditation Council for Pharmacy Education (ACPE) Standards emphasize the development of “practice-ready” and “team-ready” graduates who can thrive in the evolving landscape of pharmacy practice.¹ Foundational knowledge of commonly used medications, formulations, and drug products is listed as a key competency (Standard 1.1) and is expected of practicing pharmacists as the “medication expert” within the health care team.^1,2 Students should be able to demonstrate knowledge of brand and generic names, dosage forms, usual dosing ranges, and the mechanism of action of the top 200 medications.² Top 200 drug information is defined as knowledge of the aforementioned facts in addition to common indications and uses, drug classification(s), and patient counseling points. In addition to the key elements of foundational knowledge and the ability to provide patient-centered care, programs are encouraged to include opportunities for student independent study to foster the skills, attitudes, and values necessary for self-directed lifelong learning.² Developing self-directed learning skills in drug information to stay current with commonly used medications is an important skill to foster in the curriculum.

While instruction of top 200 drug information is an essential piece of pharmacy curricula, there are few published studies evaluating methods for teaching and assessing this knowledge.^3-6 This content is frequently taught in laboratory or recitation components of pharmacy curricula.⁷ Wilson and Scolaro assessed fourth-year student perceptions of top 200 drug information activities in a pharmaceutical care laboratory course series.³ During the course series, students were responsible for compiling information on brand/generic names, therapeutic classification, and Food and Drug Administration-approved indications. Many students used study tables or flashcards to compile and review information. Overall, 93% of students felt the exercises for learning top 200 drug information were useful, with the self-directed learning activities (eg, flashcards and review tables) ranked highest in their perceived usefulness. While students felt this content prepared them for rotations, they mentioned they had difficulty retaining the drug information.³

Methods to improve long-term retention of learned information has been extensively studied in psychological and educational literature. Studies have manipulated the timing of repeated exposures to information and then measured the resulting effects on memory retention.⁸ These studies find that learning is better when two or more exposures to information are separated in time (ie, spaced apart) than when the same number of exposures occur back-to-back in immediate succession.⁸ The learning advantage for information that is repeated in a “spaced” fashion is commonly referred to as the spacing effect.⁸ This is contrast to “massed practice” when information learned is grouped or concentrated with no delay between study sessions.⁸ The best example of massed practice is a student cramming prior to an examination. The student may perform well on that examination due to short-term retention of knowledge, however, when that same material is tested in the future, students tend to perform poorly because they did not retain the information long-term.

The spacing effect has been applied to health professions education, primarily in undergraduate and graduate medical education.^9-12 Kerfoot and colleagues reported on a novel online educational game that combines game mechanics with an evidence-based form of online education termed “spaced education (SE).”¹² Based upon two psychology research findings (the spacing and testing effects), SE delivered in an online format has been shown to improve knowledge acquisition, boost learning retention up to two years, and durably improve clinical behavior.¹² The SE game is delivered using periodic emails containing clinical case scenarios with multiple-choice questions.¹² When students submit an answer, they receive immediate feedback on the correct answer choice and an explanation.¹² The material is re-presented in a cycled pattern depending on the participant’s performance on a question, which allows for an individual pattern of SE reinforcement through adaptive game mechanics.¹² Online SE has also been successfully adapted as an educational game for teaching and assessing core content for medical students and residents.^12,13

To date, no published studies have applied the use of an online SE game to studying top 200 drug information within pharmacy education. The purpose of this study is to describe the use of an online SE game to allow for self-directed studying of top 200 drug information in a first professional year skills laboratory course.

METHODS

The University of Florida College of Pharmacy is a four-year distance-based program with campuses in Gainesville, Jacksonville, and Orlando. In fall 2015, a new curriculum was implemented to better prepare graduates to engage in advanced pharmacy practice and to achieve educational outcomes described in the 2016 ACPE Standards.¹ Along with the implementation of body-system focused courses, a longitudinal Professional Practice Skills Laboratory sequence (six semesters) was established to teach essential pharmacy practice skills in the pre-Advanced Pharmacy Practice Experience (APPE) curriculum. The first year Professional Practice Skills Laboratory (PPSL) course began in the fall of 2015 and consisted of 236 first year (P1) pharmacy students assigned to four 2-hour laboratory sections at the Gainesville campus (n=120), two 2-hour laboratory sections at the Jacksonville campus (n=44), and three 2-hour laboratory sections at the Orlando campus (n=72). Thirty students were enrolled in each laboratory section. Within each laboratory section and throughout the P1 curriculum, students were assigned in teams of 4 to 6 students. The PPSL course in the P1 year is delivered in a blended-learning format. Students view recorded lectures and readings prior to laboratory and are expected to engage in roleplay with team members or assessment of essential pharmacy skills during the laboratory session.

Top 200 drug information was established as a curricular thread to be taught and assessed within the PPSL course series. The level of drug information students are expected to learn builds each year of the curriculum and is assessed at the end of each semester with a 50 to 75 multiple choice question (MCQ) examination (Figure 1). Students were required to purchase Sigler’s Prescription Drug Cards (31^st edition, SFI Medical Publishing) prior to the start of the P1 year to help them review top 200 drug information. A list of approximately 100 of the top 200 drugs was provided in a Microsoft Excel (Redmond, WA) sheet on the course learning management system and directed students which drugs to study for that semester’s examination. Students were informed that learning top 200 drug information was a self-directed component of the laboratory curriculum and that class time in laboratory would not focus on teaching this information.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Top 200 Drug Information Curriculum and Assessment Plan.

Prior to the start of the new curriculum, the course coordinator secured internal award funds to purchase a subscription to the online SE game for use during the PPSL course. Within the online SE game, instructors design “challenges” consisting of MCQs that are delivered to students using periodic emails or app notifications (Figure 2).¹² The program is compatible with various computer platforms (eg, PCs, Macs, tablets) and may be downloaded as an app for iPhone/iPads. Upon submitting the answer, the student is immediately presented with the correct answer and rationale for the correct answer choice. Additional information about the brand/generic name and indication(s) for each answer choice are also provided as feedback to the student to reinforce learning of top 200 drug information. The number of questions presented to the student in each email, frequency of delivery (eg, every X days), number of attempts to retire a question, and re-delivery of questions after correct or incorrect answers can be adjusted by the instructor. The online SE game also allows for competition between students by displaying how the class responds to a question and tracks points for both individual performance and team performance.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Online SE Game Mechanics.

In September 2015, odd-numbered teams (n=122) in PPSL I were invited to participate in a voluntary challenge focusing on the cardiovascular drugs included in the assigned top 200 drug list. Even-numbered teams (n=114) in PPSL II were invited to a challenge focusing on neurological drugs included in the assigned top 200 drug list in spring 2016. Each challenge using the online SE game consisted of 15 MCQs on brand/generic names, therapeutic class, and indications for the assigned drugs. Points were awarded for correct answers and additional bonus points were awarded for the speed of answering a question (within a 30-second time limit). To protect student identities during the challenge and to foster team competition, aliases were assigned to both the individual and team using an adjective and animal naming convention (eg, team name = sea lion, individual name = energetic sea lion). Leaderboards displayed top team and individual participants using the assigned alias. The top overall individual and team finisher were recognized by course faculty with a small prize (monetary value approximately $20) at the end of the challenge each semester. Participation in either the fall or spring challenge was voluntary and did not contribute to the course grade. No additional incentives (eg, bonus points, dropped quizzes) were provided for participation.

The online SE game software provides reports detailed student and cohort performance to the instructor at the conclusion of the challenge. Cohort proficiency provides the percentage of students answering the initial presentation of a challenge question correctly (initial % correct), the percentage of students answering the current or most recent attempt correctly (current % correct), and the percentage improvement between the initial and current question attempt (% improvement). Student engagement in the game is also reported and classifies students as very active (responded to questions within 2 business days), active (respond to questions within 3 to 7 business days), or not started (never attempted challenge questions). The engagement report also provides the number of students who finished the challenge in its entirety (ie, the student was able to retire each question).

Following conclusion of the challenge each semester, students were invited to complete a voluntary online 16-item survey to evaluate changes in self-efficacy with top 200 drug information learning objectives in a retrospective pretest-posttest design (Appendix 1). A retrospective pretest-posttest design was chosen to minimize response shift bias which can occur with traditional pretest-posttest survey design.¹⁴ Other survey questions evaluated student perceptions of the online SE game to study top 200 drug information and overall study habits for top 200 drug information in the course. Students were invited to complete the survey via an online course announcement at the end of the semester. No incentives were offered for completing the survey.

All students in PPSL I and PPSL II completed a multiple-choice examination on top 200 drug information at the end of the fall 2015 and spring 2016 semesters, respectively. Question categories and distribution on the examination were dictated by the number of representative agents on the top 200 drug list provided each semester. The fall 2015 examination contained 50 MCQs on 97 of the top 200 drugs. Cardiovascular agents consisted of 50% of the examination (25 questions). The remainder of the examination consisted of respiratory (14%, 7 questions), pain (23%, 12 questions), topical (4%, 2 questions), women’s health (4%, 2 questions), and miscellaneous agents (4%, 2 questions). The spring 2016 examination contained 75 MCQs on the remaining 103 medications on the top 200 drug list. Cardiovascular agents (33%, 25 questions) were placed on the examination to evaluate long-term retention of this material. Neurological agents consisted of 50% of the examination (25 questions) along with endocrine (15%, 8 questions), gastrointestinal (15%, 7 questions), and infectious disease (20%, 10 questions). The examination was delivered via ExamSoft (2016, SofTest v11.0, Dallas, TX). A minimum cut score of 80% on the examination was required for students to pass the top 200 drug information curricular milestone. Students not meeting the 80% cut score were permitted to take the examination a second time.

Difference in top 200 drug examination scores between online SE game assignment (fall or spring semester), including performance on the cardiovascular agent questions, were evaluated using the Mann-Whitney U test. Differences in fall and spring examination scores across student levels of engagement (very active, active, not started) were evaluated using the Kruskall-Wallis test. Differences in student engagement levels by assigned campus were evaluated using the Chi-square test. Descriptive statistics were used to describe cohort performance with the challenge questions for each semester (initial % correct, current % correct, and % improvement). Comparison of initial % correct to current % correct were compared using McNemar’s test. Changes in self-efficacy ratings for each top 200 drug information learning objective were compared using the Wilcoxon signed rank test. For all analyses, differences were considered significant if p<.05. Kolmogorov-Smirnov tests were conducted to determine data distribution for all outcomes analyzed. For all tests analyzed, the results of the Kolmogorov-Smirnov tests indicated that the distribution of all outcome variables did not meet the normal distribution assumption (p<.001). Therefore, non-parametric tests were used with the exception of outcomes with binomial data (comparison of initial % correct to current % correct and engagement data across campuses). This study was approved by the University of Florida Institutional Review Board as exempt from review.

RESULTS

There were 122 enrolled in the online SE game in fall 2015. Twenty percent (n=24) of students were very active and completed the challenge. Sixty-three percent (n=77) were active, and 17% (n=21) did not engage in the challenge at all. Overall, those who engaged in the online SE game (n=101) completed 2404 question attempts and demonstrated a mean 12% improvement between the initial attempt at challenge questions (72%) and the current attempt challenge questions (84%) (Table 1). There was a significant difference in the percentage of participants who answered the question correctly on the current attempt vs the initial attempt for questions 1 to 9 in the fall challenge indicating gains in knowledge between the first and current attempt at a question (Table 1).

There were 114 students enrolled in the spring 2016 online SE game. Twenty-eight percent (n=32) of students were very active and completed the challenge and 45% (n=52) were active. There was a higher percentage of students (27%, n=30) who did not engage in the online SE game in the spring vs the fall. Overall, those who engaged in the online SE game (n=84) completed 1951 question attempts. There was a smaller mean improvement (5%) between the initial attempt at challenge questions (80%) vs the current attempt at challenge questions (85%) (Table 1). Only one question (question 11) focusing on attention-deficit hyperactivity disorder (ADHD) medications had a significant difference in the percentage of participants answering the question correctly on the current attempt vs the initial attempt (p=.02) (Table 1).

Students performed well on both the fall and spring top 200 drug examinations. Out of 236 students, 235 completed the fall 2015 examination. One student did not sit for the examination because they withdrew from the program. The mean score on the fall exam was 97%. One student did not successfully pass the examination because they scored below an 80%. The student successfully passed on the second attempt of the examination. Students performed similarly for the spring 2016 examination with a mean exam score of 94%. A higher proportion of students (11 out of 231 students taking the examination) did not meet the passing threshold score of 80% for the spring examination. Similar to the fall semester, each student passed on the second attempt.

When evaluating differences in top 200 drug examination scores across levels of student engagement with the online SE game, the Kruskall-Wallis test results demonstrated a statistically significant difference on fall examination scores across the three engagement levels (χ2=8.8, p=.01), while no significant difference occurred for the spring examination scores (χ2=5.8, p=.06). Follow-up procedure results showed that for fall examination scores, students who were very active in the online SE game vs those who were active (χ2=29.4, p=.01) or had not started the challenge (χ2=30.2, p=.02) performed significantly better on the fall examination scores. However, mean examination scores were high overall across each engagement level (very active mean=98%; active mean=97%; not started mean=96%). There was no difference in which semester the online SE game was assigned (fall or spring) and student performance on the cardiovascular drug question subset on the fall or spring examination (p>.05). Like the overall examination scores, the mean score on the cardiovascular drug question subset (n=25 questions) was high in both the fall (97%) and spring semesters (90%). To determine any difference in levels of engagement by campus, the 3 x 3 crosstab with Pearson chi-square test was conducted. Results showed a significant difference existed across levels of variables (p<.05). However, when follow-up procedures including residual analysis with Bonferroni adjustment were completed, all the p values were larger than the adjusted significance level of .0056, indicating no significant difference in engagement levels by campus.

Forty of the 236 students enrolled in the online SE game completed the voluntary survey (response rate=17%). Two students did not complete all survey questions, therefore the data from 38 students were analyzed to compare self-efficacy before and after use of the online SE game. Sixty percent of students had no (n=24) or very little (0 to 2 years) (n=13, 33%) experience in a community pharmacy setting. Ninety percent of students (n=36) had no history of hospital experience. For all four learning objectives associated with the top 200 drug knowledge, there was a significant (all p<.001) positive change between the pre-test and post-test, indicating that participants’ level of self-efficacy increased after use of the online SE game (Table 2). Sixty percent of students agreed or strongly agreed that the online SE game allowed them to study at times they wouldn’t normally engage in studying the top 200 drug information. Students also agreed or strongly agreed (62%) that use of the online SE game allowed them to study in places or locations they wouldn’t study (eg, while at the grocery store, movies, etc.). Sixty-eight percent of students preferred using the online SE game to the Sigler Drug Cards for studying the top 200 drug information. Eighty-four percent indicated they would like to participate in challenges using the online SE game in other courses. Student comments on aspects of the online SE game they liked most and least were analyzed for themes. Students most enjoyed the question format, immediate feedback on their answers, the competitive aspect of the challenge, and that it made studying fun. Students least enjoyed the low number of questions included in the challenge, lack of question variation (ie, challenge focused on just one drug category), infrequent question delivery, and lack of open access to questions already completed for further study.

In addition to the online SE game, 66% of respondents (n=25) studied using the Sigler Drug Cards 1 to 2 days per week and 64% (n=23) held study sessions lasting 0 to 30 minutes. Fifty-six percent of students (n=20) spaced their study of the assigned drugs each week; however, 25% (n=9) admitted to studying the top 200 drug information the week prior to the examination. Students who did not use the Sigler Drug Cards (17%, n=6) indicated they used other study methods including homemade flashcards (42%, n=14), the Access Pharmacy: Top 200 Drugs Challenge, and other study aids (39%, n=13). Students who selected other study aids most frequently entered “quizlets” as their study method.

DISCUSSION

Modern curricula within pharmacy education should provide teaching and learning methods that engage students, promote self-directed learning, and address diverse learning needs.¹ Despite the prevalence of these themes in pharmacy and health professional education, the best way to promote self-directed learning and improve student engagement in learning is not well-established.¹⁵ The use of active learning techniques and educational games have been suggested as methods to promote self-directed learning and engagement.^16,17 Educational games may be of increased interest among today’s learners, many of whom are digital natives and desire the “unforced learning” opportunities afforded by games within education.¹⁸ Although educational games are effective in engaging students, most studies that have evaluated games within pharmacy education have focused on student perception of gaming activity instead of learning outcomes or examination scores.¹⁷

This study provides an evaluation of whether the use of an online SE game contributed to improvements in top 200 drug examination scores or gains in knowledge in top 200 drug information. While there was a significant improvement in student performance on a majority (9 out of 15) of MCQs in the fall challenge, there was less improvement (1 out of 15) on the MCQs in the spring challenge. The level of student engagement in the fall challenge was significantly associated with performance on the fall top 200 drug examination; however, this finding is of debatable educational significance as mean scores in each engagement group were high (means for very active, active, and not started were equivalent to an “A” in the course grading scale). Differences in the spring top 200 drug examination scores across varying levels of student engagement were not significant, so this effect did not persist throughout the year. Overall, there was a higher percentage of participants who did not engage with the online SE game in the spring (27%) vs the fall (17%) in addition to having less improvement in performance on spring challenge MCQs. Additionally, a higher proportion of students did not successfully pass the top 200 drug examination score threshold of ≥ 80%. It is unknown what factors led to less engagement and poorer performance in the spring semester. It is possible that students at this point in the academic year were struggling with competing demands on their studying time from other courses in the curriculum, several of which were complex science-based courses.

Allocation to the fall online SE game that focused on cardiovascular agents from the top 200 drugs did not correlate with improved performance on either the subset of cardiovascular agent questions on the fall or spring examination. This finding was unexpected as the online SE game format has been associated with improved retention of clinical knowledge in medical students.¹⁹ Overall, performance on both the fall and spring top 200 drug examinations and the subset of cardiovascular questions on each examination was high; therefore, it may have been difficult to discern a difference in performance in those who engaged with the online SE game vs those who did not. The nature of the top 200 drug information knowledge assessed in the P1 curriculum is relatively straightforward (eg, brand/generic name, indication) vs the more complex content tested in published studies using the SE game.^13,19 A potential area of future study may evaluate whether use of the online SE game affects performance or knowledge retention on more advanced pharmacy topics or top 200 drug information in later years of the curriculum.

Student perceptions of the online SE game for studying top 200 drug information were positive. Despite the lack of significant correlations between online SE game use and overall examination performance, students engaged with the online SE game more than expected. This was a voluntary study resource and use was high in both the fall (83% participation) and spring (73% participation) semesters. The use of this online SE game may be preferable to today’s digital natives vs other self-directed learning formats (eg, practice examinations, researching drug information) used for top 200 drug information in pharmacy education.^3-5

A majority (93%) of respondents to the survey indicated little or no pharmacy experience. While the response rate to the survey was low and may not be representative of all students who engaged with the online SE game, it was an unexpected finding. Historically, about 30% to 40% of students admitted to the doctor of pharmacy program have some degree of pharmacy experience, therefore it was expected that respondents would indicate a higher degree of pharmacy work experience. Greene and colleagues reported that students with little or no pharmacy experience had the lowest final grades in a top 200 drug course.⁶ They suggest that course instructors identify students with little or no pre-pharmacy work experience as these students may have a more difficult time with the top 200 drug information content.⁶ It is possible that students who found the most value from using the online SE game and responded to the survey were those with less pharmacy experience. Instructors may consider offering additional study resources for the top 200 drug information to those students who lack pharmacy work experience when entering the curriculum.

From the faculty perspective, the use of the online SE game was a relatively simple program to deploy in the course. Each challenge using the online SE game was designed in approximately 4 hours by one faculty member with use of an existing bank of the top 200 drug information questions written for the course. Once the challenge was initiated, there was minimal oversight by the faculty despite the large class size (n=236) of the course. The amount of faculty time needed to deploy an educational game in a course may be an important factor to consider as the demand on faculty for improving student engagement must be balanced with other responsibilities such as clinical practice, scholarship, and service commitments.¹⁵

The online SE game requires a subscription fee based on the number of users, which may be a limiting factor for some programs that do not have funds available to purchase such educational resources. However, other online programs are available (eg, Memrise app) without a subscription fee and allow for similar game mechanics and student competition. Since the first offering of the Professional Practice Skills Laboratory course in 2015, the online SE game is still used as a study resource for the top 200 drugs. Based on student feedback from the survey and general course feedback, the number and categories of questions provided in the online SE game challenges were increased. Another instructor is planning to use the online SE game to assist students with self-directed study of pharmaceutical calculations, which is another curricular thread with milestone examinations.

There are several limitations to this study. As PPSL I and PPSL II were new courses, no historical information on top 200 drug examination performance is available for comparison to determine if the introduction of the online SE game made any difference in overall class performance on examination scores. Students were also not limited to use of the online SE game to study top 200 drug information. Therefore any effect of the online SE game on challenge or examination performance may be diluted by student exposure to other study resources. The response rate to the self-efficacy and perception survey was low and may not be generalizable to all students who used the online SE game in the course. Finally, given the high overall performance on both the fall and spring top 200 drug examinations, discerning differences in examination performance or long-term retention of drug information using the online SE game may be difficult.

CONCLUSION

The online SE game was a well-received voluntary study resource for top 200 drug information within a P1 skills laboratory course. Despite a high level of student engagement with the online SE game, it did not significantly affect top 200 drug examination performance or retention of top 200 drug information in those who used the game vs those who did not. Students using the online SE game had significantly improved self-efficacy in their ability to recognize brand, generic, and indications for top 200 drug information and preferred the online SE game to other study methods for top 200 drug information. Future studies may evaluate the use of the online SE game with more complex pharmacy topics that allow for greater differentiation of student performance due to the difficulty of the material. Additional studies may consider evaluating differences in student engagement if use of the online SE game was required or determining factors that increase student engagement in use of self-directed educational games as a study resource.