Abstract
Objective. To determine how medical literature evaluation (MLE) is being taught across the United States and to summarize methods for teaching and assessing MLE.
Methods. An 18-question survey was administered to faculty members whose primary responsibility was teaching MLE at schools and colleges of pharmacy.
Results. Responses were received from 90 (71%) US schools of pharmacy. The most common method of integrating MLE into the curriculum was as a stand-alone course (49%). The most common placement was during the second professional year (43%) or integrated throughout the curriculum (25%). The majority (77%) of schools used a team-based approach. The use of active-learning strategies was common as was the use of multiple methods of evaluation. Responses varied regarding what role the course director played in incorporating MLE into advanced pharmacy practice experiences (APPEs).
Conclusion. There is a trend toward incorporating MLE education components throughout the pre-APPE curriculum and placement of literature review/evaluation exercises into therapeutics practice skills laboratories to help students see how this skill integrates into other patient care skills. Several pre-APPE educational standards for MLE education exist, including journal club activities, a team-based approach to teaching and evaluation, and use of active-learning techniques.
INTRODUCTION
The ability to competently retrieve, evaluate, summarize, and interpret medical literature is a key skill for pharmacists. Student competence in this skill is required by the Accreditation Council for Pharmacy Education (ACPE) Standards 2016,1,2 and is identified as foundational knowledge by the Center for Advancement in Pharmacy Education (CAPE) 2013 Educational Outcomes.3 Medical literature evaluation skills components are included in several drug information education core concepts defined by the American College of Clinical Pharmacy (ACCP) Drug Information Practice and Research Network: critically evaluating medical information, distinguishing statistical vs clinical significance, and summarizing basic biostatistics and research design.4 Medical literature evaluation (MLE) can be defined as reading, comprehending, summarizing, and critiquing the medical literature. Most pharmacy professional organizations consider competence an essential entry-level skill.5-7
The invitational 2013 ACPE Consensus Conference of Advancing Quality in Pharmacy Education was designed to advise ACPE on how their accreditation standards and guidelines reflected current and future practice requirements of pharmacists.8 Attendees at this conference ranked 44 different pharmacist competencies according to impact (alignment with current and future skills needed by pharmacists) and feasibility (ease of implementation). Of the recommendations rated “high impact-high feasibility,” the top-ranked recommendation was that ACPE standards should not require a student research project but instead should place more emphasis on development of literature evaluation and interpretation skills.9
Previously published studies using survey methodology examined the drug information curriculum at schools of pharmacy in 2003 and 2004,10,11 but did not specifically identify how and when MLE skills were taught. A more recent evaluation of drug information curricula examined some aspects of MLE curricula, but not in depth.12 In this survey, most respondents (66%) noted an increased focus on evidence-based medicine in their curriculum within the previous 5 years.
Innovations in teaching MLE include incorporating active-learning strategies into lecture-based courses,13-15 providing course lecture material online rather than in the classroom,16 using rubrics to aid students in assessment of a given piece of literature,17,18 and requiring journal club participation during the advanced pharmacy practice experience (APPE) year.19 Innovations in assessment of student ability to master literature evaluation include repeated testing embedded within and throughout the curriculum,20 and assessment partway through the APPE year to measure long-term retention of previously-taught material.21 Distinguishing the standard of practice for MLE instruction and skill-building may give faculty members teaching the subject evidence on which to base curricular design.22 The purpose of this study was to determine how MLE is being taught across the United States and identify a variety of methods being used for teaching and assessing this skill.
METHODS
This study was designed as a cross-sectional analysis of data obtained from MLE instructors at US schools of pharmacy. An 18-question web-based survey instrument was developed and pilot tested by MLE faculty members at two different schools in the Pacific Northwest to refine questions, clarify wording, and estimate completion time. Participants were invited to answer all questions on the survey instrument and all responses were accepted. Three responses were open-ended to allow for thematic analysis. In addition to institutional demographic information regarding public vs private institutions and average class size, respondents were asked about where components of MLE were taught in their curriculum, who made up the team members (if the course was team-taught), and what active-learning strategies were used. The study protocol and survey was examined by the University of Washington Human Subjects Division and approved with exempt status.
In December 2013 and January 2014, attempts were made (using American Association of Colleges of Pharmacy (AACP) member data, institution directories, and networking with experiential education directors) to identify a single faculty member (to avoid duplication of responses) at each school whose primary responsibility was teaching MLE. Candidates were contacted via e-mail to confirm that they were the best person at the institution to respond to the survey. In late January 2014, all identified faculty members (n=120) were sent an e-mail announcement about the survey. One week later, the same individuals were sent an invitation and a link to the survey, with a reminder invitation sent one week afterward. The survey was open to participants for two weeks.
Most data were analyzed using descriptive statistics. If respondents stated their curriculum or teaching method was in the process of changing at the time of the survey, then the data were coded to reflect to the new curriculum. Data from the survey were compared to population data obtained from AACP enrollment data for the 2013/2014 academic year and the ACPE accreditation history for schools of pharmacy.23,24
Answers to the three open-ended questions were used to clarify and analyze answers to list questions. The open-ended questions also underwent analysis for theme identification.25 Using an iterative process, two of the authors independently identified common themes in the respondent statements, reconciled differences, improved theme descriptions, identified trigger words, and developed coding descriptions. A third author acted as verifying coder.26 Cohen’s kappa was used to determine the level of agreement between the 2 theme coders and verification coder. A kappa greater than 0.6 was considered satisfactory agreement in this analysis.27 One-way analysis of variance (ANOVA) was used to examine potential associations between the independent variables of class size and length of time instructor had been teaching the course, with the extent of active learning and method of assessment. All statistical analyses were done using R, v3.1.1 (R Foundation for Statistical Computing, Vienna, Austria).28
RESULTS
Ninety out of 127 schools (71%) responded to the survey. Contacts at seven schools could not be identified despite multiple communication attempts and were excluded from the survey invitation. Institutional data as reported by respondents are compared to data from the known population in Table 1. Two respondents did not indicate program length. Eighty-eight (98%) answered the question regarding how many years ago their school admitted its first class of students, with 15 (17%) institutions admitting their first class 0-5 years, 14 (16%) institutions admitting their first class 5-15 years, and 59 (67%) of institutions admitting their first class more than 15 years prior to the survey date. Twenty-one respondents indicated their program had a branch campus out of 29 such institutions nationally.24
Demographic Comparison of Responses with Known Population
Of the 90 respondents, most had doctoral degrees, with 72 (80%) of those in pharmacy (PharmD), 10 (11%) in research (PhD), two with both a PhD and PharmD degree, one with a PharmD and veterinary medicine degree (DVM), and one with a public health (DPH) degree. Two respondents had a master-level degree and 2 respondents a bachelor of pharmacy as their highest degree earned. Sixty-five (72%) of the respondents had completed a residency, of which 29 were drug information residencies or fellowships; 13 respondents completed 2 years of residency. Thirteen respondents had completed research fellowships. Eighty-eight (98%) answered the question regarding how many years they had been in charge of teaching medical literature evaluation at the institution—answers ranged from 0.2 to 40 years, with a mean of 8.6 years. Forty-three (49%) respondents had been in charge of the subject for 5 or fewer years, with 16 (18%), 13 (15%), 8 (9%), and 8 (9%) respondents indicating they had been in charge of the subject for 6-10, 11-15, 16-20, and more than 20 years, respectively.
Medical literature evaluation course placement in the curricula is outlined in Table 2. Thirteen respondents (14% of programs) reported the subject being taught throughout the pre-APPE curriculum. Of the 90 respondents, 44 (49%) appeared to teach the subject through a stand-alone course, while 33 (37%) indicated the course was embedded primarily in a drug information course, and 10 (11%) appeared to primarily teach the subject through a laboratory-based course. Forty-four (49%) of the respondents indicated that biostatistics was primarily taught in the same course as MLE. Answers from three respondents did not provide enough description to identify how the course was primarily taught.
Placement of Medical Literature Evaluation in Curricula
There were 85 responses to the request, “Describe the sequence of your medical literature evaluation curriculum.” Seventy responses (82%) described a more traditional format where drug information, biostatistics, research methods, and/or literature evaluation were taught as discrete subject areas at some point in the curriculum. Sixteen respondents (19%) described a progression where MLE concepts were primarily taught in a separate course after completion of coursework in drug information, research methods, and biostatistics. In 25 descriptions (29%) it appeared that MLE concepts were integrated throughout most of the curriculum (eg, three years out of a 4-year curriculum). Thirteen respondents (15%) identified specifically that MLE exercises had been integrated into modules within a therapeutics or pharmacy practice skills laboratory. Two respondents indicated that specific topics (eg, searching, meta-analysis) were taught as discrete modules or workshops at some point in the curriculum. Six respondents indicated that they were in the process of curricular change: four moving toward a more integrated curriculum, one changing the course location within the curriculum, and one offering a new elective course in the area. Six respondents indicated that they offered elective coursework in topics such as research design/advanced methods and journal club. Kappas for the curriculum theme analysis between coders and verifier ranged from 0.67 to 0.82.
Respondents from 21 programs (23%) indicated that MLE was taught by a single individual, and respondents from 69 programs (77%) indicated that the subject was team-taught at their institution. The subject was taught by at least one school-based faculty member in all but two institutions, where a volunteer faculty member was in charge of the coursework. Twenty-seven of the 69 team-based respondents (39%) indicated they use nonschool-based faculty members in the course, with 11 respondents listing volunteer faculty members, and 16 listing pharmacy residents; other team members included PhD students, fellows, librarians, and school of medicine faculty members.
Sixty-six team-taught respondents provided a description of the team structure and member roles. The thematic analysis for these responses focused on identification of broad categories for team members. Thirty-one respondents (47%) described a team of 2-4 individuals (n = 27) or an unknown number of individuals (n=4) who individually taught distinct components (eg, biostatistics, epidemiology) of the course; in this theme there was no mention of guest lecturers. Thirteen respondents (20%) indicated that the majority of the content was taught or coordinated by 1-2 team members (n=11) or an unknown number of team members (n=2) with some content taught by guest lecturers. Eight respondents (12%) described a model where virtually all material was taught by one or two individuals, with other team members used to facilitate small group learning sessions. The remaining 14 respondents (21%) did not provide enough information to determine how many team members there were nor the roles played by those team members. Twenty-two respondents (33%) reported break-out or small group sessions facilitated by team members. Six respondents (9%) specifically mentioned participation in the course by drug information service pharmacists, residents, or fellows. Kappas for the team analysis themes between coders and verifier ranged from 0.65 to 0.86, with the exception of participation in the course by drug information service personnel, which was 0.49.
Nine of 21 program instructors who did not identify their course as team-taught provided information indicating that other courses helped with MLE skill building. Specifically, four respondents noted that students did MLE skill-building exercises in a corresponding therapeutics or pharmacy practice laboratory. Thus, even though these nine respondents did not perceive themselves as part of a team, some did rely on faculty members in other classes to aid in student MLE skill-building.
All 90 respondents answered the question, “What percent of class time is devoted to active-learning strategies?” The response frequency is listed in Table 3. There was no correlation between the extent of active learning used and the class size (p=0.92), the number of years the instructor had taught the course (p=0.94), or the number of years since the school had admitted the first class (p=0.27).
Use of Active-Learning Methods
Regarding what active-learning methods were used to engage students in evaluating the medical literature, only four respondents did not list any active-learning methods, and a fifth respondent indicated active learning was used but did not list specific methods. Thirty-seven (41%) respondents mentioned some exercise involving student small groups or teams. Twenty-five (28%) respondents listed problems, activities, exercises, assignments, or discussions, with no further elaboration of what these entailed. Sixteen (18%) respondents described in-class writing activities or verbal responses to questions posed by the presenter or during breakout sessions. Active-learning methods mentioned by more than one respondent are listed in Table 4. Active-learning methods listed by single respondents include development of concept maps, information mastery hunting tool exercise, buzz groups, brainstorming, brain dump, use of de Bono’s 6 Hats Theory to dissect studies,29 process-oriented guided inquiry, teach-back, muddiest point, Jeopardy, plagiarism/improper citing activity, required visit to a drug information center, and a “mythbusters” assignment. Description of many of these learning methods are provided in Appendix 1.
Active Learning Methods
Course-related resources used (multiple responses to this question were allowed) included a physical textbook in 35 (39%) programs, an electronic textbook in 41 (46%) programs, a published review article or series of articles in 57 (63%) programs, and a course packet developed by the faculty member in 52 (58%) programs. Additional resources included primary literature, noted by nine respondents, and web-based resources noted by four respondents, with one respondent indicating self-made videos loaded onto YouTube. Two respondents also noted use of slide sets as resources.
Assessment methods listed by respondents are outlined in Table 5. There was no association between the type of assessment used and either the class size (p=0.67) or the number of years the instructor had been teaching the course (p=0.70).
Method of Evaluating Students
Eighty-nine responses were made to the question, “What has been your role or your MLE team’s role in advanced pharmacy practice experience (APPE) activities that involve medical literature evaluation?” Twenty-two respondents (24%) indicated they had not played such a role, while 27 respondents (30%) had worked with the APPE director to design one or more APPE activities. Eight respondents (9%) helped evaluate evidence-based projects submitted as part of the student’s APPE portfolio. Forty-five (50%) respondents precepted students in an APPE. Other activities included providing feedback or advice to students or preceptors about literature-related APPE assignments and providing tools or templates for students and preceptors to use while completing experiential coursework. One respondent had a longitudinal drug information APPE in which students developed and presented a 30-minute seminar.
DISCUSSION
Cole and Berensen found that medical literature evaluation was a major topic in about 70% of didactic drug information courses.11 In our study, all contacted pharmacy programs identified a course administrator for this topic, indicating that in the 10 years between the two surveys, MLE has become a major topic taught in all schools. The most common model for MLE in US schools of pharmacy curricula is as a stand-alone course (49%) taught in the second professional year (43%) in 4-year programs. These results differ slightly from a survey of drug information education conducted by Phillips and colleagues, which found that teaching literature evaluation as part of a combination course with drug information and/or biostatistics was more common than teaching it as a stand-alone course.12 Phillips and colleagues also found that teaching literature evaluation in the second and third year of the curriculum was the most common placement. The differences between the previous and current studies may be reflective of recent curricular changes. Wang and colleagues found that 10% of programs integrated drug information activities such as MLE throughout the professional program.10
In this study, 14% of respondents indicated in a “check all that apply” box that MLE concepts were taught throughout the pre-APPE curriculum, and 27% indicated teaching MLE. for at least 2 years out of a 3-year pre-APPE program. In the open-text curriculum responses, 29% described a curriculum consistent with integration of MLE throughout the pre-APPE curriculum. These numbers may be lower than the reality of how most programs’ MLE curricula are structured. The basics of study structure, classification of medical literature, and introduction to the tertiary medical literature are frequently introduced prior to a formal MLE class. The core MLE skills of how to read, process, and evaluate primary literature are acquired prior to or concurrent with the early therapeutics curriculum. The therapeutics curriculum ideally will facilitate MLE skills application, so placement of MLE in the overall curriculum will be affected by each institution’s placement of therapeutics. Although there are no comparative studies analyzing MLE placement in the curriculum, repetitive exposure to this topic may enhance student knowledge and confidence. Martin and colleagues concluded that integrating evidence-based decision making content into three years of a 4-year professional pharmacy program was successful in increasing students’ knowledge, self-reported skills, and confidence.20
In Wang and colleagues’ 2003 survey, 33% of respondents reported laboratory or recitation sections with journal club activities.10 Ten years later, our study’s survey results showed 68% of respondents were conducting pre-APPE journal club activities, indicating that use of journal clubs has become an educational standard of practice. Data from one study showed that students who had completed journal club simulations and other active-learning activities were perceived by their preceptors to have stronger skills in applying evidence-based learning to patient care compared with students who did not have this experience.30
Our study showed that the majority (77%) of programs used a team-based approach to teaching MLE, slightly higher than the 63% of drug information didactic coursework that Wang and colleagues found was team-taught.10 Advantages to using a team-taught approach include the use of multiple experts on different topics as well as their different perspectives, the ability to distribute the teaching load among the team, and the opportunity for graduate students, residents, and/or fellows to develop their teaching skills in this area.31 Potential disadvantages include differences in teaching and communication styles among team members, which forces students to adjust to varying expectations of different instructors.32
Results from our study suggest that most schools are using a progressive approach to teaching MLE. Active-learning techniques in the form of journal clubs, interactive polling, and a variety of other methods appear to be the norm, rather than the exception. The number of programs who have integrated MLE into multiple courses, particularly laboratory coursework where students apply literature to specific patient cases, seems to be a good approach to preparing students for APPEs. When evaluating students, most respondents (87%) relied on multiple measures of student performance, including performance in active-learning exercises. Active learning increases undergraduate student learning in comparison to traditional lecture-based learning.33 Medical literature evaluation is a subject where active learning can be used effectively.34
About half of the MLE course administrators in this study were relatively new to this role, while half were experienced. Being newer to teaching was not associated with either increased or decreased use of active-learning methods or assessments. Course administrators new to this role can anticipate being as successful as their more experienced colleagues but are encouraged to use active-learning methods and assessments and to form a teaching team where possible, since this appears to be the standard of practice for teaching this subject.
Respondents varied in the extent to which they had played a role in incorporating MLE into APPEs. Nearly a quarter of respondents indicated that they had not played any role in advancing these skills during APPEs. It is likely that most MLE course administrators are aware of MLE activities required in APPEs, since 50% of respondents indicated they precept an APPE, but few appeared to be actively evaluating student APPE performance in MLE-related activities. Mandatory journal clubs during APPE experiences are valuable in development of MLE skills,19 and evaluation of performance in such clubs could provide MLE course administrators with useful data about student learning in the pre-APPE MLE program. Measuring student APPE performance in MLE activities could also help students identify areas for improvement, which they might otherwise fail to identify; moreover, in their study of pharmacy residents’ biostatistics and literature evaluation knowledge, Bookstayer and colleagues found that residents’ confidence in their knowledge exceeded their examination performance.35
To the authors’ knowledge, this is the first study to specifically examine MLE curricula in US schools of pharmacy. The described curricula were similar in scope to curricula of evidence-based medicine in medical schools in the United States, Canada, and the United Kingdom,36,37 although medical schools may include more formal education in the practice setting than pharmacy education.38,39 A formal analysis of student MLE activities during APPEs should be conducted to allow measurement of evidence-based practice,40 an important educational outcome.
There are some limitations to this study. It represents a single snapshot of the MLE curricular structure and teaching methodologies in US institutions. Several programs indicated they were in the process of or considering changing their MLE curriculum. The survey did not collect data allowing characterization of how MLE course administrators at schools with branch campuses were teaching this subject. The survey did not specifically ask about topics covered in the MLE coursework. The survey questions were difficult to answer by administrators of courses with an atypical structure (such as the course structure used by the primary author), so these data were not captured well.
CONCLUSION
Medical literature evaluation is now incorporated as a core subject in all US pharmacy programs. This subject is primarily team-taught. Active learning methods are widely used, with 50% of programs reporting that 40% or more of the class is taught via active learning. The most commonly used active-learning method is article presentation at journal club, with or without guided discussion by a team member. Student performance during active learning exercises is included in determination of the student grade. Although frequently acting as preceptors for APPEs, MLE administrators and instructors are not highly involved in evaluating student performance on MLE-related required APPE exercises. Measurement of MLE activities and student performance in APPEs would provide useful educational outcomes data to aid in curricular program assessment.
ACKNOWLEDGMENTS
The authors would like to acknowledge the assistance of William Troutman, PharmD, in the development of the survey questions; Sabrina Cole, PharmD, for advice about study design; Donal O’Sullivan, PhD, in the statistical analysis of the data; and all of the MLE faculty members who responded to the survey.
Appendix
Active Learning Strategies29,41,42
- Received November 7, 2014.
- Accepted January 21, 2015.
- © 2016 American Association of Colleges of Pharmacy
