Journal Clubs During Advanced Pharmacy Practice Experiences to Teach Literature-Evaluation Skills

INTRODUCTION

The concept of journal club originates in the medical profession and dates back over 150 years.¹ Linzer describes a journal club as “…a group of individuals who meet regularly to discuss critically the clinical applicability of articles in the current medical journals.”² Given that a journal club provides an excellent forum for keeping healthcare professionals abreast of literature pertaining to their practice, improving understanding of research design and statistics, and teaching critical-thinking skills, it is widely used in medical, nursing, and pharmacy classroom and experiential education settings.^3,4

The Center for the Advancement of Pharmaceutical Education (CAPE) Educational Outcomes require graduating pharmacy students to be able to “critically analyze primary literature with regard to study design, methodology, findings, and conclusions” as well as to “integrate evidence from the literature with clinical expertise and consideration of patient preferences to draw a conclusion.”⁵ To achieve this objective, many pharmacy colleges and schools incorporate journal clubs into their classroom curriculum and advanced pharmacy practice experiences (APPEs). According to a survey by Cole and colleagues, drug-information experiences in pharmacy-degree programs incorporate journal club assignments 90% of the time.⁶ Despite the prevalence of journal clubs in the pharmacy curriculum and their use by colleges and schools of pharmacy, however, little has been published about the use of journal clubs to enhance student learning of literature-evaluation skills, as assessed by pharmacy faculty members.

Over the past decade, assessment of student learning has become an important component of pharmacy education. To meet the accreditation requirements of the Accreditation Council for Pharmacy Education – in particular, Standard 15 (Assessment and Evaluation of Student Learning and Curricular Effectiveness) – colleges and schools are required to collect and use assessment data for program and course improvement to demonstrate institutional effectiveness.⁷ The goal of this requirement is to foster development of “institutional curiosity,” a term that refers to an institution’s interest in exploring assessment.⁸ When an institution develops curiosity about assessment, its faculty members will learn more about what and how students are learning and what methods and experiences are helping them learn.

Several previous studies have documented the educational value of journal clubs, but few have used a comprehensive approach to measure the content and extent of student learning from journal clubs.^9-12 The primary goal of this paper was to provide insight into how institutional curiosity was applied to examine student learning of literature-evaluation skills using journal club competencies during APPEs. A second objective was to explore if student learning of journal-club competencies, as measured by journal club grades, varied across APPEs.

DESIGN

The third-year curriculum at the Arnold and Marie Schwartz College of Pharmacy (AMSCOP) is geared to instruct students in the critical evaluation of literature, in the context of statistical concepts and the application of clinically relevant study results to practice. In order to target a higher domain of learning, students are then expected to apply their literature-evaluation skills during the fourth year by completing 1 journal-club assignment in a required 2.5-week drug-information APPE and 2 journal club assignments in each of the 2 required 5-week internal medicine APPEs. The APPE syllabus outlines that the journal club articles should be drug related or pharmacotherapeutic in nature and that topics may include but do not have to be limited to comparative drug efficacy or effectiveness, adverse reactions/interactions, new drugs/new indications, drug stability with bioavailability and pharmacokinetic issues, and other pharmacotherapeutic, health-promotion, or disease-prevention studies (Appendix 1).

The main learning objective of all 3 journal club assignments is to improve students’ ability to effectively evaluate and communicate findings of an article while taking into consideration the study’s design, statistics, strengths and weaknesses, and its application to patient care. Additionally, journal club affords preceptors an opportunity to teach students about critically reading, interpreting, and reviewing the literature in order to add relevance to the application of evidence-based medicine to patient care.

Given the size of the student body at AMSCOP (approximately 200 students each year), 16 faculty members at 15 sites precept students for the internal medicine APPEs, and 6 faculty members at 2 sites precept students for the drug-information APPEs during the students’ fourth year. Although the learning objective of the journal club assignments is fairly standardized, pharmacy practice faculty members who are placed at different practice settings may use different formats of conducting a journal club. Formats that may be used include group discussions, formal/informal presentations, various lengths of presentations, student or facilitator-selected articles, documentation using PowerPoint (Microsoft, Redmond, WA), outlines, or answering questions. To ensure standardization, all faculty members assess the journal club using the same rubric and assign a grade based on a percentage scale ranging from 0 to 100% (A = 86.5% to 100%, B = 76.5% to 86.4%, C = 66.5% to 76.4%, D = 59.5% to 66.4%, F = 0 to 59%). For the purposes of this study, these grades served as the direct measure of student learning of journal club competencies.

To provide an indirect measure of student learning, a 21-item survey instrument was constructed by the investigators to gain insight into the students’ perceptions of improvement in journal club competencies. Five survey items based on journal club learning objectives were used to measure students’ perceptions of improvements in their understanding study design, statistics, strengths/weaknesses of the article, application to patient care, and presentation skills using a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). The items measuring value were developed from both the AMSCOP journal club assessment criteria and the published literature.^13-17 Using a 4-point Likert scale (1 = not helpful to 4 = very helpful), other items assessed the following: format used for journal club (eg, type, length, documentation required); method of documentation for the journal club assignment (eg, PowerPoint slides, outline, list of responses to questions); which APPE was completed first; student perceptions of whether the drug-information APPE would have prepared them better for journal club; and value of the classroom statistics course completed the year before APPEs. Confidence in ability to conduct journal club in the future was measured using a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). The survey instrument also allowed for open-ended, qualitative comments concerning the value of the journal club experience during APPEs. Prior to administration of the survey instrument, faculty members reviewed all items, including the 5 survey items, for face and content validity and deemed them to be clear and applicable.

Administration of the survey tool was integrated into the annual assessment conducted by the AMSCOP Curriculum Assessment Committee in May 2009. To reduce the possibility of social desirability bias, the survey instrument was distributed by a faculty member who was not involved with journal club assignments. Fourth-year students who attended the assessment day were asked to complete the paper-based, 2-page survey instrument over a 5- to 10-minute period. Students voluntarily provided consent for their survey to be used for research purposes and were assured that their responses would remain anonymous. Data that were obtained from students who graduated prior to May 2009, generated from incomplete survey instruments, or provided without consent were excluded.

The survey instrument and the protocol for data collection were approved by the Long Island University Institutional Review Board. Deidentified grades on journal clubs were collected from APPE faculty members. Because no identifiers were collected, student grades were entered in a separate database from that of the survey data and were not merged. Statistical analysis of survey results and student grades was conducted using SPSS, version 14.0, and descriptive and univariate statistics were conducted and presented. Reliability of the 5-item scale to assess improvement in journal club competencies was assessed using Cronbach’s alpha.

EVALUATION AND ASSESSMENT

All 115 pharmacy students attending the assessment day completed the survey instrument. Of these, 105 (91%) provided informed consent to participate. Approximately 50% of the participants had completed the drug-information APPE first. Table 1 describes the format and method of documentation of journal club in the drug-information and both internal medicine APPEs. Two-thirds of students reported doing formal presentations in drug-information APPEs compared with slightly more than 50% in the 2 internal medicine APPEs. Eighty percent of students reported selecting their own article in the drug-information APPE compared with approximately 50% in the 2 internal medicine APPEs. More students reported doing individualized presentations in their drug-information APPE than in their internal medicine I and II APPEs. Written outlines were the predominant method of documentation used during journal clubs across the drug-information and internal medicine APPEs. Journal club preparation time reported by the students was highly variable, with internal medicine being more time consuming. While journal club presentations were generally less than 1 hour across all APPEs, roughly 6% of internal medicine journal clubs exceeded 1 hour.

The 5-item scale used to measure student perceptions of the value of journal club was found to be highly reliable for each APPE (Cronbach’s alpha = 0.889 for the drug information APPE and 0.896 and 0.897 for the internal medicine I and II APPEs, respectively). Overall, students perceived journal clubs to be valuable in improving their understanding of different study designs, strengths and weaknesses of a study, relevance to patient care, and presentation skills. Students’ value ratings related to understanding research design and statistics were higher in the drug-information APPE compared with those of either of the internal medicine APPEs (Table 2). The journal club in the drug-information APPE was also valued more than the journal club in the internal medicine I APPE for understanding strengths and weaknesses of a study (p = 0.021). Journal club in the internal medicine II APPE was valued more for the understanding of study design and presentation skills than was the internal medicine I APPE (p = 0.013 and p = 0.038, respectively).

Because students were randomly assigned to APPEs, the influence of APPE sequence on students’ perceptions of the value of journal club for a specific APPE was evaluated. Almost 50% of the participants completed the drug-information APPE first. Students’ ratings of the drug-information journal club learning objectives regarding study design, statistics, and presentation skills were not influenced by which APPE students completed first (Table 3). However, students who completed the drug-information APPE first had a higher and significantly different mean score for the journal club learning objectives related to improvement in understanding strengths and weaknesses, and the clinical relevance of the study than did those who completed the internal medicine APPE first (p = 0.041 and p = 0.19, respectively). Student ratings on all internal medicine journal club learning objectives were not influenced by which APPE was completed first (data not shown).

Journal club assignment grades were collected for 193 students and were found to not significantly differ from each other (Table 4). Students who completed the drug-information APPE first had a significantly lower grade than did those who completed it last (mean = 82.1, 85.4: p = 0.007). Internal medicine 1 and 2 practice-experience grades were not affected by completing the drug-information APPE first (internal medicine 1, p = 0.701; internal medicine 2, p = 0.526). When data on students who completed the drug-information APPE first followed by internal medicine 1 and 2 were compared with those of students who completed internal medicine 1 and 2 followed by drug information, we found that students who completed drug information first (drug-information mean grade = 82.1, internal medicine 1 and 2 mean grades = 83.1 and 85.1, respectively) had a significantly higher learning slope (p = 0.010) than did students who completed drug information last (drug-information mean grade = 85.4, internal medicine 1 and 2 mean grades = 83.7 and 84.3, respectively).

Fifty-four students (51%) made additional comments on the survey instruments (Table 5). Two chief concerns were identified in these comments. One was that the curriculum did not adequately prepare students for journal clubs. Some comments were specific to the third-year literature evaluation/biostatistics course in which students relayed a lack of statistical understanding and preparedness to analyze study designs. These comments support the findings based on 30% of respondents perceiving the usefulness of the classroom course as “not helpful.” Additional comments pointed to the inability of professors and preceptors to teach journal club concepts, such as clinical relevance. Suggestions for improvement involved journal club training and demonstration of appropriate journal club presentations by the professor or residents, as well as more exposure to journal articles in the biostatistics course and throughout the curriculum.

The second concern focused on the variability in formats related to journal club presentation and preceptor expectations. Several students who were exposed to both formal individual and informal group journal clubs preferred informal group discussions in which students were “not put under pressure.” Almost 25% of the comments suggested that journal club was a valuable method of helping students understand the process of literature evaluation and the latest evidence-based medicine that could be applied to improving patient care.

DISCUSSION

Our study is the first to evaluate whether pharmacy students see value in journal club experiences. Students completing APPEs at AMSCOP perceived that journal club assignments improved their presentation skills and understanding of different study designs, statistics, limitations and strengths of a study, and study relevance to patient care. Our journal clubs appear to meet the educational objectives stated in Standards 2007, and students see value in the competencies outlined and delivered based on the CAPE outcomes.⁵

The second objective examined whether student-perceived value ratings related to journal clubs varied across different practice experiences, such as drug-information and internal medicine APPEs. Variation was found in students’ ratings on some items regarding the drug-information and internal medicine APPEs but not others. For instance, the journal club for the drug-information APPE helped students understand certain aspects of literature evaluation better than did the internal medicine APPE regardless of whether students completed the drug-information APPE first, suggesting that these aspects are better learned through this specific APPE. Compared with preceptors for the drug-information APPE, those for internal medicine may have focused more on clinical application and the relevance of critical evaluation of studies for direct impact on patient care than on the intricate details of study design and statistics. Because drug-information APPEs at our institution are less focused on direct patient care, more time may have been spent on the time-consuming and challenging objectives of study design and statistics during these practice experiences than in the internal medicine experiences. Because students were exposed to 5 journal clubs, they may have valued the later journal clubs more because of the experience and skill they had acquired through the previous assignments.

When evaluating the effects of the first APPE, students who completed the drug-information APPE first gave higher ratings to the value of improving their understanding of certain aspects of literature evaluation. For example, the strengths/weaknesses objective was the only 1 that was significantly different based on which APPE was completed first. Although students perceived more value overall with the drug-information APPE for this objective, they also achieved this objective with the internal medicine APPE. The drug-information APPE was rated significantly lower for this objective when completed after the internal medicine APPE, meaning that for this objective, what was perceived to have been learned in both APPEs was based more on first exposure than on type of APPE. However, students who completed the internal medicine APPE first did not rate the value of any aspects of literature evaluation differently based on sequence of the APPEs. These findings suggest that by scaffolding through a particular sequence of various APPEs (for example, drug information first), students may be better able to understand the value of certain literature-evaluation skills. This emphasizes the need for scaffolding concepts throughout the curriculum, especially during APPEs, and paying attention to the specific methods of teaching that work best for the students’ learning. To account for the possible influence of academic performance on the value ratings, comparisons of academic performance across the various APPEs were also conducted. Journal club grades improved sequentially regardless of APPE type; however, when the drug-information APPE was completed first, grades had a significantly higher learning slope than when internal medicine was first. These findings emphasize the importance of scaffolding through a specific sequence of learning exercises and repeated exposure in slightly varied settings.

Several limitations must be considered when interpreting these findings. First, the data are cross-sectional. Because the study was carried out in a single private, urban university, the results may not be generalizable to all colleges and schools of pharmacy across the United States. Also, the survey instrument examined responses of 1 class of students graduating in May 2009, raising selection bias as an issue to be considered. Because survey instruments were disseminated and collected at the study institution, social desirability should be taken into consideration as well. Comparisons between the 2 APPEs may be skewed because of the institutional requirement for students to complete 1 journal club assignment in a required 2.5-week drug information APPE vs. 2 journal club assignments in each of the 2 required 5-week internal medicine APPEs, as well as the involvement of 6 drug-information faculty members at 2 sites vs. 16 internal medicine faculty members at 15 sites. These limitations may include variability among preceptors and their subjective emphasis on various objectives for the journal club presentation. The data obtained for academic performance could not be linked to the specific students who completed the anonymous value-statements survey; therefore, links between the value statements and actual learning could not be adequately assessed. Although the learning objectives of journal club that were used to create the value statements in the survey were not validated, they were found to be highly reliable with good face validity.

Our study and assessment tools do not clearly differentiate between the definitions of strengths/weaknesses of a study. The objective of strengths/weaknesses of a study can be based on study design and statistics as well as on clinical application. Therefore, the lack of a definition for this objective in our study, coupled with the possibility that drug-information preceptors may have spent more time discussing study design and statistics than did internal medicine preceptors, may explain why the journal club in the drug-information APPE was valued more for the understanding of strengths and weaknesses of a study than in the internal medicine APPE. Additionally, our survey instrument may not have included the students’ definition of value, which is critical for interpreting the results of this study. For example, it would have been useful to determine whether student value is generated from the ability to apply concepts learned in the classroom curricula, motivation for high-academic performance, or ability to use journal club articles to problem-solve in actual practice. From an outcomes perspective, the latter would be of high interest to academia. Future studies should identify the process through which journal clubs create value for students.

There is a great deal of variability in the format of journal club presentations among AMSCOP faculty members. A review of the literature suggests that various formats are used, ranging from weekly or monthly group meetings to presenting and discussing journal articles using formal and informal presentations of varying lengths. A common requirement across different formats is that journal club members are encouraged to read the articles prior to participating in a discussion led by a moderator/preceptor.¹⁸ Although journal clubs are an integral method of teaching evidenced-based medicine, the literature does not provide a description of the most effective format, leaving decisions about format to journal club facilitators. These decisions have important implications for teaching and learning critical literature-appraisal skills.

Although this study was not designed to evaluate the effect of format on student perceptions of journal club value, implications for how format influences perception of journal club value were found. For example, larger variances in specific formats between the drug-information and internal medicine APPEs, such as individualized formal presentations vs. group informal presentations, were observed. Future studies should examine the effect of different formats of journal club on student-learning outcomes. Finally, this study did not measure the actual ability of pharmacy students’ literature-evaluation competency, but rather investigated the students' perceptions of the value of journal club assignments during APPEs.

CONCLUSION

Journal clubs are widely used to develop students’ critical-thinking skills as well as to encourage them to stay abreast of the literature in their field and analyze and connect it to their educational experiences. Based on an examination of students’ experiences with journal club across various APPEs, we conclude that they perceive journal club as a valuable assignment that helps them better understand study design and statistics, improves their ability to analyze journal articles, and connects the content to patient care while improving their presentation skills. Through multiple assignments, students’ academic performance also improved, suggesting that an examination of various learning methods in specific sequences used throughout the curriculum may be beneficial.