Impact of a Journal Club Elective Course on Student Learning Measures

INTRODUCTION

The 2013 Center for the Advancement of Pharmacy Education (CAPE) Educational Outcomes were created to guide discussions of faculty members and preceptors in curriculum planning, delivery, and assessment within colleges and schools of pharmacy.¹ The knowledge and skills identified in Domain 1 provide the underpinning for the development of the knowledge and skills for the other domains. Two of the identified learning objectives from Domain 1 and Domain 3 of the CAPE Educational Outcomes, which also serve as our institution’s educational outcomes, are the ability to “…critically analyze scientific literature related to drugs and diseases to enhance clinical decision making,” and the ability to “identify problems; explore and prioritize potential strategies and design, implement, and evaluate a potential solution.”¹ The importance of students developing these skillsets is further emphasized within Standards 2016 and the accompanying guidance document from the Accreditation Council for Pharmacy Education (ACPE).^2,3

Achievement of these educational outcomes requires both knowledge and application of evidence-based medicine (EBM). Evidence-based medicine is a crucial concept in the education of healthcare professionals, and literature evaluation represents an important step in learning to practice EBM.⁴ Journal clubs are a commonly used tool to guide students in development of the skills needed to critically evaluate the literature. Multiple studies have evaluated the impact of courses teaching EBM and literature evaluation, both in pharmacy settings and in other medical fields.^5-9 Holloway and colleagues discussed implementing additional student contact hours of EBM instruction to first- and second-year medical students, evaluating its impact using modified script concordance methodology.⁵ An evaluation module was developed to measure a student’s ability to assess five steps in the practice of EBM: generating questions, searching for evidence, critical appraisal, applying evidence, and self-evaluation.⁵ The results of the evaluation modules were then compared to a comprehensive assessment that was comprised of a mock board examination, an assessment of “work habits,” and a clinical reasoning exercise. Results from this study showed that generating questions was correlated with a student’s mock examination scores (r=.23) and critical appraisal was correlated with a student’s clinical reasoning skills (r=.31).⁵

To further foster students’ literature evaluation and critical decision-making skills beyond the required curriculum, Sullivan University College of Pharmacy and Health Sciences (SU COPHS) developed a journal club elective course. This study was a single-center retrospective observational cohort study conducted at our institution to assess the impact of the journal club elective course on longitudinal performance throughout the curriculum. Measurements of student performance included Pharmacy Curriculum Outcomes Assessment (PCOA) scores, course grades in Pharmacotherapeutics IV, and Research Design and Literature Evaluation II, and average overall scores for the acute/ambulatory care advanced pharmacy practice experiences (APPEs).

METHODS

Students at SU COPHS complete the Doctor of Pharmacy (PharmD) degree in three years rather than the traditional four years. This is made possible by organizing the first two years of the curriculum (didactic plus introductory pharmacy practice experiences [IPPEs]) into quarters, each consisting of 10 weeks of coursework and one week of final examinations, with a two-week break between quarters. Each quarter of the third professional year (PY3) consists of 12 weeks of APPEs, with a one-week break between quarters.

This retrospective study involved SU COPHS students from the class of 2015 and class of 2016. During the period of analysis, the college followed a 4.0 grading scale (A, B, C, F) for coursework, and students were provided a percent score on individual APPEs.

Instruction on the concepts of EBM and literature evaluation was initially introduced in the first professional year (PY1). Students were taught basic didactic concepts of literature evaluation in the required Research Design and Literature Evaluation I course, which was three credit hours. In the spring quarter of the second professional year (PY2) immediately prior to APPEs, all students took Research Design and Literature Evaluation II (one credit hour). This course functioned as a “capstone” for literature evaluation skills and culminated with an application exercise where all students individually presented a journal club to a small group (approximately five peers and a faculty member). This faculty-graded presentation accounted for 30% of the student’s overall course grade. Each student was assigned a specific article, and journal clubs were graded using the assigned rubric. In preparation for this final assignment, students were assigned various articles throughout the quarter to read and take a formative assessment in preparation for the class discussion.

In addition to required core classes, students took two to four credits of elective coursework during each of the first three didactic quarters of PY2 and were required to take at least eight credit hours of elective coursework during that time. Each elective course was two credit hours and met two hours once weekly. Students could enroll in a journal club elective for up to three of their four professional electives in the pharmacy curriculum (meaning students could take it more than once). Typical enrollment in the course was approximately 15-20 students per quarter.

The journal club elective utilized an active-learning approach that required each student to individually present a formal journal club presentation as well as participate in class discussions and evaluate their peers. The goal was for the students to learn to critically evaluate trials and apply primary literature to patient care. Separate trials were assigned to each student. The clinical trials selected varied by quarter, although the basic structure and format of the course remained intact. Content of the trials aligned with pharmacotherapeutics topics that students had studied in previous or concurrent courses, contributing to the formation of a scaffolding model of learning in the curriculum.

During each session of the course, students provided background information on the related disease state, delivered a journal club presentation based on a structured rubric, and responded to questions from faculty facilitators and the student audience. The rubric for the presentation included sections for the introduction (14% of the grade); the main discussion of the study type, demographics, and endpoints (24% of the grade); statistical analysis (16% of the grade); results (12% of the grade); and conclusions (24% of the grade). Communication was also evaluated and comprised 10% of the grade for this assignment. This rubric was developed by course coordinators after review of rubrics from various institutions and in consultation with other clinical faculty members at the institution.

As a part of the course, students were responsible for reading the clinical trials assigned for that week’s class and taking a quiz on these at the beginning of each class period. Students were also expected to actively participate in the discussion and were evaluated on that participation. Participation was assessed by the course coordinators documenting the frequency of student participation (asking questions, providing statements during discussion). The class also included a comprehensive didactic final examination, which covered study design, statistical analysis, results interpretation, clinical significance, and application of the evidence.

The outcome measurements chosen for this study were purposefully longitudinal in nature, rather than based on students’ achievement in the course itself or on an embedded assessment within the course. One of these outcomes was score on the PCOA that students took prior to beginning their APPEs (termed the “pre-APPE PCOA” for this analysis). The pre-APPE PCOA was administered in late April or early May of the PY2 year (during spring quarter), roughly two months prior to the start of APPEs, and was a low-stakes assessment without external motivators.¹⁰ Student PCOA data were added to other assessment data at the institution to determine at risk students per the college’s assessment plan. The PCOA is composed of 225-questions and divided into four content areas: basic biomedical sciences; pharmaceutical sciences; social, behavior, and administrative sciences; and clinical sciences.¹⁰ The examination is assessed using a scaled score model, with overall score and individual content area scores ranging from 0-700. Under the previous PCOA blueprint (which this study used), literature evaluation was a subtopic assessed in the clinical sciences content area and amounted to roughly seven of the 225 questions (3.1%). Unlike the overall and content area scaled scores, the individual subtopic scores are based on percent correct.¹⁰ The PCOA blueprint adopted in 2016 contains the same four content areas as the previous blueprint; however, literature evaluation and evidence-based medicine subtopics are spread between the clinical sciences and social, behavioral, and administrative sciences content areas.¹¹

Data for all students graduating from SU COPHS who took the pre-APPE PCOA in 2014 (class of 2015) and 2015 (class of 2016) were included in the study. Baseline characteristics collected included: gender, pre-pharmacy math and science grade point average (GPA), pre-pharmacy cumulative GPA, presence of degree on admission, first professional year (PY1) GPA, and Research Design and Literature Evaluation I course scores (Table 1). Primary outcomes included overall PCOA scaled scores as well as scaled scores for the clinical science content area and literature evaluation percent correct scores. Secondary outcomes included Pharmacotherapeutics IV overall course grades, Research Design and Literature Evaluation II overall course grades, and acute/ambulatory care APPE cumulative overall average scores. Data were analyzed using IBM SPSS version 22 (IBM, Armonk, NY). Baseline characteristics were analyzed using chi-square tests for discrete variables and Student t tests for continuous data. A p<.05 was considered significant for all statistical analyses.

The secondary outcomes chosen for this study were targeted at specific demarcation points in the curriculum. Overall course average for Pharmacotherapeutics IV (spring quarter of PY2) was chosen as a secondary outcome because students had completed all electives prior to this course. The Research Design and Literature Evaluation II overall course average was chosen because the course was required of all students and was the “capstone” for literature evaluation in the didactic curriculum. The cumulative overall average of each student’s acute care and ambulatory care APPE scores were chosen because review of North American Pharmacist Licensure Examination (NAPLEX) data at our institution showed a weak 20% correlation between this metric and passage of the NAPLEX.¹²

Primary and secondary outcomes were analyzed using hierarchical linear regression to account for potential confounders in the baseline characteristics. Hierarchical linear regression is a technique used to determine if variables of interest explain a statistically significant amount of variance in the dependent variable after accounting for all confounding variables. This type of analysis is termed a “model comparison framework” as opposed to a statistical method, as several regression models are built by adding variables to previous models in a stepwise fashion. Thus, later models purposefully include data from the models before. Each model is then reviewed to determine whether it shows a significant improvement in the R-squared (R²) value (proportion of the explained variance in the dependent variable).¹³ Hierarchical linear regression was chosen for our study to help minimize the impact of possible confounding variables.

The hierarchical linear regression model equations for this study are defined in Tables 2 and 3. For all models, F statistics were reviewed to inspect model fit (p<.05), and R² values and change in R² were reviewed to determine which model had the best fit (highest change in R²). T-statistics were used to determine if results were significant (defined as p<.05). Variance inflation factor (VIF) coefficients (VIF<10) were reviewed to ensure lack of multicollinearity (correlation between independent variables). Graphs of the residuals were reviewed to ensure that data properly fit the model.

Because hierarchical linear regression was used in this study, the resulting equations mathematically predicted performance on the outcome variable of interest (eg, PCOA scaled scores, course grade) at the individual student level, taking into account the student’s cumulative pre-pharmacy GPA and PY1 GPA. Analysis of the raw data from the outcome (ie, simply comparing grades of the students who took the journal club elective to those who did not) would not consider these baseline characteristics. Thus, raw data comparing the outcome measures of the two groups in this study were not reported. This study was evaluated and deemed exempt by the Sullivan University Institutional Review Board.

RESULTS

A total of 197 students were included in the analysis. Seventy-three of the students completed the journal club elective and 124 students did not. During the study period, only five students took the elective twice (three in the class of 2015 and two in the class of 2016). Results from these five students were included in the analysis. Baseline characteristics are depicted in Table 1. Groups were well matched, and no significant differences were found.

Tables 2 and 3 depict the hierarchical linear regression results for the primary and secondary outcomes of this study. Thirty-four percent of the variance in overall PCOA scaled scores was explained by a student’s prepharmacy GPA and PY1 GPA. Enrollment in the journal club elective explained an additional 6.3% of the variance in scores. Combined, 40% of the variance was explained by these three variables. All three variables were significant contributors to the model. After controlling for cumulative pre-pharmacy GPA and PY1 GPA, enrollment in the journal club elective was associated with an average of a 24.5-point increase to a student’s overall scaled score on the PCOA.

Approximately 23% of the variance seen in the PCOA scaled score for clinical science was explained by students’ pre-pharmacy GPA and PY1 GPA. An additional 6% of the variance was explained by enrollment in the journal club elective. Combined, these three variables explained 28.5% of the variance in the model. Only PY1 GPA and the journal club elective were significant predictors of the clinical science content area scaled scores. After controlling for pre-pharmacy GPA and PY1 GPA, enrollment in the journal club elective added 28.2 points to a student’s clinical science content area scaled score.

Neither pre-pharmacy GPA nor PY1 GPA accounted for any of the variance seen in the PCOA literature evaluation percent correct scores. Seven percent of the variance in these scores was explained by enrollment in the journal club elective, with only journal club enrollment being significant in the model. After controlling for pre-pharmacy GPA and PY1 GPA, enrollment in the journal club elective was associated with scores on the literature evaluation section of the PCOA being 11.7% higher.

Students’ pre-pharmacy GPA and PY1 GPA accounted for 44.4% of the variance seen in students’ Pharmacotherapeutics IV overall course grade. An additional 2.1% of the variance could be explained by enrollment in the journal club elective. Combined, these three variables accounted for 46.5% of the variance in the model. The PY1 GPA and the journal club elective were significant predictors in the model. After controlling for pre-pharmacy GPA and PY1 GPA, enrollment in the journal club elective added 1.9% to a student’s overall Pharmacotherapeutics IV course score.

Pre-pharmacy GPA and PY1 GPA accounted for 25.2% of the variance seen in student’s Research Design and Literature Evaluation II overall course score. An additional 2.8% of the variance could be explained by enrollment in the journal club elective, totaling 28.0% of the variance explained by these three variables in the model. Once again, only PY1 GPA and the journal club elective were significant in the model. After controlling for prepharmacy GPA and PY1 GPA, enrollment in the journal club elective added 1.9% to a student’s overall Research Design and Literature Evaluation II course score.

Pre-pharmacy GPA and PY1 GPA accounted for 10.4% of the variance seen in student’s acute/ambulatory care APPE cumulative overall average scores, with an additional 3.1% of the variance being explained by student scores in the Research Design and Literature Evaluation II course. An additional 2.5% of the variance could be explained by student participation in the journal club elective leading to 16% of the overall variance being explained by the four variables. PY1 GPA, Research Design and Literature Evaluation II course grades, and the journal club elective were all significant predictors in the model. After controlling for prepharmacy GPA, PY1 GPA, and Research Design and Literature Evaluation II course grades, enrollment in the journal club elective was associated with a 1.9% higher cumulative score on students’ acute/ambulatory care APPEs.

DISCUSSION

To our knowledge, this was the first study in pharmacy education investigating the impact of an elective course containing student-led journal clubs on multiple objective measures of student learning across the curriculum, including the PCOA. Following the publication and enforcement of ACPE Standards 2016, the PCOA is now required to be administered as a pre-APPE assessment.² However, to date, there are limited published data on PCOA scores being linked to substantive outcomes of student performance, including performance on the NAPLEX.¹⁴ With the widespread use of the PCOA in the Academy, it can be expected that further investigations will continue to associate PCOA performance with the NAPLEX and other student performance outcomes. Our study adds to this growing body of literature.

Overall scaled scores on the PCOA were an average of 24.5 points higher among students who completed the journal club elective. Journal club students’ scaled scores in the clinical science content area were an average of 28.2 points higher, with an 11.7% increase in their percent correct scores in the literature evaluation subtopic. In addition to the effect on PCOA results, our study also analyzed other outcomes such as individual student course scores in both Pharmacotherapeutics IV and Research Design and Literature Evaluation II. For students who took the journal club elective course, their overall course score improved by 2%, potentially leading to a higher letter grade in those courses. These students also performed 2% higher on their overall acute/ambulatory care APPE scores.

Several studies have examined the incorporation of components of literature evaluation (eg, journal clubs) into courses. In 2011, Bookstaver and colleagues discussed an EBM elective course which included case studies and problem-based learning, simulations of journal clubs, and “wiki” pages driven by students.⁹ Multiple clinical faculty members were involved, as well as practicing clinicians from local academic medical centers and pharmacy residents. During each week of the course, students were assigned a therapeutic topic by that week’s facilitator and two journal articles to review related to that topic. Students led a group discussion about the topic (including the articles) and posted article reviews on the course wiki page. Study investigators administered pre- and posttests to students related to their understanding of biostatistics, which showed improvement.⁹ Preceptors and APPE students who were surveyed reported perceived improved ability of students to apply EBM to patient care as compared to students who did not participate in this elective.⁹ While Bookstaver and colleagues assessed perceived performance on APPEs from both the student and preceptor perspective, no APPE performance or assessment data were reported.⁹

Hidayat and colleagues described the addition of active learning in the form of a mini-lecture, journal club, and debate in an advanced infectious disease elective course.¹⁵ In that course, peer teaching was employed in the form of mini-lectures over an assigned topic, followed by a journal club in which two students delivered 15-minute presentations, each of which were followed by 10-minute question-and-answer sessions. Grading rubrics prepared by the course coordinators were used to evaluate student performance on the active-learning exercises. Student perception data, gathered through a survey, showed significant improvement in both students’ understanding of the course topic (infectious diseases) and their perceived ability to evaluate the literature to make informed clinical judgments (p<.05).¹⁵ The journal club course in our study employed a similar activity, although ours was the entire focus of the course and integrated and reinforced the content taught in the pharmacotherapeutics sequence. Additionally, each student in our elective course created and presented to the rest of the group alone rather than in groups. Our findings of improvement in students’ PCOA scores associated with participation in a journal club elective course provides evidence of the impact of this type of active learning on educational outcomes.

Guiliano and colleagues also explored factors associated with PCOA performance using a multiple linear regression model.¹⁶ In a cohort of 142 students, factors associated with higher PCOA scores included increases in PY1 GPA and higher reading scores on the Pharmacy College Admission Test (PCAT). Factors associated with lower performance on the PCOA included the undergraduate institution students attended for pre-pharmacy coursework (identified as “Institution A” or others as a dichotomous variable) and students who preferred reading as a learning style or were identified as “accommodators” (compared to assimilators) according to the Unified Learning Style Model.¹⁶ While learning styles and PCAT scores were not included in our model, we did include admission pre-pharmacy GPA as a variable to address factors that may be present prior to a student’s admission to the program that may affect their performance on the PCOA. The combination of multiple components within the journal club elective course could be responsible for the outcomes shown, such as the individual student-led presentations (vs group), the accountability of the pre-session quizzes, and the scaffolding with pharmacotherapeutics topics. In addition, it could be that students who complete this elective are required to read and be ready to discuss 10-15 primary literature articles throughout the course’s entirety. This is more than the number of articles that students would be exposed to by only completing the required components of the curriculum. The results indicate that the benefits of completing the journal club elective course were carried throughout the didactic and experiential curriculum. Based on these data, the college’s curriculum committee recommended the journal club elective become a required “selective” for all students beginning with the 2018-2019 academic year. Additionally, other similar analyses are ongoing relative to other electives, embedded assessments, and required coursework to determine factors that may impact student success measures.

While these results are significant, there are some important limitations to note. First, this research was conducted at a single college of pharmacy with a three-year accelerated curriculum, possibly limiting its applicability to other pharmacy programs. There was a capped number of students enrolled in the elective, resulting in a limited sample size for analysis; however, the small sample provided significant results. Although only a very small population, five students (three in the class of 2015 and two in the class of 2016) were enrolled in the elective more than once, and a subgroup analysis of these five students was not performed.

There also could have been additional confounders that affected PCOA performance that were not identified. At our institution, PCOA is a low-stakes assessment with no external motivators. This study was not designed to assess whether students' PCOA scores were influenced by other external factors. Also, our study did not assess the attitude, perceptions, and motivation of the students who enrolled in the journal club elective course. It could be theorized that the students who opted to enroll in the journal club elective were more motivated because of the course delivery elements (oral presentation and journal club) that are known components of future courses, APPEs, and residencies. However, the hierarchical linear regression analysis was selected to account for variables outside of the journal club elective when explaining the amount of variance, and baseline characteristic assessments (ie, prepharmacy GPA, PY1 GPA) found no significant difference between the two groups.

Another potential limitation was that the faculty members coordinating the elective differed from quarter to quarter. Therefore, individual elements of course execution could have varied slightly by quarter. However, the overall design (including objectives, format, and basic schedule) of the course remained the same. Students can also take other elective courses that could have potentially contributed to outcome results.

As mentioned previously, this type of course analysis has become a standard component of the college’s assessment plan and is ongoing with other courses, embedded assessments, and curricular components. The journal club elective course was not intentionally designed to improve PCOA scores; however, at a programmatic level, these types of analyses are useful to determine factors that may be associated with improved outcomes for students. Other institutions are encouraged to use similar analyses in a systematic fashion to identify courses, course elements, or other factors associated with student success measures within their own curricula.

CONCLUSION

Student enrollment in a journal club elective course was associated with improved PCOA scaled scores, both overall and specifically related to clinical science and literature evaluation. Completion of a journal club elective was also associated with improved student performance in the Pharmacotherapeutics IV and Research Design and Literature Evaluation II courses as well as acute/ambulatory care APPEs. Components of the journal club elective course, including reading the primary literature, presenting journal clubs, peer-teaching, and scaffolding of therapeutic concepts are important to consider when evaluating the delivery of content across the curriculum.