Professional Benefits of a Web-based Journal Club for Critical Care Residents and Their Mentors

INTRODUCTION

In 1966, Mattingly first described participation in journal clubs as “a group of doctors meeting regularly to discuss papers of interest in the current medical journals” and as being “a regular and often compulsory feature of hospital life.”¹ As the breadth and depth of literature have grown and the roles of health care professionals have evolved, so too have the scope and formats for journal clubs. In recent years, journal clubs have expanded from the confines of an in-person, group interaction to discussions that occur through many media formats and in real time on social media.²

Evidence-based learning through journal clubs allows for health care professionals, including students, residents, and fellows still in training, to discuss and critique a journal article and apply the results to real-world practice.³ Participation in a journal club is an ideal pedagogical strategy for evidence-based learning for trainees and practitioners. When participation is limited to a geographical location, only those who are located in or travel to that location directly benefit from their understanding of the journal article and its application to practice from the discussion that occurs. However, when a web-based journal club format is used, different interpretations of and perspectives about the literature can be discussed, and the variability in quality of care between institutions may be reduced through quicker translation of new knowledge into clinical practice.⁴ For these reasons, in 2009, the Clinical Pharmacy and Pharmacology (CPP) Section of the Society of Critical Care Medicine (SCCM) transitioned its local, teleconference-based journal club to a web-based journal club.⁵

The SCCM monthly web-based journal club, facilitated by members of the CPP Section’s Education Committee, provides a concise overview and discussion of pharmacotherapy-focused research to an interdisciplinary group of critical care physicians, pharmacists, nurses, and advanced practice providers throughout the entire SCCM membership. When the web-based journal club began in 2009, presenters were critical care practitioners and critical care residents at a few institutions across the country, and the audience included students, residents, and pharmacists at these institutions. Beginning in 2010, the web-based journal club moved to a national platform and shifted from having critical care pharmacist practitioners present to almost exclusively having postgraduate year two (PGY2) critical care pharmacy residents present. At this time, moderators for the web-based journal club were predominately the preceptors at the institutions of the residents who were presenting that month or CPP Section Education Committee volunteers. A practicing critical care pharmacist now serves as the moderator. Because of the significant interest among residents in presenting and the limited number of slots, a lottery system was implemented in 2014 to determine which residents would present during the hour-long web-based journal clubs held throughout the year. Beginning in July 2015, the number of presenting residents was increased from two to three (33 throughout the year), and verbal feedback from the moderator immediately following the web-based journal club was incorporated. This provided residents with an opportunity to present to hundreds of clinicians on an international platform, receive feedback from an established practitioner, and help the critical care community remain abreast of recently published articles. The purpose of this study was to identify perceptions and benefits of the SCCM CPP Section web-based journal club among presenting residents and their mentors.

METHODS

This was a cross-sectional survey of all pharmacy residents who presented a web-based journal club through the SCCM CPP Section Education Committee, as well as all of their journal club mentors. Residents were classified as being in one of two groups: resident group 1 included residents who delivered a web-based journal club presentation between July 1, 2010 and June 30, 2015, and resident group 2 included residents who delivered a web-based journal club presentation between July 1, 2015 and December 31, 2016. The residents were separated into two separate groups to reduce the bias that less recent presenters may have had for remembering specific details of their journal club experience by focusing on long-term rather than short-term benefits of the journal club. Because the journal club format transitioned from two presenters per 60-minute meeting to three beginning July 1, 2015, this date was chosen for dichotomizing the resident groups. A single residency program director or preceptor from each institution that had one or more residents give a web-based journal club presentation during the entire study period volunteered to serve as the presentation mentor for that institution and respond to the survey.

A separate survey instrument was developed for each of the three populations of interest: resident group 1, resident group 2, and resident mentors. Each survey tool was developed according to the following steps: survey item generation, initial survey construction, pilot testing of survey items for face and content validity by 32 critical care pharmacists who were neither a study investigator or participant, incorporation of written and verbal suggestions from pilot testers based on consensus of the study investigators, and agreement from all study investigators on the final survey instruments.⁶ Components of the 16-item survey instrument for resident group 1 participants included questions about demographics, perceptions of the web-based journal club experience (using a 5-point Likert scale on which 1=strong disagree and 5=strongly agree), current and previous involvement in SCCM, professional presentations after the web-based journal club, and current and previous engagement with the web-based journal club. Resident group 2 participants were asked 15 questions about demographics, perceptions, and growth following the web-based journal club (using a 5-point Likert scale on which 1=strongly disagree and 5=strongly agree), and current or future professional involvement. Mentors answered 17 questions on demographics and preparation for, feedback on, and engagement with the web-based journal club, as well as perceptions of the benefits derived by the resident from the web-based journal club experience.

All three surveys were distributed weekly for three weeks in July 2016, and the survey instruments for resident group 2 and the mentors were distributed a second time in December 2016 to those who participated in the web-based journal club between July 1, 2016 and December 31, 2016. The survey instruments were individually distributed using an online survey software and insight platform (Qualtrics, Provo, UT). Data from each participant were aggregated such that no personal information linking specific responses to a specific participant was retained. Participation was voluntary.

Responses to each survey item were characterized using frequency and descriptive statistics. For questions that overlapped between resident groups 1 and 2, the percentage of respondents in each group who somewhat agreed or strongly agreed to a statement were compared using Fisher exact test. A two-sided p value <.05 was considered statistically significant. Microsoft Excel was used for all analyses. The research was approved by the institutional review board at Rush University, Chicago, Illinois.

RESULTS

Demographic data for resident groups 1 and 2 are presented in Table 1. Of the 93 presenters in resident group 1, 38 (41%) responded to the survey (Figure 1). Most resident group 1 respondents somewhat or strongly agreed that participation in the web-based journal club was a beneficial educational experience (92%). Respondents also indicated they somewhat or strongly agreed that the web-based journal club was a valuable part of their residency experience (79%) and that participation helped them improve their presentation skills (87%). Fewer participants strongly agreed (21%) or somewhat agreed (31%) that the web-based journal club was a valuable networking experience.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Resident Group 1 Response Rate by Residency Year

Note: Three respondents did not indicate their residency year

The majority of respondents (95%) in resident group 1 were current SCCM members at the time of the survey. The majority of respondents (80%) from this cohort somewhat or strongly agreed that participation in the web-based journal club increased their awareness of opportunities within SCCM. At the time of surveying, almost half of all respondents (49%) were currently involved in CPP Section committees, and 85% of respondents had continued to view the web-based journal club at least once per year. The majority of respondents (95%) had gone on to give another professional presentation at the local, regional, or national level since completing their residency program.

Of the 32 presenters in resident group 2 who were surveyed, 23 (72%) responded. All resident group 2 participants somewhat or strongly agreed that participation in the web-based journal club was a beneficial educational experience. Respondents also indicated that the experience helped them advance their literature assessment and clinical practice skills, communication and presentation abilities, and awareness of professional involvement opportunities. Across questions assessing these aspects of the web-based journal club, 95% of respondents somewhat or strongly agreed that participation furthered these aspects of their professional development. All of the respondents indicated that they somewhat or strongly agreed that the web-based journal club was a valuable part of their residency, and 91% somewhat or strongly agreed that the web-based journal club provided a valuable networking experience.

Resident group 2 respondents reported that the web-based journal club experience made them more confident when critically evaluating research (96%), determining clinical applications of articles (95%), developing evidence-based recommendations (95%), and educating trainees about evidence-based medicine (95%). Resident respondents indicated that participation helped them improve their capacity to communicate their assessment of articles (91%), ability to respond to questions about clinical studies (100%), and presentation skills (95%). All respondents reported that the web-based journal club presentation increased awareness of opportunities within SCCM and other professional organizations. Frequencies of individual responses are shown in Figure 2. Demographic characteristics and responses to questions that overlapped between resident groups 1 and 2 are presented in Table 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Perceptions of Web-based Journal Club Among Group 2 Resident Respondents

Note: Numbers may vary based upon response rate and questions omitted

Thirty-two of 58 (55%) mentors whose residents presented between 2010 and 2016 responded. A majority of the mentor respondents (76%) were residency program directors, while 17% were preceptors, and 7% identified as other. Most respondents worked at a community health care system or hospital (52%) or academic medical center (45%). Respondents were from the following regions of the United States: Midwest (45%), South (31%), Northeast (14%), and West (10%). Mentors precepted residents in 2015-2016 (31%), 2014-2015 (20%), and 2013-2014 (22%); the remaining 27% of respondents were mentors from 2010-2011 to 2012-2013.

Most mentors (89%) approved the article that the resident presented and felt that participation in the web-based journal club increased the resident’s knowledge of the article’s content area (68%). Some mentors (26%) felt the experience influenced patient care decisions at his or her practice site. Mentors derived additional benefits from participation, which included improved capability for providing verbal feedback on presentation content or style (53%), increased understanding of the SCCM CPP Section web-based journal club process (32%), and improved capability for providing written feedback on presentation content or style (26%).

The perceived impact of the web-based journal club on residents and mentors is displayed in Figure 3. While most mentors did not feel the web-based journal club impacted their own involvement within SCCM or the CPP Section (34%), it did somewhat or strongly affect interest in attending future web-based journal clubs (92%). Most mentors felt that the web-based journal club influenced his or her resident’s future involvement within SCCM (81%), the CPP Section (70%), and the web-based journal club (82%). Overall, most mentors (83%) felt it was extremely likely that they would have a future resident give a web-based journal club presentation.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Professional Development Associated with the Web-based Journal Club According to Resident Mentors

CPP=Clinical Pharmacy and Pharmacology, JC=journal club, SCCM=Society of Critical Care Medicine

Mentor involvement with SCCM: Participation in the web-based journal club impacted the mentor’s own involvement within SCCM

Mentor involvement with CPP Section: Participation in the web-based journal club impacted the mentor’s own involvement within CPP Section

Resident involvement with SCCM: The web-based journal club influenced his or her resident’s future involvement within SCCM

Resident involvement with CPP Section: The web-based journal club influenced his or her resident’s future involvement within CPP Section

Future resident president in web-based journal club: The mentor would have his or her future resident present a web-based journal club

DISCUSSION

Most residents who gave a web-based journal club presentation through the SCCM CPP Section indicated participation was both a beneficial educational and professional experience. Residents who more recently provided a web-based journal club reported having improved confidence in critically evaluating research, determining clinical applications of published literature, developing evidence-based recommendations, and educating trainees on evidence-based medicine. Mentors believed their residents’ participation influenced their future involvement in both SCCM and the CPP Section and were extremely likely to recommend that their future residents participate in the web-based journal club. Because no other web-based journal club programs were identified in pharmacy or other disciplines, the results and strategies used to provide this service may be applicable to multiple organizations and professions.

While both groups of web-based journal club presenters believed the presentation was a beneficial experience, more recent presenters believed their abilities to evaluate, apply, and make recommendations regarding recent medical literature improved because of the web-based journal club presentation. This increase may have been because of the shortened presentation length for group 2 participants, which necessitated a more streamlined presentation of the article, and the verbal feedback provided immediately after the web-based journal club by the moderator, which may have resonated with the presenters more than written feedback provided a month after the presentation to the residents in resident group 1. More than two-thirds of the mentors validated these beliefs, suggesting that residents increased their knowledge of the articles they presented.

Data from resident group 1 presenters and mentors suggest the experience impacted them positively in the short- and long-term. Given that the primary rationale for most students pursuing postgraduate residency training is to facilitate achieving self-actualization in their career,⁷ almost 95% of resident group 1 presenters had provided other local, regional, or national presentations since completing their training. Similarly, because residents’ knowledge in reading and interpreting research is variable but generally still developing at the beginning of postgraduate residency training,^8,9 participation in formal activities, such as the web-based journal club, creates an opportunity for the resident to grow in these vital areas.

After completing their training, residents encounter professional opportunities beyond providing presentations. Both groups of presenters and their mentors felt the resident presenters’ awareness of opportunities for involvement in SCCM increased because of their experience with delivering a web-based journal club presentation. Corroborating this, half of the presenters in 2010-2015 remained involved in the SCCM CPP Section as a member of at least one committee, and the majority had attended at least one web-based journal club presentation each year. Of note, the quality and frequency of this web-based journal club would not be possible without significant support from volunteers, many of whom are past web-based journal club presenters.

Additionally, the web-based journal club provides resident presenters the unique opportunity to network with CPP Section Education Committee members, web-based journal club moderators, and other web-based journal club presenters. While all of the presenters in 2015-2016 viewed the web-based journal club as a valuable networking experience, only half of the presenters from 2010-2015 shared the same opinion. This difference could reflect the changes to and expansion of the web-based journal club that occurred over the study period. For example, the more recent presenters had one-on-one time with the web-based journal club moderator to receive feedback following their web-based journal club presentation, which may have affected their perception of networking opportunities. The number of presenters also increased from two to three presenters per web-based journal club, allowing more interaction between resident presenters and their colleagues at other institutions before the presentations. More recent presenters entered a notably more competitive market for specialty-trained pharmacists than the 2010-2015 presenters did,¹⁰ and their greater appreciation for the importance of having an edge in this market may have informed their perceptions of the importance of networking.

Residency programs may consider this a preceptor development activity because many mentors felt their participation in assisting the resident improved their capability for providing verbal and written feedback on presentation content and style and increased their knowledge of the content area. Successful completion of other significant activities during the pharmacy residency year is more likely when residency mentors collaborate effectively with their critical care resident.^11,12 The majority of web-based journal club mentors approved the article their resident presented, and mentors’ comments indicated that many of these individuals contributed to their resident’s web-based journal club by helping the resident revise rough drafts of the presentation, discussing applications of the article to real-world patient care, and providing feedback during practice presentations.

Despite the many benefits of the web-based journal club, audience and resident participants, as well as committee members, continue to identify areas for improvement. The web-based journal club is a complimentary service to all SCCM members, which has allowed the content to reach practitioners across the world. However, its web-based nature removes the in-person dynamic, which has led to residents providing a more formal presentation that invariably decreases interaction between the presenter and audience participants. Strategies have been implemented to reduce the formality of the journal club and encourage discussion of the presentation’s content. These include the use of a chat function within the webinar system, which has facilitated robust question and answer sessions after each article, and interactive polling questions throughout each presentation, which has allowed audience members to share current practices and anticipated practice changes with each other within the allotted time for each presentation. Unfortunately, because the number of interested presenters has significantly exceeded the number of available spots, residency program directors cannot consistently incorporate this experience into their program’s learning curriculum every year. The increase in number of web-based journal club presenters each month from two to three helped to address the mismatch in interested resident presenters to available presentation slots. Although the time each presenter had to present decreased from 25 to 18 minutes, this change allowed for a greater variety of content to be provided each month and forced the presenters to discern the most relevant information they needed to convey in their allotted time. While the SCCM CPP Section has not chosen to increase the length of the monthly web-based journal club to allow for all interested residents to present, other organizations could consider that strategy. Every change to the web-based journal club has improved certain aspects of the experience, but some additional barriers or limitations could result. Ongoing assessment is essential to providing a beneficial experience for presenters and audience participants.

In addition to periodically assessing the web-based journal club format, recognizing the logistical barriers is essential before widespread implementation or expansion can be successful. We have used a team of pharmacists trained on the webinar software and non-pharmacist support staff at SCCM for additional troubleshooting, which has been necessary to ensure the web-based journal club runs smoothly. Written evaluations from audience members are an important source of feedback for residents and web-based journal club organizers; however, the number of evaluations completed each month varies, and those that are submitted often have limited comments. Verbal feedback from the moderator does allow resident presenters to receive immediate feedback on areas of strength and those in need of improvement from an experienced preceptor. A novel solution may be registering the web-based journal club to be an accredited continuing education presentation, which would increase the quantity but not necessarily the quality of the written feedback the residents receive through the use of mandatory speaker and program evaluations. It may also provide an added incentive for attendance and a potential source of revenue for the organization. Additionally, maintaining a standard, monthly date and time for the web-based journal club has helped SCCM members plan their attendance in advance. The need for process changes should be evaluated periodically to accommodate new technology and software availability as well as members’ needs and requests. Future updates may include adding themes to the web-based journal club and incorporating social media such as Facebook and Twitter to further expand discussion of articles and help advertise the programming. The use of social media may also allow members who cannot live stream the web-based journal club to participate from a mobile device.

Certain limitations were inherent in our study design, and were mitigated whenever possible. Although participation in the survey was voluntary and a response bias could have existed, high response rates suggest this bias may be limited. However, residents and their mentors who were not selected to present a web-based journal club were not surveyed because they could not be reliably identified and contacted, which prevented comparison of the residents who did and did not present and determination of the true benefits gained from participation. Additionally, audience members who viewed the web-based journal clubs were not surveyed for the impact these presentations had on their understanding of the journal articles or the care their patients received. Surveying students, postgraduate trainees, and practitioners in the future could elucidate these benefits and identify potential areas for improvement in the web-based journal club. Questions on the surveys relating to perceptions and involvement in SCCM may have been most susceptible to response bias because respondents’ contact information was gathered from the organization’s records and respondents knew the survey was being conducted by members of the SCCM CPP Section Education Committee. Additionally, because both mentors and residents could have entered the presenting resident’s name in the lottery, the interest in the web-based journal club for both parties likely varied, which allowed for a diversity of initial opinions regarding the web-based journal club experience to exist and improves the generalizability of our findings to other professional organizations. Specific questions regarding incorporation of immediate, verbal feedback from the moderator and expansion of the journal club from two to three presenters were not asked, which limits evaluation of these changes on the residents’ and mentors’ perceptions of the journal club. Despite the inherent limitations of using a survey tool, ours was validated by a cohort of pharmacists who did not participate in the web-based journal club as a resident, residency program director, or preceptor. Finally, resident participants were unable to have their responses matched with those of their mentor because of the anonymous nature of the survey.

CONCLUSION

Participation in a web-based journal club provided professional benefits to participants and their mentors that extended beyond the presentation itself. The majority of respondents experienced added value in at least one professional area. Interaction with the organization as a resident presenter or presentation mentor provided a unique experience that may have encouraged these individuals to maintain their professional involvement in the organization after the web-based journal club experience was completed. Future research should investigate the impact of moderator feedback and match responses between each resident and presentation mentor. Other professional organizations and professional programs may benefit from implementation of a similar web-based journal club.