Fostering PharmD Skills Related to Research and Quality Improvement Through Mentored Projects

INTRODUCTION

The call by the American College of Clinical Pharmacy (ACCP) for more research training within Doctor of Pharmacy (PharmD) curricula also highlights the need for pharmacists to generate new knowledge to advance the profession and improve medication safety and effectiveness.¹ The Accreditation Council for Pharmacy Education (ACPE) 2016 standards require instruction on research design, but do not specifically require research or quality improvement projects.² Accordingly, few colleges of pharmacy provide hands-on research experience to all their students. In 2007, only 15% of colleges of pharmacy required students to complete a project involving data collection,³ and in 2016, only seven colleges reported including research projects in their core curricula.⁴ Some colleges offer research electives but report lower student participation than when projects are required.⁴ Thus, the majority of PharmD graduates have little hands-on experience with research or quality improvement projects.

Research training is likely to improve PharmD students’ ability to conduct research and their literature-evaluation skills, as well as high-level cognitive skills such as problem solving, leadership, communication, and innovation, which are required by the ACPE 2016 accreditation standards.^2,5,6 Other potential benefits include improved project management skills and mentor-mentee relationships, increased marketability for jobs and residencies, improved ability to document the value of pharmacy services, and realization of increased faculty scholarly output.¹ However, the few studies evaluating required PharmD research training that include quantitative outcomes, such as publications, presentations, or measurable change in knowledge, confidence, and interest regarding research, often note that more scholarship is needed in these areas so that colleges of pharmacy can know how many of these benefits are achievable.^4,7-9 To realize these potential benefits, the University of Utah College of Pharmacy implemented required mentored student research or quality improvement projects as part of a broader curriculum revision starting with the Class of 2019. This article describes the structure of the program and evaluates the impact on and feasibility for the first graduating students to complete the program.

METHODS

In January 2017, the University of Utah College of Pharmacy began recruiting project proposals from basic science and clinical faculty members within and outside of the University of Utah, including clinical pharmacists, community pharmacists, managed care pharmacists, and public health officials. Mentors submitted project proposals using an online form. Pharmacy students could also propose their own projects if they could identify a willing mentor. A PharmD project committee was formed to provide program oversight and review proposals for feasibility. A list of approved proposals was distributed to students on March 9, 2017. Students had three weeks to rank their 15 preferred projects and five least-preferred projects. Students who proposed their own projects were automatically matched with their previously identified mentor. Remaining students were randomized to their highest-ranked project available.

Students subsequently met with their assigned mentor before their summer break following the second professional (P2) year . During that meeting, the students signed a compact adapted from the Association of American Medical Colleges’ Compact Between Biomedical Graduate Students and Their Research Advisors, which outlined the responsibilities of the students and mentors and affirmed their agreement to proceed with the project. During the summer, students started reviewing literature related to their project and completed training required by the University of Utah’s Institutional Review Board (IRB), Institutional Animal Care and Use Committee (IACUC), or Institutional Biosafety Committee (IBC). Mentors were responsible for ensuring that each project, where applicable, received approval from the University of Utah IRB and for studies involving animals were approved by the University of Utah IACUC and were conducted in accord with the Guide for the Care and Use of Laboratory Animals (eighth edition).

During the third professional (P3) and fourth professional (P4) years, the students completed four semesters of coursework related to the research project, which provided structure, deadlines, and feedback to monitor their progress and provide a support network, as students worked closely with their mentors to complete their projects. About four times a semester, mentors were required to complete rubrics or forms to assist with grading their students and document required meetings. The P3 fall semester course, Principles of Project Development (three credit hours), guided students through writing a three- to six-page project proposal based on application instructions for the ACCP Research Institute Futures Grants and the NIH National Research Service Award Fellowships. The students wrote several drafts, with each draft receiving feedback from peers, teaching assistants, course instructors, and/or their mentors. Students then participated in three semesters of Project Development Practicum (2 credit hours each), which provided students with dedicated time to complete the project as well as guidance on how to draft manuscripts and posters. In the P3 spring course, students transitioned to implementing their proposed research plan, including obtaining regulatory body approvals and collecting data. Students were also required to deliver a 10- to 15-minute oral presentation followed by a questions and answer period. During the P4 year, the practicum courses were delivered online to accommodate advanced pharmacy practice experience (APPE) schedules. In the fall, P4 students drafted the introduction and methods sections of both their posters and manuscript, following author instructions from appropriate conferences and journals. The following semester, students added the results and discussion sections. All students were required to present their poster at a poster session sponsored in the spring of their P4 year and attended by college faculty members, adjunct faculty members, and fellow students. Students were also encouraged to submit abstracts for their projects to national meetings and then present their findings if their abstract was accepted.

Each student project was eligible for up to $1000 for research supplies, $500 for consulting fees (including those charged by the University of Utah’s Research CORE services), and $300 toward travel expenses to attend a national conference to present their poster. The college also covered poster printing expenses. To receive available funding, students and mentors had to submit a short application with a budget and budget justification, which was reviewed and approved or amended by the PharmD project committee.

We conducted a research readiness survey to measure students’ confidence, attitudes, and knowledge regarding research. Originally developed to evaluate medical residents’ knowledge of biostatistics, the survey instrument had since been expanded to include measures of confidence and general attitudes toward the importance of research, and used to assess pharmacy residents.^9-11 Survey responses were collected from the class of 2019 in August 2017 at the start of the project coursework (time 1=T1), in December 2017 after completing the proposal (time 2=T2), and in April 2019 following project completion (time 3=T3). Survey completion was required, but students were informed that their answers would not affect their grade. Responses from the Classes of 2017 and 2018 were collected prior to graduation using the attitudes and confidence portions of the same survey instrument to provide baseline data for comparison. The survey included a 13-item quiz, which tested students’ knowledge of biostatistics and concepts from evidence-based medicine. Students’ knowledge score was the number of correct answers, with a possible maximum score of 13. Students indicated their level of agreement from strongly agree (5) to strongly disagree (1) for the statements found in Table 1. The attitude score was the sum of the values for each response, with possible scores ranging from 4 to 20. Students also rated their confidence on a scale from 1 (no confidence) to 5 (complete confidence) on the elements listed in Table 2. Confidence scores were the sum of the values for each response, with possible scores ranging from 10 to 50. The last survey distributed to students in April 2019 added questions designed to evaluate the program, gather feedback for future improvements, and collect information about national presentations and publications.

The changes in students' knowledge, attitudes, and confidence scores were compared over time with Friedman tests, and Wilcoxon paired tests were used post-hoc to compare specific times (T1, T2, and T3). Mann-Whitney U tests were used to compare attitude and confidence scores of the Class of 2019 at T3 to the baseline measurements of the combined Classes of 2017 and 2018. The individual items from the confidence and attitudes sections were dichotomized with responses of 4 or 5 on one hand and 1, 2, and 3 on the other. Each item was analyzed for change over time using the chi-square test of association. Statistical analyses were performed using GraphPad Prism 8 (GraphPad Software, San Diego, CA). A p value <.05 was considered statistically significant.

Project mentors were surveyed in April and May 2018 (end of P3 year) and May 2019 (end of P4 year). Mentors rated whether their time spent on the following activities was appropriate, too much, or too little: reviewing drafts, providing feedback on presentations, meeting with the student, and completing forms/rubrics. They also estimated the number of hours spent mentoring each student per semester. Open-ended questions allowed mentors to suggest program improvements. The 2019 survey added questions about overall satisfaction, benefit to the mentor, value of the learning experience, willingness to write a strong letter of recommendation for their student, and whether the project was published or presented. The University of Utah IRB determined that this project was exempt from oversight.

RESULTS

For the Class of 2019, the PharmD project committee received 69 project proposals from 43 mentors. Twenty-seven mentors were full-time faculty members at the college, nine were Utah Health Sciences faculty members, and seven were employed outside of the university. All 63 students matched a project that they had ranked, with 28 students (46%) matching with their first choice, including six students who automatically matched with their own proposed projects. Of these students, 61 (97%) completed their projects on schedule by April 2019. Two students fell behind and joined the following class of students.

Sixty-two students completed surveys in August 2017 (T1), December 2017 (T2), and April 2019 (T3), which is a 100% response rate for students still in the program (Figure 1). There were no significant changes in students’ knowledge, nor in attitude scores. The Class of 2019’s attitude scores were not significantly different from the baseline scores of the Classes of 2017 and 2018 (median of 16 vs 14 out of 20, respectively. Most items in the attitudes section showed little change (Table 1). Students’ interest in learning more about biostatistics decreased slightly, and the number agreeing that they understand the statistics that they encounter significantly increased.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Measures of Knowledge, Confidence, and Attitude Regarding Doctor of Pharmacy Student (Class of 2019) Participation in a Required Longitudinal Research Project

There was a significant increase in students’ confidence scores, p<.001. Post-hoc analysis showed a 31% higher median confidence score at T3 compared to T1 (29 to 38, p<.001). The changes from T1 to T2 and T2 to T3 were also significant (p<.001 and p =.0002, respectively). From T1 to T2, 63% of students’ confidence scores increased by more than two points, and from T1 to T3, 82% of students’ confidence scores increased by more than two points. The Class of 2019 graduated with higher confidence scores than the Class of 2017 or Class of 2018 (median 38 vs 31, respectively, p<.0001). The number of students indicating that they had greater than moderate confidence for all 10 items increased significantly over time (Table 2). Students were most confident in their ability to review and evaluate the medical literature.

The additional questions at T3 showed that 45 students (73%) felt their project would make them more competitive when applying for a job or residency. Thirty-six students (58%) presented posters at a professional meeting. Ten students submitted a manuscript to a peer-reviewed journal, and three of the manuscripts were accepted for publication. Additional students published their projects after graduation, bringing the total number of papers published to 13 (21%). On 12 of the 13 manuscripts accepted for publication, the student was the first author.

A majority of mentors and co-mentors (35 of 50, 70%) completed the mentor survey in May 2019, and 28 of 46 (61%) completed the survey in May 2018. In both 2018 and 2019, mentors reported spending a median of 15 hours per student per semester on project mentoring activities. Most mentors in both years reported spending “a reasonable amount of time” on each activity related to mentoring, ranging from 63% to 92%, depending on the year and activity. However, 30% of mentors reported that reviewing proposal drafts took too much time in 2018; however, this declined to 8% of mentors who felt this way in 2019. Likewise, 14% of mentors in 2018 and 18% in 2019 thought that completing forms and/or rubrics took too much time, and 15% in 2019 thought reviewing manuscript drafts took too much time. Further, 88% of mentors reported that they were somewhat or extremely satisfied with the final project produced, 73% thought the students’ work was beneficial to the mentor, 97% thought the project was a valuable learning experience for their student, and 79% felt they could write a strong letter of recommendation for their student based on their project mentoring experience.

DISCUSSION

A structured research program for P3 and P4 students at the University of Utah College of Pharmacy proved to be successful. Our results provide evidence that core curricular, mentored, hands-on experience with research and quality improvement projects is feasible. Furthermore, students’ confidence increased in the research process longitudinally compared to that in students who did not participate in the program. Because drug literature evaluation was included in the old curriculum and the new P2 curriculum, hands-on experience with the project increases confidence beyond what students achieved in a traditional literature evaluation class. These results are consistent which those reported in pharmacy residents.⁹ Such projects can be of high quality, resulting in increased peer-reviewed publications and presentations at national meetings.

Despite increased confidence, students’ knowledge of biostatistics as measured by the survey remained constant. A similar finding was observed in a study assessing pharmacy residents.⁹ We chose to assess students’ knowledge of biostatistics concepts using a previously published instrument, but acknowledge that this instrument may not have been the best tool to measure learning associated with projects, which emphasized application more than memorization. Students did demonstrate the ability to apply statistical concepts to their projects when applicable. Thus, participating in a project may be more effective in increasing students’ confidence and preparing them for future self-learning compared to students not participating in projects.

Interestingly, although the overall score assessing student attitudes towards research did not change significantly, there were some shifts in responses to individual questions in this domain. The slight decline in desire to learn more about biostatistics and research could indicate that some students had learned as much as they wanted to by participating in the project, as the proportion of students indicating an understanding of most statistical terms increased. Most students agreed that research is important for pharmacy as a profession, even if they were not necessarily interested in doing it themselves. Together, these results suggest that projects increase the students’ comfort with research, but may not necessarily increase the number of students pursuing research-oriented career paths.

Both mentors and students reported benefits from being involved in the research project. A majority of students indicated that exposure to research made them more competitive for jobs and residencies. Some commented that working on a project was particularly useful in job or residency interviews, helping them stand out and providing them with useful examples of problem-solving skills and ability to overcome obstacles. Three of four mentors reported benefiting from students’ work on the projects, and even more indicated satisfaction with the final products. Over half the students presented their findings at professional meetings, and as of November 2019, 13 (21%) had published their results in a peer-reviewed journal. These rates compare favorably with those published in a systematic review of pharmacy student required research projects.⁴ This was accomplished with most mentors reporting that their time commitment was reasonable.

We designed the PharmD projects program to overcome commonly noted obstacles with similar programs, such as limited faculty time, lack of interest from faculty members or students, and challenges with completing projects on schedule.^1,5,7,12-14 The program was structured to manage the faculty mentor time burden by providing feedback and instruction in dedicated coursework, and by recruiting mentors from outside of the college, which also strengthened the college’s partnerships. Having a central course that provides uniformity and feedback reduces the burden on mentors, allowing them to focus more on their area of expertise.¹² The project ranking system helped ensure that students and mentors worked on projects that matched their interests. Allowing two years for project completion and providing regular deadlines and feedback helped students finish their projects before graduation. An alternative solution to save faculty time and ensure completion would be to require completion of a research proposal only. However, there are advantages to requiring students to complete assignments in all phases of the research process, from conception to dissemination, including with providing students opportunities to acquire proficiency in higher levels of Bloom’s taxonomy.¹⁵ Programs that do not require students to move beyond the protocol/proposal stage are less likely to move their students into these higher levels of learning or for the research project to generate presentations or publications. Also, the increase in student confidence is likely to be smaller.^3,4,13

Some limitations of the program are worth mentioning. We found that non-faculty mentors were more likely to need support and guidance for issues such as study design, statistical analysis, and IRB applications. Some students had to reduce the scope of the project or their sample size to finish their research on time, but this is a real-world research experience. Identifying ideas and mentors for approximately 60 students a year is also a challenge. We are currently in the third year of this program, and so far, we have been able to match all students to projects. This may become more challenging over time, or perhaps previous years’ projects will generate new research questions for subsequent projects. Individual projects for all students may not be feasible in colleges with larger class sizes, but group projects may be. Even with dedicated coursework over two years, student availability to work on their projects was still an obstacle. Mentors in basic science departments in particular expressed challenges with the amount of time that PharmD students were available to learn laboratory techniques. In future years, mentors will be adapting the types of projects they propose to require less time in the laboratory.

CONCLUSION

A two-year core curricula capstone program in a four-year PharmD program at a public institution showed that requiring PharmD students to complete a research or quality improvement project during their P3 and P4 years was feasible, increased students’ confidence in their ability to participate in research, and increased student and faculty poster presentations and peer-reviewed publications. These findings support the consideration of the Academy that analysis, synthesis, and creation of new knowledge can be successfully implemented into core PharmD curricula.