Identifying Student Research Project Impact Using the Buxton and Hanney Payback Framework

DISCUSSION

The most important finding in this study is that student quality improvement and research projects were associated with several different types of impact for the preceptors and the practice sites, with the greatest impact in the domains of knowledge production and health and health sector benefits. These results highlight that student research is contributing to practice sites and the community. The three most frequently identified types of impact were that the projects helped the site in some way, contributed to knowledge production, and increased satisfaction for the mentor or staff. Impact was also identified for the domains of benefits to future research, informing policy and product development, and economic benefits.

When considering the specific BHPF domains, knowledge production proved to be the area with the clearest evidence. Additionally, the data collected for this domain reflected a similar presentation rate to previous work in this area.^2,8,12,13 For example, Assemi and colleagues reported project dissemination rates of 42.3% as a submitted publication, 37.8% as a poster, and 4.5% as an oral presentation.² Warholak stated that 39% of preceptors reported that the project poster was presented at least once in addition to their required presentation, and 19% were planning to present in another setting.⁸ Similarly, in this project, 39% (n=13) of preceptors reported on the questionnaire that they would be presenting the project as a poster presentation in addition to the students’ required school presentation (see Table 2).

The data from the benefits to future research domain highlight an area for improvement, as 12 preceptors agreed with the statement that “student projects [don’t] lead to the initiation of future projects.” Many preceptors and sites would benefit from using student projects to help build on existing projects or to contribute to starting new projects, but preceptors may need additional training to plan projects this way. In previous research, project mentors have reported that students completing projects on site provided an additional personnel resource.^2,8 There was little evidence to demonstrate that the student projects contribute to informing policy and product development. While this is an important area of impact, it was difficult to capture in a questionnaire or review of the student posters. We have knowledge of a specific example in this area: a quality improvement project from another cohort led to a product labeling change. However, this example would not have been easily gathered via the questionnaire or poster evaluation, but it likely would have been better captured through preceptor interviews or a longer-term study. In 2015, Assemi and colleagues reported that students enrolled in research-intensive health services policy research and pharmaceutical sciences pathways did have greater dissemination rates then students in other pathways, with a higher percentage of students publishing or presenting their findings.² However, they did not report how the nature of the student project informed policy or product development. Evidence for impact in health and health sector benefits was identified by both research methods, and this domain of impact aligns with the goals of most project objectives. Finally, the evidence for economic benefits of the student projects were captured more frequently by the preceptor questionnaire.

As stated earlier, student research projects are important to student learning and contribute to student and faculty scholarship; however, until now, it has not been clear how these students’ projects impact preceptors, clinical sites, patients, and the broader community.^1,12,14,15 With growing interest and focus from national funding agencies on the broader impacts of research, it is time for pharmacy educators to look at the numerous student projects conducted each year and begin inquiring about how these projects can better serve our communities. This study highlights that further work is needed if we are to adapt the BHPF as a framework for identifying student project impact, as it may not have captured some of the more nuanced areas of impact for some student projects.

This study serves as a starting point for schools and colleges of pharmacy seeking to describe the impact of their student projects beyond faculty time commitments, publications, presentations, and student learning. Student research projects take an incredible investment of time and resources from faculty and students but have been demonstrated to be an important learning tool for our students.¹ This study highlights that student research projects also provide contributions to the communities served by schools of pharmacy. Broader implications of impact may be that it boosts schools’ reputations, increases award nominations (eg, American Association of Colleges of Pharmacy Lawrence C. Weaver Transformative Community Service Award), continues to improve the quality of projects and activities, and, critically, demonstrates the value that schools of pharmacy bring to their communities.

Future work in this area should further adapt the BHPF and incorporate some of the broader impact areas described by the National Science Foundation, the University of Minnesota areas of impact, and the Sustainable Development Goals outlined by the United Nations.^16-18 Also, future work should incorporate additional research methods such as interviews or focus groups with project preceptors. Further, the results from the preceptor questionnaire provided slightly different responses than what was found upon review of the project posters. This is to be expected given the different methods of data collection; results reported on posters are often aimed at a specific audience and may not highlight the impact of a project, and preceptor questionnaires were subject to recall bias and asked questions in general terms (vs about specific projects). Additionally, these challenges align with previous research experience, where researchers found it difficult to accurately catalog student and faculty presentation and publication rates.¹² Publication and presentation data are difficult to collect, vary depending on the source, are often incomplete, and can be highly dependent on the study timeframe.¹²

This study had limitations that should be noted. First, the study identified examples of the impact of pharmacy student projects from a single cohort of PharmD students from a single college of pharmacy. Second, the study identified the impact of the projects done by the class of 2020 in 2021, and data collection may have occurred too early to assess the impact of some of the projects. For example, anecdotal reports from preceptors indicated that while some student projects reported no significant difference on the scientific poster, the impact was longer term and, therefore, would be seen several semesters or years after the project was completed. Thus, future studies should identify impact over a longer period of time. Third, the questionnaire response rate was limited, hence the findings related to impact are likely not representative of all the types of impact possible. Future studies should use a more parsimonious survey instrument to optimize response rates.