Assessment of Volunteer Preceptors at Colleges and Schools of Pharmacy in the United States

INTRODUCTION

Experiential education comprises approximately one third of the Doctor of Pharmacy (PharmD) curriculum.¹ While full-time faculty preceptors account for a portion of the total preceptor workforce, volunteer preceptors (persons who are employed by practice sites) play an integral role in the education of future pharmacists. This essential role is due in part to the expanded requirements of diverse experiences in a variety of settings and patient populations at lower student to preceptor ratios noted in the Accreditation Council for Pharmacy Education (ACPE) Standards 2016.¹ The accreditation standards state that programs must have quality criteria for preceptor recruitment, orientation, performance, and evaluation.¹ Furthermore, programs are expected to establish methods to assess the performance of preceptors in their ability to facilitate learning, communicate effectively, serve as a professional role model and mentor, and positively represent and advance the profession.² Neither specific guidance on how to assess preceptors nor a comprehensive list of preceptor attributes is provided, thereby challenging each program to create its own assessment criteria and procedures.

Several researchers have explored the topic of preceptor assessment. O’Sullivan and colleagues identified specific preceptor teaching-coaching, role modeling, and facilitating behaviors valued by pharmacy students and developed measures for those behaviors.³ Assemi and colleagues interviewed personnel from 29 experiential education programs to identify common practices used for measuring the quality of their programs. Based on their findings, they offered recommendations for the way in which preceptor assessment could be incorporated into an overall quality improvement process for experiential education.⁴ Larson and colleagues developed a rubric to assess what an expert panel determined to be habits of effective preceptors.⁵ Kirschenbaum and Zerilli studied the way in which colleges and schools of pharmacy assess full-time faculty preceptors.⁶ No literature has been identified to document the manner in which volunteer preceptors are assessed. The role of volunteer preceptors is critical in the education of pharmacy students, yet most volunteers enter the role of preceptorship with limited knowledge about or experience in precepting and little if any formal education in this area. As a result, and as required by the accreditation standards, pharmacy colleges and schools must dedicate significant effort and time to the development of volunteer preceptors into effective teachers, mentors, and role models.^1,7-12 The objective of this study was to identify the criteria for and methods and frequency of assessment of volunteer preceptors used by colleges and schools of pharmacy in the United States, with a specific focus on volunteer preceptors who precept students in advanced pharmacy practice experiences (APPEs).

METHODS

To develop the survey used to capture the needed data, we defined volunteer preceptors as individuals who were not full-time (salaried) members of the faculty, not co-funded faculty members (such as receiving half of their salary from the college or school and half from the practice site), and not otherwise employed by the institution for which they precepted students. However, volunteers were not excluded from the study if they had received a stipend/honorarium for precepting one or more students. The survey focused on APPE volunteer preceptors to capture individuals who precepted PharmD students in a wider variety of practice experiences and settings. Also, the intent was to avoid respondent survey burden in having to report potentially similar or duplicate information for individuals who precepted students enrolled in introductory pharmacy practice experiences (IPPEs).

A draft questionnaire was designed and pretested by persons with knowledge of the subject area at three different pharmacy colleges/schools. These individuals were asked to assess the proposed survey instrument for clarity, completeness, flow, and time needed for completion. Revisions were made to the survey instrument based on the feedback received. The resulting instrument and cover letter were then submitted to the institutional review board at Long Island University, and the project was granted exempt status.

The final online survey contained 116 items; however, participants were not required to answer all items. Instead, the instrument was developed in such a way that an individual’s response to a question determined which follow-up items were asked. The instrument included questions about the characteristics of the respondent’s institution, promotion and reappointment procedures for volunteer preceptors, abilities and attributes assessed for APPE volunteer preceptors, methods of assessment, and frequency of conducting each assessment. The attributes included in the survey were compiled using the Guidance for Standards 2016 document, in combination with attributes identified in the literature for preceptors.^2,3,6 The time required for participants to complete the survey was estimated to be 20 minutes. Respondents were asked to have access to select items as they completed the survey (eg, preceptor evaluation criteria used by students and/or other individuals, policy and procedure documents).

A list of directors of experiential education (or equivalent title) at accredited colleges and schools of pharmacy was created from a review of their websites and via personal communication. The list was further refined by eliminating institutions that were in candidate status and had not yet graduated a class of students. The survey was electronically distributed via Baseline (Campus Labs), an online survey portal, in June 2018. Survey recipients were informed that the researchers intended to collect survey responses from only one individual per program, so if the recipient was not the appropriate individual, they should reply to the investigators with the name and contact information of someone better equipped to participate on behalf of that college or school. In such cases, a request to participate was e-mailed to the newly identified individual. Three follow-up email reminders about the survey were sent to prospective participants approximately two to three weeks apart, and telephone calls were subsequently made to nonrespondents.

Baseline was used to collect data and generate descriptive statistics. Chi-square tests for independence were conducted using SPSS Statistics, version 25.0 (IBM) to ascertain whether the responding colleges and schools were representative of all institutions that were eligible to participate. Characteristics evaluated included institution type (private vs public), geographic region, years since founding, and average class size. Chi-square tests for independence also were used to determine whether there were significant differences between private and public institutions and those that were or were not affiliated with an academic medical center with respect to the preceptor abilities and attributes assessed and the methods used for such assessment. Yates correction for continuity was computed for all two-by-two contingency tables. All statistical tests were two-sided, and a p value of <.05 was considered significant.

RESULTS

Of the 140 US colleges and schools of pharmacy at the time of the survey, 132 were eligible to participate as they were accredited and had graduated at least one class of students. Responses were received from 81 of the 132 (61.4%). Not every respondent provided an answer to each question; thus, the data provided indicate the total number of respondents to each specific survey item. Eighty of 81 (98.8%) respondents reported that they evaluate the performance of APPE volunteer preceptors. As shown in Table 1, the characteristics of the US colleges/schools that provided demographic information did not differ significantly from the other pharmacy programs nationwide with respect to the percentage from private vs public institutions, geographic location, years since founding, and average class size. Thus, the responding institutions appear to be representative of all US pharmacy colleges/schools at the time the survey was conducted. The estimated number of APPE volunteer preceptors per APPE type at the responding colleges and schools are presented in Table 2.

The 80 respondents who reported evaluating the performance of APPE volunteer preceptors were asked to report on preceptor promotion and reappointment practices. Forty-one (51.3%) participants reported that preceptors affiliated with their institution could not be promoted in rank, while 39 (48.8%) reported that preceptors could be promoted. Thirty-seven of these 39 institutions (94.9%) reported that there was a specific promotion evaluation process. A majority (47 out of 80) of respondents reported that their institution had a process for reappointing preceptors. Of the 47 respondents, re-appointment was most frequently reported to occur automatically until preceptor resignation or termination (17 [36.2%]) or every few years (17 [36.2%]). Additional responses included annually (9 [19.2%]) and other (4 [8.5%]).

Information on the various criteria used by the pharmacy college or school to evaluate the performance of APPE volunteer preceptors was solicited. Seventy-nine participants answered the question on whether all volunteer preceptors of APPE students are assessed using the same criteria irrespective of practice setting or type of APPE (eg, community, acute care). Seventy-eight (98.7%) reported that their institution used the same criteria, while one (1.3%) used select criteria depending on the specific APPE.

Seventy-three institutions provided specific information on the abilities and attributes considered when evaluating an APPE volunteer preceptor (Table 3). Although the questionnaire focused on APPE volunteer preceptors, 69 of 73 (94.5%) reported that they used the same abilities and attributes when evaluating IPPE volunteer preceptors as they do for APPE volunteer preceptors. There were no significant differences found in the abilities and attributes considered by private vs public institutions. Colleges and schools of pharmacy affiliated with an academic medical center more commonly reported assessing a preceptor’s use of strategies to develop students’ critical thinking and problem-solving skills than those not affiliated with an academic medical center (χ²=4.47, p=.035). This was the only ability or attribute for which there was a significant difference based on academic medical center affiliation.

Respondents at 72 colleges and schools provided information on the methods and frequency of the assessment used to evaluate APPE volunteer preceptors. Respondents were asked to include preceptor assessment methods conducted by students, faculty, college/school personnel, site personnel, and the preceptors themselves. All respondents reported that APPE volunteer preceptors were assessed using the same methods, irrespective of the practice site or type of APPE (eg, community, acute care). The various methods used by pharmacy colleges and schools to assess APPE volunteer preceptors and the most commonly reported frequency of assessment for each method are presented in Table 4. Respondents reported using a mean (SD) of 4.1 (2.0) methods to assess volunteer preceptors. There were no significant differences in the assessment methods used by private vs public institutions. More colleges and schools affiliated with an academic medical center reported using informal feedback provided by a supervisor at the site than institutions not affiliated with an academic medical center (χ²=6.41, p=.011). This was the only assessment method for which there was a significant difference based on academic medical center affiliation.

Of the 61 respondents reporting that personnel from their college or school evaluate APPE volunteer preceptors at the practice site, 60 provided additional information on this process. They reported that a variety of individuals were involved in these evaluations, either alone or as part of a team (Table 5). Fifty-one (85.0%) respondents reported that their college or school provided guidance on the types of items to be considered in the on-site evaluation by college/school representatives. The most common items considered during an on-site evaluation were findings from an interview or discussion with students (86.3%), professional demeanor (78.4%), degree of collegiality (78.4%), review of student assignments, presentations, and/or activities (76.5%), adherence to the syllabus (68.6%), observation of the preceptor interacting with students (64.7%), observation of the preceptor in patient care areas (58.8%), comments from pharmacy staff members (56.9%), and comments from faculty members and other preceptors at the site (56.9%).

DISCUSSION

Every year, pharmacy programs rely on the support of thousands of pharmacists and occasionally other professionals to serve as volunteer preceptors in experiential programs (Table 2). These individuals are identified and recruited through many pathways but are often recruited based on their expression of interest in serving or on recommendations from individuals at a practice site.⁷ The continuous evaluation of these volunteer preceptors is necessary to ensure program expectations are met. Several reports in the literature have addressed the quality assurance methods and practices of experiential programs.^4-6 Nevertheless, to the best of our knowledge, this is the first report to detail the assessment practices used by US pharmacy colleges and schools to evaluate volunteer preceptors.

The most commonly assessed preceptor abilities and attributes were providing students with feedback, preceptor accessibility, communication skills, professional demeanor, an ability to set expectations, creating a positive learning experience, and serving as a role model. These are in line with role-modeling, teaching-coaching, and facilitating behaviors previously identified as being valued by students.³ Fewer schools reported assessing preceptor attributes related to representing and advancing the profession in a positive manner (eg, service to the community and profession, leadership). This could be due in part to the difficulty of gathering this information from volunteer preceptors, or colleges and schools may be assessing advancement of the profession in other ways not captured by this survey. The colleges and schools that participated in our study reported that the scholarship activities of volunteer preceptors were assessed less frequently than what was reported by participants in a study by Kirschenbaum and Zerilli of full-time faculty preceptors (26.0% vs 80.6%, respectively).⁶ This is not surprising given that the overall expectations for full-time faculty generally include some level of scholarly activity.

Notably, only about half of the responding programs indicated that they assess preceptors’ participation in continuous professional development related to precepting students. Considering that continuous professional development of preceptors is an accreditation requirement, we expected assessment of this attribute to have been more common among the colleges and schools responding to our survey.^1,2 It would be interesting to explore what pharmacy programs are doing to document that continuous professional development is ongoing among volunteer preceptors. The gap in assessment practices is in line with the findings of O’Sullivan and colleagues who reported that, while 89% of schools responding to their survey have experiential education faculty responsible for preceptor development, not all of them tracked participation.⁸

Despite the variable responsibilities of preceptors in different practice settings, the preceptor abilities and attributes assessed are the same. Furthermore, while this study focused on APPE preceptors, almost all respondents reported that they considered the same abilities and attributes when evaluating IPPE volunteer preceptors as they did for APPE volunteer preceptors. Though the preceptor abilities and attributes evaluated vary from program to program, no significant differences were found between private and public institutions regarding preceptor abilities and attributes they assessed. However, a significant difference was found between program affiliation with an academic medical center and assessment of a preceptor’s use of strategies to develop a student’s critical-thinking and problem-solving skills. The reasons for this finding are unknown but could be investigated.

Respondents reported using a variety of methods to assess volunteer preceptors. The most frequently reported method was summative formal student evaluations, which was expected given that this method is noted in the Guidance Document to Standards 2016.² This also was the most common method found when assessing full-time faculty preceptors.⁶ Importantly, student evaluations have been reported to be influenced by multiple factors and therefore may not be the best measure of preceptor ability.^13-16 Perhaps this is one reason why colleges and schools use multiple strategies for volunteer preceptor assessment. Using multiple assessment methods to evaluate preceptor effectiveness is consistent with the findings by Assemi and colleagues who reported on the overall quality assurance practices used by experiential programs.⁴ The second most frequently reported strategy was on-site evaluations by one or more college or school personnel. This assessment method was more frequently reported in our study than in the study by Kirschenbaum and Zerilli evaluating assessment practices for full-time faculty preceptors.⁶ Fewer programs reported using other assessment methods. For example, evaluating professional service was much less commonly reported for volunteer preceptors than for full-time faculty preceptors (12.9% vs 92.3%).⁶ The lack of widespread use of certain assessment methods might have been due to limited resources to support more robust assessment practices and the unverified value of select assessment methods.⁴ Interestingly, although there were no significant differences in assessment methods when comparing private vs public institutions, colleges and schools affiliated with an academic medical center were more likely to collect feedback from a preceptor’s supervisor at the site than institutions not affiliated with an academic medical center. The reasons for this are unclear.

As previously noted, although Standards 2016 state that colleges and schools of pharmacy must have quality criteria in place for preceptor recruitment, orientation, performance, and evaluation, specific measurable characteristics are lacking.^1,2 In the section of the guidance document dedicated to preceptor attributes, the emphasis appears to be on the role of the preceptor as a practitioner rather than as a student pharmacist educator.² For example, while the guidance notes that preceptors should demonstrate a desire and aptitude for teaching, no specific suggestions are provided to define a preceptor’s ability to facilitate learning or otherwise enhance the educational process. In addition, the only specified method to assess preceptor performance is obtaining feedback from students.² Having a more robust list of preceptor abilities and attributes, as well as suggested methods to evaluate the performance of preceptors, would be advantageous for pharmacy programs and enhance their overall preceptor recruitment, development, and assessment processes. Also, preceptors who accept students from more than one program might find it easier to deal with one set of expected abilities and assessment methods. Another potential benefit of common lists of abilities, attributes, and assessment techniques would be that pharmacy colleges and schools could collaborate and share assessment methods and data and minimize duplicative assessment efforts. The results from this survey may serve as a framework for developing such lists.

This study had limitations. Despite multiple attempts at soliciting survey participation, several programs did not participate. Nevertheless, the characteristics of our sample are reflective of the pharmacy colleges and schools across the United States (Table 1). Of those who did participate, 10% did not complete the entire survey, which may be attributed to the length of the questionnaire. The survey instrument was designed in such a way that respondents were not required to answer each question. As a result, the response rate for each item varied. Finally, as with any self-administered survey instrument, some of the participants may have misinterpreted some of the questions.

CONCLUSION

Assessment of APPE volunteer preceptors is an integral component of the quality assurance process at almost all responding US colleges and schools of pharmacy. Various preceptor abilities and attributes are considered in the assessment process, and most programs use a combination of assessment strategies, with all respondents using summative student evaluations. Data provided here can help inform programs about practices used by institutions across the United States and can be helpful for pharmacy faculty and administrators when comparing their practices to those at other institutions and perhaps implementing new ones. The results of this study also can serve as a framework for the development of standardized lists of preceptor abilities and attributes and assessment methods.