Incorporation of Professionalism Expectations and Evaluative Processes Within a College of Pharmacy

INTRODUCTION

The Accreditation Council for Pharmacy Education (ACPE) 2016 Standards established an expectation for schools and colleges of pharmacy to develop students into professionals.¹ Clearly defining and accurately assessing professionalism remain an elusive task.^2-5 The Standards’ definition of professionalism is rooted in behaviors such as trust bestowed on the profession by various stakeholders (patients, other health care providers and society).¹

The Accreditation Council for Graduate Medical Education (ACGME) describes professionalism as a core competency with an emphasis on ethical principles.^2,4 Beyond health disciplines, the legal profession refers to professionalism as high ideals.⁶ The Armed Services define professionalism within the respective service’s core values.⁷ While there are slight differences in how professionalism is defined by different professions, all professions have a clear commitment to support s development in this area.^8,9

The Standards incorporate professionalism as a core requirement (Standard 4, Key Element 4.4).¹ The basis for this inclusion was the publication of new educational outcomes from the Center for the Advancement of Pharmacy Education.¹⁰ It has long been an expectation that all pharmacy graduates develop a level of professionalism consistent with their commitment to the profession.^11,12 ACPE requires programs to exemplify and inculcate professional attitudes and behaviors, however, it defers the definition of these terms to the individual programs.¹

Literature supports the need to inculcate and assess professionalization across curriculums.^9,13 Programs are responsible for clearly defining expected professional attitudes and behaviors so that the curriculum can effectively support student development. To address this important area, the University of South Florida (USF) College of Pharmacy (COP) began considering the nature and extent of programming and corresponding assessment modalities necessary to foster consistent professionalization.

The USF COP resides in USF Health, which comprises the following disciplines: medicine, physical therapy, athletic training, physician assistants, nursing, and public health. Each of the USF Health programs includes professionalism as a student expectation within their respective student handbooks. In 2011, discussions were initiated to ensure USF Health faculty and staff appropriately exhibit professionalism. Consideration was given to include professional behaviors within faculty and staff evaluations; however, formal implementation has not yet occurred.

This article will describe how the USF COP has developed a programmatic plan to address professionalization, including the results of a performed factor analysis, and how it has informed programmatic enhancements and corresponding assessment processes. The USF Health Institutional Review Board (IRB) granted this study exempt status.

METHODS

In 2013, the USF COP began the process of identifying mechanisms to ensure continued, positive professionalization across the program. The Professionalism Task Force (PTF) was created, which included representatives from both academic departments and experiential education. The PTF decided the best initial step was to assess faculty and preceptor perceptions regarding professionalism using a factor analysis. A concerted literature review guided the determination of factors for inclusion.^11,12,14-28 After careful consideration, it was decided that the factor analysis would be derived from the Behavioral Professionalism Assessment Form – Experiential (BPAE; Table 1).²⁰ The BPAE is a validated scoring system that is used to assess the perception of professionalism in a variety of practice settings. The BPAE was chosen because it encompassed most of the professionalism attributes described in the literature.

The factor analysis consisted of four sections, the first dealt with prioritization of the 25 items contained within the BPAE. Respondents were asked to rank each item’s importance using a 10-point Likert scale (0=no importance, 10=extremely important). Section two asked respondents to rank the 10 most important factors to be included on Pharmacy Practice Experience (PPE) student evaluations. Section three asked participants to identify three factors to be included in course syllabi. Section four asked participants to identify five factors for inclusion on faculty and preceptor evaluations.

The factor analysis was administered using the Qualtrics (Qualtrics, LLC, Provo, UT) survey system. All faculty and preceptors were sent an email that included a web link to the survey. The survey was accessible February through April 2015. Data analysis occurred May through July 2015.

Factor analysis findings were used to guide subsequent PTF discussions. The original plan focused on the professionalization of students, faculty, and preceptors; however, upon further consideration, the PTF decided to expand the focus to encompass the entire program (ie, admissions, student affairs, and staff). Expansion of the plan focus was necessary as professionalization is a longitudinal, lifelong process, thus, assessment cannot be static or singular. Additional individuals were added to the PTF to represent these areas.

RESULTS

When the survey was administered, the COP consisted of 45 faculty and approximately 200 preceptors. All responses deemed usable (ie, complete response per section) were included. Demographics were not collected, thus participant background (whether faculty or preceptor) is unknown. Forty-two respondents completed the factor analysis (section one). From these respondents, 35 provided feedback on PPE (section two) and syllabi inclusion (section three), while 32 provided feedback on faculty and preceptor evaluations (section four).

The information from section one, rankings of all 25 factors, was used by faculty and preceptors to assist with the rankings in sections two through four (PPE, syllabus, and faculty/preceptor evaluations). The findings for sections two through four demonstrated a consistent pattern of distribution (Table 2). Three factors (1, 3, and 13) were selected for each of the three sections (two through four). Although 10 factors were to be identified for PPE, an additional factor (6) was included since it was ranked within one standard deviation of the other categories. Additionally, the subject matter for factor 6 directly aligns with other selected factors, is readily attainable within the experiential environment, and current evaluations include limited opportunities for assessment.

The PTF reviewed findings and discussed analysis interpretation considering obvious limitations (eg, number of respondents, respondent demographics, etc.). The loading of the factors across the sections was deemed sufficient to guide initial plan development for programmatic incorporation. The goal was to develop an inclusive plan from student entry through graduation, and to ensure faculty, preceptors and staff were prepared to serve as role models. Through these discussions, it was recognized that input from additional COP stakeholders was necessary. The COP instructional designer was consulted to identify methods for incorporation across the didactic curriculum. Similarly, representation from admissions, student affairs and human resources were added to the PTF.

DISCUSSION

The Experiential Education Office (EEO) determines the most appropriate assessment processes and accountability mechanisms for students, preceptors and sites. The PPE director and coordinators met and discussed the factor analysis, specifically, whether the identified items were being addressed within existing evaluations. Additional discussion focused on how to incorporate the new factors and ensure an equitable and valid evaluation process.

When considering how to incorporate the new factors, the PTF suggested to either use a set percentage of the student grade (eg, recommend no more than 50%) or a tiered system (eg, 10% PY1, 25% PY2, 33% PY3, and 50% PY4). Regarding grading, current PPE evaluations use a Likert scale similar to the BPAE, thus the new items could be directly incorporated. Lastly, the PTF suggested a rubric to enhance inter-rater reliability.

Professionalism is assessed during Advanced Pharmacy Practice Experiences (APPE) rotations using the System of Universal Clinical Competency Evaluation in the Sunshine State (SUCCESS) developed by the University of Florida.²⁹ The SUCCESS instrument includes several of the PPE factors noted in Table 1, albeit with a slightly different language. The SUCCESS software platform underwent changes enabling the USF COP to make minor language adjustments, yet maintain alignment with other Florida programs. For the IPPE sequence, the EEO decided to initially incorporate nine factors from Table 2 (1, 3, 4, 5, 6, 7, 10, 11, and 13) with Likert-scale grading. Percent PPE grade delineation is as follows: 25% for IPPE (PY1 - PY3), and 20% for APPE (PY4).

Language for all PPE syllabi, evaluations, and manuals were enhanced for academic year 2016-2017. Preceptors were notified accordingly via email and an online rotation management system (CORE ELMS). They were informed of the COP early intervention process and encouraged to communicate professionalism developments to their respective PPE coordinator. Another change that occurred at the start of academic year 2016-2017 was the separation of preceptor and site evaluations to allow for individualize, direct feedback. The process used to evaluate preceptor professionalism will be discussed in the faculty and staff professionalism section.

Student evaluations are reviewed at least twice per semester by the PPE coordinators to identify those who are not meeting expectations, whether academically or due to professionalism concerns. The PPE coordinators report student failure secondary to professionalism to the Office of Student Affairs (OSA).

Future efforts will focus on enhancing direct communication with preceptors and ensuring appropriate accountability. The EEO plans to share identified, persistent themes from assessments, and examples from student evaluations during preceptor development programming. Students receive training from the OSA pertaining to student groups, and faculty and course evaluations. Similarly, the EEO will educate students with regards to providing effective feedback to preceptors.

Enhancements to data collection and analysis mechanisms are inevitable. Monitoring the utility of the nine factors incorporated for IPPE will help determine whether and/or when the other two factors need to be added. Monitoring of APPE is also required because the items within SUCCESS were not an exact match to the BPAE. The utility of the instrument will be monitored to determine whether a more detailed, objective instrument is necessary (beyond the Likert scale) to assess student progression more accurately. Lastly, the EEO plans to obtain direct feedback from preceptors through practice venue-specific advisory boards.

The three factors identified for incorporation into syllabi were items 1, 3, and 13 (Table 2). The PTF reviewed the course syllabi for current levels of inclusion and found different practices. Whereas several courses allotted between 1% and 3% of the total course grade for professionalism, others incorporated a distinct number of points. It became apparent that implementation across the didactic curriculum needs to consider course format and assessment processes. A global need, regardless of course format, is to ensure sufficient weighting to cultivate appropriate professional behaviors.

Possible methods to accurately measure professionalism across the curriculum include: in-class assessments with weight counted toward the course grade, standardized point reduction for unprofessional behavior, and establishment of a professionalism grade point average (in conjunction with the academic grade point average). Consideration for how professionalism may be consistently included across the didactic curriculum include: late arrival to class, late assignment submissions, disruptive behavior during class, and lack of participation during team activities. Some of these items are already being captured (eg, late submissions are highlighted in the Canvas learning management system).

Another possible mechanism for capturing professionalism, especially accountability and responsibility, is through the use of peer and self/peer evaluations. The use of self/peer evaluations requires appropriate student training and verification by faculty. If utilized, computer-aided verification processes will need to be incorporated. Alternatively, the factors could be limited to certain courses (eg, Pharmaceutical Skills and Drug Delivery Systems sequences) or activities (eg, Pharmacy Longitudinal Research Project and interprofessional education activities) that encompass significant professionalism components.

Lastly, the PTF identified the need to enhance existing standardized language in syllabi and the student handbook in certain areas, such as examination decorum, to foster professionalization. Many pharmacy programs have incorporated detailed language, often in a step-wise manner, to promote specific behaviors. Some of these practices, however, may actually discourage independent and critical thinking. The professionalization process requires that students make active decisions and choices to attain specified expectations, and may encompass variable, but equally effective, approaches.

Based on these discussions, PTF suggestions to the Curriculum Committee (CC) included the addition of the three factors to the syllabus template as required objectives, and to the Pharmacy Longitudinal Research Project (PLRP) Manual. The suggested process for inclusion in the syllabus template is the addition of a distinct section, separate from course objectives, in which the factors are listed with brief rationale for the inclusion. Although attention to some of the factors may already be addressed in other sections, such as absence policy, purposeful reiteration would be beneficial. Similar to other course objectives, the items would be mapped to specific activities, assignments, and assessments.

Before proceeding, the PTF recommended that the CC survey course coordinators to identify where and how the factors are being addressed. Courses that already included professionalism as part of the grading process (eg, the Pharmaceutical Skills and Drug Delivery Systems sequences), will be encouraged to ensure there is appropriate data collection to monitor student outcomes.

Incorporation into the PLRP would include assessment of students and mentors. The PLRP Committee was asked to add language to the Manual commencing academic year 2017-2018. Evaluation frequency and percentage grade allotments need to be determined. It is anticipated that the mentors will be evaluated in a similar manner to faculty and preceptors, which is described in the next section. Incorporation into the didactic curriculum is anticipated to be the most challenging initiative undertaken. Careful implementation with corresponding assessments is necessary to allow for timely modifications.

Five factors were identified for inclusion in faculty, preceptor, and staff evaluations (1, 3, 6, 11 and 13; Table 2). The predominant form of feedback obtained regarding the professionalism of faculty and preceptors is from students through course, faculty, and preceptor evaluations. The OSA counsels students on how to provide constructive feedback; however, actual feedback is not assessed (as it is often provided anonymously). Although student feedback in this format is valuable, there is a need for additional perspectives. Similarly, when supervisors complete performance evaluations for staff, the incorporation of additional perspectives may offer a more balanced and complete assessment.

The PTF discussed various mechanisms for incorporating professionalism into the existing annual evaluation process. The first step involved a review of existing forms. Identified documents used to support faculty, preceptor, and staff included: data from faculty/preceptor evaluations, peer evaluations, and letters from a clinical site supervisor or an external constituent (eg, committee chair or research collaborator).

To broaden and enhance the annual evaluation process (for faculty, preceptors, and staff), the USF COP intends to include input from various stakeholders. Example assessment instruments include self-evaluation, peer evaluation, student input (eg, advisee, laboratory technician, or other), clinical site peer and/or supervisor, and research peer and/or supervisor. The respective supervisor (department chair, OEE coordinator, associate dean, etc.) will derive their summary based on the information collected.

Since several of the identified documentation mechanisms are subjective, there is a need to develop instruments that capture this type of data. Considerations include the development of rubrics similar to those used during the tenure and promotion process. The rubrics would include a detailed description of score equivalencies to enhance inter-rater reliability. The rubrics would also include an open-ended section for comments.

Department chairs are now determining how to incorporate the factors into the annual evaluation processes. The faculty annual evaluation form already incorporates self-evaluation. The faculty also provide information that facilitates the determination of teaching and service assignments to ensure appropriate work load distribution. There are mechanisms that faculty can use to secure additional supportive documentation from external constituents. The development of supplemental forms to standardize this process that includes rubrics and weighting (as described above) is ongoing.

Further discussions with the EEO are necessary to enact similar evaluative processes for preceptors. The PPE evaluation forms were enhanced to afford students the opportunity to provide direct feedback to preceptors (previously combined with the site evaluation). The PPE coordinators emphasize the need for constructive student feedback regarding preceptor professionalism and continually emphasize the importance of participation in preceptor development. The PPE coordinators also review preceptor evaluations on a continual basis depending on year (IPPE or APPE) and scheduled site visits based on identified concerns.

Future considerations include an annual review process for preceptors such as that undertaken by faculty, with adaptations depending on affiliation. Preceptors who have been afforded academic rank already undergo regular review. The review process would afford preceptors the opportunity to provide supportive documentation (objective and subjective). As noted earlier, student evaluations will be developed for PLRP mentors. The form used for faculty and preceptors will be adapted to address PLRP-specific processes.

Lastly, the staff are evaluated annually via a standardized form developed by USF Human Resources. Several factors are already included in the form, using slightly different language. The form gives the chance to assess additional parameters, which could encompass the remaining, unaddressed factors. Similar to faculty and preceptors, staff would be expected to provide objective and subjective supportive documentation. Further discussions are necessary with staff and supervisors before implementation.

When the PTF developed implementation plans for the previously discussed areas, thoughts arose regarding other opportunities that would afford more complete, programmatic coverage. Identified areas included student admissions, orientation programming, and the Student Success Program. Recent internal analyses pertaining to student matriculation and attrition identified professionalization as a focus area; thus, efforts to enhance student success need to include professionalization. These discussions also identified a need for longitudinal monitoring, which necessitates the development of prospective data collection and analysis processes.

Based on these considerations, discussions were initiated with the Office of Admissions and Admissions Committee to understand existing efforts toward determining candidate professionalism. Applicants are required to complete a supplemental application in addition to traditional PharmCAS requirements that capture data pertaining to professional attitudes and behaviors. Beginning fall 2017, the supplemental application was modified to include a professionalism essay (ie, take a moment to describe what professionalism means to you, and then also briefly describe how your non-academic experiences have prepared you to be a professional student within the College of Pharmacy). The admissions interview process uses Multiple Mini Interviews (MMI) and group activities. Professionalism behaviors addressed within each MMI station include perceived maturity and communication skills. Admissions candidates receive an average score for each of these areas, which is added to the aggregate MMI score.

The Admissions Committee plans to review the 11 items identified for PPE to determine which are already being assessed within existing admissions processes. Thereafter, the Admissions Committee will discuss how the omitted factors can be incorporated into either the holistic admissions process, interview day or corresponding evaluations. The role professionalism will play in the decision-making process requires further discussion. Lastly, consideration is being given as to whether a retrospective review would be helpful prior to prospective data collection. Discussions remain ongoing with tentative enhancement planned for academic year 2018-2019.

Another potential opportunity to address professionalism is through orientation programming. Initial discussions have focused on ensuring that consistent information regarding professional expectations is being delivered during both new and returning student orientations. Further discussion is necessary regarding the nature and content of programming and corresponding assessment processes. Similarly, the Student Success Program is a new initiative (implemented academic year 2016-2017), thus a review of current processes needs to be undertaken to appreciate existing processes and future opportunities.

The described initial efforts are directed toward the enhancement of existing COP processes. As noted earlier, other USF Health programs share an interest in ensuring the professionalization of students, faculty, preceptors, and staff. Current interprofessional endeavors across USF Health include the interprofessional education (IPE) curriculum and shared student services. The IPE curriculum includes nine modules that encompass the Interprofessional Educational Collaborative Competencies (IPEC).³⁰ Whereas pharmacy students participate in each of the nine modules across all four years of the PharmD curriculum, student and faculty participation from the other USF Health disciplines is module-dependent. How to concertedly address professionalism across the IPE curriculum is a future aspiration.

Shared Student Services currently encompass a variety of student affairs-related areas (eg, admissions, registrar, financial aid, etc.). Some, but not all, USF Health programs are involved with this initiative. Efforts are ongoing to streamline processes, which may lend itself towards additional shared programming that may include professionalism.

Lastly, a survey was administered in 2015 to qualify and quantify baseline professionalism for faculty, preceptors, staff, and students (as part of a PLRP). The results of the survey are currently being analyzed. Beyond establishing baseline values, additional objectives of the project include: the identification of development needs, question validation, and the determination of re-administration time frames. If deemed supportive and beneficial, the instrument may be incorporated into the assessment plan.

The PTF has developed a draft assessment plan to organize the processes being undertaken across the COP (Table 3). As noted, additional discussions are necessary pertaining to the development of new and/or enhancement of existing assessment instruments, administration time frames, data collection, and analysis. The goal remains to develop a programmatic quality improvement process.

CONCLUSION

The USF COP convened a PTF with the goal of establishing expectations for the COP community to ensure the culture supports and values professionalism. The original focus was to develop strategies to capture professionalization and thereafter ensure the availability of appropriate development opportunities. The performed factor analysis was successful in identifying professionalism items for inclusion into PPE, didactic syllabi, and faculty and preceptor evaluations. The PTF further broadened its original focus to encompass other related, yet similarly important areas (eg, staff evaluations, admission, and student affairs) and corresponding assessment strategies. Since professionalization cannot occur when institutional inconsistencies persist, implementation will include the enhancement of existing and development of new processes. It is hoped that the proposed programmatic plan will further support student professionalization, while fostering the continued professionalization of faculty, preceptors, and staff.