A Qualitative Analysis of Common Concerns about Challenges Facing Pharmacy Experiential Education Programs

INTRODUCTION

With the release of the American Council for Pharmacy Education’s (ACPE) draft accreditation standards for pharmacy education,¹ which emphasize preceptor development and assessment in experiential education (EE), colleges and schools of pharmacy will likely be making changes to accommodate these standards. Faculty members and administrators in EE programs will now be faced with new challenges, in addition to existing difficulties in student placement capacity created by dramatic growth in number of pharmacy schools and student enrollments in recent years.^2-8 Even though many schools have expanded and reorganized their EE programs,^9-13 further reorganization may be necessary to realign job responsibilities with the new emphasis on quality improvement, preceptor development, and interprofessional education.^14,15 Now is a good time take stock of all concerns facing EE programs so that appropriate changes can be made to meet new accreditation standards.

In 2008, Haase and colleagues reported expert recommendations for ensuring quality in experiential education in a white paper sponsored by the American College of Clinical Pharmacy.¹⁶ This report described insights from practicing clinical pharmacists, many of whom were pharmacy faculty members and preceptors. Additional insights from pharmacy preceptors about EE were reported in 2008 and 2013.^17,18 Concerns from hospital pharmacists regarding capacity for placements in the hospital setting have also been published in the past 10 years.^19-21 However, few reports have been published about what faculty members and staff directly administering EE programs perceive as concerns.²²

In 2011, Danielson et al disseminated a survey analyzing the curricular, personnel, and financial characteristics of EE programs to EE directors across the United States.¹³ As part of the survey, respondents were asked to describe the most pressing issues facing their EE programs. The results of this question were subsequently analyzed and are reported in this paper, so that additional insight from faculty members directly involved in administering EE programs could be shared publicly as many seek strategies for implementing new accreditation standards.

METHODS

In 2011, EE directors at all 118 accredited schools of pharmacy in the United States were surveyed using a web-based questionnaire (Catalyst Web Tools, University of Washington Information Technology, Seattle, WA). As part of the 35-item questionnaire, respondents were asked the following open-ended question: “What are the 2 or 3 most pressing issues your professional experience program is facing today?”¹³ Free-text answers were collected from May to December 2011 and downloaded to Microsoft Excel for analysis. Other data collected included type of institution (public or private) association with academic health centers, number of introductory pharmacy practice experiences (IPPEs) and advanced pharmacy practice experiences (APPEs), number and position of EE faculty members/staff, and class size. Class sizes were categorized as small (≤50 students), medium (51-100 students), large (101-150 students), and very large (>150 students). Identifying information about the individual completing the survey was not collected. Goal response rate was 70% (confidence 95%, margin of error 5%).^23-25 The project was approved as exempt by the University of Washington Human Subjects Division.

The investigation team included 2 faculty members (each with more than 10 years of experience in EE and 1 with training in qualitative methods) and 3 fourth-year doctor of pharmacy students. Investigators performed content analysis of the free-text responses using thematic analysis methodology informed by grounded theory.²⁶ First, the primary investigators, an EE expert (faculty member) and novice (student), identified standard themes through iterative, repeated comparison. They then independently coded the responses using the standard themes with the possibility of 1 response being categorized into multiple themes. Agreement between the investigators’ independent coding was calculated first, then common coding for the discrepancies was determined by consensus.

To check validity, additional reviewers (again, an EE expert faculty member and a novice student) independently coded the responses using the standard themes created by the primary investigators.²⁷ Consistency with investigators’ coding was first compared using percent agreement. The Cohen kappa coefficient (ƙ) was then calculated between the investigators’ and reviewers’ coding results. A ƙ>0.70 was considered satisfactory agreement in this analysis.²⁸ Final results were compared to results of a similar survey conducted in 2001¹³ to determine how concerns have changed over time.

RESULTS

Eighty-one colleges and schools responded to the survey (69% response rate), and 78 responses were received for the question of interest (66% response rate). Comparison of respondents with the known population at the time of the survey²⁹ showed that the proportions of responses received for publicly funded institutions (75%) and schools with class size 51-150 (medium 72%, large 71%) were adequate to represent these types of pharmacy colleges and schools. Since so few small schools exist anymore, results from the 1 small school that responded were combined with results for medium-sized schools. Thirty-three respondents (42%) reported being associated with an academic health center, which matches national statistics.²⁹

Eight common themes were identified as concerns (see Table 1). Good agreement was found between the investigators’ initial independent coding (mean 96%, range 86-100%) (Table 1). Mean percent agreement between investigators and reviewers for all themes was 93% (range 78-100%). Good agreement (ƙ≥0.7) was found between investigators and the expert reviewer for all themes except for the last 3 (assessment, onboarding, and support/recognition from the dean). Good agreement (ƙ≥0.7) was found between the investigators and the novice reviewer for all themes. (Table 2).

Site capacity remained as the highest concern over the past 10 years (Table 2). While slight differences were made in categorization between 2001 and 2011, concerns related to financial support for EE, workloads of EE faculty members/staff, preceptor stipends, and support/recognition by school administration were consistently reported at similar frequencies across years. New issues identified included preceptor development (18%) and procedures for placing and orienting students at individual practice sites—a process referred to as “onboarding” by many health systems (11%). Dealing with student problems was not mentioned at all in 2011.

Private institutions reported concerns about capacity and faculty member/staff workload more often than public institutions (Table 3), even though the former reported having more APPE and IPPE sites than public schools. No difference was found in number of EE faculty member and staff between public and private schools.¹³ Large and very large schools seemed more concerned about preceptor development and stipends than medium schools, and very large schools were most concerned about quality assurance compared to others.

The number of full-time equivalents and type of EE faculty member/staff seemed to be related to select concerns. Schools with fewer full-time equivalents mentioned workload and financial support more often than schools that employ more faculty members and staff. Schools without associate or assistant dean positions mentioned concerns related to workload and financial support as well as recognition from administration more often than schools with these positions. Schools without an associate or assistant director reported workload/financial support as a concern more often than schools with such positions.

Concerns related to site capacity were further divided into issues related to competition with other schools, increased class size, and difficulties with placing IPPEs or APPEs specifically (Table 1). Capacity difficulties related to competition with other schools were reported more often (40%) than difficulties associated with increased class size (12%), IPPEs (31%), or APPEs (26%).

When comparing results with number of practice sites claimed by each school, distinct differences became apparent. Those citing quality assurance as a concern reported more APPE (median 307) and IPPE (median 142) sites compared to those who did not (median 201 and 102, respectively). Responses expressing concerns about preceptor stipends reported higher numbers of APPE sites (median 380 versus 201). Contrary to expectation, those citing capacity as a concern reported higher numbers of APPE (median 535) and IPPE (median 150) sites as compared to those who did not mention this issue at all (median 212 and 100, respectively).

DISCUSSION

While the overall goal response rate was not achieved, these results were sufficient to be generalized to public institutions, schools with medium to large class size, and institutions associated with academic health centers. Since public schools with medium to large class sizes represent 69% of the entire population,²⁹ the results from this analysis can be generalized to many colleges and schools. Closer examination of the responses from underrepresented institutions did not reveal outlying differences. For example, the 1 small school respondent reported concerns consistent with other schools (ie, quality, adequate faculty members/staff in EE, and preceptor development).

Many concerns identified in this analysis were consistent with concerns raised in 2001,²² even though analysis methods were chosen to minimize influence from investigators’ assumptions or previously published conclusions. Including novices who were naïve about EE administrative functions and historical trends helped to mitigate inherent bias from the preconceived constructs that experienced EE faculty members brought to the analysis.

One difference noted was that workload of faculty members/staff in EE was combined with funding/support for EE into the same theme, whereas it was reported separately in 2001.²² Upon further examination of responses, it seemed appropriate that these issues were combined because funding and financial support was usually mentioned in reference to lack of sufficient faculty members and staff to do necessary work as opposed to need for more funding in general. And yet, the consistency of these results over time confirms that these concerns continue to be “top of mind” for EE directors.

One of the new concerns to emerge was the procedures required for placing students at practice sites, or onboarding, added significant paperwork and time constraints to EE staff when health systems began implementing new standards published by The Joint Commission (TJC) in 2011.³⁰ Some studies proposed streamlining placement procedures so students complete many of their APPEs continuously within 1 system, which reduces paperwork and improves learning outcomes.^20,31-35 These results would support use of block scheduling as a way to create efficiency.

The results also suggest an important challenge for EE programs is preceptor development, explicitly required by the 2016 draft standards.¹ Vos and Trewet suggested that a multifactorial approach of online, live, and print materials, combined with targeted preceptor development through site visits, improved preceptor performance.³⁶ This approach, although logical, is resource-intensive. Combined with results suggesting EE workload and financial support may already be suboptimal, additional resources will be required to address this growing need in experiential education. Although the number of faculty members and staff in EE programs has grown over time,¹³ these results suggest that the current full-time equivalent allocation for EE may still not be enough to meet future requirements.

That schools with an associate or assistant dean of experiential education reported fewer concerns about workload and financial support may indicate that individuals at this level are in a position to secure adequate resources for their EE departments. This trend would support recommendations made in 2005 that the ideal organization for experiential education include an associate or assistant dean as the department or program head.¹⁰ That schools without an assistant or associate director reported concerns related to workload would support that future recommendations include such a position in EE at all schools.

Concern about EE workload and financial support seemed to be inversely related to class size. More than half of medium-sized schools cited this issue, while only 33% and 15% of large and very large schools mentioned it, respectively. This trend may indicate that economies of scale are reached at a class size of more than 100 students. On the other hand, issues related to quality assurance grew with class size, which suggests that large numbers of students and sites spread faculty members and staff too thin to thoroughly evaluate all preceptors/sites regularly. For instance, private schools perceived more concerns about capacity and faculty member/staff workload when they had more sites but had no more faculty members or staff to manage them. Lack of ability to evaluate preceptor or student performance could pose difficulties in assessing sites for minimum quality standards.

Comparison of reported quality and capacity concerns with ratios of students to preceptors or ratios of students to sites could not be conducted because exact class size data were not collected. Further exploration of the potential connection between student/preceptor and student/site ratios and concerns about quality should be conducted, especially since the new accreditation guidelines recommend ratios of 3:1 for IPPEs and 2:1 for APPEs.³⁷

Many of our results are consistent with other published concerns. For instance, resources to support EE (eg, practice sites, financial support, personnel), preceptor development, and standardized assessment of EE—our second, fourth and sixth most frequently mentioned concerns—were listed in the top 20 recommendations for the draft standards by those attending the ACPE Conference on Advancing Quality in Pharmacy Education.¹⁴ Fifty-five percent of conference attendees claimed that competition with other schools (our most frequent response) was a problem, and 48% identified demands on EE staff and fiscal resources (our second most frequent response) as a problem.¹⁵ Such corroborating results help validate our analysis.^17-18 The themes raised here are legitimate concerns that must be addressed as EE moves forward.

One limitation to these results is that other questions in the survey could have impacted the responses to this question. For instance, this question contained the only free-text response and many others were related to quantitative report of the personnel employed in EE and number of sites used. Therefore, respondents may have been more likely to express concerns related to lack of the number of faculty members/staff or site capacity. Demographics of actual individuals responding to the survey were not collected and thus cannot be fully evaluated. However, the nature of the survey as a whole involved primarily reporting quantitative data about the EE program, so individual demographics would be expected to have little to no influence on responses.

Finally, it is interesting that other issues such as integration of interprofessional education (IPE) and use of portfolios to demonstrate professional development, which have been highlighted in previous publications,^38-39 were not mentioned at all in the responses. This difference could be related to one of the limitations of these results. The data was collected 3 years ago and, therefore, may not accurately reflect the current state of EE. Since the time of this survey, new educational outcomes were published by the Center for Advancement of Pharmacy Education (CAPE)⁴⁰ and fully absorbed into the draft accreditation standards.¹ The CAPE Outcomes emphasize the affective learning domain in addition to knowledge, skills, and abilities that have been traditionally assessed. The new accreditation standards require integration and assessment of IPE throughout the curriculum, use of validated assessment tools for APPEs, and recommend portfolios to promote reflection about professional development. The themes identified in our results do not include consideration of the changes needed to incorporate these different assessment tools. The new standards also separate preceptor development and quality assurance of practice sites as entirely new standards. Even though preceptor development and quality assurance were identified as concerns, they might be even greater concerns if this survey were conducted now. Therefore, further investigation is needed about the challenges of implementing the new CAPE Outcomes and ACPE Accreditation Standards over time to determine their potential impact on EE.

CONCLUSION

While site capacity continues to challenge pharmacy schools, concerns related to increased demands on EE faculty members/staff to provide preceptor development and to support onboarding procedures for students pose new problems. The consistency of these results with other evidence supports the fact that workloads of faculty members/staff, resources and financial support, and quality assurance in experiential education continue to challenge schools. Further investigation of these issues should be conducted so they are taken into account as new accreditation standards are implemented.