Development and Initial Evaluation of an Advanced Pharmacy Practice Experience Readiness Assessment Plan

DISCUSSION

This paper describes an APPE-RAP with retrospective application to a cohort of students. The APPE-RAP serves as a tool to guide focused remediation as well as inform the college of potential needs for curricular revision. Analysis of data within ExamSoft allows feasible evaluation of student-level deficits without the use of novel assessments (eg, mile marker examinations), potentially avoiding additional workload strain on faculty and students. Prior to implementing the APPE-RAP for programmatic progression decisions, our institution chose to review the data for a cohort of students that had already begun APPEs to determine appropriateness of the initial categorizations. The frequency of students with more than 50% novice ratings on individual skills was low (2.3%). Therefore, skills thresholds were revised to ensure underperforming students undergo additional remediation. The revised threshold for red categorization is now a rating of ≥1 unacceptable or ≥50% of the ratings as novice for any individual skill. Future use of the APPE-RAP will include enhanced student self-monitoring and/or earlier programmatic remediation interventions.

A review of first-attempt pass rates for skills revealed that a significant number of students required remediation to improve their ability to write SOAP (subjective, objective, assessment, plan) notes, which was suggestive of curricular deficiencies and resulted in course modification. Similar modifications were discussed but not implemented for patient counseling because the assessment of patient counseling occurred virtually due to teaching restrictions related to the COVID-19 pandemic. Several students shared that they perceived a different level of stress when learning and practicing counseling skills via distance-based videoconferencing technology and assessment in person, which may have produced spurious results.¹¹

Knowledge-based deficits were more difficult to generalize, as student-specific deficiencies varied. These findings are reflective of the overall infrequent APPE failures observed (5.4 failures per year for the last four years), but they also represent that a simple APPE-RAP designation of red may be insufficient to capture those with an increased likelihood of failing an APPE. While knowledge thresholds remain unchanged, a threshold of multiple yellow ABOs may need to be considered in addition to other supportive interventions to ensure success on APPEs.²

Using red as the risk indicator estimated a low positive predictive value and a high negative predictive value. Alternatively, being categorized as green or yellow predicted passing APPEs with good accuracy (98% negative predictive value). However, a high proportion of students categorized as red also passed APPEs (11% positive predictive value). Using the red categorization as the threshold would conservatively identify students at risk of failure, and early intervention resource allocation to such students may still be prudent.

Individual APPE-RAP reports are now provided to students to provide insight into their performance beginning in their second year of pharmacy school. This data will allow student self-monitoring and help guide academic advising efforts, potentially leading to targeted remediation prior to APPEs to improve readiness. This could prevent students categorized as red from spending a disproportionate amount of time during their APPEs on reviewing or remediating content, thus limiting their advancement of knowledge beyond the pre-APPE level. Our initial analysis demonstrates a way to identify not only those that are failing but also those that share similar deficiencies that may benefit from earlier assistance. Our goal is that early focused intervention will optimize APPE growth.

Focusing on programmatic ABOs and essential skills aligns well with published strategies.^2-6 Identifying a consistent passing threshold on item-level achievement proved difficult due to varying assessment philosophies across the college. For example, faculty may prefer to offer APPE-level examination questions to second-year students, using grade adjustments to achieve more normal distribution. However, preadjustment item-level statistics may indicate poor performance on that ability-based outcome. Instead of using an arbitrary performance threshold, average performance of the cohort was leveraged to identify relative outliers in student performance. Given that the ability for more real-time tracking and dissemination to students exists, early intervention and/or remediation options could become part of a final APPE readiness plan. Additionally, consideration has been given to adding measures of professionalism to future iterations of the APPE-RAP.²

This analysis represents the work of one institution with one cohort of students. The results might not be generalizable to other institutions. Reliability needs to be evaluated when additional cohorts complete the pre-APPE curriculum. Also, the timing of skills and knowledge assessments differed, which may have introduced issues with equal weighting throughout the pre-APPE curriculum in the APPE-RAP. However, use of relative performance criteria, rather than a preset threshold to categorize readiness, minimized these issues. To simplify data aggregation and build an initial plan, professionalism and early experiential data were not considered in our assessment matrix, but these have been previously associated with APPE failures.⁶ Additionally, not all students were assigned to a direct patient care APPE within the first two blocks, which may impact our results. Given the paucity of literature, our intention is for this approach to stimulate further discussion regarding the utility of APPE readiness to improve both individual and aggregate student outcomes.