Design and Content Validation of Three Setting-Specific Assessment Tools for Advanced Pharmacy Practice Experiences

INTRODUCTION

Health professions education has evolved substantially over the past decade as the demands and roles of various health professionals continue to change. Previous models of knowledge-based curricula have been replaced with new competency-based outcomes, which specify skills that a graduate in a health professions program should be able to demonstrate upon graduation.^1,2 This outcome-based model was predicated on the idea that demonstration of knowledge may not result in application of knowledge in the practice environment. Students must demonstrate abilities by performing under the observation of a practitioner.³ Medical education has largely embraced this competency-based model, leading to questions of how best to standardize and assess the various competencies expected from medical students.^4-6 Likewise, pharmacy education has transitioned to this model with the Accreditation Council for Pharmacy Education (ACPE) changing national accreditation standard requirements reflecting the Institute of Medicine Core Competencies.^7,8 Each pharmacy program is responsible for designing institution-specific curricula that outline the skills new pharmacy graduates should be expected to demonstrate upon graduation.⁹

Similarly, the American College of Clinical Pharmacy (ACCP) developed guidelines outlining key educational outcomes necessary for a graduating pharmacy student who plans on entering postgraduate residency training to demonstrate.¹⁰ The transition from a knowledge-based to a competency-based curriculum required several changes to be implemented in both the delivery of education and the assessment of students.¹¹ Pharmacy programs were challenged to provide objective evidence that their students were demonstrating these core competencies with increasing proficiency, and established benchmarks (minimum standards) that students must reach throughout the program. With the advanced pharmacy practice experiences (APPEs) taking place in the last year of pharmacy students’ educational training, students have been expected to demonstrate these competencies across a variety of practice settings and provide evidence of achieving minimum standards established by the profession.

The University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences (SSPPS) implemented 14 new ability-based outcomes (ABOs) as part of their curricular redesign in 2012.¹² These outcomes were established based upon the then-current ACPE Standards 2007 guidance documents, as well as substantial faculty input. These new outcomes became the foundation for assessing SSPPS student’s readiness to graduate and enter pharmacy practice. We quickly recognized, however, that the experiential assessment tool in place for preceptors to assess student performance during APPEs was inadequate to provide evidence of student competency. The tool in use was generalizable to all APPEs, regardless of setting or type. While the rubric had been revised to include the various ABOs relevant to pharmacy practice, preceptors were often misinterpreting the assessment items, and ranked a substantial number of outcomes as “not applicable” to the practice experiences offered in the program. It was unclear whether preceptors simply did not understand the terminology used in the rubric or if opportunities to perform such tasks were not available to students. Complicating this problem was faculty members’ desire to transition all previous pass/fail courses, including all of the experiential courses, to a graded A-F system. The SSPPS Office of Experiential Programs, in conjunction with faculty members, preceptors, and students serving on the school’s Experiential Education Committee (EEC), decided to systematically revise the APPE assessment tool in a data-driven manner. The goals of this revision included to contextualize outcomes for preceptors according to their practice setting, increase the reliability of student assessment by reducing the use of the “not applicable” response, and distinguish all levels of student performance through consistent summative scoring.

The focus on validating student assessments within pharmacy education has continued to grow. Examples of development and validation of tools for pharmacy student assessment have been published within pharmacy and interprofessional education literature.^13-17 Perspectives on validity, from Messick to Kane, encourage developing an argument of validity inclusive of multiple sources of evidence supporting the appropriate use of the measurement to determine appropriate consequences.^18-20 This manuscript describes a development process that promotes content validity and gathers evidence of appropriate design through distinct stages of development and implementation.

METHODS

This content validation study of tool design gathered evidence of validity from three sources: a job-task analysis of proposed student outcomes, a measurement of applicability between assessment tool and APPE setting, and acceptance of tool-generated grades by preceptors. The primary outcome of this study was to determine whether the newly designed setting-specific APPE assessment tools provided reliable and meaningful student assessments that successfully demonstrated achievement of expected skills and abilities. The context for this content validity study was limited to the three core APPE settings used in the SSPPS program: ambulatory care practice, health-system/inpatient care practice, and community pharmacy practice, inclusive of all required and elective APPEs occurring in those settings. This study was reviewed and determined to be non-human research by the Colorado Multiple Institute Review Board.

The process of developing, testing, implementing, and conducting analysis of the newly designed assessment tools encompassed five years, beginning in 2013 (Figure 1). Outcomes were initially identified and mapped from ACPE Standards 2007 and associated guidance documents, the school’s 14 overarching ability-based outcomes, and the school’s prior APPE assessment tool. The EEC formed three subject-matter expert (SME) workgroups to develop evaluative statements aligned with the mapped outcomes. Each SME workgroup consisted of at least one person representing each of the following stakeholders: experiential education administrators, clinical faculty members, non-faculty preceptors, and pharmacy students. With the exception of students, all members of the SME workgroup had significant practice experience in the setting assigned to their respective workgroup. Each workgroup was responsible for developing content (ie, student outcomes) for their assigned tool and reviewing evidence generated from subsequent validation steps. With each release of updated documents by ACPE leading to Standards 2016, SME workgroups were charged to ensure any gaps resulting from revised standards were addressed with their assigned assessment tool. A master outcome map was maintained centrally and indexed by both Standards 2007 and 2016 to ensure alignment. The SME workgroups routinely presented their work and findings to the full EEC and peer SME workgroups such that each unique tool was developed in parallel to the others. The intent of this coproduction model was to ensure consistent form and functionality of the final tools, while also ensuring no gaps in outcomes remained on the programmatic level.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Timing and Sequence of Design, Validation, and Implementation Process

Stakeholder engagement and responsibilities denoted by color:

Office of Experiential Programs in collaboration with the Experiential Education Committee (white figures),

Subject-Matter Expert Workgroups (gray figures), and Advanced Pharmacy Practice Experience Preceptors (dark gray figures).

Once all three tools were mapped with the proposed outcomes, the evaluative items for each tool’s outcomes underwent a job-task analysis by preceptors representing each setting type.^21,22 Active preceptors, defined as having an APPE student assigned to their practice during the then-current academic year (2014-2015), received an electronic survey via Qualtrics (Qualtrics Provo, UT) containing all evaluative items slated for inclusion for the tool aligning with their practice setting. Preceptors were asked to assess each item on importance and frequency of performance in their work setting. Preceptors assessed importance by responding to the question, “How important is the following task or ability to determine the student’s successful completion of your rotation?” using a 5-point scale ranging from 1=extremely important to 5=not important. Similarly, preceptors assessed frequency with which the skill was performed by responding to the question, “How often will your student(s) have opportunities to demonstrate the task or ability during your APPE rotation?” using a five-point scale ranging from 1=multiple times daily to 5=no opportunity to engage. Responses for each factor were multiplied to create a composite score for each item, with possible ranges of 1 (very important and occurring daily) to 25 (neither important nor frequent). Per governmental industry standards for job-task analyses, items receiving mean scores of 3 or below in each scale were considered to be highly critical for job performance.¹⁴ The EEC determined prior to the task-analysis that each item should achieve a mean composite score of less than 10, indicating at minimum that the item has a moderate degree of value and alignment with the environment. The full EEC would review any item exceeding this threshold and determine if the item would be removed or further refined prior to inclusion in the final tool.

The three new assessments were fully implemented for the APPE program for the graduating class of 2017 (June 2016-May 2017). A five-level rating scale was standardized across all assessment tools (Table 1). The previous assessment tool was discontinued and all existing and newly enrolled APPE preceptors were assigned an online training module (a PowerPoint presentation with voice over) regarding the new APPE assessments. All preceptors assigned as a student’s primary assessor were required to complete the training prior to the start of their first APPE. The new assessment tools were used for mid-rotation formative reviews and for summative scoring at the end of the six-week experience.

All end-of-APPE student assessments submitted for the class of 2017 were reviewed systematically for purposes of this validation study. This class was comprised of 157 entry-level PharmD candidates. Each student was required to complete, at minimum, one APPE in the community pharmacy setting, one in the ambulatory care practice setting, and at least two APPEs in the health-system/inpatient care setting, plus opportunities for up to two additional elective APPEs in any setting. All APPEs were full-time, six-week experiences (240 hours). Student assessments were compiled electronically through an online assessment and portfolio platform, E*Value (MedHub Minneapolis, MN).

Tool applicability was measured for each of the setting-specific instruments and was defined as the number of items used to rate student performance, excluding those marked non-applicable, divided by the total number of items presented to the preceptor on the assigned tool and averaged by the number of student assessments completed for each setting type. Item applicability was determined separately by taking the number of successful observations of a given item, excluding the instances it was rated as nonapplicable, and dividing it by the total number of the setting-specific APPE student assessments completed. The EEC determined an a priori goal for item applicability of 85% for all evaluative items assessed during the three core APPE settings. The EEC agreed that item applicability at or above these thresholds would indicate a tolerable use of the non-applicable response by the preceptor while allowing for variability in individual experiences. Items below this threshold would require review and adjustment.

A two-step grade submission process was developed by which the preceptors first completed the appropriate APPE assessment tool (step 1) by rating student performance on assigned outcomes using a five-level scoring rubric (Table 1). The online evaluation platform automatically tabulated the student’s performance ratings after preceptors submitted the assessment form in step 1, triggering a second assessment form, referred to as the preceptor’s grade verification form, to be made available electronically. Preceptors then opened the grade verification form, which summarized all ratings entered during step 1 and indicated the tool-generated summative letter grade to be assigned. Preceptors were required to submit this grade verification form (step 2) by either accepting the summative letter grade generated by the information provided in step 1, or suggest and justify an alternative summative letter grade. Preceptors were not allowed to alter the ratings or narrative feedback provided in step 1 even if they were dissatisfied with the resulting grade. The preceptor’s acceptance or rejection of the tool-generated grade was described using the Cohen’s kappa agreement statistic, evaluating the match between the tool-generated letter-grade and the final letter-grade assigned upon preceptor’s grade verification (step2). The EEC identified a cutoff of Cohen’s kappa >0.4, which was considered moderate agreement for each tool.^23-25 The EEC determined that tools failing to achieve this mark would be discontinued or revised and monitored.

RESULTS

The job-task analysis was completed by 131 active preceptors (a response rate of 52%), including community pharmacists (n=32), ambulatory care pharmacists (n=37), and health-system pharmacists (n=62). All 98 items developed by the SME workgroups were included in the job-task analysis, of which 95 (97%) received a combined score of less than 10, indicating moderate to high alignment with the associated practice setting. The full range of composite scores included a low of 1.08 and a high of 11.83. Scatterplots of the mean scores for individual outcomes are illustrated for each setting in Figures 2, 3, and 4. No items mapped to the community practice setting received a mean combined score greater than 10. Three items, two in the health-system/inpatient care proposed outcomes, and one item mapped to both the ambulatory care and health-system/inpatient care tool, exceeded this threshold. The SME workgroups ultimately approved all competencies to be mapped to their assigned APPE tool, with a majority of items being transposed verbatim to the final assessment tool as evaluative items. The outcomes identified as having the least alignment (combined score >10) were further refined before their addition to the final version of the assessment tools.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Job-Task Analysis of Ambulatory Care Outcomes

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Job-Task Analysis of Community Pharmacy Outcomes

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

Job-Task Analysis of Health System/Inpatient Care Outcomes

During the first year of full implementation, preceptors employed the new tools to provide 849 assessments of student performance in three core APPE settings. Each tool demonstrated a high measure of item applicability and tool applicability. The ambulatory care practice assessment tool included 30 evaluative items with a mean measure of tool applicability of 93.4% across 243 student assessments. The community practice assessment tool included 34 items with a mean measure of tool applicability of 94.1% from 243 student assessments. The health-system/inpatient care assessment tool consisted of 32 items and demonstrated a mean measure of tool applicability of 91.3% from 372 student assessments.

Each outcome item was described quantitatively using item applicability, defined as the percentage of measurements occurring across all relevant APPEs, and the frequency of student achievement of each outcome. Items within the domains of communication and professionalism were common across all settings and are presented in Table 2. The practitioner skills domain consisted of outcomes unique to each setting mapped to sub-domains of the pharmacist patient care process, medication use and systems management, professional stewardship, promotion of public health and wellness, and drug information. Item performance within this domain was summarized quantitatively by setting type: ambulatory care assessment tool (Table 3), community pharmacy assessment tool (Table 4), or the health-systems/inpatient care assessment tool (Table 5). All but three outcomes achieved the threshold of item applicability of 85% or higher. One item found in all three settings with setting-specific qualifiers, “recognize emerging issues, products, and services to improve pharmacy practice and/or public health” fell below the threshold in the ambulatory care setting (76.9%) and health-system/inpatient care (77.2%). Two additional items within the health-system/inpatient care practitioner skill domain mapped to medication use and systems management failed to achieve the predetermined threshold. These items mapped to understanding of the health system’s centralized drug distribution systems (mean item applicability=71.2%) and engaging in system-level quality improvement initiatives (75.5%).

Distribution of letter grades assigned by the assessment tool and their associated acceptance by preceptors is presented in Table 6. Preceptors accepted the tool-generated final grade recommendation for 805 (94%) student assessments across all three setting types. For ambulatory care APPEs, a total of 234 final grades were assigned. Out of these, preceptors accepted the grade generated by the tool in 216 (92.3%) instances. Considering the 18 cases (7.7%) of disagreement, the majority (16) of preceptor evaluations indicated a better grade should have been given, while two identified a lower grade was more appropriate. For community practice, there were 243 final grades, and 239 (98.4%) cases in which the preceptor accepted the tool’s calculated score. The four (1.6%) discrepancies were cases in which the preceptor felt the student should have earned a score one-letter grade lower than the calculated score. Finally, in health-system/inpatient care practice, there were 372 final grades submitted, with 350 (94.1%) of tool-generated grades accepted by the preceptor. Of the 22 cases (5.9%) with which there was disagreement, 12 preceptors felt the student earned a higher score and 10 felt the student earned a lower score than the score generated by the tool.

The Cohen’s kappa for each assessment tool was above the 0.41 threshold indicating good to very good agreement in ratings between the tool-generated and preceptor-recommended grade: ambulatory care practice, κ=0.702 (95% CI, 0.592 to 0.812); community pharmacy practice, κ=0.825 (95% CI, 0.657 to 0.993); health-system/inpatient care practice, κ=0.772 (95% CI, 0.690 to 0.853). Based on these results, the EEC recommended the adoption of the new APPE assessment tools and recommended the two-step grade verification process be adopted for all experiential-based courses.

DISCUSSION

This study describes the development and subsequent content validation of three new APPE assessment tools, all mapped to relevant outcomes within the context of tasks or abilities commonly demonstrated during three required APPE settings. The evolution of health professions’ assessments from the traditional knowledge-based model to the new competency-based frame necessitates that students demonstrate key skills and abilities. While these competencies should be demonstrated across the pharmacy curriculum, there is wide acceptance that demonstrating these skills during APPEs is a crucial part of verifying a student’s practice readiness.^26,27 However, the competencies outlined by Standards 2016 are at times broadly defined, and that demonstration of these skills can look quite different across different types of practices.⁷

This study adds to the existing literature because of the unique approach taken to develop individual assessment tools to assess student performance in APPEs, as well as the process of content validation to ensure that the tools accurately align with experiential training. Engaging practicing preceptors in the development of the tool was intentional and intended to enhance the tool’s performance and applicability. This purposeful process of engaging practicing preceptors during the design of the tools and the subsequent validation processes resulted in tools that successfully measured student performance. Considering the job-task analysis data, preceptors clearly felt that the items proposed by the workgroups were frequently demonstrated (mean (SD) scores: ambulatory care 1.84 (0.49); hospital 2.05 (0.62); community 1.77 (0.43) with a 1 indicating a skill demonstrated multiple times a day and a 2 indicating a skill performed daily. Likewise, with regards to the skill’s importance, preceptors also viewed the items as being highly important (mean (SD) scores: ambulatory care 1.59 (0.36); hospital 1.84 (0.51); community 1.55 (0.3) where 1 indicated extremely important and 2 indicated very important for the nature of each rotation. Combining these scores to attain a composite of frequency and importance demonstrated an average of 3.50 for ambulatory care, 4.54 for hospital, and 3.19 for community, all well below the predetermined threshold of 10 set by the EEC. Of note, only a few items received individual scale scores greater than 3 and combined scores greater than 10. One specific individual competency mapped to assessing the student’s awareness of emerging issues in pharmacy practice was rated consistently low in all three practice settings. However, after implementation, when setting-specific examples were included in the outcome text, the item applicability in the community setting reached 98.4%. Similar edits were less successful in the other two practice setting types for this outcome, which received item applicability rates of 76.9 and 77.2 in ambulatory care and health-system/inpatient care, respectively. Other lower-performing items identified by the job-task analysis within the health-system/inpatient care assessment tool received item applicability rates in the low 70s, despite additional edits prior to full implementation of the new tools. This suggests that these particular outcomes might be the most variable and difficult to assess in required APPEs or may require additional strategies for assessment. The congruency between high and low combined scores and subsequent preceptor ratings of item applicability demonstrates the effectiveness of the job-task analysis in being foundational in developing valid assessments.

Despite their low performance, these three lower-performing items were ultimately retained in the final versions of the assessment tools as each item was mapped directly to the school’s curriculum and assigned to the experiential education program for assessment. Additionally, tolerance for lower-performance was deemed acceptable in the context that each student would be assigned multiple opportunities to demonstrate these outcomes during their APPE training as these specific outcomes were not limited to a single required rotation. While these outcomes are valued by accreditation standards and by the school, it is unclear if alternative methods for student assessment of competency may be needed during experiential training. In the absence of alternative strategies, the decision was made to maintain these outcomes in the APPE student assessment tools. In an effort to ensure quality improvement, the EEC charged the school’s Office of Experiential Programs to monitor the performance of the new tools and determine if item applicability of these low-performing items improves with time as preceptors gain more familiarity with the tool. Alternatively, the committee could have recommended removing the three low-performing items and ensuring 100% of items achieved all benchmarks. Ultimately, and after much debate, the committee determined more was gained by maintaining the three low-performing outcomes, engaging in quality improvement process, and avoiding a gap in the curriculum outcome map.

The final source of evidence gathered to determine if the use of these tools were providing valid measurements of student performance came from the acceptance of the tool-generated grade by the student’s preceptor. This aligns with the step of evaluating the implications and decision-making which result from an education measurement, the final key aspect of Kane’s argument for establishing validity.¹⁸ Of note, the EEC chose not to include a D score for APPEs after much discussion with stakeholders. Pharmacy faculty members and students felt performing at a D level was unacceptable and equivalent to a failing grade. Therefore, the possible summative grade scores were limited to A, B, C, and F. It should further be noted that in only two cases the discrepancy between the tool-generated grade and preceptor’s desired grade was larger than one letter grade. In both cases the student was assigned a grade lower than they received; one receiving a C when the preceptor thought an A was more appropriate, and one receiving an F when the preceptor felt a B was more appropriate. Each of these discrepancies was discussed with preceptors and the Experiential Education Committee. In these cases, the school assigned the grade that the preceptor recommended, overriding the tool’s calculated score. This practice was anticipated as part of the ongoing effort to normalize grading practices across active preceptors and is a practice expected to decline with time. Because of the use of the two-step grading process, grading outliers have become identifiable. As a result, additional engagement with preceptors has become possible with the intention of decreasing variability of preceptor grading through outreach and additional training. Additionally, the school has been able to engage with students who have demonstrated low performance by offering coaching, mentoring, and remediation when necessary to reduce the likelihood of repeat poor performance.

Taken as a whole, these three sources of data provide evidence that the tools established by the EEC for these three core APPE experiences performed well with practicing preceptors, successfully documenting student competencies expected of a new pharmacy graduate. Preceptors felt that the items on each evaluation were important to their practice, and a strong majority of preceptors were able to observe students performing each outcome and rate students on that performance. Likewise, the tools accurately generated a grade correlating to the preceptor’s observation of the student’s performance 95% of the time. This is remarkable considering the high degree of variability of practice and expectations across these three different practice environments.

The setting-specific assessment strategy using unique outcomes to measure student competency offers several benefits in comparison to the prior strategy of employing one universal assessment tool generalizable to all settings. The previous SSPPS APPE assessment strategy lacked alignment in that the outcomes were not entirely relevant to each practice setting. The current approach also provides student performance data tied to rotation-specific expectations and is capable of distinguishing student performance in comparison to oneself and to peers. This assessment data can be used by students to identify areas for improvement and develop performance improvement plans, often with insight from preceptors, and anticipate opportunities to strengthen their competencies in subsequent APPEs when applicable. At the program level, SSPSS has implemented a longitudinal, scaffold approach to providing students feedback early and often, starting in their first IPPE, which aligns with the domains emphasized in the APPE tools. This vertical alignment allows students to understand and develop the knowledge, skills, and attitudes required for APPEs (and ultimately, practice) the moment they begin experiential training. This provides students with a clear understanding of what is expected of them and what they need to refine as they progress through the curriculum. An APPE remediation program has also been developed to identify and coach students based on their performance, which is only possible because of the grading system implemented.

This study offers several “take home” points that other institutions may benefit from if employed in their experiential programs. Engaging preceptors in the development of assessment tools designed for practice performance results in a tool that performs well with an extremely high rate of usability by preceptors, avoiding the “unable to observe” or “not applicable” confounders. Field-testing those tools with a small subgroup of preceptors, asking them to rate the importance and frequency each skill represents, provides further feedback regarding the alignment of the competencies expected in contemporary practice. Finally, giving preceptors the ability to challenge a calculated score, but not the ability to go back and change responses, provides valuable information to the school’s Office of Experiential Programs regarding preceptor grading styles, and provides fairness to students who may have been disadvantaged by the tool’s design. To our knowledge, this is a novel approach within experiential education that minimizes rater-bias during student assessment. This approach also encourages alignment between summative assessment scores, objective feedback, and ratings of student performance.

There are several limitations to this study. These data reflect only one year of utilization across one class of APPE students, so further analyses are necessary to continue to evaluate the tools’ performance over time. Preceptors may have disagreed with the tools’ scoring and chose not to share their disagreement with the school, possibly increasing the level of acceptance of the tools’ calculated letter grade. In calculating the Cohen’s kappa statistic, accepted grades were assumed to reflect the preceptor’s desired grade. Considering the high-stakes nature of APPE assessment and established communication between school and preceptors, we feel confident that preceptors were providing accurate information to the best of their ability. Approximately 5% of student assessments resulted in rejection of the tool-generated grade by the preceptor, indicating room for improvement in either tool design or preceptor training. About two-thirds of the instances of preceptor disagreement were associated with a desire for assigning a higher grade, with the remaining one third suggesting lower grades. Thus, unidirectional bias by the preceptors cannot be assumed in all instances of grade disagreement. Additional analyses of grading patterns may be needed to determine whether grade disagreement is associated with specific individual preceptors, unique practice sites, or level of precepting experience. From a programmatic perspective, and with an appreciation of the inherent uniqueness of each APPE, we feel that the amount of disagreement between the preceptors and suggested grades represents an acceptable amount of variability in assessment, as it is unlikely any clinical assessment tool will perform perfectly across every clinical rotation. Additional psychometric validity testing is needed to confirm subscale construction of items within domains as appropriate.

Because of the nature of assessment in APPE settings, testing for inter-rater reliability is impractical as gathering multiple independent measures of student performance by preceptors is unlikely in clinical environments. Ensuring compliance with grading norms among individual preceptors is also a challenge within experiential education as the active roster of preceptors is dynamic. Our two-step grading strategy provides flexibility of addressing outliers while also giving in-time feedback to preceptors who are identified as grading outside of the established norm. Sub-analysis of longitudinal grading practices within and between subgroups of preceptors may provide additional evidence of validity. Finally, this study was limited to those assessments used for required APPE settings, so assessments submitted during non-patient care electives need to be evaluated as well before those can be deemed valid.

CONCLUSION

Based upon three sources of data, three assessment tools mapped to ACPE accreditation standards and school-specific outcomes were successfully adopted by preceptors and generated reliable summative grades for APPE student performance. We consider the data evidence that these tools provide valid measurements of students’ performance across the various outcomes and environments included in our experiential curriculum. The strategies used to develop and evaluate these tools may be adopted within other pharmacy programs.