Measuring PreAdvanced Practice Experience Outcomes as Part of a PharmD Capstone Experience

INTRODUCTION

Pharmacy educators are faced with a difficult task in determining if students have the requisite knowledge, skills, and attitudes for advanced pharmacy practice experiences (APPE). The Accreditation Council for Pharmacy Education (ACPE) provides a set of competencies that students should know and be able to do prior to APPE.¹ However, operationalizing these outcomes into a set of educational and assessment-related activities can be challenging for pharmacy educators.

A capstone is a pedagogical technique incorporated into educational programs that require graduates to deal with continual changes and employ lifelong learning and problem-solving skills during their careers. Capstones can prepare students for challenges in the workforce by providing opportunities to apply and integrate didactic course content to real-life scenarios or problems. A capstone has been defined as a crowning event where students apply knowledge from the curriculum to new experiences requiring synthesis and integration of the knowledge.^2,3 In addition, Holdsworth et al described a capstone as “bringing together the knowledge of an academic discipline and student transition to the world of work” and summarized the key features of a capstone experience as: free-standing and authentic or “real-life,” involving out-of-class events as a component of existing courses and skill-development leading to work-readiness and/or entry to graduate studies.⁴ Across the literature, a capstone is defined as encompassing integration and application of knowledge to bridge students to their future as professionals; therefore, higher levels of cognitive performance should be the focus for the design of assessments.⁵ The strategy of authentic assessment is particularly well suited to the capstone experience. In this type of assessment, students are asked to demonstrate competence in real-world settings.⁶

Capstones have been used in multiple pharmacy schools as a method to enhance students’ critical-thinking abilities in the didactic pharmacy curriculum. Studies have demonstrated benefits of capstones in developing student ability to apply knowledge and improve their confidence.^7-11 While the use of a capstone should help further develop student knowledge and confidence, there is no data currently showing how student development from a capstone experience can be used to determine readiness for APPEs. Because a capstone provides a framework for students to apply and integrate knowledge and skills, and because ACPE defined a set of outcomes students should be able to perform prior to APPE, a capstone placed at the conclusion of the didactic program is a logical addition to the curriculum. This paper describes the development and incorporation of a capstone course into the PharmD curriculum to assess students’ preparation for APPE. The intent of the described capstone course was to provide students with the opportunity to demonstrate application and integration of knowledge from the didactic work throughout the PharmD program to simulated APPEs. Additionally, how the use of the capstone to track student learning outcomes (through authentic assessment) and use of outcomes for the purposes of curricular improvement and demonstration of accountability to external bodies is described.

DESIGN

The Ohio State University Institutional Review Board reviewed this project and determined that it was exempt.

The Ohio State University PharmD program provides a 4-year curriculum consisting of 3 years of didactic courses and introductory pharmacy practice experiences (IPPEs), followed by nine 1-month APPEs leading to a PharmD degree. A bachelor’s degree is required for entry to the program. Each class has approximately 130 students.

In 2007, a curriculum and assessment committee mapped the PharmD curriculum to our 100 ability-based, program-level outcomes,¹² ACPE’s Appendix B content, and 4 curricular threads – professionalism and ethics, communication, problem solving, and drug information. Analysis of these maps revealed significant overlaps in content between the pharmacology and therapeutics course sequences leading to a curricular revision for the 2008-2009 academic year, which included a combined pharmacology and therapeutics modular approach. The eliminated redundancy yielded 5 weeks of time in the curriculum. Careful consideration was given by faculty for how to best use this “found” time. The goals were to provide students with activities and assignments that would require them to apply and integrate knowledge and skills from across the curriculum in final preparation for APPE while enabling the college to collect program-level outcomes data. Review of the educational literature provided a framework to meet these goals in the form of a capstone course.

In 2010, with oversight, input, and assistance from the assistant dean for accreditation and assessment, faculty members from the Division of Pharmacy Practice and Administration, pharmacist practitioners at The Ohio State University (OSU) Wexner Medical Center, and the curriculum and assessment committee, the capstone director created assessments and activities to enhance and measure critical outcomes of the PharmD program. All assessments were designed to simulate core experiences students would be exposed to during APPE. In an effort to deemphasize student focus on letter grades, the decision was made that all capstone assignments would be pass/fail. Initially, the capstone was embedded into the revised modular format of the final pharmacology and therapeutics course of the third year. Beginning in 2013, the capstone course was moved into a May-term consisting of 19 instructional days as the university switched from a quarter system to a semester system. With the switch, additional flexibility in scheduling, and release of ACPE’s Appendix D,¹ the 2013 version of the capstone course was modified to consist of 11 activities constituting a total of 20 equally weighted assessments. A mixture of assignments to ensure both individual accountability and team-based activities, which allowed practice and professional communication with peers, were incorporated into the capstone. Students were assigned to groups of 6-7 to work on all team-based activities. A list of capstone activities mapped to ACPE’s Appendix D, descriptions, and assessment strategies are depicted in Table 1. In addition to the graded activities, students participated in faculty-facilitated complex patient case discussions throughout the capstone course. Students needed to pass at least 75% of assessments in order to pass the capstone course. The 75% cutoff is consistent with the NAPLEX and MPJE exams. Students that did not pass the capstone course were required to sit out of the program until the course was reoffered, consistent with the academic standing policy for all required courses.

The focus of the capstone activities and assessments was on performance of skills and demonstration of knowledge required to succeed in the APPE. Because the capstone course was designed and placed at the end of the didactic portion of the curriculum, assignments and corresponding authentic assessments were developed at the application and synthesis level of Bloom’s Taxonomy.⁵

The capstone assessments measured student performance in order to fulfill both accountability and improvement purposes. Grading for the capstone course and each individual assessment measured minimal competency. Many of the assessments, particularly the performance-based ones, were created by course faculty members then reviewed by pharmacy practitioners to assess application to practice. Criteria for passing each assessment were then developed by course faculty members before the assessment was delivered to ensure consistency. Table 1 lists the pass criteria for the assignments. After each activity, instructors assessed student performance at the cohort level by reviewing the associated rubrics involved. This review served to determine if any adjustments to the passing criteria needed to be made and if curricular issues could be identified. For example, if students universally struggled with a particular assessment, the contents of that activity were forwarded to the curriculum and assessment committee for further review and action.

To provide the reader with more detail regarding the layout of the capstone course, below are 3 of the activities in further detail.

Objective Structured Clinical Examination (OSCE) – Health-System Settings: This activity measured student skills in regard to health-system related APPEs. Students completed three 8-minute OSCE stations, each of which was assessed individually and contributed to the capstone final score. The first station was an intravenous admixture order verification. Students determined if the “pharmacy technician” appropriately prepared an IV admixture, and/or they described any errors present. For the second station, students interpreted a vancomycin or aminoglycoside drug level, determined appropriate dosing, and provided a recommendation for the next dose. Students had to provide a rationale for any dosing adjustment. The third OSCE station assessed communication skills. Students were asked a drug information question by a standardized medical resident in the inpatient setting (eg, drug compatibility, drug-drug interaction) and were allowed to use provided electronic references to determine and communicate the correct answer in a timely manner. Students were assessed on both the correct answer and their communication ability.

Therapeutic Presentation: To allow students to practice professional presentation skills, student teams were assigned a therapeutic topic to present to the entire class. Topics included both inpatient and outpatient areas of practice and often were selected to introduce a new medication, updated guidelines, or a new disease state not covered in the curriculum. Each team was assigned a pharmacist mentor for the presentation, who provided feedback and direction on literature evaluation and presentation preparation. One student was selected to present the entire 30-minute presentation to the class, instructors, and pharmacist mentor. Instructors evaluated students using a presentation rubric, with emphasis on presentation content and design. Following the presentation, all team members assisted the presenter with a question and answer session. Each student in the group received the same grade for the assignment; a minimum score of 80% (graded via rubric) was required to receive a passing grade. Following presentations, course faculty members concluded the day with a set of multiple choice questions relating to the presentation topics. Students had to score a 90% on these questions across all presentations.

Critical Literature Evaluation: For this activity, each student group was assigned 4 primary literature articles to evaluate. Forty-eight hours following distribution of the articles, student teams met with 2 faculty facilitators who conducted a 20-minute oral examination by asking specific questions about 1 of the 4 articles. Faculty members called on each student of the group individually to ensure accountability and preparedness, and then allowed other team members to add detail as needed. The faculty members assessed each team using the same literature evaluation rubric that was introduced during a drug information course earlier in the curriculum. All students on the team received the same grade for the literature evaluation activity. A passing grade was 90% based on an evaluation rubric.

EVALUATION AND ASSESSMENT

In order to document student readiness for APPEs, each of the 11 capstone activities had associated assessments. These assessments were mapped to the College’s program-level, ability-based outcomes, to a set of 4 curricular threads or themes across the curriculum, and to Appendix D of ACPE’s Standards 2.0. The capstone course was revised and moved to the May term in 2013. At this time 4 activities totaling 8 assessments were added. Tables 2 and 3 show student performance, in terms of pass rates for the capstone activities. Each activity and assessment included in the capstone course was carefully selected to help students prepare for the APPE. Thus, at the individual level, students received feedback on a set of activities that, when viewed as a package, gave them a means of quantifying their readiness for the APPE. At the program level, administrators could look across the performance of the cohort to determine the effectiveness of the curriculum in preparing students to succeed in the APPE.

The capstone course underwent continuous improvement efforts from its inception. Following the first iteration in 2010, the capstone director met with representatives from OSU University Center for Advancement of Teaching in an effort to address continuity and purpose of the course. All capstone activities were mapped to course goals and objectives. Additionally, at the conclusion of each capstone course, students were asked to complete an open-ended evaluation of individual activities and of the capstone course as a whole (Table 4). These efforts yielded consistent improvements in student perceptions of the capstone’s value. Finally, every other year, a small focus group of students met with the capstone director to provide additional feedback for course and curriculum improvements.

Compiled data from student feedback and focus group feedback served as a catalyst for revisions to the PharmD curriculum to provide better opportunities for students to prepare for capstone assignments and ultimately the APPE. Efforts were made annually to orient students to the capstone course, including time spent explaining each activity in detail and showing students how each assignment mapped to course goals and objectives. Student feedback led to improvements in communication regarding assignment expectations and assignment timing. Students have consistently expressed that there was value in each assignment and that all assignments should be included in future capstone courses. Another response to student feedback was the addition of faculty-led case discussions. These activities were not assessed but were well received by both faculty members and students as ways to provide updated material on therapeutic areas and to integrate complex patient care issues together in a simulated environment.

Evaluations of program level data led to improvements in the PharmD curriculum. For example, in 2010, only 81% of students passed the drug literature evaluation activity. This led to an investigation into how students were preparing for literature evaluation and where in the curriculum students learned these skills. The teaching methods and student performance expectations were standardized across the curriculum and consequently, 100% of students passed the literature evaluation activity in each subsequent year of the capstone.

Students provided feedback that there was a disconnect between how they were expected to document patient encounters earlier in the curriculum and how they were assessed on such documentation in the capstone course and the APPE. As a result, course directors reviewed the process of documentation throughout the curriculum, and changes were made earlier in the curriculum to help student documentation skills be more informative and concise. Linking patient encounter documentation from the classroom to IPPEs was also explored. Changes made in the curriculum regarding documentation will be assessed in the next iteration of the capstone course.

The capstone was a resource intensive course in terms of time, manpower, and money. The course director spent 400 hours per year preparing, delivering, and revising the course. In addition, the assistant dean for accreditation and assessment spent 100 hours per year preparing and assessing activities from the capstone course. In 2013, 7 additional instructors each developed and delivered 2-hour interactive reviews. Two teaching assistants assigned to the course spent 20 hours per week on the course during the course delivery and 4-8 weeks of preparation prior to delivery. In 2013, a first-year resident completed a teaching practice experience with the course director and spent the month developing activities and assessing students. Each performance-based assessment was also labor intensive and cost approximately $2500, which covered cost of rent for the skills center and fees to hire standardized patients.

DISCUSSION

Placing a capstone course at the end of the didactic portion of the curriculum can be an effective addition to a doctor of pharmacy curriculum. This placement provided opportunities for students to work collaboratively with peers to apply and integrate the skills and knowledge learned in the curriculum and demonstrate competence via authentic assessment techniques. The concentrated course design forced students to work on multiple activities simultaneously, thus preparing them for the high-level of student-generated work that is expected during the APPE. The capstone course provided these opportunities in the safe environment of the college. Additional benefits of this format emanated from the peer-learning and teamwork from group activities. Formative grading rubrics gave students valuable information about their competencies and deficiencies prior to entering the APPE. Students were encouraged to share these findings with preceptors during the APPE.

The capstone course also provided the college with valuable data for curricular improvement as well as ACPE accreditation reporting. Due to the placement of the capstone course, the college received data just before the APPE, which could be used to validate student performance on pre-APPE competence (as outlined in Appendix D¹). Additionally, the capstone provided the college with curricular areas in need of improvement. Due to the skills-based nature of several of the capstone assignments, authentic assessment of professional communication and problem-solving skills occurred in the context of difficult patient-case scenarios. The instructors also decided to emphasize skill demonstration of as opposed to “getting the right answer,” which was prevalent through much of the didactic curriculum. Finally, providing the capstone course multiple years in a row allowed the college to determine the impact curricular changes had on student retention of essential skills needed for the practice of pharmacy.

Despite the overwhelming strengths of placing a capstone course at the end of a didactic portion of the curriculum, the format is not without limitations. The capstone is a resource intensive course that requires input from faculty across the curriculum as well as the experiential team to ensure activities are appropriate. The end product is improved drastically due to this input; however, the time commitment of the course director and faculty members involved is notable. Due to time constraints and the large number of students enrolled in the course, individual presentation skills cannot be assessed by the therapeutic review presentation. The rubric is designed to focus on presentation content and material layout, rather than actual delivery, to emphasize these aspects of professional presentations without adversely impacting students based on one group member’s delivery. Despite the decision to make all assignments pass or fail, the capstone is still high stakes in that students who do not pass the required number of activities will not advance to the APPE. Additionally, offering the capstone course at the end of the didactic portion of the curriculum can lead to discovering student deficiencies or curricular revision failures at a point that is too late to rectify for individual students. Because students get a variety of experiences from the large number of APPE required for the entire class there has not been formal evaluation on change in student performance as a result of the capstone course. Further investigation into this relationship is warranted.

SUMMARY

A capstone course can be a useful addition to a PharmD curriculum when placed directly prior to the APPE. A capstone course can provide students with the opportunity to apply and integrate key curricular concepts to APPEs. Additionally, a capstone course can provide key data for the college’s curricular assessment and ACPE accreditation.