Development and Application of a Stepwise Assessment Process for Rational Redesign of Sequential Skills-Based Courses

INTRODUCTION

Schools and colleges of pharmacy are required to ensure students are ready to begin their advanced pharmacy practice experiences (APPEs). The Accreditation Council for Pharmacy Education (ACPE) Accreditation Standards and Guidelines 2011 provided guidance for the assessment of student capabilities prior to entering APPEs through adoption of the Pre-APPE Performance Domains and Abilities (Appendix D).¹ Appendix D identified the core domains and associated skills necessary for successful APPE preparedness. Schools of pharmacy were encouraged to assess and demonstrate student competency within these domains prior to advancement to APPEs, and this prompted examination of how performance competencies were taught and assessed within doctor of pharmacy (PharmD) curricula. The emphasis on assessment of APPE readiness is continued in the ACPE Standards 2016 with Appendix D transitioned to Appendix A within the Guidance Document to the Standards.^2,3

Curricular revision may be necessary to ensure schools of pharmacy are meeting the accreditation standards and keeping current with pharmacy practice. Toward that end, the American Association of Colleges of Pharmacy (AACP) commissioned an Excellence Paper on curriculum development and assessment.^4,5 Curriculum mapping is part of this process and can be useful to assess an entire curriculum to “ensure that there are no gaps or unnecessary redundancies in content, promote an integrated curriculum by showing the relationship between different content areas, and identify the types and range of assessment methods being used.”⁴ Curriculum mapping also provides an opportunity to link course content and objectives to specific learning outcomes that can be used for course or curriculum assessment.⁴ Curriculum mapping is discussed extensively in the pharmacy education literature.^4-10 Likewise, the use of focus groups in pharmacy education for formative evaluation of the learning environment and to guide development and implementation of pharmacy courses has been previously described.^11,12 However, the combined use of focus groups and curricular mapping for the assessment and redesign of sequential skills-based courses within a PharmD program has not been published.

The University of Wisconsin (UW)-Madison School of Pharmacy curriculum includes a required 4-semester pharmacotherapy sequence during the second and third years of the traditional PharmD program. Until the end of the 2013-2014 academic year, each semester of the pharmacotherapy course was composed of four credits that included a lecture component (three 1-hour lecture periods and one 1-hour discussion period per week) and a skills-based laboratory component (one 2-3 hour laboratory session per week). The laboratory component was originally designed to provide students with practical application of lecture material and to teach and assess pharmaceutical care skills.

To optimize student readiness for APPEs in concordance with the ACPE Standards, the school elected to place additional emphasis on the practice and assessment of core domain skills within the laboratory component of the pharmacotherapy courses. To facilitate this new emphasis on skills, a curricular change proposal was drafted during the summer of 2013 to split the laboratory components from the larger pharmacotherapy courses to create a 4-semester sequence of 1-credit, stand-alone laboratory courses titled Integrated Pharmacotherapy Skills (I-IV). The main pharmacotherapy courses retained the same lecture components (three lecture hours and one discussion hour per week) but were transitioned to 3-credit courses. Rationale for the curricular change included increased ability to refocus laboratory time on skill development and formative and summative performance-based assessments, greater flexibility in timing of skill instruction, increased student accountability for preparation and performance of skills, and opportunities for integration of laboratory activities with the introductory pharmacy practice experience (IPPE) courses, all supporting the overarching goal of improving student preparedness for APPEs. To ensure course transition was performed in a rational manner that targeted achievement of desired learning objectives, a systematic redesign process informed by course assessment including focus groups and mapping was employed.

The objectives of this study were to illustrate how a combination of focus groups, course mapping, and student evaluation can be successfully used to assess achievement of course learning objectives related to APPE preparedness and to describe how this assessment process can be applied to inform rational redesign of sequential courses within a PharmD curriculum. The study may be useful for those considering global assessment and/or course redesign in their respective curricula.

DESIGN

Transition of the pharmacotherapy courses’ laboratory components to the Integrated Pharmacotherapy Skills laboratory courses was approved by the University of Wisconsin for the 2014-2015 academic year. To facilitate the transition, faculty members used a 4-step process to assess the existing pharmacotherapy laboratory component and applied assessment results to the design of the new Integrated Pharmacotherapy Skills laboratory curriculum. An overview of the redesign process is depicted in Figure 1. The assessment and redesign qualified for exemption status under the UW-Madison Institutional Review Board (IRB).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Skills-Based Laboratory Curriculum Redesign Process. (APPE=advanced pharmacy practice experiences; ACPE= Accreditation Council for Pharmacy Education; UW=University of Wisconsin).

The first step of the redesign process sought to develop skills-based learning objectives related to APPE preparedness that could be appropriately assumed in the skills-based laboratories. As the primary goal of the new laboratory curriculum was preparation for APPEs, course learning objectives were crafted around achieving competency in skill performance within certain ACPE Pre-APPE Core Domains (Patient Safety, Basic Patient Assessment, Medication Information, Identification and Assessment of Drug-related Problems, General Communication Abilities, Counseling Patients [termed Patient Education in Standards 2016], Drug Information Analysis and Literature Research, and Health and Wellness).³ Skills within additional ACPE Core Domains were not included as they are extensively covered in other courses at the school. In addition, the opinions of practicing pharmacists and student pharmacists were obtained via focus groups and used to compile a list of critical skill items. These critical-skill items along with skills included in the ACPE Pre-APPE Core Domains were condensed into general skills-based categories that could be applied in the course mapping process.

During the second step of the process, pharmacy preceptor and student pharmacist perceptions were used to assess which of the skills-based learning objectives identified in step 1 enabled students to perform competently at the beginning of APPEs. According to social cognitive theory, self-efficacy or one’s perception of one’s own capabilities is a key predictor of performance.^13,14 Individuals who believe they can successfully accomplish a given task are more motivated to engage in that task and persevere in the face of adversity.¹⁵ Perceptions of pharmacy preceptors and fourth-year student pharmacists just completing APPEs were collected through a combination of focus groups and surveys. The perceptions of third-year student pharmacists just completing the final semester of the pharmacotherapy laboratory were obtained via course evaluations.

The third step of the process involved mapping the existing pharmacotherapy laboratory component to the general skills-based categories developed during step 1. The course map, along with student pharmacist feedback on course evaluations, was examined to uncover deficits in the structure and composition of the laboratory that could potentially explain why performance competence was not achieved for certain skill areas identified in step 2. These deficits became targets in the redesign of the laboratory curriculum during the fourth step of the process.

The fourth and final step determined which learning activities from the existing laboratory component should be continued in the new curriculum, which should be eliminated based on redundancy or overlap with other courses, and which novel activities should be developed to address deficits in the laboratory composition and facilitate achievement of learning objectives. Collaboration with other school faculty members, preceptors, and students resulted in development of new activities and design and implementation of mechanisms to address the structural deficits in the existing laboratory component. To assist with placement of skills, a predominant focus area was assigned to each semester that placed emphasis on one or more major skill areas to ensure sequential building of skills and achievement of learning objectives throughout the entire 4-semester curriculum. The focus areas chosen for the semesters were, in sequential order, patient communication, documentation, provider communication, and complex critical-thinking skills.

EVALUATION AND ASSESSMENT

A combination of qualitative and quantitative assessment methods including focus groups, electronic surveys, and course evaluations provided data to inform the steps of the redesign process. Three separate focus groups were conducted with the primary goal of identifying which skills are critical for students to perform competently on APPEs. A second goal of the focus groups was to determine which skills-based learning objectives, as defined by the ACPE Pre-APPE Core Domains did the existing pharmacotherapy laboratory component prepare students to competently perform and which were in need of improvement. The three focus groups included pharmacist preceptors practicing in community and/or ambulatory care settings, pharmacist preceptors practicing in inpatient and/or long-term care settings, and fourth-year student pharmacists just completing their APPEs.

Purposeful sampling was used to identify and select participants for each focus group most likely to provide quality constructive feedback regarding perceived student performance. Selection criteria for the preceptor groups included pharmacists who served as preceptors for UW-Madison fourth-year student pharmacists within the previous year and were known to faculty members as being able and willing to provide constructive feedback on the PharmD program. Twenty-four pharmacist preceptors from varied settings were invited to participate and nine attended the focus group. Five pharmacists from community/ambulatory care settings and four from inpatient/long-term care settings attended (37.5% of invited attendees), representing eight organizations. Selection criteria for the student pharmacist group included students currently completing the fourth-year of pharmacy school at UW-Madison, and who were known to faculty members as being able and willing to provide constructive feedback on the school’s courses. Eight out of 14 students (57.1%) invited to participate in the focus group were in attendance.

A semi-structured interview was used during the focus groups to explore participants’ perception of student ability to competently perform various skills while on APPEs, and the importance and relevance of those to actual pharmacy practice. The interview questions were informed by the ACPE Pre-APPE Core Domains and are displayed in Table 1. Focus group sessions lasted approximately 90 minutes and each session was attended by a laboratory faculty member who facilitated the focus group discussion and an observer who took notes. The auditory of each session was recorded with participants’ knowledge and approval.

Transcripts of the focus group sessions were reviewed by laboratory faculty members. Skills endorsed by focus group participants as critical or important for students to demonstrate competency in prior to beginning APPEs were identified and categorized into the core domains. The finalized list of 16 individual skill items from the focus groups is displayed in Table 2. Content analysis was used to identify overarching perceptions of which skills students were and were not prepared to competently perform at the start of APPEs. Table 3 summarizes the overarching perceptions of competency derived from the focus groups. Skills identified as requiring improved competence included, but were not limited to, managing complex patients, performing medication reconciliation, evaluating and applying clinical guidelines, prioritizing problems, and documenting.

All pharmacist preceptors and students invited to the focus groups were given an opportunity to provide feedback through an electronic survey prior to attending the focus group. Surveys were intended to supplement the focus group by providing an avenue for participants to provide anonymous critiques of student performance competence and laboratory activities, and an opportunity for invitees who could not attend the focus group to also contribute feedback. The survey asked participants to rank their level of agreement with student ability to perform various skills-based learning objectives at the beginning of APPEs using a 5-point Likert scale (1=strongly disagree; 5=strongly agree). The survey also included three open-ended questions focused on identifying the strengths and weaknesses of student documentation skills, as documentation was previously identified by laboratory faculty members to require modifications to better align with clinical practice. Seventeen of 24 pharmacists (70.8%) completed the survey (seven who had attended the focus group and 10 who had not). Seven out of the eight (87.5%) student pharmacists who attended the focus group also completed the survey.

Descriptive statistics were used to analyze the results of Likert scale-rated items on the surveys. Strongly agree and agree, as well as strongly disagree and disagree responses were collapsed for analysis. The level of agreement of pharmacist preceptors and fourth-year student pharmacists regarding student ability to perform a given skills-based learning objective on APPEs is presented in Table 4. Overall survey results supported perceptions derived from the focus group transcripts (Table 3) and provided additional details on performance of specific skills such as taking a medication history and performing a profile review. In general, preceptors perceived students to struggle most with items within the ACPE Pre-APPE Core Domains of Identification and Assessment of Drug-related Problems, and General Communication Abilities. Specifically, half or fewer preceptors agreed that students on APPEs could accurately prioritize conditions/drug-related problems and document in a manner that is succinct, organized, and easy for other health care providers to read and understand. Overall, the fourth-year student pharmacists had a more favorable perception compared to preceptors of their ability to perform skills-based learning objectives. However, less than half of the student pharmacists perceived they could succinctly document and confidently present a patient case.

At completion of the final semester of the laboratory, third-year student pharmacists were asked to complete a voluntary and anonymous electronic course evaluation. The course evaluation asked students to rate level of confidence in performing skills-based learning objectives using a 5-point Likert scale (1=not at all confident; 5=extremely confident). The evaluation also included two open-ended questions in which students were asked to describe what they liked most about the pharmacotherapy laboratory component and offer suggestions for improving the course overall.

One hundred fifteen out of 132 (87.1%) third-year student pharmacists completed the course evaluation. Student pharmacists rated their level of confidence in performing skills just prior to beginning APPEs; their responses are presented in Table 5. On average, students rated themselves between confident and very confident for all skills-based items. Consistent with the perception of preceptors and fourth-year student pharmacists, the lowest rated skills-based items fell within the ACPE Pre-APPE Core Domains of Identification and Assessment of Drug-related Problems, and General Communication Abilities. The highest rated items were within the Basic Patient Assessment and Patient Education domains. The same coding and categorization process used in focus group data analysis was used to identify themes in student responses to open-ended questions on the course evaluation requesting suggestions for improving the lab course. Open-ended responses regarding areas for course improvement were used to identify shortcomings within the existing laboratory curriculum to target in the redesign. Some of these themes included decreasing the amount of downtime or passive learning during laboratory sessions, increasing exposure to complex patient cases and medication regimens, offering more opportunities to practice patient and provider interactions, improving the constructiveness and standardization of feedback provided, increasing the realism and relevance of in-laboratory simulations, and diversifying exposure to various health care settings.

Three laboratory faculty members met an average of two hours weekly for 10 weeks to complete mapping of the existing pharmacotherapy laboratory curriculum. The list of 16 critical individual skill items identified and refined from focus group input (Table 2) was compared to skills presented in the ACPE Pre-APPE Core Domains and was condensed to 12 general skills-based categories that could be feasibly covered within four semesters of the laboratory course (Figure 2). To map the laboratory component, each existing learning activity in the 4-course sequence was identified and labeled with the skills-based category it covered and the corresponding level of difficulty (I=beginner, II=intermediate, III=advanced). Once all learning activities were labeled, the sequence in which skills-based categories were covered across the laboratory courses was reviewed to assess skill inclusion and appropriateness of skill progression in terms of difficulty level and number of opportunities for performance, assessment, and feedback. Appendix 1 illustrates the mapping process for five weeks of the laboratory. The complete mapping document is available upon request from the authors.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Development of Skills-Based Categories for Use in Mapping Process (APPE=advanced pharmacy practice experiences; ACPE= Accreditation Council for Pharmacy Education).

Through review of the course map, gaps, redundancies, and overlaps with other courses were identified related to instruction, performance, and assessment of various skills, with extra scrutiny placed on those for which students perceived a lack of competency prior to starting APPEs (Tables 3 and 4). Problems identified within the existing laboratory curriculum were grouped into themes that could be used to hypothesize why perceived competency was lacking in performance of certain skills. Overarching themes for improvement to be applied in the new laboratory curriculum included teaching a framework for performance of each skill in a formal and systematic way, providing students with an appropriate number of opportunities to individually perform and receive feedback on each skill with progressively increasing stakes, sequentially intensifying the complexity or difficulty level of the skill to be performed, and improving realism and relevance of learning activities to help students make the connection between classroom experiences and real-world practice.

Overall data from the focus groups, course evaluations, and course map were woven into the redesign process and utilized at several points (Figure 1). Data from the focus groups and associated focus group surveys were applied in the redesign process in two separate manners. First, they informed a compilation of critical-skill items that could be used to develop general skills-based categories to which the laboratory course could be mapped. Secondly, they identified perceived deficits in APPE student performance that could be addressed in the design of the new curriculum. Data collected from course evaluations of third-year student pharmacists were helpful in corroborating the perceptions of fourth-year student pharmacists and pharmacy preceptors regarding APPE student performance. Finally, the course mapping process generated a map to identify additional shortcomings and redundancies in the current laboratory curriculum related to purposeful building and assessing of skills. It provided a mechanism for systematically building the new laboratory curriculum by sequencing skill placement and degree of difficulty.

DISCUSSION

A stepwise process was developed that incorporated focus groups, course evaluations, and course mapping to assess and redesign a 4-semester sequence of skills-based laboratory courses. Through the process, faculty members were able to identify strengths in the original laboratory design and pinpoint weaknesses potentially contributing to suboptimal achievement of learning objectives related to skill performance competency.

Mapping at the curricular level in PharmD programs is extensively described.^4-10 Armayor et al posed four key questions to guide curricular mapping: (1) what outcomes are not being taught; (2) whether outcomes are taught in a sequential and progressive manner; (3) what learning opportunities are being used; and (4) what assessment measures are used.⁶ The mapping process described in the current manuscript applied similar questions but adds to the literature by adapting the map to an individual skills-based laboratory curriculum. Faculty members accomplished this through critical evaluation of where skills were taught and how they progressed in terms of difficulty level and number of opportunities for performance, assessment, and feedback.

Feedback from preceptors and students via focus groups afforded an additional dimension to the assessment and redesign process. A manuscript by Lee et al describes the use of focus groups composed of student pharmacists, new practitioners, and pharmacy preceptors to guide development of a new pharmacy capstone course.¹¹ Similar to the current study, participants in focus groups were asked to identify skills necessary for successful completion of APPEs. The focus groups described in the current manuscript went a step further by inquiring which skills students were prepared to competently perform on APPEs and used this as a mechanism for quality improvement within an existing course.

Overall, the assessment and redesign process produced many useful outcomes. The importance of active learning and clinical application are emphasized in the ACPE Accreditation Standards and have also been identified as vital components of pharmacy course development in previous student and pharmacist focus groups.^2,11 The current redesign process allowed faculty members to ensure in-class time was spent in active learning focused on teaching, developing, and enhancing skills. Course mapping facilitated identification of redundant and passive-learning activities that could be eliminated and nonskills-based activities more appropriate for inclusion in other courses. Previously, a portion of class time in the laboratory component of the larger pharmacotherapy course was spent reinforcing lecture material through case discussions not associated with a skill or content discussion about a specific topic. In the redesigned course, didactic material will only be reviewed in relation to performance or assessment of a skill, and case discussions will be moved to the new pharmacotherapy lecture-based course and held in large-lecture classroom discussion.

Similar to focus groups in Lee et al’s work used to inform the design of a pharmacy capstone course, the current process involved both students and pharmacists.¹¹ However, an additional strength of the current assessment and redesign process was the combination of student pharmacists at various stages of the curriculum and pharmacy preceptors from varying practice sites who provided feedback. Feedback on the course structure and composition was collected from students who had recently completed the entire 4-semester pharmacotherapy curriculum. Feedback on perception of student performance competency was obtained from third-year student pharmacists just completing didactic work and looking ahead to APPEs, fourth-year student pharmacists just completing APPEs and reflecting back on their level of preparedness, and pharmacists from a range of sites who precept students and evaluate them on skill performance on a regular basis. The perception of student pharmacists is particularly interesting given self-efficacy or self-perception is a predictor of performance and is associated with motivation to engage and persevere in an activity even under stressful situations.^13-15 Identifying skill areas in which students do not perceive themselves capable and targeting those for improvement within the didactic portion of the PharmD program may ultimately support student engagement and achievement of skills on APPEs. In the current redesign process, this was directly applicable given the ultimate goal of the new laboratory curriculum is student preparation for APPEs.

Overall, there were consistencies in the perceptions of the three groups of varying individuals. Comparing skills-based items within the ACPE Pre-APPE Core Domains, responses from all groups indicated students were best prepared to perform skills within the domains of Basic Patient Assessment and Patient Education, and least prepared items in the Identification and Assessment of Drug-related Problems, and General Communication Abilities domains. This level of corroboration between groups strengthens the authenticity of the assessment results.

The focus groups themselves provided an unforeseen benefit in encouraging communication between faculty members and pharmacist preceptors and student pharmacists. Focus group sessions demonstrated to students and pharmacist preceptors that faculty members are invested in continually growing and revising teaching methods to provide students an optimal learning experience and to achieve education outcomes. This, in turn, fostered new professional relationships and provided an avenue for collaboration on the design of learning activities to increase the realism and relevance of simulations and assignments as this was one of the constructive criticisms of the existing laboratory component.

Finally, by facilitating the evaluation and demonstration of achievement of ACPE Pre-APPE Domains, the redesign process directly aligned with UW-Madison School of Pharmacy’s assessment plan as well as ACPE standards that emphasize importance of review and quality assurance within PharmD curricula.² It ensures skills taught and assessed in the Integrated Pharmacotherapy Skills courses map to specific core domains and educational outcomes.

While executing the course assessment and redesign, several shortcomings in the process were identified. First, the course evaluation administered to third-year student pharmacists had a high response rate but did not include the exact survey items and did not utilize the same Likert scale as the survey administered to fourth-year student pharmacists and preceptors. This made it more challenging to directly compare the feedback provided by all three groups. In addition, third-year student pharmacists tended to rate their confidence for skill performance higher than was expected by laboratory faculty members, especially considering the more discerning perceptions of fourth-year student pharmacists and preceptors in focus groups. This variation may be related to unawareness of expectations for skill performance prior to beginning APPEs and performing skills in real-world situations. It highlights the importance of gathering feedback from multiple sources to illustrate the full picture of student preparedness and areas for improvement when revising a skills-based course.

While the number of focus group participants was consistent with previous focus groups used in assessing pharmacy education and the sessions were successful in initiating a targeted and detailed discussion among pharmacists and students, feedback was obtained from a relatively small number of individuals.^11,12 Sending a survey out to the wider population of UW-Madison pharmacy preceptors and fourth-year student pharmacists would have increased the number of individuals providing feedback and would have been useful to corroborate the themes derived from focus groups. Frenzel et al surveyed preceptors from 16 schools of pharmacy from across the United States to identify skills viewed as essential for student pharmacists. Communication, critical thinking, and collaboration were all deemed essential by preceptors.¹⁶ Future course assessments will use a combination of broadly administered surveys and targeted focus groups to gather feedback from fourth-year student pharmacists and pharmacy preceptors.

Laboratory faculty members served as the focus group interviewers. While this could have introduced a degree of bias into the process, laboratory faculty members possessed the greatest knowledge of the previous laboratory curriculum, which made them the most logical choice. Student pharmacists were known to faculty members, but they were no longer enrolled in a laboratory course at the time of the focus groups. Likewise, there was no status relationship between laboratory faculty members and the pharmacy preceptors that would influence participant willingness to provide candid feedback.

Finally, schools of pharmacy contemplating a similar course assessment and redesign process should consider the substantial time commitment required. Collection, analysis, and interpretation of data were performed over approximately three months, of which the mapping process was the most time consuming taking approximately 20 hours to complete. Subsequent laboratory revisions were developed and implemented across the academic year. Pharmacy residents and students with an interest in academia partnered with faculty members on projects related to the revision. This not only presented a good learning opportunity for trainees, but reduced faculty workload. Workload related to revisions in upcoming years is expected to be greatly reduced.

Future steps following completion of the redesign will include launching the Integrated Pharmacotherapy Skills curriculum with the newly developed learning activities and course structure. A specific evaluation process will be implemented to allow for continuous quality improvement of the newly designed lab curriculum. Outcome measures will be collected at completion of the 2014-2015 academic year and used as comparison points to evaluate effectiveness of the redesign and subsequent need for further revisions. Outcome measures will include student responses on course evaluations assessing opinions and preferences about laboratory courses and perceived learning, final course grades, student achievement on performance-based assessments, and student focus groups. Long-term outcomes measures will include student performance on APPEs.

SUMMARY

This study aimed to illustrate how a stepwise approach utilizing data from focus groups, course mapping, and student evaluations can be applied to systematically assess and redesign sequential skills-based courses within a PharmD curriculum aimed at preparing students for APPEs. The process allowed for identification of both strengths and weaknesses in the original course design potentially impacting perceived performance competency of critical skills that could subsequently be addressed in the design of the new laboratory courses. A similar process could be adopted by instructors and faculty members at other schools of pharmacy who intend to assess and/or revise existing skills-based courses.