Abstract
Objective. The entry-to-practice PharmD degree is designed to meet the Educational Outcomes of the Association of Faculties of Pharmacy of Canada (AFPC). We set out to evaluate how assessment strategies in a “capstone” course align with AFPC educational outcomes, their respective key and enabling competencies, and whether enough assessments exist for students to demonstrate achievement of competencies prior to embarking on advanced pharmacy practice experiences.
Methods. We mapped each assessment’s objectives, content, and methods to the key and enabling competencies of each role of a pharmacist. The number of enabling competencies mapped represents the extent to which the assessment addressed the associated key competency and broader role. Deidentified student performance data were analyzed to identify achievement of competencies despite failed assessments.
Results. Of the seven role descriptions, the roles care provider, communicator, and collaborator were the most comprehensively assessed. The roles leader-manager and health advocate were assessed to a limited extent. The role scholar was not covered to a great depth across assessments. The role professional was not represented in most assessments except for the final examination. Students with failed assessments generally had ample opportunity to demonstrate competencies through other assessments.
Conclusion. Mapping assessments to AFPC educational outcomes is an essential step to demonstrate direct evidence that students have achieved the intended learning outcomes. Our map revealed that the assessments sufficiently overlapped with most AFPC educational outcomes with a few exceptions. It is important to create multiple opportunities within a course for students to demonstrate achievement of competencies to ensure practice readiness.
INTRODUCTION
In Canada, the Canadian Council for Accreditation of Pharmacy Programs (CCAPP) sets standards for pharmacy programs regarding quality assurance and program enhancement. 1 These standards indicate that a program must be “based on an organized educational framework that facilitates development of graduates with competencies to meet the entry-level scope of practice.” 1 The educational framework must be based on the Educational Outcomes (EOs) defined by the Association of Faculties of Pharmacy of Canada (AFPC) 2 and the Standards of the National Association of Pharmacy Regulatory Authorities (NAPRA). 3 As part of CCAPP Standard 1, Criterion 1.2, pharmacy programs must produce evidence that “students demonstrate practice-readiness that enables them to provide patient care as a collaborative member of a care team before culminating direct patient care required practice experiences.” 1 In 2017, AFPC educational outcomes were updated, drawing on several international pharmacy education frameworks. 4-6 In preparation for accreditation and in recognition of the updated educational outcomes, we set out to map and analyze course assessments and students’ performance in a required course.
Over the past few years, under the auspices of the Education Office within the Leslie Dan Faculty of Pharmacy, course coordinators have updated course outlines to align with these new AFPC educational outcomes. These outcomes comprise seven major pharmacist roles: care provider, communicator, collaborator, leader-manager, health advocate, scholar, and professional. Students are expected to demonstrate achievement of these educational outcomes to graduate. The educational outcomes are further subdivided into key competencies that represent measurable behaviors, and within each key competency there are enabling competencies that denote specific skills and behaviors necessary for practice readiness. The 2017 AFPC educational outcomes, with descriptions of roles and key competencies, are provided in Table 1. While the enabling competencies are not shown in this table, they are available in the 2017 AFPC EO Report. 2 Although mapping individual course learning outcomes in relation to these new educational outcomes is important, we set out to go beyond content mapping to determine whether course assessments provide students with adequate opportunities to demonstrate achievement of practice readiness for advanced pharmacy practice experiences (APPEs). 7
Educational Outcomes Regarding Roles and Key Competencies Defined in 2017 by the Association of Faculties of Pharmacy of Canada
We chose to complete this assessment mapping in a third-year required course, namely Preparation for APPE, which all students must pass prior to commencing APPEs. A growing body of research has shown the importance of a preparatory experiential rotation course to optimize pharmacy student learning through successful demonstration of practice readiness. 8-10 This half-credit course was implemented in 2014 as part of the new PharmD program curriculum to prepare students for their culminating fourth-year experiential rotations; the course aimed to help students consolidate the knowledge, skills, and attitudes they had learned throughout the curriculum. The positionality of this course made it particularly apt to examine whether course assessments were purposeful in assessing educational outcomes and associated competencies. 11 We anticipated gaining a better understanding of whether course assessments offered students the opportunity to demonstrate practice readiness while informing us about course quality improvements. 12
This study draws on the theoretical framework of assessment blueprinting, which is defined as “the process of linking tests to learning goals.” 12 While the course coordinator already blueprinted assessments to ensure that they effectively tested course content, the process of blueprinting assessments was extended to explore whether they sufficiently mapped to intended educational outcomes. Mapping course assessments to competencies has two key advantages. First, it helps course instructors articulate course expectations and helps students understand the focus of learning experiences. Second, it enables the instructor to be strategic in their teaching activities and assessment planning, and it gives them the ability to provide direct evidence that course assessments indeed provide opportunities for students to demonstrate competencies while ensuring constructive alignment throughout the course. 13 These processes ultimately create “significant learning experiences,” whereby students have varied opportunities to demonstrate their practice readiness and achievement of educational outcomes. 14 Our study set out to address two main research questions. First, do current course assessments align with intended AFPC educational outcomes as outlined in the course learning objectives? Second, are there a sufficient number and variety of assessments for students to demonstrate achievement of AFPC educational outcomes and associated key and enabling competencies? By addressing these questions, we add to the limited but evolving body of literature in health professions education on purposeful assessment strategies in competency-based education. 15-17
METHODS
In the Preparation for APPE course (2018-2019), data from the six assessments (eg, format, weight, type) were provided by the course coordinator. An overview of the assessments and their main objectives is highlighted in Table 2. To determine whether there is sufficient overlap in assessments for students to demonstrate practice readiness for APPEs, deidentified student performance data on all six assessments was collected from the learning management system and used to identify failed assessments. A failing mark was defined as below 60% as per the Faculty Grading Policy. Approval was obtained from the University of Toronto Health Sciences Research Ethics Board.
Overview of Assessments in the Preparation for APPE Course
Assessment mapping of our six assessments was performed by two research team members using a systematic process considering the format (eg, written, video, examination), assignment/examination blueprint, and assessment rubric. For each assessment, these components were mapped to defined enabling competencies. 2 Each researcher made a dichotomous decision on whether the component did or did not fit the description of the enabling competency. Any discrepancies were resolved by consensus. In each assessment, the number of enabling competencies mapped within each key competency indicate the extent to which the key competency in each role was assessed.
RESULTS
The assessment map is shown in Table 3. Out of all seven roles, those of care provider, communicator, and collaborator were most comprehensively assessed based on the number of enabling competencies covered in at least four of the six assessments. The leader-manager and health advocate roles were assessed to a limited extent. While the scholar role was covered in most assessments, typically less than half of its enabling competencies were assessed. Finally, the professional role was not represented in most assessments, although the final examination assessed all key and enabling competencies of this role description.
Assessment Mapped to AFPC Competencies and the Student Failure Pattern
When examining student performance trends across assessments (N = 233 students), we identified that 38 students failed at least one assessment, and four of these students failed two assessments. For students who failed only one assessment (n=34), our assessment map showed that there was sufficient overlap in competency coverage with other assessments, so that students were still able to demonstrate achievement of educational outcomes. The only exception to this was when a student failed the final examination, which was the only assessment that comprehensively assessed the role of professional. However, in our analysis of students that failed only one assessment, they all passed the final examination.
For students who failed two assessments (n = 4), our failure pattern analysis in Table 3 illustrates that all students but one were still able to demonstrate practice readiness in all competencies across the other four passed assessments. For example, even though students A and D failed the patient case presentation and documentation assessments, Table 3 shows that there were opportunities in the care plan assessment and final examination to demonstrate practice readiness, due to an overlap in competencies that were not achieved in the failed assessments. An exception to this is student C, who failed the final examination, which we noted earlier as being the only assessment to cover the professional role; therefore, student C did not have the opportunity to demonstrate this competency in the other course assessments. No students failed more than two assessments, and all students passed the course.
DISCUSSION
Our mapping process revealed multiple course related opportunities for students to demonstrate achievement of most AFPC educational outcomes, including key and enabling competencies within each role statement. Within competency-based health professions education, Norcini and colleagues have discussed the value of a purposeful system of assessments, and it is important to contextualize our map findings within this framework. 15 Most assessments thoroughly covered the care provider, collaborator, and communicator roles. This purposeful overlap depicted in our map concords with the importance of ensuring that students have multiple opportunities to demonstrate practice readiness. 17 These assessments were also designed to simulate experiences that students will face in APPEs, which is consistent with recommendations by Holmboe and colleagues; these recommendations emphasize the importance of careful planning of diverse assessments that consider the clinical setting. 16
Our map revealed an assessment gap related to the roles of leader-manager and health advocate. These roles were challenging to map, as they were not explicitly assessed. This aligns with a recent review by Dennis and colleagues, which highlighted how outcomes related to advocacy and leadership are scarce in publications about APPEs in the United States. 18 Previous research has also highlighted the difficulty in assessing the health advocate role and related competencies in Canadian medical education. 19 Therefore, it is important to carefully consider various assessment approaches and identify which might be most appropriate for fully assessing these roles and their related competencies. 20, 21 Having an intentional approach, as proposed by Janke and colleagues, could better integrate the underassessed leader-manager and health advocate roles, thereby addressing assessment gaps. 22
Our map also demonstrated the importance of considering depth versus breadth of competency coverage across assessments, where depth refers to the number of enabling competencies assessed within an assessment, and breadth is the number of enabling competencies assessed across all six assessments. The professional role was only assessed in the final examination (depth), and the scholar role was assessed across all assessments (breadth). Our finding that the final examination was the only assessment that assessed the professional role reveals a gap in our assessment approach. Our student performance analysis shows one student failed the final examination, and, as such, this student did not have another opportunity to achieve the professional role competencies. Ideally, there should be opportunities across multiple assessments to demonstrate proficiency while also having focused assessments where certain outcomes are assessed to a greater extent to ensure balanced breadth and depth, respectively. To address this finding, in the next iteration of the course, we made informed adjustments to lecture content by adding professionalism lecture and case discussions, and we also adjusted the assessment blueprint.
This study has four main limitations. First, student performance on the final examination was examined holistically, and performance on individual questions was not considered. Failing the examination does not necessarily mean that the student is unable to demonstrate achievement of specific competencies, as the student may have gotten some competency-focused questions correct; the opposite also holds true, as passing the examination does not guarantee readiness in all competencies. 23 Future work should map each question to relevant competencies and evaluate student performance for each question. Second, we did not consider student perspectives of whether the assessments provide ample opportunities to achieve educational outcomes. There is value in considering both students’ and instructors’ perceptions of assessments, as they can differ, and our next steps should triangulate student feedback as well as their confidence related to practice readiness with that of faculty perspectives. 8, 10, 24-26 Third, although no student failed more than two assessments in this cohort, the possibility remains that this may be the case in future cohorts, which is why continuous monitoring of student performance in each assessment and careful blueprinting of assessments (adequately planning how assessments align and overlap with intended objectives) is required to ensure effective learning toward practice readiness. Finally, as we only considered one course, mapping all assessments across the program may provide further evidence of student practice readiness.
Despite limitations, this study extends our current curriculum-mapping work by adding an additional layer of assessment mapping to educational outcomes. Our work emphasizes the importance of designing assessments that provide sufficient opportunities for students to demonstrate achievement of practice readiness before embarking on APPEs.
CONCLUSION
Assessments in our course align with updated AFPC educational outcomes, and there are enough assessments for students to demonstrate readiness in several of the roles. Assessment mapping is an essential step to ensure that educational outcomes align with assessments in a competency-based education program. Further, it is important to provide students with multiple opportunities to demonstrate achievement of learning outcomes to help ensure practice readiness for APPEs.
ACKNOWLEDGMENTS
This project was supported by the Teaching Innovation Grant and the Leslie Dan Faculty of Pharmacy at the University of Toronto. We would like to also acknowledge the Education Advisory Committee.
- Received May 10, 2021.
- Accepted August 26, 2021.
- © 2022 American Association of Colleges of Pharmacy
REFERENCES
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.