Lessons Learned from Evaluating a Process for Systematic Curriculum Review

INTRODUCTION

The Accreditation Council for Pharmacy Education (ACPE) updated the accreditation standards for the Doctor of Pharmacy (PharmD) programs in 2016, outlining the necessity of curricular quality assurance processes for both course review and syllabi content.¹ Examples of published curricular quality improvement processes in the pharmacy and medical literature are growing, with several papers that describe processes to examine and make improvements to didactic and experiential courses.^2-7 Established, ongoing curricular review processes are imperative to ensure course materials, teaching strategies, and assessments are appropriate to adequately prepare practice-ready pharmacists. While published examples demonstrate the importance of curricular mapping to outcomes using course syllabi, formalized systematic review remains a challenge in higher education. Additional models and examples could serve as the basis for new and improved programs.

The curriculum at the Northeastern University PharmD program includes two years of undergraduate pre-pharmacy coursework and after meeting progression requirements and passing an interview, four years of professional pharmacy coursework in the P1-P4 years. Graduate direct entry students may join the program at the P1 year. Undergraduate and professional curricular affairs are managed by the faculty-led Curriculum Committee. The systematic curricular review policy and process was developed by the Curriculum Committee and approved by the faculty in 2009. The purpose of this report is to share an evaluation of the systematic curricular review procedures and the current review process as shaped by several years of experience.

METHODS

The systematic curricular review process was implemented with best practices of assessment in mind to allow for evaluation and potential modification after several years of experience.⁸ To deliver this quality improvement effort, data were collected and reviewed, feedback from several sources were sought to clarify process and workflow barriers, outcomes and impacts were learned, and suggestions for improvement after the first three-year course review cycle were solicited. Data on disposition of courses targeted for review in the first cycle were collected from the committee’s records and meeting minutes. In 2015, an anonymous electronic web-based survey (Qualtrics, Provo, UT) was developed and sent to instructors of record for all required and elective courses and to the Curriculum Committee members. Participants were asked to rate their agreement with statements about the value and impact of the systematic curriculum review process using a 4-point Likert scale (strongly disagree to strongly agree) and for their suggestions for process improvement. They were also asked to identify changes that were made to their courses as a result of the process. Two reminders were sent to encourage non-respondents. Descriptive statistics were used to evaluate the results and open-ended responses were forwarded to the curriculum committee. Additional suggestions for process improvement were compiled from the minutes of curriculum committee meetings where systematic review process improvements were discussed. This project was determined to be exempt after review by the university’s Institutional Review Board.

RESULTS

The process met the goal of reviewing courses in the PharmD curriculum with 100% of required courses reviewed during the first three-year cycle of review or as part of regular consideration during curricular revision.

Thirty respondents (88%) completed the electronic survey. Among the 30 respondents, 29 (97%) self-identified as instructors of record, of which 12 (40%) were members of the curriculum committee (past or present). Four faculty (13%) were new instructors of record and two (7%) were only responsible for an elective course, thus had not yet personally experienced systematic review. All respondents agreed/strongly agreed that systematic curricular review is an important part of curricular evaluation and 77% agreed/strongly agreed that they were familiar with the policy and procedure; 73% agreed/strongly agreed that they know where to find a copy of the policy and procedure; and 80% agreed/strongly agreed that they knew what constituted substantive changes that require curriculum committee review of an existing course. Most (89%) agreed/strongly agreed that the review has provided valuable feedback regarding the course and 74% agreed that the review process can be further improved. Of the 18 faculty who had gone through a scheduled review of their course, 13 (72%) indicated that they made course changes based on recommendations (Table 1). The most frequently suggested process modifications were to include elective courses in the process and to change the cycle for course review to once every four years.

Committee discussions revealed that there was no appropriate syllabi developed for introductory or advanced experiential courses and the review process did not include them. Syllabi were developed and approved, and experiential courses were slated into the review cycle. Several other opportunities for process improvement were identified. For example, it was common for the committee members’ report to include recommendations to the course syllabus, but as these recommendations were often not communicated to the instructor until after the course was approved, follow through to ensure implementation was challenging. Also, the workload associated with reviewing courses within a three-year cycle was burdensome to committee members and did not provide feedback on elective or experiential courses.

Current Process for Systematic Curricular Review

The process has evolved since its initial presentation and is outlined in Figure 1. The revisions included changes to information collected about the course and the focus of the discussion with the instructor, process changes about which courses are reviewed and how often, and logistical changes to how the curriculum committee responds to information about the course. A structured electronic questionnaire (available upon request) was created to allow for uniform data collection and to create an electronic archive of responses. Several changes were made to questions asked of the instructor including a prompt for discussion of the appropriateness of assessments and a question seeking reflection on how assessment data is used to make changes to the course. Additionally, a survey of active learning strategies used (available upon request) was also administered as part of the review.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Systematic Curriculam Review-Current Process.

Process changes were made so that service courses for early assurance pre-professional students were no longer routinely reviewed. Though valuable information was collected on these courses in the first cycle, it was determined that frequent re-assessment was unnecessary and burdensome to faculty from units outside the school. Required experiential courses and any undergraduate elective courses offered by the school were added to the list of courses to be reviewed. The process was changed to allow the person doing the initial data collection and review to be any faculty member oriented to the materials and process, in order to expand the pool of eligible reviewers instead of restricting that task to committee members only. As a result, standing charges for both the assessment and curriculum committees include having members oriented to the materials and review process as they join the committee. A new committee member also learns the review process by completing his/her first review together with an experienced reviewer. Lastly, instead of having the faculty reviewer compile materials, present to the committee and then ask for any revisions or discussion items to be implemented in the course, the steps of the review were reordered so the reviewer’s suggestions are made and considered by the instructor prior to the meeting. During the meeting, a revised, final version of the course syllabus is circulated and a summary of the changes made are discussed. If further revisions are recommended, the vote on the course is delayed until a revised course syllabus is available. This enhances accountability for the instructor to adopt any recommendations in a timely manner and keeps the task on the committee’s agenda until resolved.

The cycle of review was changed from three to four years and modifications were made to reduce the committee workload associated with the review and to improve implementation of recommended changes to courses and syllabi. Focus areas for discussion were also added in each of the subsequent systematic review cycles to allow the process to drive further quality improvement and still be easily modified to meet programmatic needs. For example, in the current cycle, the focus areas are use of electronic testing, standardization of the scales used to assign final letter grades in the course, and promotion of best practices for question writing on assessments.

DISCUSSION

Overall, evaluation of the systematic review process revealed positive attitudes from the faculty, a commitment to ongoing improvement, familiarity with policies and procedure, and a positive impact of the first cycle of review on courses. Survey respondents included four new instructors for required courses and two instructors of elective courses who had not personally gone through the systematic review process. The process provided the school with information needed for ongoing mapping related to curricular outcomes and content covered, assessment strategies, and instructional active learning methods used as this information can be extracted from each course syllabus and routinely transferred to curriculum maps and other summary documents as managed by the school’s Assessment Committee.

The school Curriculum Committee must ensure the quality of the entire curriculum, including appropriate content coverage. One concern for effective curriculum management is course proliferation, an increase in number of courses that sometimes results in content overlap, scheduling conflicts and increase costs.⁹ Elective course options were added to the program over time, and eventually these were also added to the systematic review process, with inclusion of additional courses resulting in additional need for course reviewers. To ensure effective curriculum management and quality, some of the principles defined by Dressel were identified and implemented including adequately define course content and resources; compare new course proposals to related courses; estimate student enrollment and prerequisite requirements; and identify fiscal implications.¹⁰ With the inclusion of all required, elective and experiential courses in the professional years of the program, courses were reviewed focusing on identifying overlap or gaps in curriculum and content.

Over the past decade, pharmacy education has shifted toward greater accountability in learning outcomes. As a result, ACPE accreditation standards outlines expectations in curriculum review and quality assurance but do not dictate how this should occur.¹ Persky and colleagues described a review process to improve required PharmD courses involving both assessment and curriculum committees.² The review process consisted of five steps, which were self-reflection by the course director, course review conducted by the review team, discussion of the team’s findings and recommendations, course director review of the team’s recommendations, and retrospective review of the course at the end of semester. This is similar to our process in that it takes into account instructor self-review and review by an outside team or faculty member. Other programs also include experiential courses in systematic review processes, including advanced pharmacy practice experiences and medical school clerkships.^4,5

Britton and colleagues described a system for curriculum review and mapping.⁶ To facilitate this process, the school designed a comprehensive, web-based curriculum management system to help course coordinators and peer review teams collect data on course policies and procedures, content and activities, skills, alignment and integration of the course within the curriculum, learning outcomes, and assessment.⁶ Based on data collected, the peer review team presented a summary report to the curriculum committee. All courses within one semester were reviewed together with presentations on a day specifically designated for the effort.⁶ Our program is similar to this example and other published course review procedures in that it uses a standardized data collection platform.⁷ However, while some information (such as the instructor’s responses to the questionnaire) can be extracted from the online survey for review, copies of syllabi and course materials are not retained as part of the online data collection process and the online system is not a searchable, robust database.⁶

One other difference between our program and others is that our program’s initial review is done by one committee member, rather than a review team. Since findings and recommendations from our program are ultimately discussed with the full committee and since the initial review focuses on syllabus content and information gathering, this single-reviewer process might streamline initial review phases where issues of school policy generally describe any revisions needed (eg, syllabus formatting). While reviewers need to be familiar with the process and curricular expectations of the school, in our experience, an orientation to review materials and participation in at least one review with an experienced committee member can provide sufficient background to the process and expectations for new reviewers. In contrast to curricular review programs that review all courses in a semester at once, our process groups courses by discipline or subject so that courses with related content can be discussed together. This allows for discussions about integration and efficiency of content covered, as well as student preparation for the next course in a series.

We offer several suggestions for other programs considering ways to develop or refine processes for systematic review of courses or entire curricula. Principles of continuous quality improvements should be built into the program framework with the initial planning (plan) and implementation (do) followed by evaluation of the process and outcomes (check) and subsequent modifications (act) to the process, policies and procedures.⁸ It is also useful to create electronic surveys for data collection and templates for communication and reports so that anyone serving as a course reviewer has a resource when comparing the course syllabus to the template, communicating with the instructor or preparing a report for the committee. Additionally, it is helpful to establish an administrative point-person to manage related processes in a given academic year. There are many tasks, such as assigning courses to the first reviewer, responding to questions, scheduling presentations in committee meetings, etc., that are best coordinated centrally to reduce redundancy and unnecessary delay.

Our evaluation of the process for systematic curriculum review was limited by several factors. The survey was not a validated tool and was not pilot tested beyond the research group before distribution. Also, as the Curriculum Committee is comprised primarily of faculty, there were likely survey respondents who were both course instructors and committee members. We did not collect extensive demographic data and were unable to compare feedback from committee members to feedback from course instructors.

The program took approximately one review cycle to establish a culture of accountability for meeting syllabus template, recognizing that different content and instructor style will be represented in course syllabi, but that certain elements (eg, programmatic outcomes covered by the course, office hours, grading criteria) are expected in all. We also encourage schools and colleges of pharmacy to adopt a culture where required PharmD courses belong to the school and to the program, rather than to one instructor. This allows for our model of ongoing review to facilitate change to improve the student experience with less resistance from instructors who feel personally challenged. Since course quality should, in theory, improve after each cycle of review, the review process can be targeted or adapted each cycle to include focus areas for discussion. This allows additional efforts to be directed at needed programmatic initiatives without creating a new faculty committee or additional task for instructors.

CONCLUSION

This evaluation of our process for systematic curriculum review found the program to be generally successful, assisting in collection of data for mapping and programmatic assessment and increasing adherence with the school syllabus template. As a result of the evaluation, several improvements were made to the systematic curriculum review process to streamline workload, improve accountability and better target which courses were reviewed. Going forward, the committee plans to evaluate the process at the end of each review cycle in order to adjust focus areas and adapt to the changing needs of the curriculum.

While ACPE sets forth standards for curricular review and quality assurance, colleges and schools of pharmacy are left to develop their own mechanisms. Sharing of processes within the literature allows for other colleges and schools of pharmacy to more easily adapt and modify such processes to implement at their institutions.