Abstract
Objective. To describe curricular mapping strategies used in analyzing and interpreting curricular mapping data and present findings on how these strategies were used to facilitate curricular development.
Design. Nova Southeastern University's doctor of pharmacy curriculum was mapped to the college's educational outcomes. The mapping process included development of educational outcomes followed by analysis of course material and semi-structured interviews with course faculty members. Data collected per course outcome included learning opportunities and assessment measures used.
Assessment. Nearly 1,000 variables and 10,000 discrete rows of curricular data were collected. Graphic representations of curricular data were created using bar charts and stacked area graphs relating the learning opportunities to the educational outcomes. Graphs were used in the curricular evaluation and development processes to facilitate the identification of curricular holes, sequencing misalignments, learning opportunities, and assessment measures.
Conclusion. Mapping strategies that use graphic representations of curricular data serve as effective diagnostic and curricular development tools.
INTRODUCTION
Curricular evaluation and revision can be an intimidating process for educators. While most faculty members within a training program may feel that they understand what their curriculum should accomplish, it is often difficult to grasp what, how, and where specific content is addressed in the curriculum, or what measures are used to assess student achievement of curricular outcomes. Thus, when curricular revision is initiated, it can become a cumbersome and often confusing process for faculty members. Frequently, curricular decisions are based on faculty members' perceptions rather than from objective curricular data because these either are not available or are not interpreted easily. In these situations, curricular changes may be implemented that either are unnecessary or ineffective because a systematic curricular evaluation process has not been performed. Curricular mapping is emerging in higher education as the curricular evaluation tool that guides and makes the curricular evaluation and revision process more streamlined and transparent for stakeholders.1 Curricular mapping has its roots in K-12 education, where it has been embraced as an effective curricular communication and decision-making tool within schools and across school districts.2–4 In academic pharmacy, curricular mapping has become the state of the art curricular measurement tool, to the point that the process is a key criteria in Standard 12 of the Accreditation Council for Pharmacy Education (ACPE) Standards 2007.5
Curricular mapping can be described simply as a technique that captures data about key curricular elements (content, outcomes, teaching and assessment strategies, etc.), and visually demonstrates the relationship among these elements so that curricular data can be interpreted and communicated more effectively. While curricular mapping is a systematic approach to curricular evaluation and revision, it is still a complex process that requires careful planning, assessment, and analysis. Higher education research literature offers several sources that outline different approaches to curricular mapping;6–14 however, strategies for using curricular mapping data to its full potential has been defined less clearly.
In 2007, a team of faculty members at Nova Southeastern University (NSU) initiated a curricular mapping project which was completed in mid-2008. The mapping process is summarized briefly in this manuscript, yet the focus of this discussion is what happens after the collection of mapping data is complete. Readers who have participated in a curricular mapping project probably will agree that upon completion of a comprehensive curricular map, a massive set of data is left that can be overwhelming to analyze and interpret.
While completing mapping data collection requires extensive time and effort, that is only the halfway point. The next phase of curricular mapping is arguably the most important: analyzing those data and facilitating understanding of the curricular map among stakeholders. The purpose of this paper is to describe the curricular mapping strategies employed by Nova Southeastern University College of Pharmacy (NSU-COP) in analyzing and interpreting curricular mapping data and present findings on how these strategies are used to facilitate curricular development.
DESIGN
In mid-2007, NSU initiated its curricular mapping project. The goal of the project was to develop a curricular quality improvement tool to serve as the centerpiece for continuous curricular evaluation and improvement. In addition, the mapping tool would gather curricular data needed to meet ACPE 2007 Curriculum Standards 9–15. For the mapping tool to be useful, it needed not only to capture and organize curricular data, but more importantly, to allow the data to be presented in an easily visualized and interpreted manner.
The first phase of this project was to identify competency-based learning outcomes for the first professional doctor of pharmacy degree. Multiple sources were reviewed (Center for the Advancement of Pharmaceutical Education [CAPE] Outcomes, 2004 CAPE Supplemental Educational Outcomes offered by the American Association of Colleges of Pharmacy [AACP], the North American Pharmacist Licensure Examination [NAPLEX] blueprint, research literature, published outcomes from other programs similar to NSU, etc), and a preliminary list of learning outcomes was compiled. Each outcome on the list was then subjected to review to ensure that the outcome was action-based and measurable. The preliminary list of potential outcomes contained more than 1,400 items; the list was culled by rephrasing outcomes to eliminate overlap or redundancy. A list of approximately 400 outcomes was submitted to the NSU faculty for review. After 3 months of review, the faculty endorsed a list of 437 competency-based learning outcomes organized into 4 major domains (Table 1).
Nova Southeastern University College of Pharmacy Educational Outcome Domain
There was concern that the college would need to design a new curriculum to achieve effective teaching of its outcomes. Rather than assume that such a radical change would be required, the college pursued the curricular mapping of the existing curriculum to guide curricular revision. The curricular mapping plan called for meeting with course coordinators to assess comprehensively how learning opportunities in their courses were aligned with the learning outcomes.
Next, the curricular mapping team defined key terms pertaining to mapping. Several classification schemes were defined a priori, such as the types of learning opportunities, the level of alignment between a learning opportunity and a learning outcome; the style of learning (active or passive) used to teach towards an outcome; the types of assessment measures corresponding to the outcome; the categories of the assessment to differentiate the content of assessment items; and the class of assessment (formative or summative). These definitions and codes are described in Appendix 1. A semi-structured interview format was devised for faculty member interviews. Each session was conducted by 2 interviewers, who had reviewed all course materials prior to the interview. At the onset of the session, the faculty members were asked to describe their courses in detail. The interviewer then described the classification scheme and answered questions. Characteristics of the courses were also recorded, such as number of students enrolled, the training year/semester when the course was offered, the use of distance learning, the instructors involved in the course, etc.
The session involved moving through each outcome systematically and discussing how the course addressed that outcome. Due to the variation of teaching methods used throughout the curriculum, generating standard questions for each session was not possible, yet the probes were essentially:
• “Let's read the outcome. Does your course address that outcome in any way?”
• “At what level of alignment would you say the learning opportunities in your course support achievement of the outcome?”
• “What types of learning opportunities are used in your course to address that outcome?”
• “Do you use active learning to teach towards that outcome? If so, please describe those techniques.”
• “How do you assess that students have learned the material related to that outcome?”
• “What kinds of items do you include in that assessment?”
• “Do students receive feedback about their performance on the assessment? Do they have opportunities to remediate, if necessary?”
Sessions typically lasted about 2 hours, often because the interview generated much discussion. Faculty members enjoyed the opportunity to describe their teaching methods in this level of detail. Initial disagreements about classification were uncommon, and when they occurred, they were resolved unanimously through discussion. During the mapping sessions, the interviewers recorded faculty members' responses on standardized data collection sheets. These documents were later transferred into a database that was developed specifically for this project. The total time required to complete the mapping of a single course was about 3 hours. Data collection for the entire curriculum was completed in about 4 months.
EVALUATION AND ASSESSMENT
Upon completion of data collection, the curricular mapping database contained nearly 1,000 variables and approximately 10,000 discrete rows of data. The mapping team next confronted the formidable question: How can the mapping data be used optimally to answer key questions about the curriculum on an ongoing basis? To answer this question, the mapping team identified the visual format that would represent the mapping data best. A search of the literature revealed a variety of representations of mapping data that included topography maps, spreadsheet depictions, and checklists.4,10,14,15 A simple, 1-dimensional map did not represent the curricular data in a way that was simple and user-friendly, therefore, multiple maps with different graphical representations were needed to illustrate visually the connections among curricular elements. To simply further the interpretation of curricular data by NSU curricular planners, the team generated solutions to the following set of key questions:
(1) Are there outcomes that are not being taught in the current curriculum?
(2) Are outcomes being taught in a sequential, progressive manner?
(3) What learning opportunities are used in the curriculum? Are these passive or active?
(4) What assessment measures are used in the curriculum? Are these formative or summative?
These questions are probably of interest to most, if not all, college curricular planners who undertake similar curricular mapping projects. The following presents how maps were used to address each of these questions.
(1) Are there outcomes that are not being taught in the current curriculum? Upon adopting learning outcomes, the college must ensure that its students are presented with opportunities to achieve those outcomes. NSU already had a curriculum in place and later adopted new competency-based learning outcomes, a different process than building a new curriculum according to a set of learning outcomes in which learning opportunities are created to address each outcome. The question became whether to jettison the existing curriculum and create an entirely new curriculum specifically designed to the new outcomes, or if revising the existing curriculum to allow for alignment to the new outcomes would be possible. This process involved matching the new outcomes to the learning opportunities of the current curriculum.
A significant component of this process was to search for curricular holes. A curricular hole is defined as an inconsistency between learning opportunities and learning outcomes, where no learning opportunity is provided for a given outcome. To assess for curricular holes, a graphical representation of the entire curricular map was created using a bar chart generated by SPSS (SPSS Inc, Chicago, IL). Figure 1 presents the raw count of learning opportunities per outcome. The set of 437 outcomes has been condensed for space. The contours of the figure demonstrate that the NSU curriculum has emphasized curricular domains A (pharmaceutical care) and D (professionalism). Comparatively, domains B (systems management) and C (public health) have been emphasized less in the curriculum.
Analysis for Holes in Curriculum
An example of a curricular hole is represented by outcome statement A2.1.5, defined as design evidence-based disease management programs for populations. In all, 37 holes were found in the NSU curriculum. Because approximately 92% of the existing curriculum was found to match the new educational outcomes, an entirely new curriculum did not need to be created.
(2) Are outcomes being taught in a sequential, progressive manner? The NSU competency-based learning outcomes have been phrased as terminal outcomes; these are tasks, activities, etc, that a student should be able to perform competently and autonomously at graduation (eg, “Assess patient adherence to prescribed medication regimens”). Students achieve outcomes through learning that is developmental and sequential. Outcome achievement typically requires that a student learns fundamental information, then learns how to think critically about that information, and ultimately implement that information in real-life environments.16 An optimized curriculum should allow for such sequential learning. Because sequence gaps or inappropriate sequencing of learning opportunities for an outcome should not exist in the curriculum, the next analysis of curricular mapping data was exploring for the possibility of sequence gaps in the curriculum.
The NSU curricular mapping project designed a classification scheme that included levels of alignment among outcomes and learning opportunities. For most outcomes, the sequence should begin at alignment level 1 (background), then progress to level 2 (applied), and ultimately attain level 3 (practice). For example, if an outcome is taught only at level 3, skipping levels 1 and 2, a sequencing gap occurs. Another example of a sequencing gap is teaching an outcome at level 2 (applied) before level 1 (background). To identify sequence gaps in the overall curriculum, a stacked area graph was generated to represent the progression/development of curricular outcomes (Figure 2). Several sequence gaps (labeled A-G in Figure 2) were identified in the NSU curriculum, as evidenced by a level of alignment in the overall curricular map being skipped (Figure 2).
Analysis for Sequence Gaps in Curriculum
Sequence gaps can be present in areas that appear to be properly sequenced at first glance (eg, Figure 2, outcome area D1.1.5), but further exploration reveals a different pattern (eg, Figure 3, outcomes D1.1.5, D1.1.5.1, and D1.1.5.2). To facilitate the efforts of curriculum planners, stacked bar graphs were developed to expose hidden sequence gaps (Figure 3).
Analysis for “Hidden Sequence Gaps” in Curriculum
Figure 3 represents the learning opportunities provided to students in the curriculum for outcomes dealing with effective communication. Upon reviewing this map, 2 types of sequence gaps were exposed: those created due to a lack of learning opportunities in a given semester or year (eg, gaps A, B, and C in Figure 3), and those that occurred due to a learning opportunity level (eg, gaps D, E, and F in Figure 3) being skipped over consecutive semesters or years. While sequence gaps due to a lack of learning opportunities are easily identified, those due to a skipping of learning opportunity levels are more difficult to pinpoint. Gap D is a good example of a gap created due to a learning opportunity level being skipped. For outcome D1, learning opportunities are first introduced at level 1 (background), followed by level 3 (applied). However, learning opportunities at level 2 (practice) are skipped and never are provided for the outcome in the curriculum.
Figure 3 shows that most of the learning opportunities for this set of outcomes were presented to students in the P2 and P4 years of the curriculum, while learning opportunities were either lacking in variety or numbers in the P1 and P3 years. Once hidden sequence gaps were identified, examining them in more detail to determine their cause was necessary.
Course level maps (Figure 4) were then created to explore more effectively the location or cause of the sequence gaps identified in Figure 3. The maps presented in Figure 4 show how sequenced courses within a college department contributed to the development of the communication-related learning outcomes. These maps can be analyzed individually (course A, B, or C) to pinpoint the role of a given course in creating curricular sequence gaps, or in succession, as presented in Figure 4, to show outcome development among courses. For example, Figure 4 displays that course A provides few learning opportunities to address attaining communication-related outcomes, while course B provides substantially more opportunities, and course C provides opportunities for virtually all communication outcomes. Moreover, learning opportunities provided by these 3 courses are limited to the background and applied levels. The lack of learning opportunities for outcomes D1.1-D1.1.5.2 in course A (Figure 4) contributes to the creation of gap A in Figure 3. Furthermore, the lack of level 2 (applied) learning opportunities for outcomes D1 in course B (Figure 4) may be responsible in part for creating gap D in Figure 3. Analysis and interpretation of mapping data for the communication-related outcomes has led to a better understanding of how the curriculum handles these outcomes. The college has used these evidence-based strategies to guide curricular revision.
Between Course Sequence
(3) What learning opportunities are used in the curriculum? Are these passive or active? Learning opportunities represent a key area of the curriculum map. Identifying the learning opportunities used in a curriculum is important not only for facilitating curricular planners' understanding of how the curriculum is delivered, but also for meeting Standard 11 of the ACPE Accreditation Standards 2007.5 Ideally, a variety of learning opportunities that provide for both passive and active student engagement should be employed, so that different student learning styles are accommodated, and developmental and sequential learning is supported.17 NSU's mapping process collected data on the type, level, and style of learning opportunities (Appendix 1). The learning opportunity levels were visualized using 2-dimensional area graphs (Figures 2 through 4), as described above.
Analysis of the learning opportunity type revealed that 8,810 learning opportunities were used in the NSU didactic and experiential core curriculum, and that these were delivered using 12 types of strategies (Table 2). The learning opportunities most often used were experiential education (31%), followed by lecture and discussion (29%), and lectures (19%). Overall, these 3 types of leaning opportunities accounted for 79% of strategies used in the curriculum.
Type of Learning Opportunities in Nova Southeastern University College of Pharmacy
In addition to understanding the type of learning opportunities, determining the style of learning opportunities used to gauge student engagement is also important. The style or nature of student engagement (ie, passive, active, or hybrid) that the learning opportunities offered in the didactic and introductory pharmacy practice experience (IPPE) curriculum was analyzed. Mapping determined that advanced pharmacy practice experiences (APPEs) were almost entirely based in active learning; therefore, APPEs were excluded from this analysis. Of the 5317 learning opportunities used in the didactic/IPPE curriculum, 34% were passive, or required little student engagement; 25% were active, requiring significant student participation and planned instruction; and 41% were hybrid. Hybrid learning opportunities were considered those that involved student interaction; however, the interaction was not planned by the instructor and was semi-structured. An example of a hybrid learning opportunity is a lecture and discussion session that includes informal discussion. Although some students may interact and fully engage in this type of learning opportunity, because it is not structured specifically to engage all students, others may not participate in the learning opportunity. Therefore, for some students the learning opportunity may be passive rather than active.
(4) What assessment measures are used in the curriculum? Are these formative or summative? Assessment and evaluation are essential components of a curriculum, allowing professors and students to gauge the acquisition of knowledge and skills towards educational outcomes. After all, providing a learning opportunity does not guarantee that students will learn the material. Assessment processes provide data about the effectiveness of learning opportunities.
Because assessment is such a pivotal and significant component of professional training, it naturally goes hand-in-hand with curricular planning. Consequently, the curricular mapping process integrated the collection of different assessment techniques used by faculty members. Yet for the purposes of curricular planning, data about assessment techniques were found presented best in aggregate.
The assessment components of the curriculum map identify the range of assessment techniques used in the curriculum. The expectation was that the college should deploy a variety of assessment strategies which should include both formative and summative measures in assessing student achievement of learning outcomes. As with the learning opportunities, NSU's mapping process linked the assessment measures to the learning outcomes and classified these by type, category, and class (Appendix 1).
In all, 5,317 assessment measures were used throughout the didactic and IPPE curriculum to assess student performance, indicating that professors routinely integrated assessment into their courses. Table 3 provides a breakdown of the 12 types of assessment strategies that were used. Quizzes (53%) were the most commonly used assessment measure, followed by written assignments (10%), and written examinations (8%). Not surprising, quizzes were used most, as this type of assessment generally is easier to develop and administer and can be used to assess diverse learning opportunities in a variety of settings. Interestingly, 10% of assessment opportunities were classified by faculty members as informal (not formally assessed). This describes the case when a professor provides a learning opportunity towards 1 or more of the educational outcomes, yet uses intuition to gauge whether the material was understood by students. For example, a professor may present new material and notice that many students seem confused; a formal assessment is not needed in this case to inform the skilled educator to cover the new material further.
Type of Assessment Measure
Most of the 5,317 assessment strategies utilized were summative (54%) in nature. A large proportion of formative assessment measures may be desirable because they provide the information needed to adjust teaching and learning in progress.
DISCUSSION
Assessment for curricular holes is an important part of the curricular evaluation and development process. Once holes are identified, their causes must be determined, and strategies must be developed to address them. In general, curricular holes, such as one presented in Figure 1, can be caused either by a deficiency in the curriculum to provide the learning opportunity, or the lack of fit of the outcome to the goal of the curriculum. Curricular planners can use at least 3 possible strategies to address curricular holes: teaching opportunities may be created to address attaining the outcome, the outcome may be revised to better fit the goal of the curriculum, or the outcome may be dropped altogether if it is deemed inconsistent with the goal of the curriculum. Through the curricular revision process, the college continues to address how best to eliminate the identified holes.
Curricular mapping using stacked graphs (Figures 2 through 4) has yielded data used to drive interpretations and continuous discussions about curricular sequencing and alignment. In addition to answering basic questions about progression of outcomes in the curriculum and among courses, stacked graphs, as presented in Figure 4, can help address other course sequencing questions such as: Are course prerequisites appropriate? How should the level of learning opportunities be sequenced among courses to meet the educational outcome effectively? For example, for outcomes D1.1 and D1.1.1 (Figure 4) an instructor in course C may assume that students have received learning opportunities in courses A and B at level 1 (background) before they are presented in course C at level 2 (applied). However, the graph shows that this is not the case. In fact, students are provided with learning opportunities for these 2 outcomes for the first time in course C. Adjustment of learning opportunities (ie, changing prerequisites) in courses A and B would be required to ensure that students are prepared for course C. Alternatively, course C could be changed to provide learning opportunities at level 1 (background) before level 2 (applied) learning opportunities are introduced.
Outcomes D1.1.2 and D1.1.3 in courses B and C of Figure 4 provide a good example of the way that learning opportunity levels should build on each other from course to course. Course B, offered in the fall semester of the P2 year, presents learning opportunities at level 1 (background) while course C, offered in the winter semester of the P2 year, presents learning opportunities at level 2 (practice). Learning opportunities in these 2 courses build upon each other progressively without skipping levels. The use of stacked course graphs in this manner aids curricular planners in developing strategies to ensure consistent outcome development among courses in the curriculum.
Figures 2 through 4 also have generated discussion about redundancy or duplication in the curriculum. A cursory glance at the maps might suggest that there are outcomes that are being taught to students repetitively throughout the curriculum. However, multiple teaching efforts towards the same outcome may have significant benefits to the student. The outcome may be taught using different techniques, contexts, perspectives, assessment methods, etc. Further, overlap/redundancy in teaching may reflect the importance of the outcome itself. Notably, input from students regarding this curricular mapping issue was unequivocal: students value having core material presented to them repeatedly throughout their training. However, the issue of curricular overlap/redundancy merits attention for at least 2 reasons. First, a saturation point may exist beyond which further teaching towards the outcome is unnecessary or ineffective. By identifying and removing true cases of redundancy that have no significant educational merit, the curriculum can be streamlined and allow holes and sequence gaps in the curriculum to be filled. Second, conflicting information pertaining to the same outcome also may be presented to students. For example, professors X and Y may teach to the same outcome in their respective courses, yet present students with contradictory information. Similarly, because these professors may not realize they are teaching towards the same outcome, they should collaborate to ensure student success. Maps similar to those used to detect curricular gaps (Figure 3 and 4) can be generated to identify and explore curricular redundancies. Analysis for curricular redundancies would center on identifying repetitive presentation of learning opportunities (count and level) for individual outcomes over the course of the curriculum, rather than searching for omissions or skipping of learning opportunity levels as with sequence gaps.
The form of analysis and interpretation of sequence gaps and redundancies presented has led to a better understanding of how outcomes are addressed in the overall curriculum and among related courses. The outcomes presented here are but one area of the curriculum. The college continues to review all outcomes using this systematic process to identify all significant sequence gaps and possible redundancies. Continuous mapping will be used to ensure that planned curricular revisions are consistent with the delivered curriculum.
The authors did not expect to detect such diversity in the type of learning opportunities that were used throughout the curriculum. The range of learning opportunities was also contrary to the perception of most faculty members who believed that a limited number were in place. Also surprising was the finding that 66% of student learning experiences involved either active or hybrid learning. These findings have generated new discussions among faculty members. A further unexpected byproduct of the mapping process has been the more widespread sharing of active-learning techniques. Curricular maps have also been used by the college to propose that small group activities be offered more often and take on greater emphasis as a learning opportunity.
Typically, assessment techniques are determined by the professor(s) coordinating the course and are not directly under the supervision of a curriculum committee. Our findings have generated discussion among faculty members about the use of assessment techniques across courses, and the potential has been indicated for more detailed mapping analyses to determine how to best deploy assessment measures in the curriculum. The assessment data collected through curricular mapping are being used within the curricular revision process to coordinate the use of assessment strategies in the revised curriculum.
In the final analysis of the NSU curriculum, the current curriculum was determined to be well aligned to the competency-based learning outcomes. The faculty members agreed that revisions to the curriculum were warranted, but it was not deemed necessary to abandon the existing curriculum and create an entirely new one. Areas in need of revision were specific, and therefore the revision process would be direct with clearly recognized goals. The curricular mapping process served as an effective diagnostic and curricular development tool. Further curricular development will focus upon answering other important questions with the help of mapping data. The next step for the NSU mapping program is to analyze assessment data with the same level of specificity as the curricular outcomes, with the hope of refining assessment techniques across the curriculum.
SUMMARY
Curricular mapping is becoming a common method used for curricular evaluation. Curricular mapping demonstrates the relationship among components of the curriculum to facilitate stakeholder understanding of curriculum delivery. However, this process can be difficult if the data collected are not presented in ways that are easily understood. A variety of formats can be used to represent complex curricular mapping data visually to facilitate understanding. Maps can be created to show deficiencies or holes, gaps or misalignments, and unnecessary redundancies in the curriculum. Additionally, mapping can be used to evaluate how learning opportunities and assessment measures are applied. Curricular data depicted in this fashion helps curricular planners better interpret the information gathered and facilitates the curricular development process. To be clear, there is no single “right” or “best” way to visually present mapping data. The authors' hope is that by sharing one college's approach, others may be inspired to build upon and improve the methods presented here.
Appendix 1. Classification Scheme for Nova Southeastern University College of Pharmacy Curricular Mapping Project
- Received July 14, 2009.
- Accepted December 13, 2009.
- © 2010 American Journal of Pharmaceutical Education
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.