Student Learning Outcomes Assessment: A Component of Program Assessment
Heidi M. Anderson, PhD, a
Deborah L. Moore, MS, b
Guadalupe Anaya, PhD, c
and Eleanora Bird, MSa
aCollege of Pharmacy, University of Kentucky bOffice of Institutional Research, Planning & Effectiveness, University of Kentucky cSchool of Education, University of California, Davis
Submitted August 31, 2004; accepted November 15, 2004; published April 20, 2005.
This is the second paper in a series on assessment that focuses on student learning outcomes assessment, a subcomponent of overall program assessment. Included are a brief history of outcome-based education and an overview of outcomes typologies. Definitions of outcomes, competencies, and objectives are presented, and distinctions between these categories are discussed. Various frameworks and models are presented to familiarize the reader with these concepts and the developmental levels of students in health professions educational programs. The paper places student learning outcomes assessment in context and highlights the steps typical of the associated process. Grading and classroom assessment techniques are discussed briefly to clarify their respective places in the assessment landscape. The paper concludes with a brief summary of the relationship of student learning outcomes assessment to institutional assessment.
Assessment efforts in higher education have evolved as a response to demands by external forces such as legislative and accrediting bodies asking for better accountability evidence.1 Further, an executive order required all federally approved accrediting organizations to include evidence of institutional outcomes in their criteria for accreditation.2 This mandate requested that each program describe what its graduates are able to do (outcomes) and provide evidence that they have demonstrated these abilities (assessment). Thus, accrediting bodies and institutions of higher education now focus significant attention on the implementation of outcomes assessment plans within their areas of purview.
Although experts in the area of assessment suggest that assessing student learning is only one of many possible measures of an institution’s outcomes, it is a critical way of demonstrating institutional effectiveness.3 In light of the general public’s growing demand for accountability and the internal shift to a learning-centered instructional paradigm, the focus on student learning outcomes assessment has become something of an “industry standard” for higher education. Further, throughout this assessment movement, definitions, interpretations, and viewpoints have varied and reflect the disciplinary or work contexts of the professionals involved in supporting the assessment endeavors. New terminology has emerged: student learning outcomes, outcomes assessment, competency, and student learning outcomes assessment. This new terminology has caused many in the field to pose a number of questions: What are outcomes? What is outcomes assessment? How do outcomes differ from objectives and competencies? What is outcomes assessment’s relationship to institutional assessment? Does this differ from student learning outcomes assessment?
The first paper in this series presented a primer about the emergence of assessment in higher education and detailed the various levels of higher education assessment: institutional, program, and individual. This paper, the second in the series, will focus on a subcomponent of overall program assessment: student learning outcomes (SLO) assessment. Readers may use the following as an outline for this article:
Define outcomes and describe the origin of outcome-based education;
Discuss the relationship of outcome-based education to health education;
Clarify student learning outcomes terminology;
Explain the differences between outcomes, competencies and objectives;
Describe taxonomies that can be used to create student learning outcome statements and learning objectives;
Clarify the purpose of student learning outcomes assessment;
Articulate the steps in student learning outcomes assessment;
Explain the relationship to institutional assessment;
Explain differences between grading and assessment; and
Define classroom assessment and discuss its use and role.
Definition of Outcomes and Outcome-Based Education
Outcomes are the consequences or results associated with instructional experiences; the end results of institutional, program, or curricular goals. Assessment specialists point out that language used by instructors to convey desired abilities and performances will also vary with respect to intended and actual outcomes; intended outcomes are those desired and planned, whereas actual outcomes are those that have been achieved. In sum, a learning outcome can be useful at various levels: the course, the college, the program, or the institution. Davis summarizes these concepts: “An outcome is a culminating demonstration of learning; it is what the student should be able to do at the end of a course. Outcome-based education is an approach to education in which decisions about the curriculum are driven by the exit learning outcomes that the students should display at the end of the course.”4
Harden and colleagues further assert:
In outcome-based education, product defines process.5Outcome-based education can be summed up as “results-oriented thinking” and is the opposite of “input-based education” where the emphasis is on the educational process and where we are happy to accept whatever is the result. In outcome-based education, the outcomes agreed [upon] for the curriculum guide what is taught and what is assessed.5
Thus, the emphasis of outcome-based education is not on acceptance of any “results” but on efforts to produce the specific results based on predetermined agreement realized through consensus.
Once outcomes are articulated, an institution or a program writes statements that reflect the student learning outcomes they plan to measure. These student learning outcome statements guide the teaching, learning, and assessment processes and will reflect what the program expects of their students when completing the program. Various characteristics of student learning outcome statements by Maki are described in Table 1.6
Brief History of Outcome-based Education
In the United States, Spady is credited with developing much of the outcome-based education movement of the 1980s.4,5,7 Spady stated that “outcome-based education (OBE) means clearly focusing and organizing everything in the educational system around what is essential for all students to be able to do successfully at the end of their learning experiences. This means starting with a clear picture of what is important for students to be able to do, then organizing the curriculum, instruction, and assessment to make sure this learning ultimately happens.”8 Spady suggests “in OBE, exit outcomes are a critical factor in designing the curriculum: you develop the curriculum from the outcomes you want students to demonstrate, rather than writing objectives for the curriculum you already have.”9 Spady defined outcomes as “the actual results of learning that students visibly demonstrate. They involve the integration and application of content, competence, and confidence in actual performance settings when or after formal instructional experiences are over.”8
In 1985, the General Professional Education of Physician (GPEP) Report, published by the Association of American Medical Colleges (AAMC) about medical education, challenged medical schools to ensure each student had the knowledge, skills, values, and attitudes that all physicians should have (GPEP).10 This report prompted the United Kingdom General Medical Council to examine medical education. In 1993, their report, Tomorrow’s Doctors, suggested much needed reform of the United Kingdom undergraduate medical curriculum.11 Although updated in 2001, the United Kingdom report still relies on identification of knowledge, skills, attitudes, and behaviors of new medical graduates as a result of an outcome-based curriculum.12 These reports were the impetus that led to the beginning of outcome-based education in medical schools.
Types of Student Learning Outcomes
In detailing steps for planning assessment, Ewell explains that there are many different types of student learning outcomes.13 He also notes that various types of outcomes can be classified into several typologies: cognitive, affective, psychological, behavioral, within-college outcomes, and after-college outcomes. Ewell states that:
The distinction between cognitive and affective outcomes is a distinction between gains in knowledge and changes in attitudes or values. The distinction between psychological and behavioral outcomes is a distinction between changes occurring inside a student’s head and changes that can be observed directly during or after college. Finally, the distinction between within-college and after-college outcomes is a distinction about when the outcome occurs.13
Ewell further asserts that all of the typologies have a number of features in common that help researchers make choices about what to investigate and why. Since an institution’s assessment program is linked to its mission, the outcomes selected for study should also be driven by this mission. In addition, these various types of outcomes may be combined in a number of ways to respond to the institution’s hypotheses about its programs and institutional effectiveness.
This discussion highlights the necessity for each institution, college, school, or program to determine the specific student learning outcomes that they desire to measure within their own programs. For example, in the United States, the Association of American Medical Colleges (AAMC) developed a set of broad learning outcomes to guide medical schools in establishing their own training programs.14 These broad outcomes cover cognitive, affective, and behavioral aspects that represent what a medical student should possess at the time of graduation. They set forth guidelines that each medical school can use to establish its own program outcomes and objectives.
In discussions about developing specific learning outcomes, often this question is asked: How many student learning outcomes (SLOs) should a program have? Assessment experts have responded by recommending that programs adopt only a few educational outcomes to focus on in a period of time.13,15,16 For example, as a guideline, Nichols recommends “between three and five statements of intended educational outcomes [SLO] be identified for each academic program in the department, though there is nothing magical about these numbers.”16 Although he makes this recommendation in the context of annual reporting, Nichols urges that these statements represent the important outcomes or global outcomes that span several courses and that they be used with course-specific objectives.
Why focus on only a few outcomes at a time? The answer to this question has to do partly with a 1-year timeframe. For each learning outcome a number of more detailed objectives are needed to support it. Further, there should be at least one way to determine (assess) whether the objective is accomplished. Thus, having many outcomes requires a large and expansive assessment system so burdensome that professionals might desire to abandon the process altogether. Colleges and schools tend to be more successful in assessment when they focus their efforts on a handful of priorities in a single year. This is not to say that overall programmatic outcomes are never revisited. In fact, a comprehensive plan would involve selecting a limited number of specific priorities from the broader array of outcomes for emphasis each year. In addition, the college or school would revisit some complex or difficult areas several times within a multi-year period.
While this section addresses student learning outcomes, a college or school of pharmacy will also have other program outcomes that are of interest. Although a school or college may concentrate on assessing a few outcomes at a time, it is efficient to define the broad range of outcomes for the entire program. This will alleviate areas of duplication but more importantly assure that all the outcomes are aligned with the college’s or school’s overall program missions. For example, at one school the emphasis on pharmaceutical care was so narrowly defined that it led to a neglect of practice management outcomes; this negatively influenced hiring and had undesirable long-term curricular consequences.
One of the earlier health professions schools to adopt the approach of focusing on only a few outcomes at a time or on broad outcomes was Brown University School of Medicine, which defined 9 abilities for students completing its competency-based curriculum (Table 2).17 Also, AACP initiated various mechanisms through the Center for the Advancement on Pharmaceutical Education (CAPE) to help colleges and schools of pharmacy transform their curricula.18 The CAPE Educational Outcomes are a product of this initiative. These outcomes include several professional practice-based as well as general ability outcomes. Recently, these CAPE Educational Outcomes were condensed to 3 global domains with accompanying outcomes and competencies.
Other colleges and schools have adopted general outcomes such as these: communication, problem-solving, and lifelong learning or professional outcomes (eg, patient care, management, and medication preparation). A program could adopt a limited set of global outcomes and have a more detailed set of outcome statements that undergrid the global outcomes. The new CAPE outcomes reflect this principle; for example, within the domain of pharmaceutical care, broad outcomes include both a patient-centered and a population-based focus to pharmaceutical care. These new CAPE outcomes emphasize assessment at the program or curricular levels, while the former CAPE outcomes illustrated specific course-level outcomes as well. Table 3 uses one of the CAPE outcomes, pharmaceutical care, to provide a schematic for considering this framework. The table shows the overall global domain of pharmaceutical care, suboutcomes, and the integrated competencies. Thus, depending upon its assessment plan and goals, a program can focus on a comprehensive assessment of only a few outcomes at a time in the beginning or to meet specific needs. Basically, the program should ensure that all critical goals are addressed in some systematic way over a multiyear span of time. This does not mean to monitor everything annually. The program can prioritize and expand on those areas that are critical/essential and then revisit them periodically. Also the goals should change to address the results of and recommendations from accreditation, program, strategic planning, or other reviews.
As discussed previously, outcomes are the achieved results or the actual consequences of what students demonstrate or accomplish in a course or program. Competency, another term in assessment and medical literature, also reflects a shift in focus from learning to actual performance. The idea of competencies evolved within the context of workforce development. The general strategy is to recognize industry leaders or bodies affiliated with specific occupational areas and have these groups identify national standards of performance for professionals in their fields. In the broadest sense, competency means being able to function in a work context. In contrast to objectives, which reflect the educational context of teaching and learning, competencies are what individuals should be able to perform independently as a professional in the workplace. Further, the educational modes should represent authentic contexts that reflect practice. Although competencies reflect development of occupational skills, the critical outcome is the transferability of these skills into new work contexts.
In addition to articulating the competencies or outcomes of the professional, some models within health professions education delineate the developmental sequences leading to the competencies or outcomes. Competency-based education assumes that learning to become a professional is a progression through stages. For example, Chambers and Gerrow stress that “because novices, beginners and competent individuals are learning in different ways, the best educational experience for each of them will differ.”19 Further, since the students are learning in different ways the assessment methods should be matched with the particular teaching method; hence, competency-based evaluation. Thus, the educational process requires different teaching, learning, and assessment formats appropriate for students at each developmental level, several of which are suggested by Chambers and Glassman and described in Table 4.20
Miller has proposed another description of competency that includes developmental milestones at various levels in health professions education.21 In his framework he defines student competence from a basic foundation level to a more advanced level and uses a pyramid to depict this framework of 4 levels (knows, knows how, shows how, and does). See Miller and Winslade for additional details about this framework.21,22
In their model, Yip and Smales are explicit in labeling the developmental stages of health care professions students.23 The authors emphasize that “competency is not achieved immediately, but is gained in stages” (Table 5). They assert, “competency-based education is based upon the early identification, at the course-planning stage, of clearly specified outcomes of learning. Statements of competence and attainment define what students are expected to learn.”23 In health care education, students advance from the novice to the competent stage. The length of educational and instructional time spent in each stage varies. For example, depending on the content, the novice stage may last only a week, so lecturing longer than that would not be as educationally helpful as another approach. Thus, to document progression of student learning throughout these stages, competency-based assessments should be designed as a series of assessments conducted at appropriate points in the curriculum rather than only at the end of the program.
Hager and Gonczi further suggest that using an integrated approach to defining competency within the professions broadens the concept and places the assessment in a contextual framework.24 They state that the language of competency denotes a relationship between knowledge, skills, and abilities. Yet, competencies reflect broader integration of various levels that are exhibited by a practitioner. Our expectations of practitioner performance vary depending upon the individual’s experience. “A feature of this integrated approach is that it avoids the problem of a myriad of tasks by selecting key tasks or elements that are central to the practice of a profession. The main attributes that are required for the competent performance of these key tasks or elements are then identified.”24 Through experience the practitioner achieves greater integration of knowledge and skills and can apply or adapt them as needed according to the situation or the environmental context. To achieve this approach, adopting a reduced set of global outcomes that focus on integration of abilities rather than adopting a large number of outcomes is logical.
For example, problem solving, analysis, and communication are attributes necessary to practice pharmacy; however, these characteristics are highly context dependent. Consequently, to teach or assess these attributes out of context is ineffective. Thus, using an integrated approach allows more effective learning and provides flexibility in the assessment process. In pharmacy, the recently revised CAPE outcomes represent this trend toward integrated competencies.18 Recall Table 3 displays the pharmaceutical domains and organizes them into suboutcomes and indicates the integration of knowledge, skills, attitudes that are needed to practice as a competent pharmacist.
Distinction Between Outcomes and Objectives
The terms instructional objectives and learning outcomes are often used interchangeably, and both are concerned with educational intent and the product of educational processes. One of the main differences between a learning outcome and an objective is that a learning outcome describes how the student will integrate knowledge, skills, and abilities in the performance of a complex role. The language of learning outcomes reflects the more independent and self-directing role expected of graduates in their professional lives after graduation. In general, outcomes tend to be broader in scope than objectives. Outcomes focus on what the learner should get out of the program or be able to do as a result of the instructional activities within the program.
The term “objective” is most frequently used to describe discrete skills or bits of knowledge that are specific to a given instructional context. Objectives tend to focus on the instructor’s goals in the course, module, etc, rather than his or her goals for the end of a sequence of courses. Intended learning objectives may be classified as cognitive, affective, developmental, or skill or performance related. Cognitive learning objectives are focused on knowledge of the subject matter and skills. For example, a knowledge of the basic chemical principles of medicinal products represents a cognitive learning objective. Developmental objectives may include cognitive and affective aspects, such as critical thinking, problem solving, professionalism, and ethics. These objectives may be assessed through the educational experiences, student developmental factors (ie, attitudes, values, or beliefs) or through alumni (ie, workplace preparation, job satisfaction, performance, etc).
Paradigm Shift in Outcomes and Objectives
In another paradigm shift, the language of outcomes and objectives has shifted from a focus on instructional delivery to a focus on student learning and performance. In this shift, the phrase “student-centered” is used to describe “instruction” and “assessment planning.”
Harden explains that in medical education “attention has moved from an emphasis on the education process to a consideration of the product and the expected learning outcomes of the students’ studies.”25 Thus, Harden argues that although the terms outcomes and objectives are often used interchangeably, there are 5 distinctions between instructional objectives and learning outcomes that have implications for the curriculum developer, the teacher, and the student:
the detail of specification;
the level of specification where the emphasis is placed;
the classification adopted and interrelationships;
These 5 key distinctions are explained in more detail in Table 6. In sum, the trend in health care outcome-based education has shifted toward a focus on assessing outcomes or competencies, and health education programs have adopted the new assessment terminologies, taxonomies, and frameworks.
Finally, throughout the educational literature there are numerous debates about the differences between objectives/outcomes and competencies, various frameworks and models. In fact, Spady included a list of terms (Table 7) that people confuse with outcomes. Spady asserts that none of these terms either singly or in combination conforms to the definition of outcomes (as defined previously in this paper).8 Assessment experts suggest that each institution should agree on language and models that fit their needs. After those decisions and choices are made, the next step will be to familiarize faculty members and students with the adopted language or model. Palomba and Banta assert, “the exact language faculty use is not important. It is important that faculty reach agreement about what graduates of their programs are expected to know and be able to do and express these intended results with enough precision to guide the selection of assessment instruments.”26 The purpose of this previous section on assessment terminology, concepts, frameworks, and models has been to introduce the reader to the various areas and, further, to recommend that as educational programs in the health professions embrace assessment, they develop a common language and framework tailored to their specific needs and programs.
Taxonomies to Create Outcomes and Objectives
In a recent text, Suskie reminds faculty members that taxonomies can be helpful in developing outcome statements or learning objectives.15 These classification schemes may be used to help structure and organize outcomes/objectives as cognitive, behavioral, or affective. They assist faculty members in thinking broadly and comprehensively and avoiding omissions or gaps in their efforts. The next section describes various taxonomies that a college or school of pharmacy may want to use in developing or revising its student learning outcomes and objectives.
Perhaps best known and most frequently used is the framework developed by Bloom and his associates.27 Bloom’s work addresses 3 different learning domains; however, use of the cognitive domain is much more widespread than that of the affective and psychomotor domains. Rooted in higher education for over 40 years, Bloom’s taxonomy has been helpful in ensuring breadth of instruction and assessment. More recent taxonomies include those articulated by Anderson,28 Marzano,29 and Biggs and Collis.30 Anderson revised Bloom’s taxonomy by adding a second dimension to each of Bloom’s 6 cognitive domain categories.28 He crossed each of the types of knowledge categories with 4 types of cognitive processing levels (declarative, conceptual, procedural, and metacognitive) into a new formula matrix consisting of 24 cells. This model recognizes contextual factors that impact learning and performance.
In 2001, Marzano expanded Bloom’s framework to 6 levels of mental processing: (1) retrieval, (2) comprehensive, (3) analysis, (4) knowledge utilization, (5) metacognition, and (6) self-system thinking.29 This expanded model also has 8 categories of skills: focusing, information gathering, remembering, organizing, analyzing, generating, integrating, and evaluating. Marzano developed this model to build on Bloom’s work, adapting it to reflect current thinking of cognitive psychology, education, and philosophy.
The taxonomy of Biggs and Collis, known by its acronym SOLO, which stands for Structure of the Observed Learning Outcome, is perhaps more familiar outside of the United States.30 The SOLO taxonomy, describes how learner performance and understanding of a subject grow in complexity. This taxonomy has 5 levels: (1) prestructural level where the student is acquiring information but makes no sense of it; (2) unistructural level where the student begins to make obvious connections; (3) multistructural level where the student begins to make a number of connections but misses the meta-connections between them; (4) relational level where the student is now able to appreciate the significance of the parts to the whole; and (5) extended abstract level where the student is not only making connections in the subject area, but also transferring those/that knowledge to other principles and concepts. In developing this model, Biggs and Collis recognized that learners come from a diverse set of experiences and interact with a wide range of tasks, yet typically learn through a similar structural sequence.30
Finally, Suskie urges, “Although these and other inventories are helpful in developing outcomes statements, selecting from them is not an arbitrary process.”15 The focus of an institution’s outcome statements must be driven by its mission. For example, some programs may build on the cognitive domain and use Bloom’s taxonomy, while others may elect a different philosophy, such as a developmental model for which one of these other models would be more easily used.
Other Intended Outcomes
Although the previous section discussed student learning outcomes, a college or school of pharmacy may have concerns about other outcomes that also indicate the success of its students. The National Postsecondary Education Cooperative (NPEC) developed a taxonomy that included educational success as one of the components.31Educational success refers to retention/persistence, educational attainment, time required to attain the degree, time remaining until the degree is awarded, and satisfaction. The taxonomy also includes job placement, licensure, job satisfaction, initiative, and leadership. In addition, programs may want to conduct assessments about promotion rates, employer satisfaction, and performance after graduation. Whatever outcomes are selected should be driven by the college’s or school’s mission statement and overall program goals.
Ewell states “outcomes assessments can range in content from cognitive development in college to career achievement after graduation and in technical complexity from simple surveys of currently enrolled students to carefully designed longitudinal assessments employing sophisticated psychological measurements. Thus, in order to make an informed choice, the researcher must be clear about which kinds of outcomes to look at and why.”13 An institution must be clear about the purpose of its assessment endeavors. In addition, pharmacy colleges and departments should clarify the terms they plan to use in the process and agree upon the goals and research questions that will be answered by their assessment data. A critical feature of the process includes determining how the assessment data will be used.
Pace asserts that “almost everyone will agree on the importance of two student outcomes: the acquisition of knowledge and the development of intellectual skills. What do students learn? How well do they think?”32 Thus, in order for assessment to make sense in pharmacy education, the designers of the assessments should be quite explicit about the type of outcomes under consideration. Although faculty members will continue to debate cognitive outcomes, articulating outcomes associated with the values of the professional are sometimes more challenging. Issues of ethical conduct, for example, are often absent in the lists of competencies or expected outcomes.
Pace urges us to:
…take the point of view that the purpose of outcomes assessment is learning. What do you want to learn? How will you learn it? And, what will you do after you have learned it? …Design a program of outcomes assessment as a continuing, ongoing enterprise. This does not mean that every aspect of every program should be evaluated all the time…It is probably more feasible to consider an evaluation cycle, perhaps lasting five years, that corresponds to student [cohorts], or to major changes in college programs.32
Thus, a program must define its research questions about learning and link the assessment process to these questions. Further, the process should be continuous and revised periodically.
Student outcomes assessment includes several steps: (1) create learning outcome statements; (2) design teaching/assessments to achieve these outcome statements; (3) implement teaching/assessment activities; (4) analyze data on individual and aggregate levels; (5) reassess the process.
Spady recommended using a “design down” approach to create outcomes.8 Design down means to begin curriculum and instructional planning by indicating where you want students to ultimately end up and building back from there. Using this approach faculty members first identify and adopt the exit outcomes for the curriculum. The outcomes for each year in the curriculum are derived from these exit outcomes. This process is then repeated for individual courses and ultimately the various instructional activities within each course. The outcomes for the levels, courses, and learning activities should be aligned with and contribute to the exit outcomes. The entire process is done in a carefully structured manner. A caveat in the process is that faculty members must be willing to replace or delete those outcomes that do not significantly contribute to the overall exit outcome. This design down process compels the college or school to determine what is truly essential for students to accomplish in the limited amount of time given. Spady maintains that using this process is preferable to “having teachers and students cover more and more material at an increasingly superficial level, with no assurance that the culminating” outcome is the result.8
Stated another way, in writing outcome statements, first think of what you expect the student to be able to do after your material (ie, lecture, reading assignment, etc) has been presented. Then answer the question, “What should they be able to do as a result of working with or through the material (ie, hearing the lecture, reading the article, etc)?” A common approach is to complete this sentence: “At the end of this course/module/program, students will be able to (fill in the blank).” Note, the significant word is do, and the key in drafting learning outcomes is to use active verbs that convey observable behavior associated with performance competencies. Outcome statements should use action verbs to describe student learning or behavior. Avoid selecting words that are “fuzzy” (unclear or open to interpretation). Mager offers suggestions on how to recognize and correct these types of abstract words in outcome statements.33 Finally the outcome statements should be realistic, achievable within the time available, and written clearly without jargon. The following are examples of vague and effective outcomes statements.
Too vague: Demonstrate drug information skills. More effective: Locate drug information and critically evaluate its validity and appropriateness. Too vague: Explain chemical properties of drugs. More effective: Explain chemical properties of the newer generation of non-steroidal anti-inflammatory drugs (NSAIDs) (ie, cox-2 inhibitors) introduced after the traditional NSAIDs. Too vague: Test a chemical hypothesis or theory. More effective: Design an experiment to test a chemical hypothesis or theory.
Step 2: Design Teaching/Assessments to Achieve Outcomes
Next, design or modify teaching strategies that align with learning outcomes. The relationship between instruction and student outcomes assessment is further depicted by Gronlund in Table 8.34 Gronlund suggests that we “expand our concern to a teaching-learning-assessment process, with assessment as a basic part of the instructional program. As with all instructional activities, the main function of assessment is to improve learning and it can contribute to this end in a number of ways.”34 An orientation toward assessment contributes to student motivation by providing short-term goals, clarifying learning tasks, and providing feedback.35 As described in an earlier section, assessment can also facilitate retention and transfer of learning through a focus on more complex learning outcomes. Assessment results also help faculty members identify elements of instruction that are more productive with respect to desired learning gains.
To get underway, a first task is to select or develop assessment methods that correspond to the questions being asked about the learning performance. One can select appropriate assessments methods that provide various types of data: formative/summative, direct/indirect, or quantitative/qualitative data. These terms are also described by Boyce on the AACP Web site: http://www.aacp.org/Docs/MainNavigation/Resources/5416_pharmacyprogramassessment_forweb.pdf.36,37
The next article in this series will focus on specific assessment methods.
When selecting or designing the assessment, several criteria about the methods must be taken into consideration: validity, reliability, educational impact, feasibility, and acceptability of the assessment method. These criteria are also discussed in great detail in a previous article.22
Step 3: Implement Teaching/Assessment Activities
Step 3 requires implementing the teaching and assessment methods designed in Step 2. Student outcomes involve interrelated dimensions of student development in several domains: cognitive, affective, behavioral, and psychological. Thus, multiple assessment methods should be planned over the duration of a program. It might be helpful to consider those elements necessary to shape learning for a single student and then broaden the plan to encompass considerations for all learners within a pharmacy college or department. For example, an organization begins at micro-level outcome indicators and progresses to macro-level outcome indicators. These are actually poles along a continuum, anchored on one end by assessment of a student(s) and at the other by assessment of an institution(s).
Micro level. Assessment at the micro level focuses on the needs of the individual student. Professors conduct assessment at this level when they grade student assignments and administer classroom quizzes and examinations. Data are recorded and compiled for each student and reports of individual student performance are submitted.
Macro level. Assessment at this level shifts attention to the performance and successes of educational programs and results are typically aggregated across groups of learners. Assessment projects generate evidence that programs are adhering to standards and program goals, as well as to the institutional mission. These data can paint a profile of program successes and outline areas in need of attention or improvement. Previous publications provide an expansive discussion of assessment formats: objective structure clinical examinations, progress testing, surveys, focus groups, interviews, portfolios, and embedded assessments.22,36,37
Step 4: Data Analysis and Report Preparation
After implementing the assessment strategies, the next step is to analyze the data and prepare the individual and aggregate reports. Assessment data analysis depends on the research questions that were asked at the beginning. Assessments may be collected to answer various questions that describe student learning, or explore, predict, or explain learning and teaching. For example, it may be important to determine general concerns such as which teaching methods facilitate student learning and which hinder student learning, or more specific concerns such as how can student problem-solving be enhanced and whether critical thinking can be measured longitudinally.
When preparing associated reports on assessment, consider the needs and interests of the intended audience. Keep the report simple and use visuals (diagrams, tables, graphs, figures, etc) as needed to clarify and explain data. Be sure to disseminate the report and ask for feedback. Finally, indicate how the data will be used (ie, to improve instruction, aid decision-making, etc). The third article in this series will further describe a data collection method using a data audit and analysis format for various kinds of reports.
Step 5: Reassess the Process
No assessment process is complete until it is re-evaluated. The completion of one cycle will shape and signal the launch of a subsequent cycle. It is this type of continuity between assessment and teaching activities that maximizes their value. At the end of the cycle, determine next steps to build on strengths or ameliorate weaknesses through a reflection on the cycle as a whole.
Student outcomes assessment relates to institutional assessment as one part of measuring the overall program effectiveness. For example, program effectiveness may be for institutional operation and academic success, securing state funding, in meeting accreditation requirements, and in meeting the needs of employers. External agencies constantly request that colleges and universities assess institutional effectiveness and student learning. Institutional assessment focuses on how well the institution is achieving its missions and goals. Because student learning is a significant element of the institution’s mission, the assessment of student learning should also be a significant element in the overall institutional assessment plan. Outcomes assessment can also be completed after students graduate from the program. Results from these assessments can offer suggestions for program improvement and effectiveness.38 However, institutional assessment will also include focusing attention on other mission areas such as research, scholarship, and operations/services. To determine overall institutional effectiveness a full range of assessment areas are required.
In conclusion, faculty members may argue that they already engage in assessment by assigning student grades and requesting advice from students about their teaching techniques. They are correct; grading and classroom feedback are other types of assessments that are focused more specifically at the course level. There are a few unique differences between grading, classroom assessment techniques, and overall assessment of student learning for the program.
Grading Versus Assessment
Although grading is an innate process in the classroom, grades alone are not sufficient evidence of student learning for meaningful assessment of a course’s effectiveness. Suskie suggests that faculty members “imagine applying for tenure or promotion and providing your students’ grades as sole evidence of your teaching effectiveness and what your students have learned. Would your tenure and promotion committee accept grades as sufficient evidence?”15 Grades focus on individual students and are generated based on student accomplishments in a specific course. Thus, examinations and assignments may measure levels of mastery in course-specific content rather than on broad learning outcomes. Also, grades may not reflect all of the learning experiences. Often faculty members use various criteria to establish grades that may or may not involve only learning. For example, some faculty members give credit for attendance or class participation as part of the grade as an attribute of professionalism. Although participation may indicate professionalism, its scope is broader than student learning. In another example, simply determining that the average student GPA for a course is 3.2 does not tell the faculty member which learning objectives the students mastered and which require further instruction. Finally, course grades may reflect short-term learning of information, which tends to be forgotten after the examination is completed.
For these reasons experts argue that grades alone are not sufficient evidence of student learning. Effective assessment of learning requires taking/conducting more measures than course grades. The emphasis of assessment of learning is to focus on broad outcomes and objectives that cut across courses in the curriculum and place importance on integration of learning and long-term retention.
Classroom Assessment Techniques
Classroom Assessment Techniques (CATs) developed by Angelo and Cross are periodic assessment activities used to determine how well students are learning in the classroom.39 These techniques allow faculty members an opportunity to monitor student learning prior to assessments associated with grading (ie, quizzes or examinations) and adjust their teaching strategies appropriately to enhance learning. The techniques are simple, non-graded, anonymous, in-class tools that provide feedback on the teaching-learning process. The goal of these techniques is to offer useful information for faculty members to improve their teaching.
Angelo and Cross suggest that in many professional areas, students success depends on their ability to translate specialized information into language their patients will understand. The Direct Paraphrasing CAT is an assessment technique designed to measure student understanding of concepts and theories. “In this CAT, students are directed to paraphrase part of a lesson for a specific audience and purpose, using their own words.”39 A faculty member could use this CAT as a complement to the examination to determine whether students have the ability to explain pharmaceutical care and the pharmacist’s role to various audiences such as patients, physicians, or other health professionals. Results could help determine whether additional classroom time is needed to develop or reinforce the concept.
Another example of a CAT that is often used by pharmacy faculty members is the “Muddiest Point CAT.” Angelo and Cross state that this is the simplest and most efficient Classroom Assessment Technique. The purpose of this CAT is asking students to write down a quick response to one question: “What is the muddiest point in (blank)?” The focus of the Muddiest Point assessment might be a lecture, a discussion, a homework assignment, etc. This CAT “provides information on what students find least clear or most confusing about a particular lesson or topic. Faculty use that feedback to discover which points are most difficult for students to learn and to guide their teaching decisions about which topics to emphasize and how much time to spend on each.”39 Angelo and Cross assert that although the Muddiest Point CAT is the simplest to use, responding to it requires some higher-order thinking. Rouseff-Baker and Holm state, “As faculty experience immediate classroom improvements that result from the use of CATs, they become increasingly convinced that this is indeed a worthwhile process. ….CATs can be invaluable tools that help make important links between classroom, program, department and institution-wide assessments, engaging both students and faculty along the way.”40 Classroom assessment techniques are unique tools that can give faculty members quick feedback on how effectively the teaching/learning process is going in their classes. But CATs can be adapted for program assessment as well, by aggregating the results across the curriculum. Thus, by focusing on an individual faculty member’s classroom, CATs can help faculty members become engaged and involved in the larger assessment process.
This paper focuses on student learning outcomes assessment, a subcomponent of overall program assessment. It was written to address the interest of faculty members, administrators, legislators, and employers in enhancing student learning and development. The review of the development of Outcome-based Education (OBE) underscores that well thought-out educational activities are guided by the attainment of clearly articulated learning outcomes. Definitions of outcomes and competencies are presented with outcome-based education (OBE) as the point of reference. Several definitions of competencies are found in the literature and generally refer to specific abilities or skills. However, the use of global statements of learning is recommended in learning outcomes assessment. This is particularly important in accountability contexts; outcomes statements are much more manageable (in number and content) global statements of what students are to learn.
The authors wish to express their sincere gratitude to Stephanie Aken for her library assistance in completing a search of the literature indexes for this paper. The authors also wish to thank Belinda Morgan for her editorial assistance on this manuscript.
1.
Anderson HM, Anaya G, Bird E, and Moore DL.A review of educational assessment. Am J Phar Educ. 2005;69: Article 12.
2.
US Department of Education. Secretary’s Procedures and Criteria for Recognition of Accrediting Agencies. Federal Register. Washington, DC. 1988:53: 127, 25088-99.
4.
Davis MH. Outcome-based education. J Veterinarian Med Educ. 2003; 30:258-63.
5.
Harden RM, Crosby JR and Davis MH. AMEE guide no. 14: outcome-based education: part 1- an introduction to outcome-based education. Med Teach. 1999; 21:7-14.
6.
Maki PL. Assessing for Learning: Building a Sustainable Commitment Across the Institution. Sterling, Va: Stylus Publishing; 2004: 60.
7.
Glatthorn AA. Outcome-based education: reform and the curriculum process. J Curr Super. 1993; 8:354-63.
8.
Spady WG. Outcome-Based Education: Critical Issues and Answers. American Association of School Administrators. Arlington, Va; 1994:1;8;50;76.
9.
Spady WG. Organizing for results: the basis of authentic restructuring and reform. Educ Leadership. 1988; October:4-8.
10.
Physicians for the twenty-first century: the GPEP report. Association of American Medical Colleges. Washington DC; 1984.
11.
General Medical Council. Tomorrow’s Doctors: Recommendations on Undergraduate Medical Education. London, General Medical Council. 1993.
12.
General Medical Council. Draft Recommendations on Undergraduate Medical Education. London: General Medical Council; 2001. Available at: http://www.gmc-uk.org/med_ed/tomdoc.htm. Accessed August 16, 2004.
13.
Ewell PT. Assessing Educational Outcomes. New Directions for Institutional Research, 47. San Francisco, Calif: Jossey-Bass; 1985: 1-5.
14.
Association of American Medical Colleges. Report 1: Learning Objectives for Medical Student Education. Guidelines for Medical Schools. Medical School Objectives Project. Washington, AAMC; 1998.
15.
Suskie L. Assessing student learning: a common sense guide. Bolton, Mass: Anker Publishing Company, Inc; 2004:14.
16.
Nichols JO, Nichols KW. The Departmental Guide and Record Book for Student Outcomes Assessment and Institutional Effectiveness. 3rd ed. New York: Agathon Press; 2000: 19.
17.
Smith SR, Dollase R. AMEE guide no 14: Outcome-based education: part 2—planning, implementing and evaluating a competency-based curriculum. Med Teacher. 1999; 21:15-22.
19.
Chambers DW, Gerrow JD. Manual for developing and formatting competency statements. J Dent Educ. 1994; 58:351-6.
20.
Chambers DW and Glassman P. A primer on competency-based education. J Den Edu. 1997; 61:651-66.
21.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990; S63-7.
22.
Winslade N. A system to assess the achievement of doctor of pharmacy students. Am J Pharm Educ. 2001; 65: 363-92.
23.
Yip H-K, Smales RJ. Review of competency-based education in dentistry. Br Dent J. 189:324-7.
24.
Hager P, Gonczi A. General issues about assessment of competence. Assess Eval Higher Educ. 1994;19:3-17
25.
Harden RM. Learning outcomes and instructional objectives: is there a difference? Med Teacher. 2002; 24: 151-5.
26.
Palomba CA, Banta TW. Assessment Essentials: Planning, Implementing and Improving Assessment in Higher Education. San Francisco, CA: Jossey-Bass Publishers; 1999:4.
27.
Bloom BS Engelhart MD Furst EJ, Hill Wh & Krathwohl D Taxonomy of educational objectives: The cognitive domain. New York: McKay, 1956.
28.
Anderson LW. A Taxonomy for Learning, Teaching, and Assessing. New York: Addison Wesley Longman; 2001.
29.
Marzano RJ. Designing a new taxonomy of educational objectives. Corwin Press, Inc. 2001.
30.
Biggs JB and Collins KF. Evaluating the quality of learning: the SOLO taxonomy Structure of the observed learning outcome). New York: Academic Press. 1982.
31.
US Department of Education, National Center for Education Statistics, 1997, Student Outcomes Information for Policy-Making, NCES 97-991, prepared by Pat Terenzini for the Council of the National Postsecondary Education Cooperative Working Group on Student Outcomes from a Policy Perspective. Washington, DC: US Department of Education.
32.
Pace CR. Perspectives and problems in student outcome research. In Ewell PT. Assessing Educational Outcomes. New Directions for Institutional Research, 47. San Francisco, CA: Jossey-Bass; 1985: 1-5.
33.
Mager RF. Goal Analysis. 3rd ed. The Center for Effective Performance Inc.; 1997:23-41.
34.
Gronlund NE. Assessment of student achievement. 7th ed. Boston, Mass: Allyn and Bacon; 2003:3.
35.
Huba ME, Freed JE. Learner-Centered Assessment on College Campus. Shifting the Focus from Teaching to Learning. Boston, Mass: Allyn and Bacon; 2000:8.
38.
Meyer SM. Student assessment: a component of providing quality education. Am J Pharm Educ. 1994; 58: 106-107.
39.
Angelo TA, Cross KP. Classroom Assessment Techniques. 2nd ed. San Francisco, Calif: Jossey-Bass Publishers; 1993: 232,154.
40.
Rouseff-Baker F, Holm A. Engaging faculty and students in classroom assessment of learning. In: Serban AM and Friedlander J ed. Developing and Implementing Assessment of Student Learning Outcomes. San Francisco, Calif: Jossey Bass; 2004:33.
Corresponding Author: Heidi M. Anderson, PhD Address: Assistant Dean, Education Innovation, College of Pharmacy, University of Kentucky, Lexington, KY 40536 Tel: 859-257-5802 Fax: 859-257-6153 E-mail: hande2@email.uky.edu
Anderson HM, Moore DL, Anaya G, Bird E.
Student Learning Outcomes Assessment: A Component of Program Assessment.
Am J Pharm Educ. 2005; 69(2):article 39.
