Abstract
Those in pharmacy education who are tasked with assessment may be overwhelmed by deadlines, data collection, and reporting, leaving little time to pause and examine the effectiveness of their efforts. However, assessment practices must be evaluated for their impact, including their ability to answer important questions, use resources effectively, and contribute to meaningful educational change. Often assessments are implemented, but then attention is diverted to another assessment before the data from the former assessment can be fully interpreted or used. To maximize the impact of assessment practices, tough and uncomfortable decisions may need to be made. In this paper, we suggest an approach for examining and making decisions about assessment activities and provide guidance on building high-impact assessment practices, evolving or “sunsetting” low-impact assessment practices, and managing mandated assessment.
Since the 2009 American Association of Colleges of Pharmacy Curricular Summit, curricular transformation has been openly encouraged and pursued among colleges of pharmacy in the United States.1 The release of the 2013 Center for Advancement of Pharmacy Education Outcomes2 and Accreditation Council for Pharmacy Education’s Standards 20163 added to the fervor for curricular redesign and evolution. With this aim in mind, decisions for change should be based on evidence, and transformation in education should be guided by assessment.
This work is not easy. While resources have been developed to support formative assessment strategies4 and guide the development of assessment leads,5 colleges and schools may nonetheless struggle to develop their assessment operations. Ever-increasing mandates for evidence of program effectiveness have driven demands for more metrics and benchmarks. In addition to pharmacy accreditation requirements, universities have additional reporting obligations. Collectively, there is a seemingly endless list of required data that sometimes makes the work of assessment feel like little more than checking off boxes.
Pharmacy faculty members, committees, and administrators should integrate assessments responsibly, use continuous quality improvement processes, and work to establish a culture of assessment within pharmacy schools.6 Yet, this is easier said than done. Intellectually, we know that gathering and managing reams of data does not create the culture we seek. And though technology may seem like an answer to managing the workload, it may simply make it easier to ask faculty members to supply ever-increasing levels of minutia. As assessment enterprises grow and mature, there may be the appearance of progress because people are working, analyses are conducted, and reports are produced. However, despite the flurry of activity, is it possible we are still not answering important questions that help us drive evidence-based improvement and change? Has assessment practice become, as Gilbert suggests, “similar to surgeons patting themselves on the back for taking out tumors without checking to see if their interventions are affecting mortality rates”?7
As organizations, pharmacy schools are susceptible to getting caught up in the assessment routine, churning through endless cycles of data collection and reporting. Thoughtful decisions must be made to foster sustainable and impactful assessment and to aid the movement towards a culture of assessment. The academy needs well-intentioned and well-designed assessments with attention given to implementation, fidelity, and quality improvement. We should be asking how we can refine and improve assessment processes. To aid in asking important questions to ultimately drive positive educational change, we outline an approach here for examining and making decisions about assessment activities. Given a recent commentary in higher education assessment that calls for clarity in the language of assessment,8 we also provide working definitions for the concepts introduced.
Ask Meaningful Questions
Colleges and schools need to strive toward assessment that is meaningful, valuable, and pragmatic.9 Simply gathering data is not sufficient. Good assessment begins with asking pivotal questions. Like posing good research questions, assessment that makes a difference is based on asking questions that hold meaning to the people involved in the process. These criteria apply for assessments conducted at the individual (eg, student) or program level, as well as those that are formative or summative. If the task is not serving a purpose, we should reconsider whether it needs to be done.
Identify Assessment Practices
Many routines are used to help accomplish the work of assessment. However, these routines do not all lead to meaningful change or improvement. A mechanism is needed for evaluating assessment activities and making decisions that allow an assessment program to be both effective and sustainable. We believe this involves isolating and naming our assessment practices. We posit that an “assessment practice” is a process, procedural sequence or system that accomplishes a specific goal related to curricular or programmatic improvement, utilizes specific strategies, tools and techniques, and is implemented and repeated with intention and discipline.
Having both a strategic approach and tools are key. We can have a good strategy, but lack effective tools for delineating actionable data. We can also have good tools, but lack a strong strategy for using them. For instance, an annual “data day” might employ the strategy of convening stakeholders, including external parties, for an onsite event to review results from curricular assessments. Tools or techniques, such as the use of data placemats, may then be used to encourage interaction with the data and their interpretation.10 When used intentionally, routinely, and effectively, a “data day” with all of its associated strategies and tools, is an assessment practice. In the frenzy of everyday operations, assessment practices may not be regularly reviewed; however, these practices are critical to the impact of assessment. So what makes an assessment practice impactful?
Build High Impact Assessment Practices
Identifying, listing, and describing assessment practices is not enough. We need a lens through which to evaluate our investment in this work. The concept of high-impact practices (HIPs) in the context of student learning has been conceptualized in undergraduate education. Specifically, HIPs have been defined as “an investment of time and energy over an extended period that has unusually positive effects on student engagement in educationally purposive behaviors.”11 Examples of HIPs include capstone experiences, service learning, undergraduate research experiences, and internships. By their nature, HIPs involve interacting with faculty members and peers about substantive matters, experiencing diversity, reflecting and integrating learning, responding to frequent feedback, and discovering relevance of learning through real-world applications.12,13
Just as there are high-impact educational practices associated with student learning, we suggest that assessment within our colleges and schools requires examination with an eye to identifying high impact assessment practices (HIAPs). The notion of high impact assessment practices would allow us to sort through assessment routines, looking for excellence. Building from the undergraduate literature,11 a HIAP would involve an investment of time and energy in the repeated implementation of an intentional set of assessment strategies and tools, which results in substantive, positive effects on stakeholder engagement and educationally meaningful, relevant improvements in curricula, the student experience, or the program.
In short, HIAPs answer critical questions and motivate change. A HIAP encourages the use of data for dialogue and debate and ultimately affects stakeholders (eg, students, faculty members) and/or the curriculum or program. Its implementation results in discernable, transformative, and memorable action being taken. While one-time or initial use of an assessment practice provides good input on its potential and promise, experience with an assessment practice (ie, repeated use) is needed to determine its impact. In addition, an assessment practice’s utility must be judged in context. A collection of successful strategies and tools used in a different environment may or may not work.
As an example, a key question may be whether a new curriculum requires a reasonable and appropriate student workload. A HIAP might focus on student workload monitoring. Several data sources (tools) might be utilized, such as faculty estimates of study time (assessed via survey tool) and student documentation of actual out-of-class time (logged by a sample of students). The assessment practice would also include the strategies by which data were reviewed (eg, an ad hoc “workload committee” evaluates the information). Decisions can then be made to “right size” workload to minimize peaks and troughs within a single semester. As needed, repeated cycles of data collection and use would be conducted, allowing for improvement and progress on the variable under consideration (ie, student workload).
As any HIAP is used, it may need to evolve to increase its utility (eg, refining measures, strengthening participation, employing triangulation, addressing reliability issues, experimenting with reporting methods, scaling up stakeholder involvement). For example, it might be determined that measures of student stress are needed, along with hours spent, in order to more fully appreciate and interpret the effects of student workload.
A common HIAP is identifying required learning assessment activities at strategic places in the curriculum (eg, pre-APPE). This collection of assessments can do “double duty” by providing program-level data on the success of the curriculum, as well as identifying students who are struggling so that assistance and remediation can be provided. As an example, these assessments could be conducted at the end of each didactic year and might include a 50-item written examination and performance-based assessments that assess program-level learning outcomes. When faculty members and students are actively involved in interpreting the data and meaningful educational changes result, this practice may be considered high-impact.
Evolve or Sunset Low Impact Assessment
When assessment practices fail to hit the mark, we should routinely determine what went wrong. Best practice would dictate that assessing our assessments for their impact should be part of the process. It may be possible to evolve assessment practices to be more effective, perhaps even making them be high impact. Tables 1-3 build from available literature to assist committees or administrators in self-evaluating impact and identifying methods to elevate impact further.
Design- and Development-Related Questions for Committees and Administrators to Use in Evaluating the Impact of an Assessment Practice
Implementation-Related Questions for Committees and Administrators to Use in Evaluating the Impact of an Assessment Practice
Evaluative Questions for Committees and Administrators to Ask When Examining the Impact of an Assessment Practice
A lack of impact is likely attributable to a variety of reasons. Perhaps data are collected but not shared publicly with relevant stakeholders. Maybe there is insufficient dialogue allowing for solid and meaningful data interpretation eg, how much weight do we give this finding? What additional evidence is needed to better understand this issue? Maybe there is not enough discussion and debate about the data’s implications and the options for change, eg, what alterations could inch us toward a stronger outcome? Perhaps the measure is not as sensitive or specific as we need it to be.
Raising the “impact” of an assessment practice may be possible. For instance, a change in the instrument or the method of administration may be needed. A proper analysis may also be lacking; performing a more targeted sub-analysis may provide us with the information we need. A technique (eg, student focus groups) may not be meeting the desired intent (eg, quality input on the curriculum) and may need redirection, replacement, or supplementation with a new technique or tool. Perhaps the critical part that is missing is consistent closure of the assessment loop; procedures can be put in place to reevaluate and ensure that data are shared, discussed, interpreted, and acted upon.
Salvaging a low-impact practice is not the only option. Assessment practices can also be retired. In particular, we must be careful about assessing that which we have neither the will nor the ability to change.14 If the initial question prompting the assessment has been answered, the practice may no longer be needed and may be retired or “sunset.” For example, if student workload in the curriculum has been optimized after several years of monitoring and adjustments, the assessment practice can be sunset. In Good to Great, author Jim Collins argues that strong organizations have a “stop doing” list.15 It requires courage to channel resources differently and disciplined people to make an honest and diligent effort to understand the current reality and confront it through dialogue and debate (not blame or coercion).15
Sunsetting an assessment practice can be an uncomfortable decision. Many educators are understandably interested in preserving the possibility of examining historical data. They anticipate that at some point they may want to look for a trend, demonstrate improvement, or document growth. However, the willingness to retire or revise low-impact assessments is essential to allow for investment in other key assessment areas. In addition, we must be sensitive to the burden that assessment practices place on students and faculty and staff members, ensuring we are intentional and strategic in what we are doing and how we use the resulting data. The utility and cost-benefit of an assessment must be considered. To help faculty members let go of low impact assessment, asking them how the data will be used going forward can be helpful. If the answer is “We have always collected it,” or “We may need it,” then administrators or other faculty members must convince them that it is okay to let those measures go. Creating a mechanism for warehousing the item (or technique) in case it needs to be instituted again in the future may help some faculty and staff members to accept sunsetting assessment practices.
Manage Mandated Assessments
There are fundamental differences between those practices carried out for accountability (eg, annual monitoring of licensure examination scores) and those carried out for the purpose of improvement. Pharmacy educators have a responsibility to the Department of Education and to the public to show accountability for what we say we do. However, balancing assessment for accountability with assessment for improvement is necessary. Assessment for improvement helps us to do what we are accountable for doing even better and allows us to reach a higher bar. Fundamentally, we feel more invested in the outcome and more likely to be motivated to make changes when we answer questions with local meaning that can lead to real impact. Keeping the purpose of these two types of assessments front and center can help prevent getting caught up in the required assessments for accountability without leaving time or energy for assessing what matters most to pharmacy programs.
Several strategies can be used to change the time and effort spent on mandated assessment. One approach is to spend just enough time on assessment for accountability and to meet accreditation requirements (eg, pass rate or score on a standardized measure such as licensure rate) and then move on. Another approach is to elongate timelines and conduct certain assessments every 2 to 3 years versus every term or annually (eg, curricular quality surveys). Furthermore, it may not be necessary to conduct some assessments at every curricular touchpoint. For instance, one of our programs has deliberately not implemented assessments at every interprofessional didactic or experiential encounter, but more strategically. The result is purposeful measurement at times in the curriculum when the student cohort would have reached a level of achievement. A “slow it down” list may help to release resources and control the rate of data acquisition to more usable amounts.
Finally, creative efforts may help to elevate the impact of some mandated data collection by linking a new assessment strategy into the process that allows for necessary improvement. For example, several programs in the academy recently reported finding limited use in the reports provided for the Pharmacy Curriculum Outcomes Assessment (PCOA), making it difficult to use the data to make informed decisions about curriculum revision.16 Adding a simple step may prove valuable, such as interviewing students who just completed the examination to capture their experience. Their response to a question such as, “Is there content on the exam that is not covered or reinforced in the curriculum?” can be quite informative. This approach, which was used in one of our programs, revealed that the calculations content on the PCOA was covered in the curriculum, but some items were only covered in the first year of the program. This information allowed for a targeted curriculum revision that built in additional calculation touch points in the second and third years.
Facing the Mountain: A Call to Action
In order to prevent faculty and staff burnout, it may be helpful to periodically ask the assessment committee and/or assessment staff to examine the impact of each practice in the assessment portfolio, evaluating its ability to aid in meaningful educational change. Suggestions for balancing assessment for accountability and assessment for improvement can be made. A list of low-impact practices can be generated, along with recommendations for either increasing their impact or sunsetting them altogether. Approaches for fostering HIAPs and managing mandated assessments can also be included in this evaluation. While this approach does not eliminate the workload and staffing challenges related to assessment, it can help to focus energy and attention.
Identifying and evolving assessment practices requires diligence on several fronts. This work cannot be realized without leadership. The assessment leads must have a vision for the assessment program as a whole and work to involve others so that this becomes a shared vision.5 Developing a culture of assessment within our institutions requires that leaders build trust among those involved in assessment, a shared language around assessment, and research-based guidelines that can orient our assessment activities and serve as criteria for evaluating assessment plans and efforts.14 In addition, HIAPs happen within a context and culture, which requires the design, implementation, and evaluation of assessments to be uniquely tailored to meet local needs.
Now is the time to challenge ourselves to define the tangible benefits of our activities and the need for, and usefulness of, each data point that we collect. How do we judge what truly matters, when enough is enough, or perhaps more importantly, when some practices should be abandoned? Colleges and schools cannot continue to do it all. We should not be on an endless journey of climbing mountain after mountain, day after day. Instead, we should set our sights on the highest peak and make each day of climbing a meaningful step towards our goal. We have reached a point of maturity where examination of our assessment practices is needed so that our efforts lead us towards targeted, quality, and sustainable assessment. We encourage faculty members to identify high-impact practices to assist not only their own institution, but also the academy at-large, and to share their findings.
ACKNOWLEDGMENTS
The authors wish to acknowledge the insight and encouragement of 2016-2018 Big Ten Academic Alliance Pharmacy Assessment Collaborative (BTAA-PAC) members in developing this manuscript.
- Received December 20, 2018.
- Accepted April 28, 2019.
- © 2019 American Association of Colleges of Pharmacy
