A Novel Question-based Framework for Writing Assessment Plans to Facilitate Their Understanding and Acceptance in Pharmacy Education

To the Editor. For over half my 40 years in pharmacy education the word “assessment” and the phrase “assessment planning” have prompted my colleagues’ eyes to glaze over and their ear canals to close. Two colleagues even told me those words activated their autonomic nervous system and immediately tightened several sphincters. I was no different.

The articles I read and the discussions on assessment I participated in left me with the feeling I understood what assessment was all about. Unfortunately, this feeling of, and confidence in, my understanding always had a relatively short biological half-life. Eventually, I proceeded directly to dismay. Working with different sets of colleagues through the dreaded Standard 3 and Standard 15 of the Accreditation Council for Pharmacy Education Guidelines required rescue by an assessment specialist to write an assessment plan because our collective level of understanding of what was needed had decayed to nearly zero.¹

Why was it so difficult for my colleagues and me to wrap our minds around assessment and assessment plans? We all had plenty of experience in academia. We all progressed through academic ranks successfully and seemed fairly intelligent. I read numerous assessment plans from other institutions but they left my mind empty. This was especially troubling to me because I served as an academic administrator for most of my career and knew assessment was an increasingly important component of education. Because of my training as a pharmacologist, I immediately sought a biological answer. Had my early exposure to assessment vocabulary sensitized my immune system to the point I developed antibodies to neutralize my brief periods of understanding? Had my exposure to intermittent barrages of a vocabulary and ways of thinking different from that in my comfort zone caused tachyphylaxis or down regulation of the assessment receptors in my central nervous system?

To complicate matters, I remembered the attitudes and comments from my nonassessment-type faculty colleagues over the years. “Assessment is something to be done by someone to meet some requirement, and they should do it and not bother me with it.” Or even worse, “What is the dean doing hiring those people down there in the assessment office when he should be hiring a faculty member to help teach and do research?” Finally, “Assessment people are taking over education, and before we know it they will consume 90% of our budget.” I wondered how we could create the required “culture of assessment” with attitudes like this among our faculty. I would be surprised if similar situations did not exist somewhere in most, if not all, of our colleges and schools of pharmacy.

While pondering possible solutions to this situation, I recalled conversations held long ago with 2 of the most astute women in pharmacy I know. Dr. Linda Strand, then at the University of Minnesota, once talked about the importance of a common understandable vocabulary during a conversation about pharmacists becoming contributing members of a healthcare team. Pharmacy could not participate effectively in healthcare teams unless we used and understood the vocabulary of those teams, she said. It was obvious to me that vocabulary was part of the problem for faculty members not trained in assessment. If you doubt this, walk up to your basic science or practice colleagues and ask them if they do formative or summative assessments of their departmental mission and see what response you get. Then ask them if they agree with the philosophy and direction of their own college assessment plan and see if the response to that question is any more lucid. My prediction is the response to both questions would be something analogous to “And why should we waste our time on that?” Although educating faculty members about assessment vocabulary would probably help their understanding and acceptance of assessment and adopting a culture of assessment, I am fairly certain the short half-life principle would still apply. Understanding common vocabulary is important, but that is not going to happen by pharmacy education faculty members learning the assessment vocabulary. Instead, assessment documents must be written using a vocabulary that pharmacy faculty members understand.

The second astute woman who influenced me was Dr. Barbara Brandt from the University of Minnesota. We had frequent and incredibly energizing conversations about making education more effective. She pointed out the importance of framing what you want students to learn within the background, context, and interests of students. We joked about teaching math to 9-year-old boys interested in baseball by putting as much math education within the framework of baseball statistics as possible. The correlation to educating pharmacy students meant putting principles within the framework of application to medications and diseases. For faculty members to accept assessment and assessment planning, this meant putting an assessment plan within a context faculty members could appreciate, consistent with their training and way of thinking, that used terms in major sections to which they could relate. This also meant doing something different from the way assessment specialists usually communicate with faculty members because basic science and clinical faculty members are not trained and do not think the way assessment specialists are and do. Different hemispheres are involved in the training of each— sometimes different cerebral hemispheres, but it could just as easily be different global hemispheres.

I knew intuitively what assessment and writing assessment plans were trying to ultimately accomplish - improvement. Assessment tries to answer the question “How are we doing?” What we need to do in assessment planning is list the “doing what?” issues, word them appropriately, and organize them according to the important items in our missions.. Logically, we should frame assessment plans in a format that asks some of the most critical questions. Questions should be designed to include the range of what we need to know to track our educational, research, and service missions, and what is required for accreditation. They should reflect what assessment plans are supposed to accomplish in plain language readily understandable by the faculty, staff, and students. Such a format should more readily facilitate development of a “culture of assessment” or, at least, decrease resistance to doing so.

Suggestions for the most critical questions to be answered and some examples of the types of data that might be collected for each question are:

• • Are we attracting and admitting students with the greatest likelihood for success? Admissions data (ie, grade point average (GPA), Pharmacy College Admissions Test), work experience, interview score, personality or critical thinking tests, demographics, applicant surveys correlated with PharmD GPA, grades in specific courses, student awards, Pharmacy Curriculum Outcomes Assessment (PCOA), North American Pharmacist Licensure Examination (NAPLEX), and Multistate Pharmacy Jurisprudence Examination, etc.

• • Are our learning experiences sequenced correctly and delivered optimally, and do they cover the most appropriate material? Student surveys on courses and instructors, end-of-year student surveys correlated with American Association of Colleges of Pharmacy (AACP) graduating student surveys, course director meetings, alumni, and preceptor surveys.

• • Are we providing the best environment for the professional development of our students, staff, and faculty? Student Affairs and Library surveys; faculty, staff, and student satisfaction surveys; faculty and staff evaluations; and peer reviews of teaching correlated with AACP graduating student surveys.

• • Are we advancing health care and our profession? Faculty and staff data from performance evaluations (ie, peer-reviewed publications; impact factors proposals submitted; grants funded; patents; invited presentations; national awards and recognitions for research, teaching, and service; and development of new and improved practice models).

• • Are we serving society and our profession? Faculty and staff data from performance evaluations and student portfolios (ie, community and public service projects; number of individuals served; number of student sand faculty members involved; service on college, university, national association or clinical site committees; and elected officer positions in national organizations).

• • Are our students developing the knowledge base, skills, attitudes, and behaviors we desire and are needed by society and the profession? Course learning outcomes correlated with course individual readiness assurance tests, group readiness-assurance tests, quizzes, homework assignments, examinations, preceptor evaluations, and OSCEs. Program learning outcomes correlated with portfolio examinations and PCOA results. College learning goals correlated with PCOA results, NAPLEX scores, preceptor and employer surveys, and post-graduation experiences (ie, residencies, fellowships, and board certifications.)\

• • Do we have a culture of assessment for improvement and are our improvement processes successful? Faculty, staff, and student surveys to be created; AACP surveys.

Conceptualizing this question format for framing and organizing assessment plans immediately and effectively made assessment comprehensible and relevant to me. It suited my logical and linear thinking patterns, which I believe are common in other pharmacy faculty members. It allowed me to reframe the responses to Standard 3 and Standard 15 in a way that did not require extensive formal assessment training, although I would not proceed without an assessment specialist to validate those responses. It has lowered my titer of “assessment antibodies” and effectively counteracted the down regulation of my central nervous system assessment receptors. I am more accepting of a culture of assessment. I believe this would apply to other faculty members as well. The questions listed can easily be modified or deleted, or additional questions can be added to make them more relevant to an individual institution.

I believe the question framework described above would clarify assessment and assessment planning for our faculty. Being concerned that I may be an outlier and that I may unknowingly be violating some basic assessment principle (or even several principles) with this thinking, I shared this question-based assessment planning format with 7 basic science and clinical faculty members from 3 colleges and schools of pharmacy. They all thought it was a great idea. I also shared this format with 2 individuals with extensive formal assessment training at 2 institutions and they reacted the same way. They felt the format did not violate the principles of good assessment and it “captured the essence of institutional effectiveness.” In a moment of temporary cerebral ischemia or a moment of delusions of grandeur induced by anoxia, I modestly decided the question format for assessment planning described in this communication is analogous in simplicity and elegance to the description of the structure of DNA written 60 years ago by Watson and Crick, but probably will have less eventual impact.² I hope the academy finds it useful.

• © 2013 American Association of Colleges of Pharmacy