An Assessment Model for Multidisciplinary, Team-Taught Integrated Pharmacy Courses

INTRODUCTION

Assessment is a major component of pharmacy education in the revised accreditation standards of the Accreditation Council for Pharmacy Education (ACPE).¹ In ACPE guidelines 2.0, standard 15 places additional focus on learning objectives, assessment, and evaluation to improve programmatic outcomes. Also recognizing the importance of assessment, the American Association of Colleges of Pharmacy (AACP) started a special interest group for assessment in pharmacy education in 2010.² There are many assessment models that are implemented at various pharmacy colleges and schools based on their curricular needs.^3-4 ACPE highly recommends curricular integration among disciplines for the effective use of knowledge gained by pharmacy students, and pharmacy educators recognize the benefits of integration, resulting in the implementation of several approaches.^5-7 In general, there is a long-standing perception that basic sciences courses are not well received by pharmacy students because they do not see the direct application or use of these concepts in clinical practice. Integration of basic sciences in clinical courses is expected to address this issue by providing a stronger foundation for the use of basic science knowledge. Several pharmacy colleges and schools across the country have begun integrating basic science subjects – most commonly pathophysiology, pharmacology, and medicinal chemistry - with clinical sciences in various combinations based on curricular design.

Although interdisciplinary integration of courses offers obvious opportunities, it also presents challenges, including: (1) the possibility of content redundancy and delivery discrepancies resulting from multiple instructors teaching the same disease states; (2) coordination of multiple instructors (in some instances, 10 or more) for a single course; (3) continuity of the course content among instructors teaching the same disease state; (4) overlap in examination questions; and (5) variation in correct responses on assessment items resulting from different instructors teaching overlapping concepts.

Faculty members teaching integrated courses also face challenges, such as heavy preparation time, the potential for content overlap in lectures, and redundancy in examination questions. Teaching in integrated courses requires heavier-than-usual preparation time for faculty members. In addition to ensuring that they have overall knowledge about each disease state being taught, faculty members also must narrow their teaching focus to their respective areas, which requires knowing what material other instructors are covering.

Another challenge for faculty members is the potential for content overlap with other instructors in the preparation of lectures. It may not be possible to overcome this challenge entirely, given that content from each of these disciplines is tightly connected and, in some instances, there is a need for overlap so the concepts as they relate to each discipline can be better explained. Even if multiple instructors team-teach at the same time in a classroom, if a member of the team leaves or does not teach in subsequent years, the teaching structure of the course may collapse until a replacement team member is trained. Finally, redundancy in examination questions (ie, test items that cover the same content) is the most significant concern for faculty members as it has the greatest potential to impact outcomes/students’ performance in the course.

We addressed several of these challenges in Integrated Pharmacotherapy V with the effective use of course assessment to avoid overlaps in content and examination questions, to ensure seamless continuity of content among instructors teaching same disease state, and to provide data that can be used to further improve the course. The major objective of the current study was to design and implement an assessment model that can assist in the effective delivery of multidisciplinary team-taught pharmacy courses.

DESIGN

At the Texas A & M Health Science Center, College of Pharmacy, all disease states in the curriculum are taught in an integrated format. In each of these courses, physiology, pharmacology, medicinal chemistry, and pharmacotherapy are taught sequentially for a given disease, as there are no stand-alone basic sciences courses in the curriculum covering disease states. There are 8 such integrated pharmacotherapy courses within the PharmD curriculum. The assessment model in the current study was implemented in the fifth integrated pharmacotherapy course during fall 2009. Integrated Pharmacotherapy V covers the major diseases in psychiatry and addiction, including schizophrenia, depression, bipolar disorder, anxiety-insomnia, attention-deficit/hyperactivity disorder, and addiction. This course was delivered in 5.5 weeks and included 8 lectures per week and 5 examinations with multiple-choice questions.

This study involved students enrolled in the college of pharmacy in the fall of 2008 and 2009. The similar composition of the 2 classes, including age, gender ratio, ethnicity, and average GPAs, was assumed to provide reasonably good comparisons. Several sequential steps were implemented in IPT V during fall 2009, including the writing of detailed learning objectives, matching of all examination questions with learning objectives, pre-examination reviews, student self-assessment of the learning objectives, quantitative assessment of learning objectives through examinations, post-examination reviews, course-embedded quantitative assessment, and objective assessment of subdisciplines in the course.

The director of assessment trained all instructors on writing learning objectives through a workshop. To keep all faculty members on the same page, a handout on writing learning objectives, examination questions, and other course-related instructions (available from corresponding author on request) was sent to all faculty members teaching the course. All instructors teaching a particular disease topic (pathophysiology, pharmacology, medicinal chemistry and/or therapeutics) wrote topic-specific detailed learning objectives that collectively represented all content taught in the classroom. The instructor team met and reviewed learning objectives prior to the course to avoid possible replication of the material by faculty members. Before a topic was delivered, the complete set of objectives was provided to the students, who were informed that they were expected to master these outcomes in order to perform well on examinations.

Course material was divided in such a way that 1 or more complete disease states were covered in every examination. This was to ensure that students studied all disciplines at the same time to improve their appreciation and application of material among the various disciplines. All examination questions were written in a multiple-choice format, with 1 or more of the learning objectives represented by each question. On average, 4 to 5 multiple-choice questions were written for each lecture delivered. Instructor teams comprised of physiology, pharmacology, medicinal chemistry, and therapeutics faculty members met once again to review the examination questions to ensure they were readable and understandable to all instructors; there was coordination among faculty members regarding questions (to avoid instances in which, for example, questions from 1 instructor identified the answer to another instructor’s questions); the questions represented the objectives provided to the students; and the majority of the objectives were assessed in the examination.

There was a 50-minute session before each examination, which was held outside the class meeting time and was attended by faculty members who would be teaching material that would be on the forthcoming examination. Pre-examination reviews were driven by learning objectives, in that instructors presented the learning objectives to students and answered any questions they had regarding the concepts behind them. After each examination, students were asked to perform a subjective self-assessment of the learning objectives (sample survey available on request from the corresponding author), scoring their perceived mastery on a scale of 1 to 5, on which 1 = strongly disagree to 5 = strongly agree. These data provided information regarding student perceptions about and their mastery of the objectives. Processed survey data were provided to all instructors who wrote the objectives.

All questions were derived from 1 or more learning objectives provided to students. Student performance on the individual examination questions was matched to an unbiased assessment of learning objectives, which helped in identifying which learning objectives were assessed and how students performed on them (Table 1). The data collected were then sent back to the faculty members for possible student remediation on those objectives on which they underperformed or felt less confident in performing.

Postexamination reviews were 50-minute sessions held outside class meeting times. The examination questions were presented to students along with the learning objectives from which they were derived. Students had an opportunity to challenge the examination questions and/or learning objectives for any perceived discrepancies. Faculty members were also allowed to review material for low-scoring learning objectives in the students’ self-assessments and/or examinations. This approach allowed for instant use of the assessment data for the current class of students rather than delaying their use until the next course offering.

Course-embedded quantitative assessment was performed by matching questions from all examinations, including the comprehensive final examination, with course learning objectives (Table 1). Data from the embedded assessment (objective assessment data) helped to examine how well students mastered the learning objectives provided for the entire course, which ones may need to be emphasized further in subsequent delivery of the course, and how extensively each objective was assessed.

Student performance in each individual subject area was evaluated for all examinations in the course to identify deficiencies within the general content areas of physiology, pharmacology, medicinal chemistry, and therapeutics (Table 2). In the process of collecting item analysis data for each individual examination question, an examination bank was developed. The concept of an examination bank is based on the premise that if questions are written to effectively test a learning objective, they should be stored for future use with other cohorts of students.

All of the above steps except for pre- and post-examination reviews were part of the assessment model and facilitated the effective implementation of the integrated multidisciplinary team-taught course. Pre- and post-examination reviews assisted with this process but were not necessarily part of the assessment model. The Internal Review Board of Texas A&M University approved this study.

EVALUATION AND ASSESSMENT

Seventy-one students in the fall 2009 class were included in this assessment process. Student learning was measured in multiple ways throughout the assessment process and, on each learning objective, was measured using both qualitative and quantitative assessments based on student perceptions and selective examination data, respectively. Qualitative and quantitative examination data for all learning objectives provided to students are presented in Table 1. The data in this table demonstrate how well students understood the learning objectives and their performance on examinations for the same learning objectives. For example, as shown in Table 1, student perceptions for the schizophrenia section of the course ranged from 3.0 to 4.6 on a 1-to-5 rating scale, with 1 being least agreeable and 5 being strongly agreeable. Student performance on examinations for each of the learning objectives for schizophrenia ranged from 56% to 97%. Table 1 also provides information about learning objectives that were not assessed. Table 2 illustrates the distribution of grades by subdiscipline and by topic, which range from 71.7 % to 95.6%. The effectiveness of this exercise was measured by (1) selective examination score comparisons, (2) a comparison of student evaluations of the course and student open responses obtained during the collection of data from the qualitative assessment of learning objectives, (3) comments from student and faculty focus groups, and (4) course coordinators' perceptions.

Based on a comparison of the examination data from the classes of fall 2008 and 2009, the first 2 examinations showed significant improvement of more than a letter grade in student performance from the previous year. However, student performance on the remainder of the examinations was similar or lower compared with that of the previous year. The course evaluations for fall 2009 improved in every aspect compared with those of 2008 (Table 3). Student evaluations were compared using the Mann-Whitney U-Test. Although several survey statements indicated improved student perceptions of the course compared with those of the previous year, none of the differences were significant. However, students believed that this course contributed significantly to their professional growth and also perceived Integrated Pharmacotherapy V to be 1 of the best-integrated courses based in part on the improved teaching methods.

Three major themes related to learning objectives, lecture recordings, and the integration of basic sciences with therapeutics emerged from the course evaluations and focus group. Learning objectives seemed to work as a guide for students in preparing for the examination. As all questions were based on learning objectives, understanding them improved students’ performance on the examinations. Students commented that they appreciated lectures being recorded and posted on Blackboard (Blackboard Inc., Washington, DC) because it allowed them to review concepts after the lectures. Students also recognized better integration of the basic sciences and clinical content, despite the challenges associated with it. Redundancy in the content presented by instructors teaching the same disease state, which seemed to be especially applicable to the pathophysiology section, was one of the major challenges. This problem was effectively rectified through the use of learning objectives and faculty discussions prior to the delivery of each topic.

Faculty members who taught this course found the number of meetings to be demanding. All team members had to commit to spending more time because of the integrated teaching approach. Interaction between faculty members and students in general seemed to increase with the integrated sequential courses, with much of this increase occurring through the electronic media. The embedded quantitative course assessment shows all the learning objectives that were assessed and the extent to which they were assessed. This information enabled the identification of objectives that had not been tested as well as those on which students performed poorly, providing data that can be used to help improve teaching and testing when the course is offered again.

DISCUSSION

Assessment for a team-taught integrated course was effectively implemented in this study as evidenced by the course evaluations and the qualitative and quantitative measures of the learning objectives. Assessment started with the construction of the detailed, measurable learning objectives by all faculty members who were teaching the course. Learning objectives played a major role in achieving effective integration. Writing learning objectives and using them in a course is not a new concept to the academic community, and there are several resources available to construct them.^11-13 However, writing effective learning objectives that can be used in an assessment process is still a challenge to many faculty members teaching in pharmacy colleges and schools. One of the reasons could be that pharmacy faculty members are not trained educators and may require faculty development in this area. The course in this study was completely driven by learning objectives, which faculty members were trained to write prior to the study. The belief is that when students understand and master the learning objectives, they learn course content better and are able to successfully answer examination questions.

We used Bloom’s and Fink’s taxonomies^14,15 in constructing learning objectives to cover all levels of learning. The learning objectives were provided to the students prior to the introduction of each new topic being taught. During lectures, the instructors highlighted these objectives, usually by listing the specific objectives in the respective presentation slides. Examination question construction was based on the objectives and every question was derived from the list of objectives provided to the students. Students were given an opportunity to ask questions to clarify their understanding of learning objectives during pre-examination reviews. During post-examination reviews, students were presented with the examination along with learning objectives from which each question was derived. At this time, they had an opportunity to challenge the questions and/or objectives.

Learning objectives served as a guide to students in preparing for the examination and focusing on the most important material. The success of this approach is evidenced by the examples provided in Appendix 1, which show improved student performance on concepts that proved difficult for the 2008 students. Pre- and post-examination reviews were scheduled outside class meeting times. A qualitative assessment of all objectives was performed by the students after each examination to obtain the overall class perception of the concepts being taught during that period. Objectives usually covered a single disease state with respect to pathophysiology, pharmacology, medicinal chemistry and therapeutics. Assessment data were sent to the corresponding instructors with the scores (1-5 scale) matched to their learning objectives for the purpose of identifying which concepts were or were not mastered by the class. Instructors had an opportunity to go over those points during postexamination reviews or during their subsequent lectures. Qualitative assessment and embedded quantitative assessment data provided in Table 1 show that there are some learning objectives on which students both performed low in examinations and scored low in their perceptions evaluations. Some objectives were not tested throughout the course and a few were tested extensively. Thus, based on the nature and importance of the content covered by these objectives, appropriate changes to instructional delivery and testing may be adopted for the next delivery of the course. Objective quantitative assessment of examination data (Table 2) also shows student weakness in the general subject area. The process of assessment described in this paper provides data that can be used to inform ongoing improvements in the integration of material and course delivery over time.

We were not able to use examination grades to measure the effectiveness of the assessment activity because multiple changes were initiated in the subsequent course delivery during fall 2009. One of these changes involved poorly written questions from the fall 2008 examinations for which several answers were accepted as correct, leading to improved class averages. When the course was delivered during fall 2009, more than half of the instructors had changed, and the course was delivered in a block format (5.5 weeks) compared with fall 2008, during which it was delivered over an entire semester. Additionally, Integrated Pharmacotherapy V was delivered more systematically during fall 2009; ie, multiple changes had been incorporated, including faculty orientation on how to write meaningful learning objectives, improvement in examination question writing, lecture recordings, and systematic assessment. However, on the questions for which students were provided with learning objectives, student performance improved. Several examples from medicinal chemistry are shown in Appendix 1. The course evaluations and responses from student and faculty focus groups indicated that course delivery was much more effective in fall 2009 compared with the year before.

Because this 3-credit-hour course was delivered at a quick pace with 8 lectures per week, it was very challenging to organize several meetings each week to go over learning objectives and examination questions and to perform surveys on learning objectives after each examination. Coordinating this process took a significant amount of time, and, because of schedule conflicts, it was often challenging to find time when it was possible for all faculty members to meet. However, these difficulties were effectively addressed with the use of assessment personnel who assisted in compiling and matching examination data with learning objectives and generating and evaluating surveys.

When written in a meaningful way, learning objectives help not only in effectively delivering the integrated course with minimal overlaps among faculty members but also facilitate curricular mapping. In this study, we mapped assessment data with learning objectives, which, in turn, could be mapped to Appendix B of the ACPE’s accreditation standards and guidelines for the pharmacy program leading to the doctor of pharmacy degree¹ and the college’s terminal outcomes to identify redundancies and/or gaps in the entire curriculum. Effectively written learning objectives not only help standardize the course from year to year, which is especially helpful if team members are periodically replaced, but also make it easier for new faculty team members to understand their role in a team-taught course.

CONCLUSION

Courses that are simultaneously taught by multiple instructors pose a challenge, especially when they involve the teaching of 1 disease state sequentially from multiple perspectives. Some of the common student concerns we observed in such courses include overlap in the content delivered by the teaching team, use of discipline-specific abbreviations, content discrepancies, repeated examination questions, and lack of true integration among the faculty members. At the same time, these courses are time-intensive for faculty members. These issues were addressed through the writing and effective use of detailed learning objectives, which resulted in improved outcomes in every aspect and provided data that can be used to further improve the course. This model is highly reproducible if followed systematically and can be implemented effectively in multidisciplinary, team-taught courses independent of periodic changes in instructors.