Quantitative Assessment of Assisted Problem-based Learning in a Pharmaceutics Course

INTRODUCTION

In 1993 the Commission to Implement Change in Pharmaceutical Education recommended that pharmacy schools undergo major curricular reform to include an educational process that encourages learning to produce independent and self-directed individuals.¹ The American Association of Colleges of Pharmacy (AACP) Center for the Advancement of Pharmaceutical Education (CAPE) Educational Outcomes (revised 2004) integrated general abilities into 3 practice functions that pharmacy graduates should possess. The general abilities are those first recognized by the 1993 Commission and include critical thinking, problem solving, communication, social responsibility, social interaction, and self-directed learning.² Guideline 11.2 of the Accreditation Council for Pharmacy Education (ACPE) Standards 2007 states that the development of critical thinking and problem-solving skills should be supported through the application of guided group discussions and case studies.³ To comply with these recommendations, many colleges and schools of pharmacy have adopted a problem-based learning (PBL) teaching approach.⁴

Consistent with these goals, we have used PBL and case studies in our pharmaceutics courses at the School of Pharmacy, University of Southern California, since the early 1990s.⁵ Our approach is best described as “assisted” PBL, because we do give some didactic lectures in the course, but we also use all the common elements of a PBL course, including student-centered learning, group-based work, case studies, and facilitation of group discussions by faculty.⁵ Variations of the PBL method have been described, but the debate about how and whether PBL can be adapted to scientific and medical courses continues.^6–11 Moreover, objective and quantitative evaluation of the effectiveness of PBL in pharmacy education has been limited, as pointed out by Cisneros et al in their 2002 review.⁴ The publication of this review prompted us to think carefully about how such an evaluation might be performed. At that time, we had acquired 10 years of experience teaching pharmaceutics through a PBL approach and we had established a case study-based course that was workable for a large class size.^5,12 Our previous papers provided descriptive and qualitative evidence for the effectiveness of elements of the PBL approach, but fell short of an objective and quantitative comparison of PBL with a didactic teaching approach.^5,12 Thus, these papers did not meet the challenge proposed by Cisneros et al regarding the need for a more objective evaluation of the effectiveness of PBL.⁴

In 2002, due to a growing class size and limitations on faculty time, we changed the format of our pharmaceutics course to a mixture of assisted PBL and didactic teaching. In doing so, we also recognized that the new format provided an opportunity to meet the challenge of objective evaluation of PBL. To do this, we asked the questions: “Do students learn course material better in a didactic lecture format or a PBL format?” and “What are the short-term (weeks) and medium-term (months) effects of PBL on learning?” The newly designed course format allowed quantitative assessment of these questions based on an internal control group. After 7 offerings of the course in this format, from 2002 to 2008, we now provide evidence that teaching pharmaceutics in a case study-based PBL format has a positive short-term effect on student learning.

DESIGN

Course Structure

In the fall 2002 semester, we implemented a version of our PBL pharmaceutics course in which we were able to compare the performance of students who received case study material through PBL and didactic lectures. The same model has subsequently been used in every fall semester until 2008, providing 7 years of comparative data. The PBL elements and manner of implementation were based on our 10 years of experience (before 2002) in previous offerings of the course.^5,12 Here, we describe the course structure (2002–2008) with brief comments on components described previously^5,12 and a focus on the new elements of the course design that are relevant for the comparison of PBL and didactic teaching.

The main activities in the pharmaceutics course (2002–2008) are summarized in Table 1. The course was given in the first semester of the PharmD curriculum. The course activities reflect the need for (1) introduction of the subject to the students through lectures; (2) development of student learning through case studies; (3) facilitation of this learning through small group discussions led by faculty members; and (4) evaluation of student learning through case study reports, verbal interviews, and midterm and final examinations. This combination of activities reflected an assisted PBL approach that we had found effective for a large class.^5,12,13 The course grade was based on 25% for the case study (15% from a group-based report and 10% from verbal and peer assessments), 25% for the midterm examination, and 50% for the final examination. An example case study is given in Appendix 1.

• Download figure
• Open in new tab
• Download powerpoint

Table 1.

Course Activities by Group in a Study Assessing the Effectiveness of Assisted Problem-based Learning (PBL) Compared to a Didactic Approach in a Pharmaceutics Course

The class was divided into case-study groups of about 8 students each; we used 26 groups in 2002–2004 (designated A-Z) and 24 in 2005–2008 (A-X). The allocation of students to groups was random and the students could not choose their group colleagues. Development of an appreciation of the dynamics of group work was an important element of the course, and best achieved if students had to work with colleagues who they did not necessarily know well. As shown in Table 1, all students attended lectures in weeks 1–4 as a basic introduction to the material for the first half of the semester. A case study was assigned in week 3. Only groups A-M (A-L in 2005–2008; hereafter, we use PBL1 to indicate the students in these case-study groups) produced a written report on the case study, while groups N-Z (M-X in 2005–2008; PBL2) were encouraged to complete the case study, but not required to produce a report. During weeks 5 to 7, the PBL1 students participated in group work, which was facilitated by small group discussions led by faculty members and teaching assistants. The case study report was due as a group report in 3 separate parts in weeks 5, 6, and 7. During weeks 5 to 7, the PBL2 students attended further lectures, referred to as “case-study lectures,” that were aimed at reinforcing the concepts taught in weeks 1–4 and discussing the details of the case study. These lectures were essentially the same discussion as that given to the PBL1 students, but in a larger class without the same level of personal interaction. Hence, while the PBL1 students were working in a PBL environment, the PBL2 students received the same material through didactic lectures. This allowed evaluation of the effectiveness of PBL compared to the didactic lecture approach based on performance on the midterm examination in week 8. In the second half of the semester, all students attended lectures in weeks 9–11. The PBL2 students then studied this material in weeks 12–14 in a PBL format, while the PBL1 students received didactic case-study lectures.

Course Elements

Class Lectures.

The material for each half of the semester was introduced in weeks 1–4 and weeks 9–11 (Table 1). The 2-hour lectures were attended by all students and provided a definition of the scope of the material to be covered in the case studies. The students were provided with the key objectives for each lecture prior to the first meeting of the class.

Case Studies.

Two case studies were given in the semester, but each group of students produced a written report in either the first or second half of the semester (Table 1). The case study presented a complex problem with a workload and level of difficulty that required the case to be solved by groups of students. Thus, working as a team became a necessity. The course faculty members provided general guidance to the students about group interactions, but the students themselves were allowed to determine the most effective approach for their group. The case study was designed to promote learning and transfer of information among students and to encourage students to seek information from diverse and non-assigned sources.

The case studies were structured so that the first 2 weeks were spent mainly searching for literature data and, when these data were not available, developing approaches for their estimation. Methods for estimation were discussed in the lectures. The first part of the case study was due as a group report in week 5 of the semester (Table 1). The second and third parts of the case study required the students to apply their data to a problem of pharmaceutical relevance involving formulation or delivery of drugs. These parts were due in weeks 6 and 7. Dividing the case study report into 3 separate parts due on different dates was effective because it ensured consistent work over the course of the case study; helped the students develop clarity of thinking by separating a complex problem into manageable parts; and allowed for timing of faculty-led discussion sessions that focused on specific areas of the case study, and thereby facilitated the process most effectively.

Case Study Discussion Groups.

Faculty-led discussion groups were held twice each week for each case-study group (PBL1 students in the first half of the semester and PBL2 students in the second half of the semester). These discussion groups were 40 minutes to 1 hour long, and designed to guide the students in their learning. Knowing they had to write a case report served as a strong motivation for the students to participate because the focus of the discussion was on issues that needed to be addressed in a particular section of the report. The level of sophistication of the discussions increased as the case study progressed.

Case Study Lectures.

Faculty-led case study lectures were held twice a week for students who were not in the discussion groups (PBL2 students in the first half of the semester and PBL1 students in the second half of the semester). Therefore, while half of the class worked in the case study-based PBL environment described above, the other students were given 6 additional lectures, each 2 hours long, on material relevant to the case study (Table 1). These lectures generally included a discussion of material already presented in the earlier lectures given to the entire class, followed by application of these ideas to the case study. Hence, we viewed these sessions as an attempt to lecture the case study material to the students. This represented a didactic approach to the delivery of the material in which we emphasized critical thinking but did not require the students to write a case study report.

Case Study Interviews.

At the end of the case study, each case-study group was interviewed by a faculty member (Table 1). The entire group was present for the interview, but individual questions were directed to a specific student, with some additional general questions open for anyone in the group to answer. The interview period lasted for an average of 5 minutes for each student in the group. Each student was asked to answer questions on a specific area of the case study and the interviewer began by asking which student would take a given area. To encourage and reward broader understanding of the case study, as a student showed the ability to answer simpler questions, the questioning was broadened. The students were not advised of the areas on which the interview would focus or which student would be responsible for a particular area prior to the interview. Therefore, it was important for all students to have a complete understanding of all parts of the case study, and not just the sections(s) on which they worked.

Case Study Grading.

The case study score was worth 25% of the course grade, including a score of 15% for the case report (each student in the group received the same score) and an individual score of 10% that was further broken into a peer score (2%), a discussion score (3%), and an interview score (5%). The case study was viewed primarily as a learning experience, and the group report was graded with this in mind. Errors were pointed out but not heavily penalized. For the peer score, each student provided a score for their group colleagues and these scores were averaged. The discussion score was based on student participation in regular faculty-led group discussions held over the period of the case study. The interview score was given for a formal faculty interview of each student conducted at the end of the case study, as described elsewhere.¹³

Examination Grading.

From 2002 to 2008 the same 3 instructors (the 3 authors) randomly graded the essay questions on the midterm and final examinations. For example, instructor 1 might have graded questions 1 and 4, instructor 2 might have graded questions 2 and 3, etc. Grading was independent from grader to grader: each grader determined the answer and an appropriate grading scale, and there was no discussion of the scores until all grading was completed.

Class Size.

The method chosen for many of the course components was constrained by the size of the class (Table 2). An average of 188.6 students completed all elements of the class in the years from 2002 to 2008.

• Download figure
• Open in new tab
• Download powerpoint

Table 2.

Number of Students Enrolled in a Pharmaceutics Course in Which the Effectiveness of Assisted Problem-based Learning (PBL) Was Compared to a Didactic Approach

EVALUATION AND ASSESSMENT

Midterm Examination

In the first half of the semester, PBL1 students performed the case study in a PBL format, while PBL2 students received the case study material through didactic lectures (Table 1). Following this, a midterm examination was administered to all students. The examination represented 25% of the total course grade and comprised 4 or 5 questions worth a total of 250 points. These questions were focused on an aqueous solution formulation problem and addressed molecular weight, degradation kinetics, pKa, stability and solubility, and buffer solution properties. The questions indirectly reflected the contents of the case study, in that they dealt with similar concepts to those discussed in the case study, but the molecules on which the examination was based differed from those used in the case study. The examinations included knowledge, content and fact-based questions that required students to draw structures, count atoms, recognize and use equations; and higher order thinking questions that required interpretation and integration of multiple concepts, following Bloom's taxonomy of learning.¹⁴

To evaluate the short-term effect of PBL on student learning compared to a didactic approach, we compared the midterm examination scores for PBL1 and PBL2 students. The midterm examination was held about 1 week after completion of the first case study (performed in PBL format by PBL1 students). For comparison of data from multiple years, examination scores were converted into percentages. Statistical comparison of the scores was performed using a 2-tailed Wilcoxon-Mann-Whitney test in SAS (SAS Institute Inc., Cary, NC) with a p value less than 0.05 considered significant.

The results for the midterm examinations from 2002 to 2008 showed a marked trend for improved performance by the PBL1 students (Table 3). In 6 of the 7 years, those students (who received the material through a PBL format) outperformed the PBL2 students (who received the material through didactic lectures), based on the difference between the means of the raw scores, with this difference reaching a significance level of p < 0.05 in 3 years. The midterm examination score for each student was converted into a percentage and these data were averaged over the 7-year period. These results indicated that the PBL1 students (n=675) scored an average of about 3% higher than the PBL2 students (n=645) and that the difference was significant (73.7% ± 14.7% vs. 70.8% ± 15.4%, P = 0.0008).

• Download figure
• Open in new tab
• Download powerpoint

Table 3.

Comparison of Pharmacy Students' Midterm Examination Scores in a Study Assessing the Effectiveness of Assisted Problem-based Learning (PBL1) Compared to a Didactic Approach (PBL2) in a Pharmaceutics Course

Final Examination

In the second half of the semester the PBL2 students learned the case study material in a PBL format, while the PBL1 students received the same material through didactic lectures. The final examination was comprehensive, worth 50% of the course grade, and comprised essay-style questions worth a total of 500 points. The examination required knowledge of concepts from aqueous solution, suspensions and emulsions, and tablet formulation and some basic knowledge of dissolution and absorption. The final examination had a mixture of knowledge-based and higher order questions, but with greater emphasis on integration of material.

The scores for PBL1 and PBL2 students on the final examination, which was held about 1 week after completion of the second case study (performed in PBL format by PBL2 students) and about 2 months after completion of the first case study, were compared. The overall examination results over the 7-year period are presented in Table 4. Similar to the midterm examination results, these data showed marked trends that indicated a better performance by the PBL2 students (those who spent the second half of the semester studying in a PBL format), although with lower statistical significance than for the midterm examination. The PBL2 students had higher scores in all years except for 2002. Conversion of the score for each student into a percentage and comparison of these normalized data over all years indicated that the PBL2 students scored an average of about 0.7% higher than the PBL1 students, but the difference was not significant (72.0% ± 14.0% (PBL1) vs. 72.7% ± 13.7% (PBL2); p = 0.38).

• Download figure
• Open in new tab
• Download powerpoint

Table 4.

Comparison of Final Examination Results for a Study Assessing the Effectiveness of Assisted Problem-based Learning (PBL) Compared to a Didactic Approach in a Pharmaceutics Course

The data for the final examination are complicated by the inclusion of questions that drew from material from the first half of the semester. Therefore, the data were divided into scores for questions that focused exclusively or mainly on material from the first half of the semester (studied in a PBL format by PBL2 students) and the second half of the semester (studied in a PBL format by PBL1 students) (Table 5). The scores on the final examination for questions on material from the first half of the semester showed no clear trend and the differences between the groups were within 1% in 6 of the 7 years. Scores for the PBL1 students were slightly but not significantly higher than those for PBL2 students in the normalized cumulative 7-year data for these questions (83.2% ± 13.8% (PBL1) vs. 82.7% ± 13.8% (PBL2), P = 0.51). The scores for material from the second half of the semester reinforced the findings from the overall scores for the final examination, with PBL2 students outperforming PBL1 students in all years except 2002. The normalized cumulative 7-year data showed that the PBL2 students scored an average of about 1.4% more than the PBL1 students on this subset of questions, but the difference did not reach significance (63.1% ± 19.6% (PBL1) vs. 64.4% ± 19.1% (PBL2), P = 0.19).

• Download figure
• Open in new tab
• Download powerpoint

Table 5.

Pharmacy Students' Scores on Final Examination Focused on Material from the First and Second Half of the Semester in a Pharmaceutics Course in Which the Effectiveness of Assisted Problem-based Learning (PBL) Was Compared to a Didactic Approach

DISCUSSION

The case-study based PBL approach is used widely in the teaching of basic and clinical sciences, but there have been few long-term quantitative assessments of its effectiveness. Most previous studies of PBL in pharmacy education have been descriptive reports using qualitative measures such as student and faculty perceptions regarding the extent of learning, usually assessed using a questionnaire.⁴ An objective and quantitative comparison of PBL with a didactic lecture approach requires elimination or minimization of the many variables that might influence the comparison. These include everything from group dynamics to instructor bias when designing tests to measure PBL effectiveness. It is difficult to control all these variables and various papers have pointed out the problems associated with studying PBL.^4,6,15–18 However, the course design described in the current study incorporates an internal control group of students receiving didactic teaching for comparison with a group of peers taught in a PBL format. Given this approach and the long-term nature of the study, we believe that our results provide quantitative evidence for a short-term benefit of PBL on learning outcomes.

The most direct evidence of improved learning using assisted PBL emerged from the higher average score of about 3% on the midterm examination achieved by the students who studied the tested material in a PBL format. This difference is perhaps even more significant than it first appears, because the students who received didactic lectures were strongly encouraged to problem solve, and the case study lectures were, in reality, a large group discussion session in which the problems were described and solved through a joint effort of the lecturer and the students. Hence, the data suggest that even taking this approach to didactic lecturing does not produce the same level of problem-solving skills as that achieved in an assisted PBL approach (that is, one focused on self-directed learning through case studies and discussion groups). The significant difference in midterm scores is all the more remarkable given the large number of variables in the process underlying these data, since each annual mean is calculated over the performance of about 90 students of varying ability, and the scores are based on independent grading of multiple examination questions by different instructors.

The results of the final examination are more difficult to interpret. In general, the students who worked in a PBL format in the second half of the semester obtained slightly higher scores, but the difference between the 2 groups of students was not significant. A sub-analysis of scores for questions that focused on material from the second half of the semester indicated that students who received this material in a PBL format scored about 1.4% higher. These data provide further support for a short-term educational value-added effect of PBL, although the yearly differences in means are smaller than those for the midterm examination scores. In this comparison, the students who received the material didactically do not provide an ideal control group since they already have training in the PBL method. However, the smaller difference in performance may indicate that students who have experience in PBL improve their higher order thinking ability in general, and that this caused the more similar performance on the final examination. This provides some evidence for persistence of the PBL effect in the medium term and suggests that higher order thinking can be transferred to new material.

The more similar performance on the final examination between the student groups might also have been a result of the relative difficulty of the examinations or their relative importance for the overall course grade. The final examination was more difficult than the midterm examination in that more material was examined and, most importantly, the questions were less direct on the final examination. On the final examination the questions were generally broader and required several steps to answer, whereas those on the midterm examination were more direct. The final examination also counted for a higher percentage of the course grade (50%), and this may have influenced performance. There was some variation in the absolute score from year to year in the midterm and final examinations. This variation is partly caused by the challenge of writing examinations of the same level of difficulty each year. This is made more difficult because we encourage the students to look at previous examinations as a learning tool. These examinations and suggested answers are made available to the class by the faculty members.

In implementing the course format, we were concerned that dividing the class in half might place 1 group of students at a disadvantage. We did our best to mitigate this concern by ensuring that the case-study lectures reflected the small group discussion sessions that were held for groups who were performing the case study. The same faculty members participated in both discussions and lectures, and therefore we believed that all students were receiving the same information. Students who worked in a PBL environment in the first half of the semester scored slightly higher on the midterm examination. Although we view this difference as significant in showing the added learning effect of PBL, it is only a small difference in the context of the overall grade for the course. The distribution of scores on the final examination did not differ greatly between the 2 groups of students, and the grades for the 2 case studies (data not shown) were similar. The mean GPAs for the PBL1 and PBL2 students over the 7 courses were 3.07 and 3.04, respectively, and the annual GPAs never differed by more than 0.1 between the 2 groups. Therefore, students were not placed at a disadvantage by being in a particular half of the class.

For the purpose of comparison of PBL with didactic teaching, the ideal design would be for one half of the class to learn in a PBL environment throughout the semester, while the other half receives lectures only. However, as shown by our results, this approach would result in half the class receiving a less than optimal course. Splitting the class so that half of the students learn in a PBL environment in each half of the semester avoids this problem, but still allows for a comparison of PBL with didactic teaching. The control group is not ideal, since there is interaction between students learning in each environment, as well as interactions between students in different case study groups. However, we believe that these interactions are positive academically and reflect the typical practice of a PBL course.

Our overall goal is to promote an educational process that encourages self-directed learning and produces a pharmacist who can solve problems and is knowledgeable and competent in pharmaceutics. One element in this approach is to instill the idea that complex problems have multiple, potentially correct answers, and this principle is at the heart of our PBL approach. Therefore, an understanding of how effectively the course material is learned using the PBL format is of importance. Based on a quantitative analysis of teaching outcomes over 7 years, we conclude that PBL has a short-term effect on learning and that this effect may persist over the medium term and may be transferable to new material. These results provide a strong argument for teaching of basic sciences in a PharmD curriculum in a PBL format. We believe that the course components described in this work and in our previous papers^5,12,13 provide sufficient detail for adaptation of the method to teaching of pharmaceutics in most academic environments. With modifications, the approach is applicable to teaching of most basic science courses in the PharmD curriculum. The learning advantages of PBL are likely to be reinforced in a curriculum in which multiple courses are taught in this format.

ACKNOWLEDGMENTS

We are grateful to Drs. Curtis Okamoto, Tim Gallaher, Melina Bayramyan, and Brian Sutch for their work with discussion groups in the course.