A Concept Mapping Activity to Enhance Pharmacy Students’ Metacognition and Comprehension of Fundamental Disease State Knowledge

INTRODUCTION

In the current age of rapid advancement of medical practice, pharmacy students must acquire skills to become critical thinkers and self-directed, lifelong learners. An important component in the learning development of pharmacy students is metacognition. Metacognition is “knowledge and cognition about cognitive phenomena as well as monitoring of one’s own memory, comprehension, and cognitive enterprises.”¹ More simply stated, metacognition is “thinking about thinking.”¹ A fundamental skill of metacognition is accurate self-assessment; utilization of this skill by Doctor of Pharmacy (PharmD) students has clearly been associated with improved classroom performance. Two studies involving pharmacy students found that high-performing students demonstrate more accurate self-assessment of their abilities compared to low-performing students, and are better able to identify incorrect items on an examination.^2,3 A study evaluating pharmacy students’ confidence level and willingness to seek help with answering drug-information questions found they were overconfident and less willing to ask for help with unfamiliar topics.⁴

Pharmacy graduates should be able to examine and reflect on their personal knowledge, skills, and abilities; therefore, it is imperative that pharmacy curricula implement learning strategies using metacognition. One increasingly prevalent strategy is the use of concept mapping. Concept maps are graphical representations depicting concepts related to one another, and are used as an educational technique to improve understanding of course material, integrate didactic and experiential knowledge, and encourage higher-order learning.⁵ Concept mapping activities have demonstrated improved metacognitive accuracy and increased critical-thinking skills in assessments evaluating nurses and pediatric medical residents.^6-8 Concept mapping is currently being used in pharmacy curriculums to teach disease states and the Pharmacists’ Patient Care Process, as well as for assessment purposes.^9-11

Performing concept mapping may improve pharmacy students’ metacognition through utilization of judgement of learning¹² and reflection. For example, using concept maps as a study tool requires students to judge their own learning and reflect on it, both of which may improve the accuracy of students’ self-assessments, which is an important element of metacognition. The aim of this study was to evaluate whether participating in concept mapping and receiving feedback on their concept map performance would enhance pharmacy students’ ability to self-assess their mastery of therapeutic topics.

METHODS

The study included second professional year (P2) pharmacy students in a case-based learning pharmacotherapy course, the second of five standalone courses in the pharmacotherapy series. This course served as a transition to problem-based learning. The course ran parallel to a patient care laboratory course where students met weekly in assigned small groups of eight or nine students. The laboratory activities for the course supplemented the case-based learning method used in the pharmacotherapy course lectures. Student laboratory groups were assigned at random to include a distribution of high- and low-performing students. Laboratory groups remained the same throughout the semester. In the pharmacotherapy course, a new disease state was introduced at the start of each week through independent pre-class readings. Readings from Pharmacotherapy: A Pathophysiologic Approach,¹³ the primary literature, and/or clinical practice guidelines were assigned by the content expert who was giving the lecture. Following the pre-class reading, students met in their assigned laboratory groups, each of which was led by a resident teaching assistant. The day following each laboratory session, students attended the pharmacotherapy course lecture. The students completed a pre-class quiz covering foundational disease state knowledge prior to participating in case-based learning led by the content expert. This study included course lectures on 14 disease states and pre-class quizzes from the course.

We pseudorandomized the 14 disease states covered in the course lectures into one of three study conditions based primarily on the predetermined course schedule and content expert lecturers. The conditions dictated how the course topic was studied in the patient care laboratory session. For condition 1, students created group concept maps for which a teaching assistant provided review and feedback using an expert key. For condition 2, students created group concept maps but the teaching assistant provided no formal feedback. For condition 3, no in-class activity specific to the study was conducted. A condition was assigned to each of the 14 disease states. Condition 1 was implemented for the following lecture topics: anemia of chronic kidney disease (CKD), rheumatoid arthritis, heart failure, and asthma. Condition 2 was implemented for the following lecture topics: liver disease, hyperlipidemia, and urinary tract infection. Condition 3 was implemented for the following lecture topics: fluid and electrolytes, bone mineral disease of CKD, gastrointestinal disease, osteoarthritis, osteoporosis, seizures, and upper respiratory infection. A within-subject design was used for the study, with students serving as their own control as all students in their assigned laboratory group covered the same disease state with a specific study condition each week.

During weeks when condition 1 or 2 was implemented, students were given 45 minutes during laboratory to create a concept map within their assigned small group. Students were instructed to include 50-70 terms within their concept maps in the following six domains: pathophysiology/etiology, signs/symptoms, diagnosis, goals of care, treatments, and monitoring/follow up. Using their notes from pre-class readings, groups had the option of developing a concept map on a whiteboard or electronically using a word mapping software.

After concept mapping the disease states assigned to condition 1, students received feedback via review of a key that had been approved by a content expert. The students were given five minutes to review the concept map key after which they participated in five minutes of individual reflection. The reflection was guided by predetermined questions which students used to assess how well they mapped the specified domains to the level of the key. The individual reflection was followed by group discussion or reflection for 10 minutes. For disease states assigned to condition 2, the small groups developed a concept map but were not given feedback via review of a content expert’s key nor time for self- and group-reflection. Teaching assistants were instructed to remain hands-off during the concept mapping activities and facilitated the reflections only by asking the predetermined questions.

Regardless of which condition had been assigned for the week, on the day following the patient care laboratory, prior to the beginning of the large group lecture on the topic, students completed a multiple-choice pre-class quiz. The quiz assessed the foundational disease state knowledge that students were expected to have gleaned from pre-class readings and which should have been further reinforced through concept mapping for conditions 1 and 2 or reinforced using the students’ preferred study methods for condition 3.

In order to assess students’ metacognition skills, prior to completing the quiz, they were required to predict the number of quiz questions they would answer correctly. Two measures of metacognition were used.¹⁴ The first was bias, a measure of under-confidence or overconfidence, which was calculated by determining the difference between the predicted score and the actual score (ie, predicted score minus the actual score). Negative scores indicated under-confidence (ie, the actual score was greater than the predicted score), while positive scores indicated overconfidence (ie, the predicted score was greater than the actual score). The second measure was absolute bias, which is a measure of predicted quiz performance accuracy, calculated as the absolute value of bias.

Conditions were compared using a paired t test. For multiple comparisons, a Bonferroni adjustment was performed for three comparisons. To elucidate time effects, a general linear model was applied using repeated measures. Significance was set at p<.05 (p<.017 for multiple comparisons). Effect size was calculated, when appropriate, using Cohen d with d<0.5 as a small effect, 0.5≤d≤0.8 as a medium effect, and d>0.8 as a large effect. This study was approved by the University of North Carolina at Chapel Hill Institutional Review Board.

RESULTS

Of the 143 pharmacy students enrolled in the course, seven did not consent to have their data included and were excluded from analysis. Another 19 were excluded for having incomplete data (defined as missing >20% of data, eg, some students did not predict their grades for multiple quizzes). Of the 136 students included, 12 students reported using concept mapping as an independent study method for at least one of the seven disease states in condition 3 (business as usual).

Because of the difference in the number of topics within each condition (four topics in condition 1, three topics in condition 2, and seven topics in condition 3), data for conditions 1 and 2 were initially combined (seven topics) and compared to data for condition 3 (seven topics), thus comparing group concept mapping to the business-as-usual control to create two equal comparators. When students engaged in group concept mapping, their quiz performance was significantly better (small to moderate effect) compared to their performance in the business-as-usual control condition [94 (5)% vs 91 (6)%, d=0.57, p<.001]. However, there was no significant difference in the students’ predicted scores on quizzes [83 (10)% vs 82 (10)%, d=0.08, p=.06]. We then examined the under-confidence and overconfidence of students within each condition. When students engaged in group concept mapping, they showed a small but significantly higher degree of under-confidence compared to the business-as-usual control [-11 (10)% vs -9 (11)%, d=0.18, p=.002]. However, there was no difference in their accuracy [15 (9)% vs 14 (8)%, d=0.10, p=.1].

This analysis was repeated by examining the three conditions: group concept mapping with feedback (four topics), group concept mapping without any feedback (three topics), and business-as-usual control (seven topics). The greatest difference in quiz performance was observed with group concept mapping without feedback (95%, d=.36 vs feedback, d=.66 vs control), followed by group concept mapping with feedback (93%, d=.36 vs control), and lastly the business-as-usual control condition (91%). There were no differences between conditions for students’ predictive scores. For topics in which students completed the group concept mapping and received no feedback, they showed more under-confidence (-12%) compared to the control condition (-9%, d=.26). In addition, the topics for which students received no feedback after group concept mapping resulted in less accuracy of prediction (14%) compared to the control condition (11%, d=.31) (Table 1).

Because metacognitive judgments are impacted by practice, potential time effects were examined but yielded no significant linear trend for quiz scores (Figure 1) or prediction of scores over time. However, the second quiz demonstrated significantly lower predictions than any other quiz (Figure 1). There was no linear trend in bias, which is a measure of overconfidence or under-confidence, but students started at and remained under-confident throughout (Figure 2A). When examining accuracy, there was a near significant linear trend (p=.06) showing improved accuracy over time (Figure 2B).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Quiz performance and students’ predicted quiz performance over the course of the semester. Group concept mapping with feedback is gray, group concept mapping without feedback is white, and business-as-usual-controls are black.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Metacognitive changes in under-confidence (A) and accuracy (B) between actual and predicted quiz scores in pharmacy students who participated in a concept mapping activity to enhance metacognition and comprehension of fundamental disease state knowledge. Group concept mapping with feedback is gray, group concept mapping without feedback is white, and business-as-usual-controls are black.

DISCUSSION

These results demonstrated improved quiz performance when pharmacy students participated in concept mapping, but concept mapping did not impact students’ metacognition. Despite an increase in actual quiz scores in conditions 1 and 2, students consistently predicted the same scores across all three conditions. The greater discrepancy between actual and predicted quiz score was evident by greater bias and absolute bias when comparing scores for both conditions 1 and 2 with those for condition 3, though only changes in scores under condition 2 met statistical significance. However, it is interesting to consider whether students’ poorer accuracy in predicting score could reflect improved metacognition. Underpredicting quiz scores to a greater degree in the concept mapping conditions could indicate that students had more awareness of the material they were unfamiliar with, and self-awareness is an important component of metacognition.

Although quiz performance was significantly higher when students completed concept mapping compared to the business-as-usual control, it is important to consider if this difference is significant enough for students to engage in concept mapping independently. A score of 94% vs 91% on a 10-point quiz may not represent a meaningful difference to students. Perhaps this was demonstrated by only 12 students reporting use of concept mapping for at least one disease state assigned to condition 3. The low rate of independent concept mapping in condition 3 could also be attributed to this being an unfamiliar study technique, and students feeling more comfortable using their previous study methods. Alternatively, students found other study techniques that were more efficient or did not prioritize concept mapping for a pre-class assignment when managing their other coursework.

Future study directions should include the impact of concept mapping on comprehensive examination performance, long-term retention, integration of information, and self-regulated learning. Should concept mapping be used more in the future, a challenge posed to educators is ensuring standardized and objective evaluation of students’ maps.¹⁵ Kolar and colleagues developed the grading tool that we utilized, though it is nonspecific and subject to different interpretations by different evaluators.¹⁰

Though our study ultimately showed positive results from pharmacy students using concept mapping, it did have limitations. Disease states were not distributed evenly among the three conditions, which could lead to potential sampling bias and variations in topic difficulty. We were unable to counterbalance the topics because of predetermined course schedules and the need to align topics with patient care laboratory activities. This limitation is typical of a study designed around a real world, preexisting pharmacy course. Quizzes were developed by the respective content experts for each disease state, further contributing to variation in difficulty. Additionally, the number of questions included in the quizzes varied; however, all quizzes were normalized to 10 points. Finally, students made metacognitive judgements about their overall quiz performance (a global judgement); therefore, results may have differed if students made topic-level judgements (eg, pharmacology, therapeutics, epidemiology, etc) within the quiz.

This study does have several strengths. Students served as their own control in this within-subject design. The within-subject study design improved our ability to assess individual students’ progress throughout the different conditions. It would be unethical to randomize students to different learning conditions in a core pharmacotherapy class where the intent is for all students to have similar learning experiences. Data analysis was adjusted for time to account for students becoming better at predicting quiz scores with more practice as the semester progressed. Additionally, students were assigned at random to the small laboratory groups they worked in to develop their concept maps, promoting an even distribution of historically high- and low-performing students.

CONCLUSION

Pharmacy students must acquire the skills needed for them to become critical thinkers and self-directed, lifelong learners; however, teaching these skills has proven to be challenging for pharmacy faculty. In this study, concept mapping activities improved students’ quiz performance but did not affect their metacognition as measured by bias and absolute bias. Future studies are needed to assess the utility of concept mapping in pharmacy education and students’ perceived barriers.