Abstract
Objective. To evaluate the metacognitive abilities of pharmacy students and determine whether introducing the concept along with team-based learning (TBL) enhances metacognition.
Methods. Pharmacy students completed a Metacognitive Awareness Inventory (MAI) and a low-stakes pretest during the first class that evaluated students’ knowledge about the therapeutic concepts that would be taught through TBL. The same questions were administered on the comprehensive final examination for the course. For each of the course assessments, students were asked to indicate their understanding of the topic and predict their performance. Actual performance was measured as a result of each assessment.
Results. The pre-MAI composite score was 77.3%. Scores significantly improved by the end of the course to 84.6%. There were significant differences in both declarative knowledge and conditional knowledge when evaluating performance groups. Students in the middle performance group demonstrated the greatest ability to predict their performance on the final examination. Though these were not significant, students in the low group overestimated their performance, while students in the high group underestimated their performance. Baseline grade point average was the only factor predictive of the final examination score and the final course grade.
Conclusion. Pedagogies such as TBL may support development of metacognitive skills in pharmacy students. However, intentional guidance provided by an instructor is required to improve pharmacy students’ regulation of cognition skills.
INTRODUCTION
Pharmacy students are expected to master a plethora of information and develop into lifelong learners. The importance of student learning awareness is supported by the Accreditation Council for Pharmacy Education’s (ACPE’s) Standards for the Doctor of Pharmacy Degree, which emphasizes the importance of actively engaging learners and promoting self-directed learning.1 Although the ACPE standards do not specifically address metacognition, the guidance acknowledges that life-long learning requires individuals to self-assess their learning needs. Metacognition broadly refers to how individuals think about thinking. It can be described more specifically as “the ability to monitor thinking to use skills and strategies appropriately to achieve a desirable outcome,” which is crucial in controlling and guiding thinking.2,3 Metacognition consists of three components, including knowledge, monitoring, and control.4 It accounts for the students’ awareness of how they learn (metacognitive knowledge and beliefs), how they assess their learning and performance (metacognitive monitoring), and their ability to regulate their understanding (metacognitive control). Students at the lowest performance levels tend to overestimate their abilities compared to high- and mid-level performers (Dunning-Kruger Effect).5 Metacognitively aware learners are more strategic, predict their performance more accurately, and excel further than unaware learners.4,6,7 These students use their study time most efficiently, narrowing their focus to specific areas and maximizing their preparedness.8 In meeting the goals of a pharmacy program, students benefit from the use of metacognitive skills to solve new problems and appropriately evaluate their understanding of additional information.9 A key instructional strategy for metacognitive skills is “learning through guided experience.” This is achieved when teaching activities focus on both cognitive and metacognitive processes while performing a task.8,10 Team-based learning (TBL) is a pedagogy that provides a scaffolding, which may enhance metacognition. In TBL, students have the potential to improve their awareness of how they learn, assess their learning by predicting their performance, and build more context-independent knowledge and skills through cooperative learning. During TBL, students complete an individual readiness assessment test (IRAT) followed by a team readiness assessment test (TRAT) using the Immediate Feedback Assessment Technique (IF-AT). When employing the IF-AT, students use a scratch-off card, which makes the assessment process part of the learning cycle and allows them to finish the assessment knowing the correct answers.11 While discussing the TRAT and then completing in-class activities, teams reinforce their knowledge and problem-solving skills through deliberate cooperative processes. As this process is repeated multiple times in a semester, the students gain practice in preparing and then monitoring their level of preparation (ie, metacognitive knowledge, beliefs, and monitoring). Stewart and colleagues reported that students who were given the ability to self-assess using practice quizzes, which provided immediate feedback, saw an improvement in their examination scores.12 Feedback was given in a way that encouraged students to provide the rationale for each correct and incorrect answer. We hypothesize that the same benefit may be gleaned from the TBL process because the TRAT involves discussion within the group and rationale is frequently provided when a question is answered incorrectly. We are not aware of any reports of the impact of the TBL process on metacognition or on the ability of students to assess their understanding when they are given repeated opportunities to see if their expected performance matches their actual performance.
The purpose of this study was to evaluate the metacognitive skills (including metacognitive knowledge, beliefs, and monitoring) of pharmacy students within a Pharmacotherapeutics of Cardiovascular Disease course, and to determine if introducing metacognition and performing a metacognition inventory along with team-based learning can enhance metacognitive knowledge/beliefs and monitoring. These findings will enhance our ability to promote life-long learning skills in pharmacy students who will be entering a profession that requires these skills for continuing professional competence.
METHODS
Study Participants
This study involved all students enrolled in The Principles of Cardiovascular Therapeutics Course at a research-intensive public university. This was a modular course that included medicinal chemistry, pharmacology, and therapeutic concepts. There were 11 different instructors, and the course was a seven-week four-credit hour course offered at the end of the second professional year of the Doctor of Pharmacy (PharmD) program. The course incorporated six TBL sessions for selected therapeutics material, each of which required students to complete pre-work. At the beginning of a TBL session, students completed an IRAT in paper format, followed by the same questions administered to the team using the Immediate Feedback Assessment Tool scratch-off cards (Epstein Educational Enterprises, Cincinnati, Ohio). The scratch-off cards provided students with immediate feedback regarding their performances. Students spent the remainder of the two-hour class session solving case-specific problems using the Michaelsen’s 4S model of TBL (ie, problem significant to students, same question, specific answer, simultaneous report).13
This study included a low-stakes pretest during the first class that evaluated students’ knowledge about the therapeutic concepts that would be taught through TBL. Students did not review their performance on the pretest. The same questions were administered on the comprehensive final (posttest) examination for the course, all of which used the LockDown Browser (Respondus Learning Tool for Learning Systems). During the first-class session, metacognition was introduced as a concept that aids in developing effective study habits and improving course performance. Afterward, the Metacognitive Awareness Inventory (MAI) was administered as a survey through the Blackboard Learning Management System (Blackboard, Washington, DC), which provides only aggregate data and not student level data.7 The survey was distributed again at the end of the semester through the Respondus Lockdown Browser, which does provide student-level data. The MAI provides a validated and reliable initial test of a student’s metacognitive awareness, knowledge, and beliefs. Using this tool helps students identify areas for improving their own metacognitive skills. It can also help faculty identify lower-performing students who display comprehension monitoring deficiencies.6 The MAI is a 52-item survey that addresses knowledge of cognition and regulation of cognition. Knowledge about cognition is divided into procedural (four questions), declarative (eight questions), and conditional (five questions). Regulation of cognition is divided into planning (seven questions), information management strategies (10 questions), comprehension monitoring (seven questions), debugging strategies (five questions), and evaluation strategies (six questions).
For each of the course assessments (pretest, comprehensive final, and TBL IRAT), students were asked to indicate their understanding of the topic and predict their performance on a scale of 0 to 100%. Actual performance was measured as a result of each assessment. Additionally, students were asked to indicate the amount of time they spent preparing prior to each TBL session.
Following approval by the Wayne State University Institutional Review Board, data were extracted and collected using Excel 2013. The following information was collected: grade point average, final course grade, performance on the course pretest and posttest, IRAT scores for each session, MAI pretest and posttest, prediction of performance at each assessment, and self-assessed understanding of the material.
Data Analysis
Metacognitive knowledge and beliefs were measured using MAI data.4,6 Pre- and post-MAI data were compared using composite class scores with the chi-square test. Pearson correlation was used to determine the relationship between the MAI and scores on the final examination. Metacognitive monitoring was measured using performance predictions in two different ways to yield more concrete conclusions. First, monitoring accuracy was used to compare global judgments (examination performance) and concept judgements (IRAT performance) with student performance. Bias was calculated to provide measures of overconfidence or under confidence, and absolute bias was calculated to provide an estimate of how discrepant the judgment magnitude was from actual performance.14,15 Global bias was calculated by subtracting the actual test score from the student’s predicted test score. Concept bias was calculated by subtracting the IRAT score from student’s predicted IRAT score. Absolute bias was calculated for global and concept bias by computing the absolute value of the respective bias score. Relative accuracy was also used to provide correlations between student judgments and performance.4,8 The Pearson correlation coefficient (Pearson’s r) was used to evaluate the accuracy of predictions across the course. These were then compared using the coefficient of determination (R2) to determine if there were any changes in accuracy across the assessments. Linear regression was also used to evaluate the relationship between self-assessment of knowledge, hours spent studying, and actual performance. Students were divided into three performance groups (high, medium, and low) of equal size based on their final examination scores to determine the predictive accuracy for each performance group. This was done to validate the Dunning-Kruger effect.5 Two-way repeated measures analysis of variance was used to determine predictive factors for the final examination scores and final course grades. All data analysis was completed using JMP Software, version 11 (SAS Institute, Inc). Data were presented as mean±SD unless otherwise noted. Statistical significance was defined as p<.05.
RESULTS
One hundred students were enrolled in the course. More than half the class (66%) was female, and the mean grade point average at the beginning of the course was 3.2 (±0.4). Results were organized according to the following three research questions: What is the metacognition of pharmacy students? What is the impact of TBL on metacognition in pharmacy students? What factors impact accuracy of predictions (performance group, hours spent studying, etc)?
The pre-MAI composite score was 77.3% and improved significantly by the end of the course to 84.6% (p<.001). The class significantly improved in all four categories for the knowledge of cognition by the end of the course; however, planning was the only category in which there was a significant improvement for the regulation of cognition (Table 1). There were significant differences in both declarative knowledge and conditional knowledge when evaluating performance groups. For declarative knowledge, there was a difference of 1.3±0.4 between top and bottom performers and 1.1±0.4 between top and middle performers (both p values less than .01). Conditional knowledge had a difference of 0.5±0.2 between top and bottom performers and 0.4±0.2 between top and middle performers, with the p values of .03 and .04, respectively.
Pharmacy Students’ Composite Scores on the Metacognitive Awareness Inventory Before and After Participating in Team-based Learning to Promote the Development of Metacognitive Awareness and Monitoring
The mean pretest global bias for the class was -12±20 (indicating under confidence), and the mean posttest global bias was -11±12 (also indicating under confidence). Absolute global bias was 20±12 for the pretest and 12±11 for the posttest. Concept level bias was -9±24 (underconfident) on the pretest and -7±19 on the posttest (underconfident). Absolute concept level bias was 22±13 for the pretest and 13±15 for the posttest. Relative accuracy for each of the assessments, calculated using the coefficient of determination (R2), are shown in Table 2 (increasing R2 values indicated that the accuracy of predicted performance increased with each TBL quiz). The R2 also increased from the pre-examination to the final examination, though the R2 value was not significant for the pre-examination. Self-assessment of knowledge only correlated to actual performance on the fourth and fifth TBL sessions (R2 0.1 and 0.04, respectively; p<.05).
Regression Analysis of Pharmacy Students’ Predicted Performance Using Actual Performance as the Outcome Variable to Evaluate the Relative Accuracy of Metacognitive Monitoring
Students in the middle performance group demonstrated the greatest ability to predict their performance on the final examination (R2 0.1, p<.02). Though not statistically significant, students in the low performance group overestimated their performance, while students in the high-performance group underestimated their performance (Figure 1). There was no correlation between students’ reported hours of preparation and actual performance, but it did correlate with students’ predicted performances on all TBL assessments. Pearson correlations were 0.04, 0.08, 0.16, 0.025, and 0.13, respectively (p<.05). The MAI was not predictive of the final examination score or the final course grade. Baseline GPA was the only factor predictive of the final examination score and the final course grade (R2 0.5 and 0.7, respectively; p<.001).
Mean Performance and Student Predicted Performance on the Final Examination by Performance Groups
DISCUSSION
The major components of metacognition include knowledge and beliefs about cognition (awareness of one’s strengths and weaknesses), monitoring of cognition (ability to monitor performance), and regulation of cognition (ability to implement strategies to increase understanding and performance).7 Our study demonstrates that students can improve their ability to predict performance (confidence) with practice. However, in this study, on average, the students’ regulation of cognition did not improve significantly with practice. This may be improved with intentional guidance through an instructor-provided plan of study, but further studies are needed to better understand its effectiveness on metacognitive regulation. We noted improvement in the pharmacy students’ overall composite metacognition and knowledge of cognition over a semester. This improvement was not seen in regulation of cognition, which seems to be consistent with most studies evaluating the major components of metacognition.
Karpicke and colleagues evaluated study strategies in undergraduate students to determine their metacognition. The students were proficient in their knowledge about cognition but were unable to regulate and monitor their own learning.16 In another study, researchers evaluated pharmacy students’ ability to predict examination performance and identify individual incorrect items. The students accurately predicted their performance but had limited success in identifying which items they missed.17 Persky and Dinsmore demonstrated that student’s metacognitive judgements do not correlate with improved regulation of cognition, reinforcing the limitation in students’ abilities to regulate their learning.18 These findings suggest that developing methods to enhance learning (regulation) is not an innate skill and requires intentional interventions within the curriculum.
Team-based learning has been proposed as a successful method to improve regulation of cognition. Richmond suggested that the immediate feedback students receive during TBL with a TRAT may increase metacognitive awareness, allow students to calibrate their cognition in real time, and increase regulation of cognition.19,20 Several studies have also emphasized the value of immediate feedback for monitoring of learning.21,22 This study demonstrated that TBL had an impact on prediction performance over the semester, which could suggest that students were evaluating their learning. Our findings support that students predict their performance better over time with TBL. In contrast, regulation of cognition did not improve in our study, with the exception of planning. Planning is the initial step in regulation of cognition and is heavily supported by the TBL structure of completing pre-work to prepare for assessments. Managing information, comprehension, and debugging strategies are higher-level skills and require a greater level of self-regulation and may require more guidance.23
Although student predictions improved with practice, not all students were equally proficient at predicting their performance. In fact, the students with the lowest performance expressed overconfidence, while those with the highest performance expressed under confidence. Though this may sound like a paradoxical finding, this concept is well elucidated in the literature. Kruger and Dunning found that participants with performance in the bottom quartile had overestimated their performance because of a lack of metacognitive skills. When participants improved their metacognitive skills, they were no longer overconfident and were able to recognize previous errors. Similar to our study, these investigators also found an underestimation of performance by high performers.5 Previous research also demonstrates that performance is closely related to high-quality, deliberate practice.24 Such practice requires regulation of cognitive tasks, which could explain why the amount of time that students reported they spent studying did not correlate with improved performance. While students who spent more time studying predicted they would do better on assessments, the lack of higher-level metacognitive skills found in this study further supports the need for deliberate instructional methods to further develop regulation of cognition.
There were several factors that could be confounders in our study, suggesting that differences were not just a result of the TBL and metacognitive activities. First, the volume of pre-work and difficulty of content differed from one TBL session to the next. Upon further analysis, a pyramid learning model was applied for the TBL sessions because some of the previous material served as a foundation for newer topics. Pyramidal sequential processes can either enhance or break down performances depending on students’ foundational knowledge. Next, when students were asked to report their preparation, they were not provided with a clear definition of what constituted “time spent studying.” As such, some students reported studying over 20 hours per day, which is unlikely. Finally, there was missing data for one of the quizzes and the pre-MAI did not provide student-specific data. Although the missing quiz data does not deter from the overall improvement in predictions, the inability to compare pre- and post-MAI data did prevent further analysis of factors that may improve MAI scores over the curriculum.
Despite these limitations, valuable information was gained from this study. Metacognitive skills are emphasized as a critical part of pharmacy curriculum. Accreditation standards promote self-awareness, self-directed lifelong learning, and continuous professional development (standard 4 and 9 of ACPE’s Standards 2016).1 These are key factors that are essential to the metacognitive process and help students regulate their activities over time to reach desired goals. As such, faculty need to be diligent in enhancing learners’ metacognition in the classroom by aligning metacognitive processes with cognitive processes. Previous researchers have recommended activities that can improve regulation of cognition, including self-questions to promote monitoring and evaluation of cognition, thinking out loud, examination assessments, and written feedback.2,25 Regulation may also be improved within TBL sessions, first by asking students to predict their performance with immediate feedback. This allows students to more accurately assess their preparation and knowledge of cognition. Additionally, by creating intentional opportunities to improve cooperative learning during TRAT and in-class activities, students can observe the judgements of team members or faculty experts and revise their own thinking. These strategies may be particularly helpful for low-performing students.
CONCLUSION
Developing metacognition skills in student pharmacists promotes their life-long learning skills in a profession that requires these skills for continuing professional competence. In this study, student pharmacists demonstrated improvement in knowledge of cognition but lacked changes in regulation of cognition. Pedagogies such as TBL can support development of these skills. However, deliberate and intentional guidance is required to improve regulation of cognition skills.
- Received April 15, 2020.
- Accepted October 16, 2020.
- © 2021 American Association of Colleges of Pharmacy