Abstract
Objective. To determine whether third year Doctor of Pharmacy students’ self-reported use of optional supplemental material impacted their ability to accurately predict their performance on a low-stakes assessment.
Methods. An instructor created optional supplemental material in the form of an online quiz. Students were asked to report whether they used the supplemental material and to predict and postdict their performance on an in-class assessment. The relative accuracy of the predictions and postdictions as well as the assessment grades and overall course grades were compared between students who reported using the supplemental material and those who reported not using the supplemental material.
Results. More than half of the students (60%) reported using the supplemental material. Most students underpredicted their performance on the in-class assessment, but there was no difference in the accuracy of predictions based on supplemental material use or non-use (-1.2 vs -1.0) or on the postdictions (-1.3 vs. -1.0). Students who reported using the supplemental material performed better on both the low-stakes assessment (7.7 vs 7.2 out of 10) and overall in the course (87.0% vs 84.9%).
Conclusion. Pharmacy students’ self-reported use of optional supplemental material does not appear to impact their ability to accurately predict their performance on a low-stakes assessment.
INTRODUCTION
Metacognition is higher-order thinking that involves knowledge of cognition and also regulation of cognitive processes related to learning.1 Metacognition comprises self-reflection of one’s knowledge, goals, strategies, and results.2 This is a significant phenomenon because it can apply to every aspect of life including reading, oral skills, writing, language acquisition, memory, attention, social interactions, personal development, and education.3 When students perceive their ability as higher than it is (overconfident), then they may terminate studying too early and not reach the desired level of competency.4 Those without developed metacognitive skills lack the ability to recognize their insufficiencies and tend to overestimate their performance.4 Students with more self-awareness can leverage their metacognitive skills to perform better.2 The development of metacognition in pharmacy students is especially important as these learners will need to continue to improve their knowledge and skills through independent study and practice even after graduation to achieve and maintain competence in pharmacy practice.
While not explicitly addressed, the importance of including metacognition in the Doctor of Pharmacy (PharmD) curriculum appears inherent in the 2016 Accreditation Council for Pharmacy Education (ACPE) Standards definition of self-awareness (Key Element 4.1): “The ability to examine and reflect on personal knowledge, skills, abilities, beliefs, biases, motivation, and emotions that could enhance or limit personal and professional growth.”5 Incorporation of metacognitive skills into the PharmD program is vital as they are important skills for students and postgraduates as they transition from novice practitioners to more seasoned pharmacists.
The study of metacognition within pharmacy education is a growing area of interest, particularly in ways of assessing, developing, and providing learner feedback on metacognition.4,6-8 Fox and colleagues aimed to develop metacognitive skills in pharmacy students as a method to increase self-awareness.7 The study used calibration, the correlation between student judgement of performance and actual performance, as a method of assessing and developing metacognition. The researchers asked students to provide posttest predictions, known as postdictions, on a series of multiple-choice examinations to determine whether continued feedback on postdicted vs actual performance could improve accuracy of the postdictions.7 While students were overall poor postdictors of their performance, with most of the class being underconfident, the number of overpredictions was significantly reduced between the first and last examination of the course.7 Fox and colleagues also found that higher performing students were more likely than lower performing students to be underconfident.7 Similarly, Schneider and colleagues asked second year students to postdict their performance on a summative examination as well as identify those questions they felt certain they had answered incorrectly.8 They found that most students significantly underpredicted their examination performance and higher-performing students were more accurate in their postdictions.8 These studies indicate that PharmD students are likely underconfident about their academic performance in general, and this is consistent with findings in other educational settings, ie, that high performing students tend to be underconfident in their academic performance.9 However, the studies by Fox and colleagues and Schneider and colleagues studies were limited to assessing only postdictions and not predictions, and both used high-stakes assessments for the calibration efforts. These studies also did not assess factors that might help identify students who displayed better calibration other than higher academic performance.
The studies by Fox and colleagues and Schneider and colleagues, in conjunction with ACPE’s standard 4.1, led us to question what other factors impact pharmacy students’ metacognitive skills in the form of calibration accuracy. Our prior work demonstrated that there was a correlation between students who demonstrated better self-regulated learning and students who accessed faculty member-developed optional supplemental course material.10 We hypothesized that student use of supplemental material would not only correlate with better self-regulated learning, but also with better metacognition in the form of calibration accuracy. If demonstrated to be correlated, pharmacy educators could use this important finding to help students develop their metacognition in the form of calibration. Therefore, the purpose of this study was to gauge the relationship between student-reported use of supplemental material and accuracy of calibration (both prediction and postdiction) on a low-stakes assessment in a PharmD therapeutics course.
METHODS
This study was conducted in two successive spring semesters (2018 and 2019) in a college of pharmacy at a large public university. This prospective study was voluntary and invited participation from all third professional year PharmD students across the college’s three campuses who were enrolled in the required course Patient Care 8: Complex Patients. Two hundred twenty-nine students and 260 students were invited in 2018 and 2019, respectively. Students were given access to supplemental material, which consisted of a set of 36 optional practice questions designed by three subject matter expert faculty members in a module on critical care. The supplemental material, in addition to the required learning material, was made available to students one week before a scheduled active learning session that included a low-stakes assessment known as an individual readiness assurance test (iRAT). The required learning materials included instructor-led video lectures and readings. The supplemental practice questions were uploaded to Quizlet (quizlet.com),11 a mobile and web-based application that allowed students to study information using flashcards and fill-in-the-blank questions as well as other learning tools. The supplemental content included recall questions on critical care topics, such as sepsis and sedation. The students could access both required and supplemental materials repeatedly if desired. Further details about the supplemental material have been previously published.10
The iRAT consisted of 10 multiple-choice questions based on the topics covered in the required material and reinforced in the supplemental material. Each question was worth one point for a possible total of 10 points. The iRAT was completed using the quiz function in the course’s learning management system. Before the iRAT, students were asked in an electronic survey designed in Qualtrics if they used the supplemental material and to predict how many points they would score on the iRAT. They were then given 10 minutes to complete the iRAT using the electronic learning management system. After completing the iRAT, they were asked to postdict how many points out of 10 they earned. The prediction and postdiction results were then compared to actual iRAT scores based on self-reported use of the supplemental material. Performance in the course overall based on self-reported use of the supplemental material was analyzed as well. Overall course grades were determined by performance on six iRATs, six team readiness assurance tests, two examinations, and three additional assignments. All students who registered a prediction and/or postdiction were included in the analysis. The only students who were excluded were those who were absent from the ALS in which the iRAT was administered and those who submitted neither a prediction nor a postdiction.
The difference between predicted iRAT scores and actual iRAT scores, as well as overall course grades were compared on the basis of use/non-use of the supplemental material, with one group being the students who used the supplemental material and the other group being the students who did not use the supplemental material. An equal variance two-sample t test to determine differences in relative accuracy between groups was performed. Relative accuracy was calculated as the difference between the predicted iRAT score and the actual iRAT score, where a negative number would indicate underconfidence in performance and a positive number would indicate overconfidence in performance. For all statistical analyses, JMP was used (jmp.com), with a 2-sided p-value less than .05 considered to be statistically significant. The study was deemed exempt by the local institutional review board.
RESULTS
Across both semesters, 489 students were invited to participate in the study. Most of the students in the cohorts identified as female (59%). Most of the invited students identified as White (42%), followed by Hispanic (22%), Asian (20%), Black (8%), Native American/Hawaiian (2%), and those not reporting ethnicity (6%). The mean (SD) age of the invited students was 25 (4.1) years. Of the 489 students, 453 students registered either a prediction or postdiction or both (93% participation). Of these, use of the optional supplemental material was reported by 273 (60%) students. Use of the supplemental material by students resulted in significantly higher iRAT scores (7.7 vs 7.2 out of 10; p=.004) (Figure 1). Overall, students underpredicted their iRAT scores with a mean relative accuracy of prediction (predicted score minus actual score) of -1.2 in the supplemental material group vs -1.0 in those who did not use supplemental material (p=.47) (Figure 2). The mean relative accuracy of postdiction was -1.3 in the supplemental material group vs -1.0 in those who did not use supplemental material (p=.) (Figure 3). Students who self-reported use of the supplemental material had significantly higher performance in the course overall (87.0 vs 84.9 out of 100; p<.001) (Figure 4).
Pharmacy students’ performance on an Individual Readiness Assurance Test (iRAT) based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the subject matter (n=453). The horizontal line represents the mean score of all students on the iRAT.
Pharmacy students’ relative accuracy of prediction of Individual Readiness Assurance Test (iRAT) score based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the material (n=453). The horizontal line represents the mean relative accuracy of all students on the iRAT.
Pharmacy students’ relative accuracy of postdiction of Individual Readiness Assurance Test (iRAT) score based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the material (n=418). The horizontal line represents the mean relative accuracy of all students on the iRAT.
Pharmacy students’ overall course grade based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the material (n=453). The horizontal line represents the mean course grade for all students.
DISCUSSION
This study examined the relationship between student-reported use of optional supplemental material and calibration ability on a low-stakes assessment in a therapeutics course. This study is important because development of calibration ability as a form of metacognition could lead to improved self-awareness and self-regulated learning. Overall, students were mostly underconfident when asked to predict and postdict their performance, but we did not find a difference in calibration accuracy between the group that reported using the supplemental material and the group that did not. However, like our results in a previous study, students who reported using the supplemental material performed better on the low-stakes assessment and in the course overall.10
Calibration as a method of metacognitive development has been investigated in PharmD students in prior studies. Like Schneider and colleagues and Fox and colleagues, we found that pharmacy students are generally underconfident when asked to predict or postdict their performance on either low-stakes assessments or high-stakes assessments, which is consistent with metacognition literature that suggests higher-performing students tend to be underconfident as compared to lower-performing students.7-9 This slight under-confidence may also be a result of student concern for patient safety. While these studies looked at the difference in calibration for higher vs lower performing students, we added the additional analysis layer of student-reported use of supplemental material, albeit not finding a difference based on supplemental material use.
The ability for students to predict their performance accurately can depend on a number of factors. High levels of stress as well as task difficulty have been shown to decrease metacognitive processes and accuracy.9,12 Therefore, we chose to attempt to correlate supplemental material use and calibration on a low-stakes assessment rather than a high-stakes assessment to try to minimize the risk of inaccurate calibration due to stress and a difficult examination. However, the heavy course load in the PharmD curriculum, stress outside of the coursework, and/or the difficulty of critical care material may have still impaired the students’ ability to calibrate well. Additionally, gender identification appears to play a role in confidence and metacognitive accuracy, with individuals identifying as female being less confident.13 Given that more than half of the students identified as female, gender identification could have potentially played a role in the underconfidence shown in the study.
The strengths of this study include the large sample size and diversity of student cohort. Limitations of the study include the reliance on students to try to accurately predict and postdict their scores on a single assignment. If students were given additional calibration opportunities, their calibration ability might improve, as it did in the study by Fox and colleagues.7 An additional limitation lies in relying on student self-report of using the supplemental material. Some students may not have wanted to admit to using (or not using) the material for a variety of reasons.
Given these findings that student-reported use of supplemental material does not seem to correlate with better calibration on a low-stakes assessment, future research is needed to determine if repeat calibration attempts on low-stakes assessments will lead to more accurate calibration. Additionally, further research is needed regarding different ways of assessing and improving student metacognition and self-awareness.
CONCLUSION
In this study, the use of instructor-created supplemental material was significantly associated with improved performance on a low-stakes assessment and in the course overall. However, the majority of students were underconfident before and after the assessment, and the use of supplemental material was not associated with better calibration. Although improvement in pharmacy students’ calibration skills was not observed, students who self-reported use of the supplemental material performed better, indicating enhanced learning. Further studies surrounding repeated opportunities for pharmacy students to practice calibration and other methods of developing metacognitive skills are needed.
- Received February 24, 2021.
- Accepted May 28, 2021.
- © 2021 American Association of Colleges of Pharmacy
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.