Skip to main content

Main menu

  • Articles
    • Current
    • Early Release
    • Archive
    • Rufus A. Lyman Award
    • Theme Issues
    • Special Collections
  • Authors
    • Author Instructions
    • Submission Process
    • Submit a Manuscript
    • Call for Papers - Intersectionality of Pharmacists’ Professional and Personal Identity
  • Reviewers
    • Reviewer Instructions
    • Call for Mentees
    • Reviewer Recognition
    • Frequently Asked Questions (FAQ)
  • About
    • About AJPE
    • Editorial Team
    • Editorial Board
    • History
  • More
    • Meet the Editors
    • Webinars
    • Contact AJPE
  • Other Publications

User menu

Search

  • Advanced search
American Journal of Pharmaceutical Education
  • Other Publications
American Journal of Pharmaceutical Education

Advanced Search

  • Articles
    • Current
    • Early Release
    • Archive
    • Rufus A. Lyman Award
    • Theme Issues
    • Special Collections
  • Authors
    • Author Instructions
    • Submission Process
    • Submit a Manuscript
    • Call for Papers - Intersectionality of Pharmacists’ Professional and Personal Identity
  • Reviewers
    • Reviewer Instructions
    • Call for Mentees
    • Reviewer Recognition
    • Frequently Asked Questions (FAQ)
  • About
    • About AJPE
    • Editorial Team
    • Editorial Board
    • History
  • More
    • Meet the Editors
    • Webinars
    • Contact AJPE
  • Follow AJPE on Twitter
  • LinkedIn
Research ArticleRESEARCH

Impact of Supplemental Material Use on Student Metacognitive Monitoring and Calibration

Lindsey M. Childs-Kean, Jennifer Rodriguez, Aaron O. Thomas and Stacy A. Voils
American Journal of Pharmaceutical Education November 2021, 85 (10) 8614; DOI: https://doi.org/10.5688/ajpe8614
Lindsey M. Childs-Kean
aUniversity of Florida, College of Pharmacy, Gainesville, Florida
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jennifer Rodriguez
aUniversity of Florida, College of Pharmacy, Gainesville, Florida
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Aaron O. Thomas
bUniversity of Florida Information Technology, Gainesville, Florida
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Stacy A. Voils
aUniversity of Florida, College of Pharmacy, Gainesville, Florida
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • PDF
Loading

Abstract

Objective. To determine whether third year Doctor of Pharmacy students’ self-reported use of optional supplemental material impacted their ability to accurately predict their performance on a low-stakes assessment.

Methods. An instructor created optional supplemental material in the form of an online quiz. Students were asked to report whether they used the supplemental material and to predict and postdict their performance on an in-class assessment. The relative accuracy of the predictions and postdictions as well as the assessment grades and overall course grades were compared between students who reported using the supplemental material and those who reported not using the supplemental material.

Results. More than half of the students (60%) reported using the supplemental material. Most students underpredicted their performance on the in-class assessment, but there was no difference in the accuracy of predictions based on supplemental material use or non-use (-1.2 vs -1.0) or on the postdictions (-1.3 vs. -1.0). Students who reported using the supplemental material performed better on both the low-stakes assessment (7.7 vs 7.2 out of 10) and overall in the course (87.0% vs 84.9%).

Conclusion. Pharmacy students’ self-reported use of optional supplemental material does not appear to impact their ability to accurately predict their performance on a low-stakes assessment.

Keywords
  • learning assessment
  • metacognition
  • calibration
  • supplemental material

INTRODUCTION

Metacognition is higher-order thinking that involves knowledge of cognition and also regulation of cognitive processes related to learning.1 Metacognition comprises self-reflection of one’s knowledge, goals, strategies, and results.2 This is a significant phenomenon because it can apply to every aspect of life including reading, oral skills, writing, language acquisition, memory, attention, social interactions, personal development, and education.3 When students perceive their ability as higher than it is (overconfident), then they may terminate studying too early and not reach the desired level of competency.4 Those without developed metacognitive skills lack the ability to recognize their insufficiencies and tend to overestimate their performance.4 Students with more self-awareness can leverage their metacognitive skills to perform better.2 The development of metacognition in pharmacy students is especially important as these learners will need to continue to improve their knowledge and skills through independent study and practice even after graduation to achieve and maintain competence in pharmacy practice.

While not explicitly addressed, the importance of including metacognition in the Doctor of Pharmacy (PharmD) curriculum appears inherent in the 2016 Accreditation Council for Pharmacy Education (ACPE) Standards definition of self-awareness (Key Element 4.1): “The ability to examine and reflect on personal knowledge, skills, abilities, beliefs, biases, motivation, and emotions that could enhance or limit personal and professional growth.”5 Incorporation of metacognitive skills into the PharmD program is vital as they are important skills for students and postgraduates as they transition from novice practitioners to more seasoned pharmacists.

The study of metacognition within pharmacy education is a growing area of interest, particularly in ways of assessing, developing, and providing learner feedback on metacognition.4,6-8 Fox and colleagues aimed to develop metacognitive skills in pharmacy students as a method to increase self-awareness.7 The study used calibration, the correlation between student judgement of performance and actual performance, as a method of assessing and developing metacognition. The researchers asked students to provide posttest predictions, known as postdictions, on a series of multiple-choice examinations to determine whether continued feedback on postdicted vs actual performance could improve accuracy of the postdictions.7 While students were overall poor postdictors of their performance, with most of the class being underconfident, the number of overpredictions was significantly reduced between the first and last examination of the course.7 Fox and colleagues also found that higher performing students were more likely than lower performing students to be underconfident.7 Similarly, Schneider and colleagues asked second year students to postdict their performance on a summative examination as well as identify those questions they felt certain they had answered incorrectly.8 They found that most students significantly underpredicted their examination performance and higher-performing students were more accurate in their postdictions.8 These studies indicate that PharmD students are likely underconfident about their academic performance in general, and this is consistent with findings in other educational settings, ie, that high performing students tend to be underconfident in their academic performance.9 However, the studies by Fox and colleagues and Schneider and colleagues studies were limited to assessing only postdictions and not predictions, and both used high-stakes assessments for the calibration efforts. These studies also did not assess factors that might help identify students who displayed better calibration other than higher academic performance.

The studies by Fox and colleagues and Schneider and colleagues, in conjunction with ACPE’s standard 4.1, led us to question what other factors impact pharmacy students’ metacognitive skills in the form of calibration accuracy. Our prior work demonstrated that there was a correlation between students who demonstrated better self-regulated learning and students who accessed faculty member-developed optional supplemental course material.10 We hypothesized that student use of supplemental material would not only correlate with better self-regulated learning, but also with better metacognition in the form of calibration accuracy. If demonstrated to be correlated, pharmacy educators could use this important finding to help students develop their metacognition in the form of calibration. Therefore, the purpose of this study was to gauge the relationship between student-reported use of supplemental material and accuracy of calibration (both prediction and postdiction) on a low-stakes assessment in a PharmD therapeutics course.

METHODS

This study was conducted in two successive spring semesters (2018 and 2019) in a college of pharmacy at a large public university. This prospective study was voluntary and invited participation from all third professional year PharmD students across the college’s three campuses who were enrolled in the required course Patient Care 8: Complex Patients. Two hundred twenty-nine students and 260 students were invited in 2018 and 2019, respectively. Students were given access to supplemental material, which consisted of a set of 36 optional practice questions designed by three subject matter expert faculty members in a module on critical care. The supplemental material, in addition to the required learning material, was made available to students one week before a scheduled active learning session that included a low-stakes assessment known as an individual readiness assurance test (iRAT). The required learning materials included instructor-led video lectures and readings. The supplemental practice questions were uploaded to Quizlet (quizlet.com),11 a mobile and web-based application that allowed students to study information using flashcards and fill-in-the-blank questions as well as other learning tools. The supplemental content included recall questions on critical care topics, such as sepsis and sedation. The students could access both required and supplemental materials repeatedly if desired. Further details about the supplemental material have been previously published.10

The iRAT consisted of 10 multiple-choice questions based on the topics covered in the required material and reinforced in the supplemental material. Each question was worth one point for a possible total of 10 points. The iRAT was completed using the quiz function in the course’s learning management system. Before the iRAT, students were asked in an electronic survey designed in Qualtrics if they used the supplemental material and to predict how many points they would score on the iRAT. They were then given 10 minutes to complete the iRAT using the electronic learning management system. After completing the iRAT, they were asked to postdict how many points out of 10 they earned. The prediction and postdiction results were then compared to actual iRAT scores based on self-reported use of the supplemental material. Performance in the course overall based on self-reported use of the supplemental material was analyzed as well. Overall course grades were determined by performance on six iRATs, six team readiness assurance tests, two examinations, and three additional assignments. All students who registered a prediction and/or postdiction were included in the analysis. The only students who were excluded were those who were absent from the ALS in which the iRAT was administered and those who submitted neither a prediction nor a postdiction.

The difference between predicted iRAT scores and actual iRAT scores, as well as overall course grades were compared on the basis of use/non-use of the supplemental material, with one group being the students who used the supplemental material and the other group being the students who did not use the supplemental material. An equal variance two-sample t test to determine differences in relative accuracy between groups was performed. Relative accuracy was calculated as the difference between the predicted iRAT score and the actual iRAT score, where a negative number would indicate underconfidence in performance and a positive number would indicate overconfidence in performance. For all statistical analyses, JMP was used (jmp.com), with a 2-sided p-value less than .05 considered to be statistically significant. The study was deemed exempt by the local institutional review board.

RESULTS

Across both semesters, 489 students were invited to participate in the study. Most of the students in the cohorts identified as female (59%). Most of the invited students identified as White (42%), followed by Hispanic (22%), Asian (20%), Black (8%), Native American/Hawaiian (2%), and those not reporting ethnicity (6%). The mean (SD) age of the invited students was 25 (4.1) years. Of the 489 students, 453 students registered either a prediction or postdiction or both (93% participation). Of these, use of the optional supplemental material was reported by 273 (60%) students. Use of the supplemental material by students resulted in significantly higher iRAT scores (7.7 vs 7.2 out of 10; p=.004) (Figure 1). Overall, students underpredicted their iRAT scores with a mean relative accuracy of prediction (predicted score minus actual score) of -1.2 in the supplemental material group vs -1.0 in those who did not use supplemental material (p=.47) (Figure 2). The mean relative accuracy of postdiction was -1.3 in the supplemental material group vs -1.0 in those who did not use supplemental material (p=.) (Figure 3). Students who self-reported use of the supplemental material had significantly higher performance in the course overall (87.0 vs 84.9 out of 100; p<.001) (Figure 4).

Figure 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 1.

Pharmacy students’ performance on an Individual Readiness Assurance Test (iRAT) based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the subject matter (n=453). The horizontal line represents the mean score of all students on the iRAT.

Figure 2.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 2.

Pharmacy students’ relative accuracy of prediction of Individual Readiness Assurance Test (iRAT) score based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the material (n=453). The horizontal line represents the mean relative accuracy of all students on the iRAT.

Figure 3.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 3.

Pharmacy students’ relative accuracy of postdiction of Individual Readiness Assurance Test (iRAT) score based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the material (n=418). The horizontal line represents the mean relative accuracy of all students on the iRAT.

Figure 4.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 4.

Pharmacy students’ overall course grade based on student-reported use of supplemental material intended to improve their confidence in their knowledge of the material (n=453). The horizontal line represents the mean course grade for all students.

DISCUSSION

This study examined the relationship between student-reported use of optional supplemental material and calibration ability on a low-stakes assessment in a therapeutics course. This study is important because development of calibration ability as a form of metacognition could lead to improved self-awareness and self-regulated learning. Overall, students were mostly underconfident when asked to predict and postdict their performance, but we did not find a difference in calibration accuracy between the group that reported using the supplemental material and the group that did not. However, like our results in a previous study, students who reported using the supplemental material performed better on the low-stakes assessment and in the course overall.10

Calibration as a method of metacognitive development has been investigated in PharmD students in prior studies. Like Schneider and colleagues and Fox and colleagues, we found that pharmacy students are generally underconfident when asked to predict or postdict their performance on either low-stakes assessments or high-stakes assessments, which is consistent with metacognition literature that suggests higher-performing students tend to be underconfident as compared to lower-performing students.7-9 This slight under-confidence may also be a result of student concern for patient safety. While these studies looked at the difference in calibration for higher vs lower performing students, we added the additional analysis layer of student-reported use of supplemental material, albeit not finding a difference based on supplemental material use.

The ability for students to predict their performance accurately can depend on a number of factors. High levels of stress as well as task difficulty have been shown to decrease metacognitive processes and accuracy.9,12 Therefore, we chose to attempt to correlate supplemental material use and calibration on a low-stakes assessment rather than a high-stakes assessment to try to minimize the risk of inaccurate calibration due to stress and a difficult examination. However, the heavy course load in the PharmD curriculum, stress outside of the coursework, and/or the difficulty of critical care material may have still impaired the students’ ability to calibrate well. Additionally, gender identification appears to play a role in confidence and metacognitive accuracy, with individuals identifying as female being less confident.13 Given that more than half of the students identified as female, gender identification could have potentially played a role in the underconfidence shown in the study.

The strengths of this study include the large sample size and diversity of student cohort. Limitations of the study include the reliance on students to try to accurately predict and postdict their scores on a single assignment. If students were given additional calibration opportunities, their calibration ability might improve, as it did in the study by Fox and colleagues.7 An additional limitation lies in relying on student self-report of using the supplemental material. Some students may not have wanted to admit to using (or not using) the material for a variety of reasons.

Given these findings that student-reported use of supplemental material does not seem to correlate with better calibration on a low-stakes assessment, future research is needed to determine if repeat calibration attempts on low-stakes assessments will lead to more accurate calibration. Additionally, further research is needed regarding different ways of assessing and improving student metacognition and self-awareness.

CONCLUSION

In this study, the use of instructor-created supplemental material was significantly associated with improved performance on a low-stakes assessment and in the course overall. However, the majority of students were underconfident before and after the assessment, and the use of supplemental material was not associated with better calibration. Although improvement in pharmacy students’ calibration skills was not observed, students who self-reported use of the supplemental material performed better, indicating enhanced learning. Further studies surrounding repeated opportunities for pharmacy students to practice calibration and other methods of developing metacognitive skills are needed.

  • Received February 24, 2021.
  • Accepted May 28, 2021.
  • © 2021 American Association of Colleges of Pharmacy

REFERENCES

  1. 1.↵
    1. Livingston J
    . Metacognition: an overview. https://eric.ed.gov/?id=ED474273. Accessed November 11, 2020.
  2. 2.↵
    1. Fadel C,
    2. Bialik M,
    3. Trilling B
    . Four-Dimensional Education. Boston, MA: Center for Curriculum Redesign; 2015.
  3. 3.↵
    Theories of Learning in Educational Psychology: John Flavell. Metacognition Theory. https://www.demenzemedicinagenerale.net/images/mens-sana/Theories_of_Learning_in_Educational_Psychology.pdf. Accessed November 11, 2020.
  4. 4.↵
    1. Persky AM,
    2. Lee E,
    3. Schlesselman LS
    . Perception of learning versus performance as outcome measures of educational research. Am J Pharm Educ. 2020;84(7):Article 7782.
    OpenUrl
  5. 5.↵
    Accreditation Council for Pharmacy Education. Accreditation standards and key elements for the professional program in pharmacy leading to the Doctor of Pharmacy degree. February 2015. https://www.acpe-accredit.org/pharmd-program-accreditation/. Accessed November 11, 2020.
  6. 6.↵
    1. Medina MS,
    2. Castleberry AN,
    3. Persky AM
    . Strategies for improving learner metacognition in health professional education. Am J Pharm Educ. 2017;81(4):Article 78.
    OpenUrl
  7. 7.↵
    1. Fox L,
    2. Piccorelli A,
    3. Bruch D
    . Practicing metacognition through the use of repeated posttest predictions. Curr Pharm Teach Learn. 2019;11(6):630-634.
    OpenUrl
  8. 8.↵
    1. Schneider EF,
    2. Castleberry AN,
    3. Vuk J,
    4. Stowe CD
    . Pharmacy students’ ability to think about thinking. Am J Pharm Educ. 2014;78(8):Article 148.
    OpenUrl
  9. 9.↵
    1. Bol L,
    2. Hacker DJ
    . Calibration research: where do we go from here? Front Psychol 2012;3:229.
    OpenUrlPubMed
  10. 10.↵
    1. Voils SA,
    2. Childs-Kean LM,
    3. Thomas A
    . Relationship between pharmacy students’ use of self-regulated learning strategies and course outcomes. Am J Pharm Educ. 2019;83(10):Article 7566.
    OpenUrl
  11. 11.↵
    Learning tools & flashcards, for free. Quizlet. https://www.quizlet.com/. Accessed November 11, 2020.
  12. 12.↵
    1. Reyes G,
    2. Silva JR,
    3. Jaramillo K,
    4. Rehbein L,
    5. Sackur J
    . Self-knowledge dim-out: stress impairs metacognitive accuracy. PLoS ONE. 2015;10(8):e0132320-e.
    OpenUrl
  13. 13.↵
    1. Ariel R,
    2. Lembeck NA,
    3. Moffat S,
    4. Hertzog C
    . Are there sex differences in confidence and metacognitive monitoring accuracy for everyday, academic, and psychometrically measured spatial ability? Intelligence. 2018;70:42-51.
    OpenUrl
PreviousNext
Back to top

In this issue

American Journal of Pharmaceutical Education
Vol. 85, Issue 10
1 Nov 2021
  • Table of Contents
  • Index by author
Print
Download PDF
Email Article

Thank you for your interest in spreading the word on American Journal of Pharmaceutical Education.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Impact of Supplemental Material Use on Student Metacognitive Monitoring and Calibration
(Your Name) has sent you a message from American Journal of Pharmaceutical Education
(Your Name) thought you would like to see the American Journal of Pharmaceutical Education web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
2 + 0 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Impact of Supplemental Material Use on Student Metacognitive Monitoring and Calibration
Lindsey M. Childs-Kean, Jennifer Rodriguez, Aaron O. Thomas, Stacy A. Voils
American Journal of Pharmaceutical Education Nov 2021, 85 (10) 8614; DOI: 10.5688/ajpe8614

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Impact of Supplemental Material Use on Student Metacognitive Monitoring and Calibration
Lindsey M. Childs-Kean, Jennifer Rodriguez, Aaron O. Thomas, Stacy A. Voils
American Journal of Pharmaceutical Education Nov 2021, 85 (10) 8614; DOI: 10.5688/ajpe8614
Reddit logo Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • INTRODUCTION
    • METHODS
    • RESULTS
    • DISCUSSION
    • CONCLUSION
    • REFERENCES
  • Figures & Data
  • Info & Metrics
  • PDF

Similar AJPE Articles

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Trends in the Number of Authors and Institutions in Papers Published in AJPE 2015-2019
  • Comparison of Suicidal Ideation and Depressive Symptoms Between Medical and Pharmacy Students
  • A Large-scale Multicenter Study of Academic Resilience and Well-being in Pharmacy Education
Show more RESEARCH

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Keywords

  • learning assessment
  • metacognition
  • calibration
  • supplemental material

Home

  • AACP
  • AJPE

Articles

  • Current Issue
  • Early Release
  • Archive

Instructions

  • Author Instructions
  • Submission Process
  • Submit a Manuscript
  • Reviewer Instructions

About

  • AJPE
  • Editorial Team
  • Editorial Board
  • History
  • Contact

© 2023 American Journal of Pharmaceutical Education

Powered by HighWire