Review ArticleREVIEW

Measuring Metacognitive Knowledge, Monitoring, and Control in the Pharmacy Classroom and Experiential Settings

Michelle L. Rivers, John Dunlosky and Adam M. Persky
American Journal of Pharmaceutical Education May 2020, 84 (5) 7730; DOI: https://doi.org/10.5688/ajpe7730

Abstract

Objective. To provide a user’s guide on measuring metacognition in authentic contexts so that educators and researchers can explore students’ metacognition with an aim towards improving their students’ metacognitive processes and achievement.

Findings. Metacognition can be measured in a variety of ways depending on whether the interest is knowledge, monitoring, or control. These methods include surveys, assessment of student predictions versus their performance on examinations, or investigating students’ decisions during their learning process.

Summary. Metacognition refers to people’s knowledge about and regulation of their cognitive processes. These aspects of metacognition are important for supporting students’ success in academic and experiential settings. In particular, students who recognize successful learning strategies can accurately monitor their own progress and make effective study decisions that are more likely to help them meet their learning goals. Thus, measuring metacognitive knowledge, monitoring, and control can help educators identify struggling students who may benefit from interventions to improve their metacognitive processes.

Keywords

INTRODUCTION

Riley begins studying for an examination she will take tomorrow in her pharmacy course. She starts by diligently reviewing her notes from class. As she flips through her notes, she feels confident that she understands infusion kinetics and decides she does not need to devote any more effort to learning that topic. However, she keeps getting the concepts of hepatic clearance and renal clearance confused. To overcome this confusion, Riley decides to reread the section of her textbook about these topics. During the examination, Riley is feeling a bit anxious because although she studied the evening before, she still feels that she does not fully understand hepatic and renal clearance. She completes her examination but before turning it in, she goes through each question and puts a check next to the ones she is uncertain about so she can look up the answers later. Based on this rough analysis, she estimates that she will receive a score of about 80% on the examination. She plans to begin studying earlier for future examinations so that she can space out her study times and use different strategies in hopes of improving her learning and performance on the next examination.

Defining Metacognition

In the aforementioned scenario, Riley is using different aspects of metacognition. Metacognition is generally defined as “thoughts about one’s own thoughts,”1 or more specifically, “one’s knowledge concerning one’s own cognitive processes and products or anything related to them.”2 Metacognition consists of three primary components: knowledge, monitoring, and control (Table 1).3 Metacognitive knowledge refers to facts and beliefs about how we learn, such as our knowledge about the effectiveness of learning strategies or our efficacy in our ability to learn (ie, self-efficacy4). Riley presumably believed that studying the night before an examination (ie, cramming) and rereading were effective learning strategies, but after taking the examination, she realized that these strategies may have been limited. Metacognitive monitoring refers to evaluating the process of learning or current state of knowledge. Riley engaged in monitoring to discover that she felt confident in her understanding about infusion kinetics (and less confident about hepatic clearance) and to estimate her performance on the examination. Metacognitive control is the regulation of learning activities. Riley decided when she was going to study, what strategies she would use, and after the examination, how she would study differently for future examinations. Each of these aspects of metacognition can limit or enhance learning depending on the quality of the students’ knowledge, monitoring, and control processes, –how students regulate their learning can make a real difference in their ultimate success.

View this table:
Table 1.

Aspects of Metacognitiona for Educators to Consider When Developing Strategies to Improve Student Learning

Most educators have probably encountered students like Riley who are using metacognitive knowledge and monitoring to inform how they will control their learning and test taking. Some students’ study habits will be sophisticated, while others’ study habits will be much less so. The latter may have difficulties monitoring their progress and even believe that some ineffective strategies are effective, which can undermine their learning. Accordingly, this review offers some advice on how to assess students’ metacognition by providing a review of the kinds of questions that can be asked (and empirically answered) about students’ metacognition within authentic educational settings, including the typical measures of metacognition and the limitations associated with each. Our goal in reviewing these measures was twofold. First, we wanted to help educators and discipline-based education researchers5 assess metacognition, which can inform instruction. If, for example, an educator discovers that many students tend to be overconfident with respect to their knowledge of particular concepts, the educator can alert their students to not underestimate the difficulty of the content and might decide to spend more time teaching those concepts. Second, measuring metacognition across time will allow educators to evaluate whether an intervention (eg, a change in instruction) improves student metacognition (for advice on conducting instructional research in a classroom, see Dunlosky and colleagues6). Unlike other factors related to achievement (eg, intelligence), metacognition is not hardwired and is malleable: students’ inaccurate beliefs can be changed, they can learn to use effective strategies, and they can be trained how to improve their monitoring and control of learning.7 Thus, finding ways to measure and improve students’ metacognitive knowledge, monitoring, and control can lead to insights into how to improve their achievement.

To help foster research, a list of questions that can be asked about metacognition, both within classroom and experiential settings, is presented in Table 2. Also, although some concepts are illustrated in the context of a pharmacy course, the measurement of metacognition is relevant to any course. Thus, if an educator encounters a particular question worth addressing, this overview can provide some basic research tools to begin answering it within a course, whether it focuses on pharmacy, chemistry, physics, psychology, or other subjects. For each question, we included representative articles that sought to answer the question and which provide in-depth illustrations on how one might approach answering a particular question. By no means is this list comprehensive. Instead, it identifies some of the questions that we and others have been interested in pursuing, both within laboratory settings and in authentic educational contexts.

View this table:
Table 2.

Potential Questions for Educators to Ask About Each Aspect of Metacognition When Developing Strategies to Improve Student Learning

Measuring Metacognitive Knowledge and Beliefs

Research focused on learning has identified the relative effectiveness of different study strategies.8,9 However, do students know which strategies are most effective and use them consistently? This type of question can be answered by assessing students’ metacognitive knowledge. In the aforementioned scenario, the student believed reviewing her notes and rereading her textbook would lead to effective learning. Unfortunately, such relatively passive strategies are less effective than strategies that require the learner to take on a more active role, such as practice testing10,11 or self-explanation.12 This example demonstrates one reason to measure students’ knowledge and beliefs: students can be misguided and have misconceptions about what works best,13 perhaps because many students are not formally taught the basic principles of learning and how to study effectively.14 Accordingly, assessing students’ knowledge and, if necessary, providing guidance to them about when and how to use effective strategies may benefit their achievement.

Another important aspect of metacognitive knowledge pertains to students’ beliefs about their ability to succeed. For instance, as students progress through a course, are they more confident in their ability to learn new material and perform well on examinations? As students advance in their experiential curriculum, do they feel more confident in completing their professional activities (eg, entrustable professional activities), such as educating patients about their medications or identifying patients’ medication-related problems? These questions refer to students’ self-efficacy or people’s beliefs about their ability to succeed in specific situations.4 Self-efficacy is important because it is related to effective self-regulation, ie, students who are confident in their ability to succeed are more likely to set achievable goals and use effective strategies to reach them.15 Doing so ultimately leads to higher achievement, as supported by the relationship between self-efficacy and grade-point average.16 To reiterate an important point: self-efficacy pertains to a student’s belief about his or her ability and not to the student’s actual ability. Although people’s beliefs about their ability may be related to their actual ability, beliefs and performance can also be misaligned. That is, an intervention can enhance students’ efficacy yet have no impact on their performance. Thus, measures of self-efficacy should be viewed as beliefs about ability and not used as surrogates for ability or achievement. We recommend measuring both students’ self-efficacy (a belief) and performance, so that the degree to which students’ beliefs align with their performance can be explored. We return to this issue of measuring judgment accuracy under the section measuring metacognitive monitoring.

Given the importance of metacognitive knowledge and beliefs, how can they be assessed? Metacognitive knowledge and beliefs are typically measured through questionnaires, such as those listed in Table 3. Descriptions of other questionnaires measuring study habits, skills, and attitudes have been published elsewhere.17 Questionnaires are useful because they can be easily modified to meet a specific research goal and can generate informative (and extensive) data sets in a brief session. For instance, several questionnaires have multiple scales, some of which tap students’ knowledge about strategies, their use of strategies, and their self-efficacy for learning. Accordingly, an instructor may be interested in only a subset of the questions pertaining to those scales that are most relevant to the question(s) being addressed. Once equipped with information about students’ metacognitive knowledge, an instructor can address knowledge gaps and potentially use the responses to understand why particular students are struggling. With respect to motivating students to adopt more effective strategies, the good news is that instructing students about effective strategies appears to lead to students’ greater endorsement of them.13,18

View this table:
Table 3.

Popular Questionnaires Aimed at Measuring Students’ Knowledge About Strategies, Efficacy, and Control of Learning

Before administering any questionnaire, we suggest reviewing the properties of each scale and thinking critically about what each one is assessing. Some questionnaires combine outcomes from multiple questions into a single scale, but sometimes the answers to a specific question may be of most interest. For instance, one scale of the Motivated Strategies for Learning Questionnaire19 combines items relevant to both ineffective and effective strategies. We would not expect use of ineffective strategies to predict performance in the same way that effective strategies would, so using this learning strategies scale would not make sense if instructors are only interested in students’ knowledge about effective strategies. Therefore, it is worth carefully considering how each question of a scale is relevant to one’s assessment goals. With this recommendation in mind, note that a full discussion of the challenges associated with using questionnaires20 is beyond the scope of this review.

Measuring Metacognitive Monitoring

Assessments are administered to evaluate student progress toward achieving learning objectives, which allows instructors to assign grades (ie, summative evaluation21). However, assessments can also provide instructors, preceptors, and students with formative evaluation or information about students’ understanding of the course material.22 Educators can use this information to modify their instruction to target more difficult concepts, and doing so comprises one of the most effective teaching practices.23 Students can also use this information to identify concepts that they have not yet learned. Students can receive targeted feedback about what concepts they are struggling with, or they can discover this for themselves by monitoring their performance. Note that accurate metacognitive monitoring is crucial for students to gain this formative evaluation. For instance, Riley’s confidence in her understanding of infusion kinetics influenced her decision to stop studying that topic. If she overestimated her understanding, then she would have benefitted from studying infusion kinetics more thoroughly. That is, if students are overconfident in their knowledge of a concept, they may not spend enough time studying.24 Under-confidence may also be problematic, because students may spend too much time on material that they have already learned well, which may lead them to use their limited study time inefficiently. Ultimately, accurate monitoring can support the most efficient and effective control of learning.

Monitoring can be measured at different phases of learning in the classroom or in experiential settings as illustrated in Figure 1.25 If instructors are interested in students’ level of understanding during learning (or acquisition), instructors should have students judge their current understanding before attempting to demonstrate what they have learned. For instance, instructors could ask students, “How confident are you that you understand infusion kinetics?” If instructors are interested in students’ level of understanding after learning but before they answer questions on an examination, instructors could have students predict their upcoming performance on an examination: “How confident are you that you will correctly answer questions about infusion kinetics on the exam?” Finally, if instructors are interested in students’ level of understanding after completing an examination, instructors could have students estimate how many questions they accurately answered on an examination after completing it (ie, have them make retrospective judgments of their performance). In a clinical setting, instructors could ask students to judge their ability to perform particular tasks (ie, perform a comprehensive medication review) either before or after doing so and receiving feedback from their preceptor.

Figure 1.
Figure 1.

Types of Monitoring Judgments and Control Decisions and When They Can be Measured

  • Note: Judgments can also vary in their level of specificity (item, category, global) at each time point. Inspired by Nelson and Narens.25

As implied by the aforementioned examples, monitoring can also be measured at different levels of specificity. At the global level, students judge their performance for all questions over an entire examination. At the concept level, students judge performance on particular concepts or topics, such as judging how well one will perform on all questions pertaining to the concept of renal clearance. At the item level, students judge whether they correctly answered a particular question on an examination. Most research on metacognitive judgments has focused on the global or item level and relatively little research has focused on the concept level. The relative lack of information about how well students can judge their concept-level knowledge is unfortunate because monitoring at this level is particularly relevant to providing useful formative evaluation. For instance, a student accurately judging that she answered about 70% of the items on an examination correctly (global level) will not necessarily help her figure out what specific concepts she does not understand. In contrast, accurately judging that her test performance was lower than desired because of her failure to perform well on questions about specific concepts (eg, infusion and nonlinear kinetics) would allow her to more effectively guide her restudying.

Collecting monitoring judgments from students can be easily incorporated into a classroom or clinical routine. For example, instructors could include a prediction cover sheet on examinations or have students provide confidence ratings as they answer quiz questions. A supervising clinician could have students assess their ability to complete tasks within a particular domain (eg, answering drug information questions). An example of a cover sheet (that involves predicting examination performance) that has been developed for a course entitled “Foundations in Pharmacokinetics” is provided in Appendix 1. This cover sheet was used to obtain monitoring judgments at both the global level and concept level, with the latter focusing on concepts being tested on the upcoming examination. A similar sheet could be used after the examination to obtain students’ judgments about how well they performed on the examination. Of course, at what level (and at what stage of learning and testing) students’ monitoring is assessed will depend on the questions being addressed.

Once students have made judgments about their knowledge or test performance, how does one estimate their monitoring (or judgment) accuracy? Measuring students’ level of judgment accuracy can help instructors discover the extent to which students can accurately estimate their own knowledge and whether any change in instruction influences their judgment accuracy. Monitoring accuracy involves comparing judgments with performance, and two types of accuracy, ie, absolute and relative accuracy, highlight different aspects of the relationship between students’ judgments and their performance.

Absolute accuracy refers to how well students can estimate their actual level of performance. Absolute accuracy can be measured in three ways: bias, absolute bias, and calibration. Each measure captures a different aspect of how well the judgments match performance, and each one is first computed at the level of each student. Example values based on a hypothetical student are presented in Figure 2. In this figure, judgments and test scores at both the global and concept level are included, which pertain to the cover sheet shown in Appendix 1. This student’s global prediction was 85, with the concept-level predictions ranging from 75 (for hepatic clearance) to 95 (for infusion kinetics).

Figure 2.
Figure 2.

Judgments and Examination Performance of a Hypothetical Student

  • Note: The global judgment does not need to be identical to the mean concept-level judgment, and, in most cases, mean examination performance will not equal the mean performance across topics (eg, they will not be equivalent if some questions tap topics that are not represented by the judged topics or if a different number of questions tap each topic on the examination). Resolution was computed as a gamma correlation between concept-level judgments and performance (see text for alternative measures).

Bias is the mean level of judgment minus actual performance, with positive values indicating overconfidence and negative values indicating under confidence. As shown in Figure 2, this student demonstrated overconfidence for their global judgment, with bias being +10. For bias at the concept level, the mean judgment (in this case, M=85) is compared to actual performance, and the student’s judgments were also on average overconfident at the concept level (bias=+5). Note also that the student was underconfident for one of the concepts (ie, single dose intravenous kinetics). Averaging across all students’ bias scores (which would be a standard approach to presenting descriptive analyses) can be misleading, because a mean of 0 (which would appear to be perfect absolute accuracy as measured by bias) could result from large discrepancies in both directions. That is, some students may show extreme overconfidence and others could show extreme underconfidence, yet the bias could average to 0. We recommend analyzing frequency distributions to evaluate whether excellent (close to 0) bias is resulting from averaging across bias values from both over and underconfident students. Another way to resolve this potential problem is to compute the absolute value of bias, which is called absolute bias. By computing the absolute value, then over and underconfidence across students will not reduce the magnitude of the overall bias, so absolute bias will provide a better estimate of how discrepant the judgment magnitude is from actual performance.

Another measure of absolute accuracy is called calibration, and it is based on an analysis of an entire calibration curve that maps percent performance as a function of increasing judgments. To construct a calibration curve26 and to estimate the corresponding measures of calibration, it would be ideal to have more observations per student than would be typically collected for concept-level judgments. Thus, calibration analyses would be most appropriate for judgments collected at the individual item level, such as when students provide a confidence judgment for each of their answers for an examination with many questions.

To estimate absolute accuracy, students must make judgments using the same metric as task performance because it would not be appropriate to subtract values that are from different metrics (eg, a judgment made on a seven-point Likert scale is not comparable to examination performance measured on a percent scale). We recommend having students make judgments on a percent scale (eg, estimate the percentage of questions they will answer correctly) because this scale makes intuitive sense to students and is a typical metric used for scoring examinations.

Relative accuracy (also referred to as resolution) refers to students’ ability to discriminate between different levels of performance across items or concepts. Many different measures of relative accuracy have been proposed, each with their own strengths and weaknesses.27-33 One common way to estimate resolution is to compute an intra-individual correlation between students’ judgments and their performance. A strong, positive correlation is observed when a student can accurately judge the likelihood of correct performance on one item (or concept) relative to another. In Figure 2, resolution was computed as a Goodman-Kruskal gamma correlation,30 and for the hypothetical student, resolution was close to perfect (which, for a correlation, would be a value of +1.0). Other correlations (eg, Pearson r) can be used as well.31 By contrast, others have argued that the correlational approaches provide more biased estimates of resolution as compared to measures based on signal detection theory.33 Our recommendation is to evaluate whether each measure supports the same conclusion by computing as many measures of resolution as viable given the structure of the data.

Measuring Metacognitive Control

Consider Riley once again. After taking her first examination, she plans to study even more in the future. Because she would eventually like to practice pharmacy professionally, her goal is to master all the material she is taught so that she can apply it when she is working in the field. Given this goal, she makes a schedule outlining when and what she will study each day. As she studies and judges her progress, she revises her plan based on her understanding of particular concepts. She also decides that she is going to use some new study techniques that promise to promote long-term retention.

Riley’s plan demonstrates a few key points about metacognitive control. First, her studying is influenced by her specific goals. According to all theories of self-regulated learning based on information-processing models,34-38 an effective learner develops goals and plans as to how best to obtain them. Students vary in their learning goals (eg, master all the material or simply pass a class), and those goals influence how they prepare for study, such as what they study, when and how they study, and how long they study. Students’ decisions on what and when to study will be influenced by other factors as well. In one survey, students reported focusing on topics they found interesting and whatever assignment was due the soonest.14 In addition to using surveys to assess when (and how) students prepare for examinations, technology allows for additional opportunities for instructors to measure patterns of students’ study behavior. For instance, researchers39,40 have assessed when students accessed and submitted assignments through a learning management system, which provided an objective measure of students’ learning behaviors such as academic procrastination. If these measures indicate students are driven by what is due soonest, it may mean that they are waiting until the last minute to cram for important examinations. If so, educators could consider administering low-stakes quizzes each week to encourage students to study more frequently.

Second, control decisions also include the kinds of study techniques that students explicitly adopt to learn material. In other words, Riley is demonstrating control when she decides to stop rereading her material passively and switch to using a more effective strategy, such as successive relearning.9,41,42 The surveys discussed above will provide some insight into students’ beliefs and use of some strategies, and brief checklists can be used to assess students’ use of effective (and ineffective) techniques and how they may change their use of these techniques across a course.14

A final point concerning Riley’s control of her study is that her monitoring is likely to be intricately linked to her control.43,44 In particular, one function of monitoring in this context is for Riley to identify material that she is struggling to learn. Riley can then decide how to engage differently with the less well-understood material, such as by studying it more, choosing a different strategy for studying it, or seeking help from a peer or instructor. In fact, one of the main reasons that cognitive researchers have been interested in metacognitive monitoring is because students use their monitoring to make decisions about how to control their subsequent study. The rationale for such an interest is based on the importance of accurate monitoring for effective control,44 which we suspect is intuitively plausible. If Riley’s judgments are perfectly accurate, then when she judges that she does not yet understand one topic and will not correctly answer questions on an examination, then in fact (because her judgments are accurate) she actually does not yet understand that topic and does need to study it more.

In summary, if students are inaccurately judging their learning, and in particular, if they are overconfident in their knowledge of a topic, their poor judgments could lead them to underperform. If after an examination you ever had a student say, “I was sure I knew all the material, so why did I perform so poorly?” then you have already run into the problem of misjudgment and how overconfidence can make students fall short of their learning goals. Again, using techniques to help students inform themselves about how well they understand content (eg, administering mini quizzes for use in formative evaluation) could be useful. Furthermore, with the methods described above for how to collect and evaluate judgment accuracy, educators could explore the degree to which any intervention (eg, mini quizzes) is actually improving student’s ability to accurately judge their knowledge.

SUMMARY

Most learning takes place outside of the classroom, where students have limited guidance about what and how to study and are faced with ample opportunities to become distracted. Thus, effective self-regulated learning (guided by students’ metacognition) is critical for students to reach their academic goals. Discovering the extent to which students know and use effective learning strategies, believe in their ability to succeed, and successfully monitor and control their own learning can inform instruction.

Appendix 1. Sample Cover Sheet for Examinations Used to Assess Student Monitoring at the Global and Concept Level (modified from Hartwig and Dunlosky)53

Table

Footnotes

  • Received June 11, 2019.
  • Accepted October 31, 2019.

REFERENCES

  1. 1.↵
    1. Flavell JH
    . Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. Am Psychol. 1979;34(10):906-911. doi: 10.1037/0003-066X.34.10.906.
    OpenUrlCrossRef
  2. 2.↵
    1. Resnick L
    1. Flavell JH
    . Metacognitive aspects of problem solving. In Resnick L ed. The Nature of Intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates. 1976.
  3. 3.↵
    1. Dunlosky J,
    2. Metcalfe J
    . Metacognition. Thousand Oaks, CA: Sage. 2008.
  4. 4.↵
    1. Bandura A
    . Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191-215. doi: 10.1037/0033-295X.84.2.191.
    OpenUrlCrossRefPubMed
  5. 5.↵
    National Research Council. Discipline-based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. Washington, DC: National Academies Press; 2012.
  6. 6.↵
    Dunlosky J, Morehead K, Zamary A, Rawson KA. From the laboratory to the classroom. In: Otani H, Schwartz BL, eds. Handbook of Research Methods in Human Memory. Abingdon, VA: Routledge; 2019. doi: 10.4324/9780429439957-23.
  7. 7.↵
    1. Medina MS,
    2. Castleberry AN,
    3. Persky AM
    . Strategies for improving learner metacognition in health professional education. Am J Pharm Educ. 2017;81(4):78. doi: 10.5688/ajpe81478.
    OpenUrlAbstract/FREE Full Text
  8. 8.↵
    1. Dunlosky J,
    2. Rawson KA
    1. Dunlosky J,
    2. Rawson KA
    . How cognitive psychology can inform evidence-based education reform. In Dunlosky J, , Rawson KA, eds. The Cambridge Handbook of Cognition and Education. Cambridge: Cambridge University Press; 2019. doi: 10.1017/9781108235631.
    OpenUrlCrossRef
  9. 9.↵
    1. Dunlosky J,
    2. Rawson KA,
    3. Marsh EJ,
    4. Nathan MJ,
    5. Willingham DT
    . Improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychol Sci Public Interest. 2013;14(1):4-58. doi: 10.1177/1529100612453266.
    OpenUrlCrossRefPubMed
  10. 10.↵
    1. Roediger HL,
    2. Karpicke JD
    . The power of testing memory: basic research and implications for educational practice. Perspectives on Psychol Sci. 2006;1(3):181-210. doi: 10.1111/j.1745-6916.2006.00012.x.
    OpenUrlCrossRefPubMed
  11. 11.↵
    1. Stewart D,
    2. Panus P,
    3. Hagemeier N,
    4. Thigpen J,
    5. Brooks L
    . Pharmacy student self-testing as a predictor of examination performance. Am J Pharm Educ. 2014;78(2):32. doi: 10.5688/ajpe78232.
    OpenUrlAbstract/FREE Full Text
  12. 12.↵
    1. Dunlosky J,
    2. Rawson KA
    1. Renkl A,
    2. Eitel A
    . Self-explaining: learning about principles and their application. In Dunlosky J, , Rawson KA, eds. The Cambridge Handbook of Cognition and Education. Cambridge: Cambridge University Press; 2019. doi: 10.1017/9781108235631.022.
    OpenUrlCrossRef
  13. 13.↵
    1. McCabe J
    . Metacognitive awareness of learning strategies in undergraduates. Mem Cogn. 2011;39(3):462-476. doi: 10.3758/s13421-010-0035-2.
    OpenUrlCrossRefPubMed
  14. 14.↵
    1. Kornell N,
    2. Bjork RA
    . The promise and perils of self-regulated study. Psychonomic Bull Rev. 2007;14(2):219-224. doi: 10.3758/BF03194055.
    OpenUrlCrossRefPubMed
  15. 15.↵
    1. Boekaerts PR,
    2. Pintrich PR,
    3. Zeidner M
    1. Schunk DH,
    2. Ertmer PA
    . Self-regulation and academic learning: self-efficacy enhancing interventions. In: Boekaerts PR, , Pintrich PR, , Zeidner M, eds. Handbook of Self-regulation. San Diego, CA: Academic Press; 2000.
  16. 16.↵
    1. Robbins SB,
    2. Lauver K,
    3. Le H,
    4. Davis D,
    5. Langley R,
    6. Carlstrom A
    . Do psychosocial and study skill factors predict college outcomes? a meta-analysis. Psychol Bull. 2004;130(2):261-288. doi:10.1037/0033-2909.130.2.261.
    OpenUrlCrossRefPubMed
  17. 17.↵
    1. Credé M,
    2. Kuncel NR
    . Study habits, skills, and attitudes: the third pillar supporting collegiate academic performance. Perspect Psychol Sci. 2008;3(6):425-453. doi: 10.1111/j.1745-6924.2008.00089.x.
    OpenUrlCrossRefPubMed
  18. 18.↵
    1. Ariel R,
    2. Karpicke JD
    . Improving self-regulated learning with a retrieval practice intervention. J Exp Psychol Appl. 2017;24(1):43-56. doi: 10.1037/xap0000133.
    OpenUrlCrossRef
  19. 19.↵
    1. Pintrich PR,
    2. Smith DAF,
    3. Garcia T,
    4. McKeachie WJ
    . Reliability and predictive validity of the motivated strategies for learning questionnaire (mslq). Educ Psychol Meas. 1993;53(3):801-813.
    OpenUrlCrossRef
  20. 20.↵
    1. Krosnick JA
    . Survey research. Ann Rev of Psychol. 1999;50(1):537-567. doi: 10.1146/annurev/psych.50.1.537.
    OpenUrlCrossRef
  21. 21.↵
    1. Bloom BS,
    2. Hasting JT,
    3. Madaus GF
    . Handbook on Formative and Summative Evaluation of Student Learning. New York, NY: McGraw Hill; 1971.
  22. 22.↵
    1. Peterson EBP
    1. Allal L
    . Assessment and the Regulation of Learning. In Peterson EBP ed. International Encyclopedia of Education. Vol. 3. Oxford, UK: Elsevier; 2010.
  23. 23.↵
    1. Hattie J
    . Visible learning: A Synthesis of Over 800 Meta-analyses Relating to Achievement. London, UK: Routledge; 2008. doi: 10.4324/9780203887332.
    OpenUrlCrossRef
  24. 24.↵
    1. Dunlosky J,
    2. Rawson KA
    . Overconfidence produces underachievement: inaccurate self-evaluations undermine students’ learning and retention. Learn Instruct. 2012;22(4):271-280. doi: 10.1016/j.learninstruc.2011.08.003.
    OpenUrlCrossRef
  25. 25.↵
    1. Bower GH
    1. Nelson TO,
    2. Narens L
    . Metamemory: A theoretical framework and new findings. In Bower GH, ed. Psychol Learn Motivation. Vol 26. New York: Academic Press. 1990.
  26. 26.↵
    1. Keren G
    . Calibration and probability judgements: conceptual and methodological issues. Acta Psychologica. 1991;77(3):217-273. doi: 10.1016/0001-6918(91)90036-Y.
    OpenUrlCrossRef
  27. 27.↵
    1. Fleming SM,
    2. Lau HC
    . How to measure metacognition. Frontiers in Human Neuroscience, 2014; 8:443. doi: 10.3389/fnhum.2014.00443.
    OpenUrlCrossRefPubMed
  28. 28.↵
    1. Masson ME,
    2. Rotello CM
    . Sources of bias in the Goodman-Kruskal gamma coefficient measure of association: implications for studies of metacognitive processes. J Exp Psychol Learn Mem Cogn. 2009;35(2):509-527. doi: 10.1037/a0014876.
    OpenUrlCrossRefPubMed
  29. 29.↵
    1. Murayama K,
    2. Sakaki M,
    3. Yan VX,
    4. Smith GM
    . Type I error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective. J Exp Psychol Learn Mem Cogn. 2014;40(5):1287. doi: 10.1037/a0036914.
    OpenUrlCrossRefPubMed
  30. 30.↵
    1. Nelson TO
    . A comparison of current measures of feeling-of-knowing accuracy. Psychol Bull. 1984;95:109-133. doi: 10.1037/0033-2909.95.1.109.
    OpenUrlCrossRefPubMed
  31. 31.↵
    1. Schraw G
    . A conceptual analysis of five measures of metacognitive monitoring. Metacog Learn. 2009;4(1):33-45. doi: 10.1007/s11409-008-9031-3.
    OpenUrlCrossRef
  32. 32.↵
    1. Dunlosky J,
    2. Rawson K
    1. Griffin TD,
    2. Mielicki MK,
    3. Wiley J
    . Improving students’ metacomprehension accuracy. In Dunlosky J, , Rawson K, eds. The Cambridge Handbook of Cognition and Education. Cambridge: Cambridge University Press; 2019. doi: 10.1017/9781108235631.
    OpenUrlCrossRef
  33. 33.↵
    1. Higham PA,
    2. Higham DP
    . New improved gamma: enhancing the accuracy of Goodman-Kruskal’s gamma using ROC curves. Behav Res. 2019. doi: 10.3758/s13428-018-1125-5.
    OpenUrlCrossRef
  34. 34.↵
    1. Ariel R,
    2. Dunlosky J,
    3. Bailey H
    . Agenda-based regulation of study-time allocation: when agendas override item-based monitoring. J Exp Psychol Gen. 2009;138(3):432. doi: doi:10.1037/a0015928.
    OpenUrlCrossRefPubMed
  35. 35.↵
    1. Hacker DJ,
    2. Dunlosky J,
    3. Graesser AC
    1. Dunlosky J,
    2. Hertzog C
    . Training programs to improve learning in later adulthood: helping older adults educate themselves. In: Hacker DJ, , Dunlosky J, , Graesser AC, eds. Metacognition in Educational Theory and Practice. Mahwah, NJ: Erlbaum; 1998.
  36. 36.↵
    1. Metcalfe J
    . Is study time allocated selectively to a region of proximal learning? J Exp Psychol Gen. 2002;131(3):349-363. doi: 10.1037/0096-3445.131.3.349.
    OpenUrlCrossRefPubMed
  37. 37.↵
    1. Thiede KW,
    2. Dunlosky J
    . Toward a general model of self-regulated study: an analysis of selection of items for study and self-paced study time. J Exp Psychol Learn Mem Cogn. 1999;25(4):1024-1037. doi:10.1037/0278-7393.25.4.1024.
    OpenUrlCrossRef
  38. 38.↵
    1. Hacker DJ,
    2. Dunlosky J,
    3. Graesser A
    1. Winne P,
    2. Hadwin A
    . Studying as self-regulated learning. In Hacker DJ, , Dunlosky J, Graesser A, eds. Metacognition in Educational Theory and Practice. Hillsdale, NJ: Erlbaum. 1998.
  39. 39.↵
    1. Miyake A,
    2. Carruth NP,
    3. Gustavson DE,
    4. Lurquin JH,
    5. Kane MJ
    . A classroom study of individual differences in procrastination: assignment submission times as objective measures of academic procrastination. Paper presented at: 59th Annual Meeting of the Psychonomic Society; 2018November; New Orleans, LA.
  40. 40.↵
    1. Taraban R,
    2. Maki WS,
    3. Rynearson K
    . Measuring study time distributions: implications for designing computer-based courses. Behavior Research Methods, Instruments, & Computers. 1999;31(2):263-269. doi: 10.3758/BF03207718.
    OpenUrlCrossRefPubMed
  41. 41.↵
    1. Dunlosky J,
    2. Rawson KA
    . Practice tests, spaced practice, and successive relearning: tips for classroom use and for guiding students’ learning. Scholarship Teach Learn Psychol. 2015;1(1):72. doi: 10.1037/stl0000024.
    OpenUrlCrossRef
  42. 42.↵
    1. Rawson KA,
    2. Dunlosky J,
    3. Sciartelli SM
    . The power of successive relearning: improving performance on course exams and long-term retention. Educ Psychol Rev. 2013;25(4):523-548. doi: 10.1007/s10648-013-9240-4.
    OpenUrlCrossRef
  43. 43.↵
    1. Ross BH
    1. Dunlosky J,
    2. Ariel R
    . Self-regulated learning and the allocation of study time. In: Ross BH, ed. Psychol Learn Motivation. Vol. 54. New York: Academic Press; 2011. doi: 10.1016/B978-0-12-385527-5.00004-8.
    OpenUrlCrossRef
  44. 44.↵
    1. Son LK,
    2. Metcalfe J
    . Metacognitive and control strategies in study-time allocation. J Exp Psychol Learn Mem Cogn. 2000;26(1):204. doi: 10.1037/0278-7393.26.1.204.
    OpenUrlCrossRefPubMed
  45. 45.
    1. Garavalia LS,
    2. Scheuer DA,
    3. Carroll CA
    . Comparative analysis of first-and third-year pharmacy students' perceptions of student-regulated learning strategies and motivation. Am J Pharm Educ. 2002;66(3):219-222.
    OpenUrl
  46. 46.
    1. Smith L,
    2. Saini B,
    3. Krass I,
    4. Chen T,
    5. Bosnic-Anticevich S,
    6. Sainsbury E
    . Pharmacy students' approaches to learning in an Australian university. Am J Pharm Educ. 2007;71(6):120. doi: 10.5688/aj7106120.
    OpenUrlCrossRefPubMed
  47. 47.
    1. Smith L,
    2. Krass I,
    3. Sainsbury E,
    4. Rose G
    . Pharmacy students' approaches to learning in undergraduate and graduate entry programs. Am J Pharm Educ. 2010;74(6):106. doi: 10.5688/aj7406106.
    OpenUrlAbstract/FREE Full Text
  48. 48.
    1. Yorra ML
    . Self-efficacy and self-esteem in third-year pharmacy students. Am J Pharm Educ. 2014;78(7):134.
    OpenUrlAbstract/FREE Full Text
  49. 49.
    1. Hartwig MK,
    2. Dunlosky J
    . Study strategies of college students: are self-testing and scheduling related to achievement? Psychonomic Bull Rev. 2012;19(1):126-134. doi: 10.3758/s13423-011-0181-y.
    OpenUrlCrossRef
  50. 50.
    1. Hagemeier NE,
    2. Mason HL
    . Student pharmacists' perceptions of testing and study strategies. Am J Pharm Educ. 2011;75(2):35. doi: 10.5688/ajpe75235.
    OpenUrlAbstract/FREE Full Text
  51. 51.
    1. Persky AM,
    2. Hudson SL
    . A snapshot of student study strategies across a professional pharmacy curriculum: are students using evidence-based practice? Curr Pharm Teach Learn. 2016;8(2):141-147.
    OpenUrl
  52. 52.
    1. Suda KJ,
    2. Bell GC,
    3. Franks AS
    . Faculty and student perceptions of effective study strategies and materials. Am J Pharm Educ. 2011;75(10):201. doi: 10.5688/ajpe7510201.
    OpenUrlAbstract/FREE Full Text
  53. 53.↵
    1. Hartwig MK,
    2. Dunlosky J
    . Category learning judgments in the classroom: can students judge how well they know course topics? Contemp Educ Psychol. 2017;49:80-90. doi: 10.1016/j.cedpsych.2016.12.002.
    OpenUrlCrossRef
  54. 54.
    1. Rivers ML,
    2. Dunlosky J,
    3. Joynes R
    . The contribution of classroom exams to formative evaluation of concept-level knowledge. Contemp Educ Psychol. 2019;59:101806. doi: 10.1016/j.cedpsych.2019.101806.
    OpenUrlCrossRef
  55. 55.
    1. Foster NL,
    2. Was CA,
    3. Dunlosky J,
    4. Isaacson RM
    . Even after thirteen class exams, students are still overconfident: the role of memory for past exam performance in student predictions. Metacog Learn. 2017;12(1):1-19. doi: 10.1007/s11409-016-9158-6.
    OpenUrlCrossRef
  56. 56.
    1. Persky AM,
    2. Dinsmore DL
    . Metacognitive changes and sources of confidence judgements in health professions classroom learning. Curr Pharm Teach Learn. 2019;11(4):338-345. doi: 10.1016/j.cptl.2019.01.005.
    OpenUrlCrossRef
  57. 57.
    1. Benjamin LT,
    2. Cavell TA,
    3. Shallenberger WR
    . Staying with initial answers on objective tests: is it a myth? Teach Psychol. 1984;11(3):133-141. doi: 10.1177/009862838401100303.
    OpenUrlCrossRef
  58. 58.
    1. Fischer MR,
    2. Herrmann S,
    3. Kopp V
    . Answering multiple‐choice questions in high‐stakes medical examinations. Med Ed. 2005;39(9):890-894. doi: 10.1111/j.1365-2929.2005.02243.x.
    OpenUrlCrossRef
  59. 59.
    1. Ferguson KJ,
    2. Kreiter CD,
    3. Peterson MW,
    4. Rowat JA,
    5. Elliott ST
    . Is that your final answer? relationship of changed answers to overall performance on a computer-based medical school course examination. Teach Learn Med. 2002;14(1):20-23. doi: 10.1207/S15328015TLM14016.
    OpenUrlCrossRefPubMed
  60. 60.
    1. Shatz MA,
    2. Best JB
    . Students' reasons for changing answers on objective tests. Teach Psychol. 1987;14(4):241-242. doi: 10.1207/s15328023top1404_17.
    OpenUrlCrossRef
  61. 61.
    1. Chu Y,
    2. Palmer A,
    3. Persky AM
    . Assessing metacognition in the classroom: student help-seeking behavior. Curr Pharm Teach Learn. 2018;10(11):1478-1487. doi: 10.1016/j.cptl.2018.08.011.
    OpenUrlCrossRef
  62. 62.
    1. Finley JR,
    2. Benjamin AS
    . Adaptive and qualitative changes in encoding strategy with experience: evidence from the test-expectancy paradigm. J Exp Psychol Learn Mem Cogn. 2012;38(3):632.
    OpenUrlCrossRefPubMed
  63. 63.
    1. McDaniel MA,
    2. Blischak DM,
    3. Challis B
    . The effects of test expectancy on processing and memory of prose. Contemp Educ Psychol. 1994;19(2):230-248.
    OpenUrlCrossRef
  64. 64.
    1. Szpunar KK,
    2. McDermott KB,
    3. Roediger HL
    . Expectation of a final cumulative test enhances long-term retention. Mem Cogn. 2007;35(5):1007-1013.
    OpenUrlCrossRefPubMed
  65. 65.
    1. Weinstein CE,
    2. Palmer D,
    3. Schulte AC
    . Learning and Study Strategies Inventory. Clearwater, FL: H & H Publishing, 1987.
  66. 66.
    1. Weinstein CE,
    2. Palmer DR
    . User’s Manual for Those Administering the Learning and Study Strategies Inventory. Clearwater, FL: H & H Publishing, 2002.
  67. 67.
    1. West C,
    2. Sadoski M
    . Do study strategies predict academic performance in medical school? Med Ed. 2011;45(7):696-703. doi: 10.1111/j.1365-2923.2011.03929.x.
    OpenUrlCrossRef
  68. 68.
    1. Chemers MM,
    2. Hu LT,
    3. Garcia BF
    . Academic self-efficacy and first year college student performance and adjustment. J Ed Psychol. 2001;93(1):55. doi: 10.1037/0022-0663.93.1.55.
    OpenUrlCrossRef
  69. 69.
    1. Gredler ME,
    2. Garavalia LS
    . Students' perceptions of their self-regulatory and other-directed study strategies: a factor analysis. Psychol Reports. 2000;86(1):102-108. doi: 10.2466/pr0.2000.86.1.102.
    OpenUrlCrossRefPubMed
  70. 70.
    1. Bartoszewski BL,
    2. Gurung RA
    . Comparing the relationship of learning techniques and exam score. Scholarship Teach Learn Psychol. 2015;1(3):219. doi: 10.1037/stl0000036.
    OpenUrlCrossRef
  71. 71.
    1. Blasiman RN,
    2. Dunlosky J,
    3. Rawson KA
    . The what, how much, and when of study strategies: comparing intended versus actual study behaviour. Memory. 2017;25(6):784-792. doi; 10.1080/09658211.2016.1221974.
    OpenUrlCrossRef
  72. 72.
    1. Gurung RA,
    2. Weidert J,
    3. Jeske A
    . Focusing on how students study. J Scholarship of Teach Learn. 2010:28-35.
  73. 73.
    1. Yan VX,
    2. Thai KP,
    3. Bjork RA
    . Habits and beliefs that guide self-regulated learning: do they vary with mindset? J App Resh Mem Cogn. 2014;3(3):140-152. doi: 10.1016/j.jarmac.2014.04.003.
    OpenUrlCrossRef
  74. 74.
    1. Agarwal PK,
    2. D’Antonio L,
    3. Roediger HL,
    4. McDermott KB,
    5. McDaniel MA
    . Classroom-based programs of retrieval practice reduce middle school and high school students’ test anxiety. J App Resh Mem Cogn. 2014;3(3):131-139. doi: 10.1016/j.jarmac.2014.07.002.
    OpenUrlCrossRef
  75. 75.
    1. Karpicke JD,
    2. Butler AC,
    3. Roediger HL
    . Metacognitive strategies in student learning: do students practise retrieval when they study on their own? Memory. 2009;17(4):471-479. doi: 10.1080/09658210802647009.
    OpenUrlCrossRefPubMed
  76. 76.
    1. Wissman KT,
    2. Rawson KA,
    3. Pyc MA
    . How and when do students use flashcards? Memory. 2012;20(6):568-579. doi: 10.1080/09658211.2012.687052.
    OpenUrlCrossRefPubMed
Back to top

In this issue

Print
Download PDF
Email Article
Citation Tools
Measuring Metacognitive Knowledge, Monitoring, and Control in the Pharmacy Classroom and Experiential Settings
Michelle L. Rivers, John Dunlosky, Adam M. Persky
American Journal of Pharmaceutical Education May 2020, 84 (5) 7730; DOI: 10.5688/ajpe7730
Reddit logo Twitter logo Facebook logo Mendeley logo

Jump to section

Keywords