When first-professional year pharmacy students fail examinations, traditional measures such as Pharmacy College Admission Test (PCAT) scores and overall undergraduate or science grade point average (GPA) often provide little insight into reasons for low achievement. Students who fail often have several things in common: (1) they do not monitor their learning, ie, they do not identify what they know and do not know before a test; (2) they spend inordinate amounts of time reviewing material they already know and not enough time studying information they do not know; and (3) they do not know if their study strategies are “paying off” until after an examination.
Metacognition is a term used to describe skills involved in monitoring learning and making changes in either how or what one studies. There are several common indicators of inadequate metacognitive skills. Text material that is heavily underlined or highlighted, suggests that a student does not know how to identify relevant information. For example, in a textbook used by students in this study, over a page is devoted to the cytoskeleton, including information about the size of each component.1 Students without adequate metacognitive skills could not use the course study guide to identify relevant information such as parts of the cytoskeleton and the function of and processes related to each. Another indicator of inadequate metacognitive skills is resorting to reciting textbook information verbatim when asked to describe a structure or process, suggesting that study strategies are limited to memorizing without understanding relationships. For example, reciting memorized information verbatim when asked to describe action potentials in skeletal, cardiac, and smooth muscle. Another indicator of inadequate metacognitive skills is a poor non-technical vocabulary. For example, students who do not know the meaning of the word sequestering, which is central to pharmacological mechanisms involving metals, cannot understand how such mechanisms work.
Since prior achievement does not always correlate with achievement in pharmacy school,2 among the first indicators that students do not have or use appropriate metacognitive skills are low scores on their first examinations. The objectives of this pilot study were to identify skills related to metacognition, construct a diagnostic test of these skills, and beta-test the diagnostic instrument by administering it to entering pharmacy students. This study was the first step in developing a valid and reliable diagnostic tool or test, called a Learning Skills Checkup which would serve as an early warning system for identifying students with poor metacognitive skills. Students who were identified as not having relevant metacognitive skills could then be counseled to attend workshops or individual sessions to help them develop these skills before the first round of tests. Although opportunities for developing these skills have been available to students at this institution for many years, generally such services are more widely used by students with good metacognitive skills than by students who are struggling.
The pilot study was designed to address 4 questions: (1) Since all items on each part of the test were intended to assess only one skill, did all items on each part of the test actually assess the same metacognitive or information processing skill? (2) Was the test reliable and did items discriminate between the students who did and did not have each skill? (3) Since the metacognitive skills test was based on physiology content and only some of the students had taken a physiology course prior to entering pharmacy school, were test results influenced by prior knowledge? (4) Did the diagnostic test do what it was designed to do: provide a measure of the impact of a variable called metacognitive skills on achievement? If so, higher scores on each part on the test should be reflected by higher scores on the criterion measure: the first examination in physiology.
Several terms have been used to describe the activities involved in checking understanding and making changes based on the results of self-checks. Although psychologists and educators have been aware of the importance of these types of activities for almost a century, according to Brown,3 use of the term metacognition to describe this activity is generally attributed to the pioneering work of Flavell 4 who issued a call for studying the impact of metacognition on learning. Readers who are interested in more information about metacognition should consult either Wienert and Kluwe 5 or Hacker. 6 Terms which have emerged from the field of cognitive psychology are metamemory, metacomprehension, and calibration of comprehension. Maki uses the term metacomprehension to describe the process of monitoring learning from text.7 According to Maki, much of the metacomprehension research has used an error-detection paradigm in which errors are detected in text with missing or incorrect information. Calibration of comprehension is described by Otero as a measure of the relationship between how well readers think they understand text vs. how well they actually can answer questions about it. 8 The term metacognition is used by educational psychologists to address the complexity of this type of activity.
Attempts have been made to identify variables that influence metacognition, beginning with research by Flavell,4 who believed that metacognition was influenced by 3 variables: the learner (self), the task, and the strategy. Among these variables are learner characteristics such as self-perception, verbal skill and ability, motivation, learning task variables, and context or situational variables, with task difficulty being especially important.9 Davidson and Sternberg discuss the importance of identifying relevant information and of forming mental maps or representations.10 The relationship between motivation and metacognition has been the subject of numerous studies since students are more likely to monitor their understanding of information that interests them. One study that provides evidence of the relationship between motivation and comprehension monitoring is a confirmatory factor analytic study of the metacognitive and motivational components of self-regulation. 11 Among the efforts to identify components of metacognition are those of Weinert who identifies 2 variables: evaluation, in which a problem is identified, eg, a student realizes that she does not understand something; and regulation, in which the student takes measures to increase understanding, such as studying more or using different study strategies. 12
Metacognitive activities involve mental activities, which by their nature cannot be observed directly. Therefore 2 methods of inquiry are often used in metacognitive studies. In the first, students evaluate their understanding in terms of feeling-of-knowing (FOK), judgments-of-learning (JOL), or ease-of-learning (EOL) judgments.13 A representative study of this type is that of Tobias and Everson, in which students’ abilities to correctly estimate what they know and do not know were compared to measures of academic achievement. 14 In the second, responses to self-report instruments, such as the Metacognitive Awareness Inventory, are used to assess comprehension monitoring skills. 15
Although the impact of comprehension monitoring or metacognition has been part of the learning research literature for almost a century and has been the focus of systematic study since the mid-1970s, a missing element in the research literature has been the identification of skills that are actually needed to monitor comprehension. This study addresses a call by Flavell to “try to discover the early competencies that serve as building blocks for subsequent acquisitions rather than merely cataloging … lacks and inadequacies.”4
METHODSDue to its information processing demands, one of the most difficult courses for P1 students at this institution is physiology, especially for those who have been accustomed to rote memory learning. A specialist in communicative disorders assisted in identifying skills thought to be related to metacognition in the course (Table 1). Although metacognition has both motivational and cognitive components, the focus of this study was on assessing cognitive skills involved in monitoring understanding.
Selections from the textbook for the course were used in constructing test items.1 Information about the skills included on each part of the original test, the number of items and possible points for each, is summarized and prioritized in Table 1 based on their relevance in the course, where 2 primary methods of delivering information were lectures and diagrams. The number of possible points exceeds the number of items since with the exception of part 5 on vocabulary, items were scored according to the number of required elements in a correct answer with 1 point given for each correct element included. For example, the item requiring students to visualize a sarcomere was scored on 10 elements for a total of 10 points (Appendix 1).
To introduce students to the types of items on the test, less complex structures that were easier to visualize and that were likely to be more familiar to students (eg, a neuron) were presented first, with subsequent items increasing in difficulty. Included in Appendix 1 are instructions for part 1 of the test, “Visualizing Spoken Information,” an actual item from this part of the test, and the elements expected in a correct answer.
The study was reviewed and approved by the institutional IRB. The test was administered during the first week of fall semester. Although the results of prototype testing with individuals suggested that the test could be completed in a 50-minute class period, preliminary test administration activities such as explaining the purpose of the exercise, test instructions, and responding to student questions, significantly reduced actual time remaining for taking the test. After the first part of the test was completed, it became obvious that most students would only have time to complete the first 2 parts of the test: visualizing lecture material and interpreting diagrams. Therefore, only these 2 skills were included in the pilot administration.
Due to the limited availability of raters, all items were scored by the principal investigator. For the pilot study, an arbitrary criterion of 70% correct, with more formal standard-setting procedures to be established in future phases of the study. Each student was provided with an individualized report detailing strengths and weaknesses. Students who scored lower than 70% on either part of the test were advised that although many things could influence scores on this exercise, they should try to improve their study skills in these areas.
An answer was scored as “1” in a spreadsheet if it included expected information and “0” if information was omitted or incorrect. A similar scheme was used to score other responses and an overall score (total possible = 37) was computed for each student. Scores on the first physiology examination were also recorded and the data were uploaded to SPSS, version 12.
Since items on each part of the test were designed to assess only 1 type of skill, eg, visualizing or interpreting diagrams, each reading selection or diagram used in constructing test questions had to meet 2 criteria: (1) present a unidimensional test of the skill being tested, ie, answering a question based on a diagram or text should involve the use of only 1 type of metacognitive skill; and (2) be similar in complexity to other items on that part of the test. Although the number of participants was small (N = 78), exploratory factor analysis (SPSS) was used to examine the dimensionality of items on each part of the test.
SPSS was also used to evaluate test reliability (Cronbach's Alpha) and provide item-total correlations. SPSS was also used to identify cut points for the top and bottom 25% of scores on the test (<22.75 and >30.24). Crosstabs was used to compare the number of desirable response patterns (answered correctly by students in the top 25% of the class but missed by those in the bottom 25%) to the number of undesirable response patterns (missed by students in the top 25% and answered correctly by those in the bottom 25%). A discrimination index was then computed for each item by subtracting the number of undesirable from the number of desirable response patterns. Comparison groups used in computing discriminating indices are shown in Figure 1, with desirable and undesirable response patterns are illustrated by lines A and B respectively. For the purpose of evaluating item discrimination indices relative to total test scores, an item was considered to be discriminating positively if the number of desirable response patterns was at least twice as large as the number of undesirable responses.
SPSS was also used to identify cut points for the top and bottom 25% of scores on the first physiology examination (<65.63 and >81.99) and crosstabs analyses were also used to compare the number of desirable/undesirable responses to each item by students who scored in the top 25% on this examination.
Since test items were based on information which should have already been familiar to students, another concern was the extent to which test scores might be confounded by prior knowledge. The Kolmogorov-Smirnov test for normality was used to evaluate the score distribution of each group. In order to further address the question of whether the test did what it was designed to do, ie, assess the effect of a variable called metacognitive skills on achievement, ANOVA was used to compare the average physiology examination 1 scores of students with and without prior physiology coursework and who scored above and below the class mean on parts 1 and 2 of the test.
RESULTSDue to the small number of participants, the stability of factor analysis results is questionable (Question 1). However, preliminary results suggest that the information processing demands of some reading selections were indeed multidimensional, ie, answering items based on these selections involved more than one skill. For example, the 10 required elements in the reading selection pertaining to the sarcomere (Appendix 1), were all assumed to be related to the same skill: visualizing spoken information. However, the clustering of factor loadings for elements 1-6 and elements 7-10 suggested 2 different skills. A close inspection of this item confirmed the results of factor analyses: Elements 1-6 pertain to visualizing static structures while elements 7-10 pertain to visualizing a process: that of muscle relaxation and contraction.
The overall reliability (Cronbach's alpha) of the Learning Skills test was 0.83 (Question 2). Since each element required in a correct answer was scored as either correct (included) or incorrect (not included), the SPSS reliability program produced item-total correlations for each of these elements. The item with the lowest item-total correlation required students to visualize the parts of a neuron. This item did not discriminate among students, since most answered the question, thereby almost eliminating variability. Items with the 2 highest average item-total correlations were those pertaining to visualizing a sarcomere (r = 0.42) and ion channels (r = 0.39), both of which involved visualizing processes.
Of the 37 scored elements on the test, 29 met the criterion for discriminating positively between students in the top and bottom 25%, while 8 did not. Four of the 8 items with low discrimination indices pertained to the neuron, which most students in all score groups answered correctly. The other 4 elements that did not discriminate positively pertained to interpreting diagrams.
When achievement on Examination 1 was used as the criterion measure for determining whether test items discriminated appropriately, items fared less well. When the criterion for an appropriately discriminating item was that the number of desired responses should be 1.5 times as large as the number of undesired responses, only 50% of the scored elements met that criterion.
When the score distributions of two groups of students, those with and without prior physiology coursework, were compared, the Kolmogorov-Smirnov statistic for the group without prior coursework was 0.102 (p= 0.20); while this statistic for the “prior coursework” group was 0.123 (p= 0.07). These results suggest that neither distribution differed significantly from normal. The average diagnostic test score of students with prior physiology coursework (N = 47) was 26.2 ± 6.2, while the average test score of students without a prior physiology course (N = 31) was 26.8 ± 5.8 (Question 3).
The average physiology examination 1 scores of students with and without a prior physiology course and who scored above and below the class mean on each part of the test are presented in Table 2,Table 3 (Question 4). As shown in Table 2, when criterion scores were analyzed relative to low/high scores on part 1 of the test, the average scores of the 3 groups were similar: groups 1 and 2 = 70.8, group 3 = 72.2, group 4 = 76.6.
As shown in Table 3, the average physiology examination 1 scores of groups 1-3 were also very similar: 71.3, 70.5, and 71.4, respectively. However as in part 1, only the average score (76.3) of students in group 4, (those who had taken a prior physiology course and scored above the mean on this part of the test), was considerably higher than those of students in the other 3 groups.
DISCUSSIONMetacognition has 3 components: skills used in monitoring, actual monitoring activities, and making changes based on the results of monitoring. The focus of most studies pertaining to metacognition has been on methods of assessing the impact of monitoring activities and not on the actual cognitive skills involved in monitoring. One contribution of this study is its focus on this much-neglected area. Although limited in scope due to test administration time, preliminary data from this pilot study suggest that the test did assess skills other than prior knowledge. Data resulting from this study also provided some evidence of the impact of 2 types of metacognitive skills on achievement: visualizing spoken information and interpreting diagrams.
Although data suggested that test scores were not unduly influenced by prior knowledge, one unexpected finding of the study was the seemingly differential effect of metacognitive skills on students who had and had not had a prior physiology course. Data in Table 2, Table 3 suggest that the impact of metacognitive skills on examination 1 scores was greatest for students who had taken a preprofessional physiology course. One interpretation of these results is that metacognitive skills may have more of an impact on retention than on initial learning.
A major outcome of this pilot study was insight gained into developing and administering this type of diagnostic tool. One of the most important considerations in instrument construction is the information processing requirements of selections used in constructing test questions: questions pertaining to processes will result in the most useful information. Both from the standpoint of ease of scoring and test reliability, when developing questions that test students’ ability to interpret diagrams, objectively scored questions are preferable to open-ended items. Instead, objective questions should be constructed over parts of a process. For example, when revising the item pertaining to protein synthesis, the open-ended statement/item “Identify the main points of the diagram,” will be replaced by questions such as “Where does the process start?,” and “How many paths can a protein take before it leaves a cell?” By numbering or lettering each “protein synthesis” path, questions can be constructed about differences in the paths, eg, 2 pertain to proteins that leave the cell, while 1 does not. Other considerations are beta-testing and test-administration time. Although beta testing with 2-3 individual students, as was done in this study, provides useful information about revision needs, beta testing with as large a group as possible provides even more useful information. Ideally diagnostic testing should be done before classes begin and time should not be limited to one class period. In subsequent administrations of similar diagnostic tests in other professional programs at this institution (nursing and medicine), about 90 minutes has been required for most students to complete a test similar to the one outlined in the Test Plan Table in Table 1.
A practical limitation of this type of assessment is the time required for scoring each test. In order to be useful, students must receive feedback very quickly. In this pilot study, results were available 2 days after the test was administered. Reliably scoring 78 tests within these time constraints was a Herculean task and may have introduced some scorer bias. Although measures were used to reduce this type of bias, such as all tests being scored by 1 person using stringent criteria for judging answers, the scoring of constructed-response items remains a potential source of error variance.
Our plans are to revise the diagnostic test. Questions with item-total correlation coefficients below 0.35 were targeted for revision and/or replacement in future versions. Items that appear to be influenced by prior knowledge will be replaced in future revisions. Although pilot testing of this diagnostic instrument did not suggest any confounding by prior knowledge, using content that should already be somewhat familiar to students does have the potential for this type of confounding so additional study of possible confounding effects of prior knowledge is needed. In this pilot study, the decision was made to err on the side of background information enhancing learning skills instead of obscuring them.
Due to the limited number of skills included on the diagnostic test in this pilot study, another focus of future studies should be the identification of other cognitive skills related to metacognition, eg, skills such as distinguishing between relevant and irrelevant information or knowledge of learning tasks, ie, what one must know or be able to do to demonstrate that he or she knows something. Another needed focus is whether such skills are course specific. Preliminary findings of other studies being conducted at this institution suggest that skills used in metacognition vary considerably from course to course. Although visualizing structures or processes may be very important in a physiology course, it may be less important in therapeutics, where more relevant metacognitive skills are condensing and organizing to identify similarities and differences in treatment regimens for closely related disorders.
An interesting finding resulting from this pilot study was a seemingly differential effect of both metacognitive-related skills on initial learning and retention. Although these results may have been influenced somewhat by the large standard deviation of examination scores, we will continue to investigate this possible differential effect.
Although a follow-up session also provided students with suggestions for improving skills in each area, it did not provide opportunities to practice these skills. Moreover the extent to which some students took advantage of this feedback to improve their skills or the extent to which any subsequent improvement may have influenced scores in the criterion measure is not known.
Finally, some would argue that this type of assessment is not needed. They assume that students who do not have appropriate metacognitive skills when they enter pharmacy school will somehow eventually develop them. Analysis of end-of-course achievement data of this group strongly suggests otherwise. Of those students who scored in the bottom 25% on the first physiology examination, 50% were still in the bottom 25% at the end of the course. These data suggest that unless metacognitive skills deficits are identified and remedied, students who do not already have these skills will not develop them on their own.
Assistance in developing learning skills has been available for the students at this institution for several years. However, until this study, there was little information about either the specific types of skills needed in this P1 course or the effect of not having these skills. These data will permit us to be more proactive in identifying students who do not have these skills and in providing structured programs, thereby remedying learning skills deficits before they negatively influence achievement throughout pharmacy school.
CONCLUSIONThis study provided a shift in the focus of metacognitive research from methods of assessing the impact of monitoring to one of identifying skills actually used in monitoring. Preliminary data from this pilot study suggest that 2 cognitive skills related to metacognition, visualizing lecture information and interpreting diagrams, are independent of content, and that these skills may have more impact on retention than on initial learning. More importantly, it resulted in some guidelines for developing more valid measures of the skills related to metacognition. If administered during the first few days of the semester, measures such as these will be useful in identifying students who may benefit from structured interventions to improve their study skills. The value of this type of assessment is expected to increase through continued revisions of the test. As revisions are made in test items and more time is available for testing, we will be able to better assess the relative contributions of different types of skills and their impact on academic achievement.
