Factors Affecting Student Time to Examination Completion

INTRODUCTION

Self-regulation is the ability to act in one’s long-term best interest.¹ Self-regulation is a critical influencer of time-to-complete a given task (eg, an examination) because self-regulation directs motivation and attention (eg, what material needs attention).^2,3 One’s ability to self-regulate is an interaction of prior experiences, personality, environment, and ability to monitor one’s own thinking (ie, metacognition). There is very little research on factors that determine the self-regulation of students during an examination. This exploratory analysis examines potential factors that might influence how long it takes students to complete a high stakes, cumulative examination with time to completion being a measure of self-regulation.

Learners perform best when they approach a learning situation with prior knowledge related to the situation.^4,5 With more prior knowledge, a learner can detect and recognize features, generate a solution and perform more quickly and accurately; this is the nature of expertise.^4,5 As such, students entering an examination with high prior knowledge should complete examinations more quickly. In this study, prior knowledge was assessed with a cumulative, low stakes assessment that occurred two weeks prior to the final examination.

As learners’ progress toward expertise, they gain more knowledge and develop better metacognitive monitoring allowing them to judge how well they know a given fact or can complete a given skill. Metacognition is a central practice in self-regulated learning. The accuracy of metacognitive monitoring can be affected by various factors, including problem or test difficulty, background or prior knowledge, and desired performance level.^6-8 More metacognitively accurate individuals may have extended testing times due to increased processing demands – that is, they engage in more thinking when faced with a problem.^9,10 For example, students may use how fast they could answer a question as an anchor in determining how well they did on an examination. However, the speed of which answers come to mind do not relate to performance and some evidence suggest the slower answering times were correct more often.^11,12 Even if students had some level of metacognitive monitoring during the learning process, they may not apply it in a testing environment.^7,13 These students could be faster or slower in completing an examination depending on their metacognitive ability. In this study, metacognitive accuracy was assessed by bias and absolute bias. Bias is the difference between the predicted score and actual performance and includes both directionality (ie, under- or over-confident) and magnitude of that directionality; absolute bias captures the magnitude or accuracy only. It is expected that learners with higher degrees of overconfidence (predicted score > actual score) may proceed more quickly through an examination compared to students with stronger degrees of underconfidence (predicted score < actual score).

Time to complete an examination requires time management skills and these skills may be the driver for the link between personality traits and academic achievement.¹⁴ Personality research has established a five-factor structure, which has the dimensions of openness to experience, conscientiousness, extraversion, agreeableness, and emotional stability (or neuroticism).¹⁵ Numerous studies have examined the impact of the five-factor personality assessment on academic achievement, and of these studies, conscientiousness correlates the most with academic achievement.^16-20 It has been hypothesized that this relationship is due to the use of better time management strategies, and there is a positive relationship between higher levels of time management skills and higher levels of conscientiousness.^21,22 Openness to experience appears to be positively correlated to academic achievement.²⁰ Emotional stability (or neuroticism) and achievement have shown a negative correlation, suggesting that elevated emotional instability places learners at the risk of reduced academic achievement. Agreeableness and extraversion have shown mixed and inconclusive results in relation to academic achievement.^20,23As such, there may be a relationship between the five-factor personality traits and time management during examination completion. In this study, the five-factor personality was used because of its reliability, validity and prior research investigating its relationship to academic achievement.²⁴

Taken together, the authors examined the impact of prior knowledge, current knowledge, metacognitive accuracy and personality on time to complete an examination. The findings from this study can be used to help develop better study or test taking strategies for students or allow for better time allocation for examination taking.

METHODS

Participants were student pharmacists enrolled in the Doctor of Pharmacy (PharmD) program at a large, public university in the southeastern United States. In that program, pharmacokinetics was a required 3-credit course in which students met once a week for three hours and course format has been described previously.²⁵ During a two-year period, 295 students were enrolled in the course, spread across two campuses that were synchronously video-conferenced. Upon admission to the program, the average age of students was 23 years (range 19-50), and approximately 65% were female. The mean grade point average upon admission was 3.5 (out of 4.0), the mean Pharmacy College Admission Test (PCAT) score was approximately 87%, and approximately 80% of students had a prior degree.

The course consisted of two parts. Part one used team-based learning (TBL) and occurred over the first 10 weeks of the semester. There were five main sections (pharmacodynamics, single dose kinetics, approach to steady-state, violations of the one-compartment model, and physiologic influences on clearance) that accounted for nine different topics. Each section started with the readiness assurance process (ie, individual assessment followed by team-assessment) and the remaining time dedicated to cases. Part two of the course occurred during the last five weeks of the semester and involved students completing one out-of-class and one in-class case both of which integrated the course material.

Prior Knowledge. Four 36-question cumulative assessments were administered during the course. These examinations were used to improve learning through the “testing effect” and allow students to improve their metacognitive monitoring. Each of these examinations sampled from the nine major topics covered in the course. The first three assessments were unique based on a random drawing of four questions from each of nine question pools (corresponding to the nine topics), for a total of 36 questions on each test. Each pool contained 12 to 20 questions which had been used in previous years of the course. Over 90% of the questions were at the level of application or higher according to Bloom’s Cognitive Taxonomy.²⁶ These first three examinations were administered online and were low stakes. The first examination (week 1) provided a baseline of performance; the second examination (week 9) occurred at the completion of content (week 9); the third examination (week 14) occurred midway through review and integration activities occurring in the second part of the course. This third assessment was used as the measure of prior knowledge.

Current Knowledge Assessment. The fourth examination was the final examination (week 16) that consisted of newly created questions compared to the questions within the question pools. This examination was administered in-class and employed an answer-until-correct format (IF-AT, Epstein Educational Enterprise) in which students received immediate feedback on the accuracy of each response.²⁷ Students used the same answer-until-correct format for the readiness assurance process during the TBL section of the course (weeks 1 through 10). The final examination was used as the measure of current knowledge. Scores were examined with and without partial credit for two reasons. The first reason is the first three examinations had no partial-credit, thus it is easier to compare scores across examinations. The second reason is learners would have extended testing times when correcting errors.

Self-regulation Time to Completion. During the final examination, either the students or the instructor recorded the clock time the examination was submitted. Duration to complete the examination was based on this recorded clock time.

Metacognitive Monitoring. At the start of each of the four exams, students made a global prediction of their performance: “On this assessment, I expect to receive a __%”. A bias score was calculated (predicted score – actual score) to determine degree of overconfidence (positive values of bias) and underconfidence (negative values of bias).²⁸ An absolute bias score (absolute value of the difference between the predicted performance and actual performance) was calculated, determining the students’ metacognitive accuracy. ²⁸ Students were asked to make these judgments on each examination to help develop their metacognitive monitoring skills. For this study, only the measures of bias or absolute bias on the final examination were used.

Personality Assessment. Prior to the semester, students completed the five-factor personality (eg, Big Five, NEO-I) test that was freely available (http://www.outofservice.com/bigfive/) to assist in team assignments. Scores were recorded on a 0% to 100% scale with higher numbers indicating a stronger trait.

All variables are summarized in Table 1. Data were graphed to examine the relationship between time to complete the examination and each factor of interest. Based on visual inspection, data was analyzed both as an entire cohort and according to completion time tertiles to discern regional effects. Spearman correlations were performed on all variables using SPSS software (IBM Corporation, Armonk, NY). A one-way ANOVA with Tukey post-hoc test was used to compare tertiles. Finally, multiple linear regression was performed using entry criteria on the entire cohort and each tertile. This study was deemed exempt from review by the Institutional Review Board. Statistical significance was set at p<.05.

RESULTS

There were 266 participants who completed the data sets. The full correlation matrix can be found in Appendix 1. Overall, time to complete the final examination was negatively correlated with the final examination performance with and without partial credit (Table 2). Time to complete the final examination also was positively correlated with agreeableness and metacognitive bias. These findings suggest students who completed the examination quicker performed better, had a lower degree of the agreeableness trait and had higher degrees of underconfidence/lower degrees of overconfidence However, there were some regional differences. In the fastest tertile, there were significant negative relationship with time to complete the final examination and final examination score with and without partial credit; there was no relationship with personality or metacognitive monitoring. In the middle tertile, agreeableness was associated positively with time to complete the final examination. For the slowest tertile, there was a significant negative association with final examination completion time and performance with and without partial credit.

Each time tertile was compared for significant differences between each variable of interest (Table 3). The fastest tertile completed the examination on average an hour earlier than the slowest third and approximately a half-hour earlier than the middle third. The fastest students performed less than half a grade better than the slowest students. There was no differences in personality or metacognitive monitoring between any of the tertiles.

Finally, a regression analysis was performed on each time tertile and the entire cohort (Table 4). For the entire cohort, agreeableness and current knowledge significantly predicted time to complete the final examination, explaining 14% of the variability; these results parallel the correlation analysis. Metacognitive accuracy, as measured by absolute bias, predicted time to complete the examination for the early completers only. However, agreeableness predicted time to complete the examination for the middle tertile. Finally, current knowledge and metacognitive bias predicted time to complete the examination for the slowest tertile.

DISCUSSION

This is one of the first studies that examined self-regulation as a function of the time to complete an authentic, in-class assessment. Factors including current and prior knowledge, metacognitive accuracy and personality traits were examined. All three factors seem to play a role in the self-regulation of completing an examination.

The first hypothesis was students with a higher level of prior knowledge based on in-semester assessments would complete the task quicker. Prior knowledge did not affect time to complete the examination. Even though prior knowledge had no relationship with examination completion time, current knowledge seemed more influential. Overall, the authors found that current knowledge predicted time to complete the examination. The discrepancy between the effects of prior and current knowledge could be caused by students studying the two weeks leading up the final examination. This acute study and practice lead to additional learning or a higher level of retrieval strength and thus, higher overall performance. These learners may move through examination questions with more confidence because answers come to mind more quickly, reducing time per question and overall review time.

The authors hypothesized that metacognitive monitoring played a role in examination time and had limited effects. The limited results suggest that students who tend toward overconfidence (or less underconfidence) completed the task more slowly. This may relate to their use of study strategies prior to the examination. Typically, the lowest-performing students tend to show the greatest overconfidence (ie, the “unskilled-and-unaware” effect) even though this was not true in this study – there was no statistical difference in metacognitive bias or accuracy.²⁹ However, this overconfidence can have negative effects on the efficacy of students’ short-term study behaviors by underpreparing for examinations or massing their practice instead of spacing it appropriately.^29,30 If students massed their practice, they would have high retrieval strength for the knowledge and skills but low storage strength, and thus, lower long-term retention.³¹ Their ability to cram can lead to high performance. However, the metacognitive measures had limited predictive utility. This may be due to the answer-until-correct format of the final assessment. Students were aware of how they were scoring as they went through each question, eliminating time that would have been spent second-guessing or re-reading questions after providing an original answer. Judgment of their understanding was provided externally as feedback immediately after answering a question and thus was not dependent on internal metacognitive monitoring. In addition, students practiced their metacognitive judgments and became more accurate despite the presence of inter-student variability (data not shown). The metacognitive monitoring may have been a more important factor if students did not practice their judgments over the course of the semester and did not receive external feedback from the examination format.

The final factor investigated was personality. Within this study, only agreeableness seems to play a role. Individuals high in agreeableness are characterized as kind, appreciative, generous, cooperative and friendly; those that score low are characterized as fault finding, quarrelsome and thankless.²⁴ This trait may explain time to complete the examination based of its relationship to social desirability, self-regulation, and boredom.^32-37 On one hand, the “social do” may dictate when students should turn in their examination rather than when they were completed with the examination – that is, “I should turn it in after a certain time but I should not be the first to turn it in.” Students may feel social pressure on the appropriate timing to submit an examination. This might be modulated by the inverse relationship between agreeableness and boredom with high agreeableness showing less boredom and may persist through a task.³³ In addition, agreeableness is related to self-control and effortful control. Thus, individuals with a higher agreeableness trait can modulate the emotional controls and persist through tasks like completing an examination.³⁶ Thus in this study, students with higher levels of agreeableness were slower in completing their examination because of social desirability and their ability to modulate emotions, which may result in a slower completion time.

The factors explored explained a moderate amount of the variability and that human behavior is often quite complex. There are three potential confounders within the study: problem difficulty, working memory capacity, and metacognitive practice. The first issue is problem difficulty or how easy or hard a problem is. Hoffman and colleagues found that differences in metacognitive prediction depends on problem difficulty with metacognitive prompting, and self-efficacy having a stronger interaction for more complex math problems.³⁸ This may or may not be a factor given the high practice levels of students prior to the final examination leading to more automaticity and high overall performance. Another potential factor is working memory capacity. In one study, higher working memory capacity was correlated with faster improvements in response time.³⁹ Working memory capacity was not investigated within the study but may explain another section of the variance within the study. Finally, students practiced their metacognitive judgments, thus all students became more accurate over time. It is unclear if the results would change if students did not practice these judgments, which could lead to larger differences in accuracy and larger degrees of underconfidence and overconfidence.^40,41

CONCLUSION

Current knowledge, and to a lesser extent, pro-social behavior (ie, agreeableness) influenced examination completion time. Metacognitive monitoring had limited predictability in time to complete the examination. Regions of the performance versus time relationship (eg, slow vs fast) may be governed by different drivers of self-regulation and these relationships may be complex. In general, instructors may have to help students develop behaviors to overcome natural tendencies in their personality to enhance performance. More importantly, time-limited examinations may negatively affect students with certain personality traits.