The Impact of Eliminating Backward Navigation on Computerized Examination Scores and Completion Time

INTRODUCTION

Computer-based testing has quickly replaced paper-based testing to become the standard in higher education. ^1-3 Computer-based testing provides a number of options that were not possible with paper examinations including: question and answer randomization, missing answer reminders, and elimination of backward navigation. Elimination of backward navigation restricts examination takers’ ability to return to questions they have already answered. Students are not permitted to skip a question and return. Additional computer coding of the examination, when activated, will not allow students to leave a question without giving a response.

Potential benefits to restricting backward navigation include improved examination security and the ability to include a series of questions where the previous question may provide the answer to a subsequent question. Several articles that address methods to improve security with computer-based testing have been published, most of which focus on access to the examination material and how the examination is administered.^2-5 The literature suggests that removing a student’s ability to revisit questions on an examination is generally unpopular with examinees; however, there is limited information regarding the impact that this decision has on test efficiency and performance of graduate or professional students. ^6-9 The added imperative to provide online and virtual education created by the 2019 SARS-CoV-2 pandemic, has increased educators’ concerns about the computerized testing process. The lack of literature addressing the impact of restricting backward navigation makes it difficult for educators to decide whether or not to implement the feature.

One benefit to prohibiting backward navigation is that doing so provides an opportunity for students to gain experience taking an examination in which reverse navigation is not allowed as this is a restriction placed on both the North American Pharmacists Licensure Examination (NAPLEX) and the Multistate Pharmacy Jurisprudence Examination (MPJE), standardized tests that are required for pharmacist licensure in the United States. Studies to date have indicated that removing backwards navigation on examinations may impact performance at the elementary- and middle-school levels but has not been shown to have a significant impact on test performance at the high school and college levels.^10,11 The purpose of this evaluation was to determine whether the elimination of backward navigation led to changes in students’ examination score or the time required to complete the examination in a four-year Doctor of Pharmacy (PharmD) program when compared to student performance on the same examination from the previous year when backwards navigation was allowed. At the time the decision was made to restrict backward navigation, these two potential negative impacts were identified by both faculty members and students as undesirable. Because of the lack of sufficient literature to determine whether these potential negative impacts occur, this evaluation was developed to monitor both change in score on identical questions and the time to complete the examination.

METHODS

Backward navigation was available for examinations in all courses prior to the 2019 academic year. Prior to the beginning of the fall 2019 semester, the decision was made to limit backward navigation for examinations in courses originating in the University of Nebraska Medical Center College of Pharmacy Practice and Science. Six examinations, two in each of the first, second, and third professional (P1-P3) years, were identified in courses where backward navigation had been eliminated. The pedagogy in the P1 courses consisted mainly of lecture with active-learning methods mixed throughout. The P2 and P3 courses used team-based learning with a mix of active learning through case presentations and other simulated patient interaction exercises. Each examination given in the fall of 2019 was compared to a similar examination given in the fall of 2018. Examination content, topic presentation order, and lecturer were largely the same between the two years. While new lecturers were rare in the courses studied, in all instances, these instructors created new examination questions. Therefore, none of these questions would have been included in our primary analysis of identical questions from year to year.

The primary comparison of interest was the change in mean score for a subset of identical questions given on both 2018 and 2019 examinations. Questions were required to have the same question identification number and revision number on the 2018 and 2019 examinations to be included in the primary analysis. The examination software used assigned a new revision number for any change in the question or in the answers, including spelling corrections or replacement of distractors (incorrect answer). Questions with identical stems but different revision numbers were excluded from evaluation. Mean scores for the total examination (new and previously used questions) were also compared. Only multiple-choice and true/false questions were included from the analysis. Essay questions were rarely used on the six examinations and excluded from the analysis.

One examination given in the P3 year was reorganized in 2019. Assessment of learning in the infectious disease section of the pharmacotherapy course was divided into two examinations for the 2018 semester. The same content was divided into three examinations for the 2019 year. Because the 2019 examination questions were included in the first two examinations given during the 2018 year, a direct comparison of overall examination time and total examination score is not possible. A comparison of change between the 2018 and 2019 examinations for identical questions (primary outcome) was performed.

Three student demographic variables were collected: age at the beginning of their P1 year, sex, and grade point average (GPA) at the end of the P1 year.^12,13 These variables were included in a linear regression model to control for differences in student characteristics. The P1 GPA represents the cumulative grade point average of all courses completed during the first year of the four-year program.

The average time students spent answering each question was compared for each examination by dividing the total student examination completion time by the total number of questions. Median, 90%, and maximum completion times were also reported.

All examination data were extracted from ExamSoft testing software (Dallas, TX). The Student t test was used to compare age, P1 GPA, examination scores, and average time spent on examination questions. The chi-square test was used to evaluate differences in sex. A p value of <.05 was considered statistically significant. All statistical comparisons were performed using Stata 14.2 (StataCorp. 2015. College Station, TX). The UNMC Institutional Review Board determined that this project did not constitute human subject research.

RESULTS

Student age, sex, and P1 GPA are reported in Table 1. Student demographics were similar between 2018 and 2019 students for each of the P1-P3 classes. The primary outcome of interest was change in score for a subset of identical questions included on both the 2018 and 2019 examinations. Change in mean score ranged from +3.3% (+0.6 questions) for the P1 Pharmacy and Healthcare examination to -2.6% (-0.5 questions) for the Pharmaceutical Care examination (Table 2). Four of the six mean examination scores were slightly lower after eliminating backward navigation; however, none of the changes were significant. Because there was only one identical true/false question on all six examinations combined, no analysis could be done based on question type.

Changes to total examination scores were also evaluated. The total examination score for the second nutrition examination given in the P2 year was 4% higher after elimination of backward navigation (78% vs 82%; p=.02). No significant differences between total examination scores were observed for any other examination (Table 3).

Change in time to complete the examinations was evaluated using four metrics. First, mean time spent on each question was calculated to provide a normalized measure, so examinations with different numbers of questions could be compared. Mean time students spent on each question was significantly reduced for the no backward navigation group on two of the five examinations compared (Table 4). Median, 90% student completion, and maximum examination time was evaluated to determine whether slower examination takers had sufficient time to complete the examination.

Age, gender, and student performance were postulated as potentially confounding variables by students and faculty members.^12,13 The P1 GPA was used as a proxy for student performance. Even though no differences in demographic characteristics exist (Table 1), linear regression models were conducted as planned to control for potential confounding. Regression models were only created for the primary outcome, which was change in score for identical questions (Table 5). After controlling for student characteristics, changes between the 2018 and 2019 examinations either remained the same or moved closer to the null when compared to the unadjusted analysis. No changes in adjusted scores were significant.

DISCUSSION

This evaluation did not find a negative impact on pharmacy student examination performance after eliminating backwards navigation. Across courses in the first three years of the professional program, average student scores on a set of identical questions did not decrease as a result of eliminating backwards navigation. Similarly, the average time spent per question did not increase as a result of eliminating backwards navigation. These findings are important because they represent the two primary unwanted and unintentional consequences of restricting students’ ability to revisit test questions.

Students had two hours to complete the P1 examinations evaluated in this study. The four remaining examinations were three hours in length. We did not see any evidence that restricting backward navigation resulted in longer examination completion times. There are no institutional guidelines defining examination length. Examination length is determined by each course or section coordinator. Assurance that time to complete the examinations would not be increased was important to both students and faculty members.

When faced with the elimination of backwards navigation, it is reasonable to expect student anxiety and concern, particularly in a rigorous discipline such as pharmacy, where a slight decrease in examination performance can halt a students’ academic progress. At many schools and colleges of pharmacy, courses are only offered once per year, which means that failure may result in a “lost” year for the student. Even for students at no risk of failing a course, a difference of a few points on their examination score may result in a lower course grade, which can affect their ability to attain scholarships and residency placement. With such serious stakes, it is easy to understand student anxiety about any process or procedural change that could negatively impact their performance. This evaluation was intended to determine if a negative impact was seen following the elimination of reverse or backwards navigation during an examination administered online. It was conducted in part to quell student anxiety about this change and in part to offer reassurance to faculty members who shared similar concerns.

Changes to how an examination is administered are of particular concern when they run counter to how students were trained to take a test. Many students have been taught to skim through the entire examination before starting to answer questions or to work through an examination answering the questions they know immediately while skipping questions that require more time and returning to them later.¹⁴ This approach is intended to give students confidence and help them manage their time by not lingering on difficult questions to the point that they have to rush through other questions. Because examinations are a means of measuring knowledge rather than a method of evaluating testing strategies, it was important that scores and testing duration not change when compared to those students who used backwards navigation. A similar study by Caetano and Pawasauskas likewise showed no overall impact on student performance on items or questions when backwards navigation was prohibited; however, that study did not compare student examination performance between class years before and after the new rule was added as we did.¹¹

The benefits of eliminating backwards navigation from an examination have to be weighed against the impact on student performance as well as their concerns and anxiety. With computer-based testing, faculty members have opportunities to assess learning that are not possible with paper-based tests. One advantage of eliminating backwards navigation is the ability to use questions that build on the content of previous questions without giving away the answer to the previous question. For example, in a series of questions regarding a specific patient, the first question might test the student’s ability to correctly assess the patient’s medical condition. The student would need to select the correct answer, cellulitis, from multiple choices, eg, cellulitis, osteomyelitis, and others. The next question might ask the student to select the most appropriate antibiotic to use in treating the patient’s cellulitis. The student would then know that the correct response to the previous question was “cellulitis” and, using backward navigation, could return to the previous question and change her answer if she had not selected “cellulitis.” If the student had not been able to use backwards navigation, the instructor in this hypothetical case could have better assessed the ability of the student to treat cellulitis regardless of whether the student had answered the first question correctly. However, with backward navigation available, the instructor would be forced to construct all subsequent questions in the series so that the correct answer, “cellulitis,” would not be divulged. The student, who may have thought the case involved osteomyelitis rather than cellulitis, and therefore had selected quinolone in the second question as the correct antibiotic to use, might then select “monitor the patient’s response to the quinolone antibiotic” as the answer to a third question on long-term treatment. Without backwards navigation, however, the student is not at risk of this “waterfall” of missed questions occurring because they got the first of the three questions wrong. The instructor would be able to specify that the patient’s condition was cellulitis in question 2, regardless of whether the student had answered incorrectly on question 1. The student’s ability to choose the correct antibiotic (eg, β-lactam) and correct monitoring of the patient could then be assessed. In this scenario, the removal of backward navigation not only provided a better assessment of the student’s knowledge of individual concepts covered in the examination, but also avoided having one wrong answer impact the student’s responses to subsequent questions where she may have known the correct answer.

Another advantage of eliminating backwards navigation is related to examination integrity. Creating reliable and valid assessments, particularly multiple-choice questions, is a time-consuming process so it is reasonable that precautionary measures should be taken to protect those questions. While students excused to leave the examination room to go to the restroom or for other reasons have always had the opportunity to seek out answers to questions they have seen on the examination, current technology has made it easier to get information about the examination material and more difficult for examination proctors to monitor. While eliminating backwards navigation does not remove the potential for a student leaving the classroom to review material, it does reduce the number of answers that can be changed upon their return. Likewise, given the requirements of both Title IX and the Americans with Disabilities Act, restricting a student from leaving the classroom during an examination is not always possible.

Restricting backward navigation may also benefit examination integrity within the classroom. When backward navigation is available, an examination taker may see an answer selected on another student’s computer screen and navigate forward or backward to change their answer. With backward navigation eliminated, the student cannot change responses to questions already completed, and cannot skip ahead in the examination to find a question not yet encountered. Enhanced examination integrity is a benefit to students as well. Students are anxious, not only about their own performance during tests, but also the performance of classmates. With scholarships and grant dollars becoming more limited, confidence that examination scores and grade point averages will not be impacted by academic dishonesty is a relief to students, as well as to faculty members.

Finally, the ability for instructors to mirror the testing protocols used for both the NAPLEX and MPJE, which prohibit backwards navigation, will provide students with multiple opportunities to practice taking tests with this restriction. While the primary goal of all pharmacy educators is to train high-quality practitioners, students’ ability to successfully pass both the NAPLEX and MPJE is critical to being licensed.

The current study has several limitations. The study did not evaluate the impact of eliminating backward navigation on student test anxiety. When the new policy was announced, student leaders expressed their concerns about both test performance and anxiety. The concerns were particularly high among second- and third- year pharmacy students, for whom this would be a change in testing procedures from previous years. For the first-year pharmacy students, this was likely viewed as less of a disruptive change. In an effort to be transparent and allay fears, course coordinators agreed to monitor and communicate examination performance relative to past years. While this likely did not eliminate all student concerns, anecdotal evidence suggests that anxiety and concerns went down over the course of the semester.

This evaluation consisted almost exclusively of multiple-choice questions, and no essay or short-answer questions were evaluated. While graduating class years are generally comparable, we did control for differences in GPA at the end of the second semester of the program, sex of the student, and student age. There may be other student and/or faculty characteristics that were not included in our analysis that might impact examination performance. For example, comradery and study habits were not evaluated, for instance. The impact on the subgroup of students who were given more time to complete the examination was not analyzed because of the small sample size.

Despite its limitations, our study reports on data not previously available regarding the impact of changing navigation through a computerized examination at the level of a professional, graduate program. As computer-based testing becomes more common and faculty members have more decisions to make about the assessment features they use, it is important to consider the potential impacts on student performance. It is also important to consider timing and communication of any assessment changes. Clearly communicating the reasons for changes in testing procedures to students may not eliminate all concerns, but it can help to reduce some of the confusion, anxiety, and potential animosity. Other approaches include transitioning the change gradually with each incoming first-year class, instead of changing examination procedure across all years of the program at once. Finally, giving students an opportunity to take a low-stakes or practice examination that restricts navigation may provide students an opportunity to familiarize themselves with the changes without the pressure of a high-stakes summative assessment.

CONCLUSION

Changes to evaluation processes and procedures in PharmD program should assess the potential unintended consequences on learner performance. In this initial evaluation of the impact of eliminating backwards navigation in a professional pharmacy program, a model was developed for assessing the impact on examination scores and test time. Across all three years of the didactic pharmacy curriculum, no adverse effect on scores or testing time was found.