Comparison of Answer-Until-Correct and Full-Credit Assessments in a Team-based Learning Course

INTRODUCTION

A variety of methods are used to grade assessments in courses that use team-based learning (TBL). Pharmacy educators use of the Immediate Feedback Assessment Technique, or theIF-AT (Epstein Educational Enterprises, Cincinnati, OH), on tRATs and on team examinations.^1-7 In Allen and colleagues survey of faculty members from colleges’ and schools of pharmacy in the United States, 74% of respondents who had implemented TBL were using the IF-AT forms for assessment.⁸ The IF-AT card is a response form with a preset answer key designed to be used with multiple-choice questions.^9-11 Students work in teams to determine the best answer to an assessment item and scratch the area under the answer they think is correct. A star is revealed when students answer correctly, and a blank box is revealed when students answer incorrectly. Following an incorrect answer, students have the opportunity to scratch the area under another answer choice until the correct answer is revealed. This method of answer-until-correct allows for partial credit to be awarded for each item of the assessment. It also provides immediate feedback for students and faculty members.

Of those who report using IF-AT forms in TBL sessions, only 2 specify using a partial grading method.^3-4 However, the grading method is not compared to other methods in these reports. Persky and colleagues assessed the impact of IF-AT forms compared to traditional assessment methods (ie, multiple-choice, true/false, short-answer) for individual course examinations in a pharmacokinetics course not using TBL.¹² They found no differences in student performance but did find that students preferred IF-AT forms over traditional assessment methods. Published comparisons of student performance in pharmacy education using IF-AT forms for team assessments in TBL courses do not describe the impact on the quality of team interactions.

The purpose of this study was to assess the impact of awarding partial credit to team assessments using an answer-until-correct method compared to traditional methods of grading on both student performance and quality of team interactions.

METHODS

Team-based learning was implemented in 2008 in an elective course titled Ambulatory Care.¹³ Since this time, the course has been offered annually during the fall semester to third-year student pharmacists and has used TBL as the sole instructional method. The course directors consistently incorporate the core elements of TBL in the course structure.¹⁴ During the 2013 fall semester, IF-AT forms were introduced to the course to be used as the assessment tool for tRATs.

We conducted a retrospective longitudinal study that included students enrolled in the ambulatory care elective in the 2010, 2011, and 2013 fall semesters. The control group (full-credit) consisted of students enrolled in the course in the 2010 and 2011fall semesters. The course used a traditional, full-credit assessment method for tRATs at those times. Students from those course offerings were identified as the control group because they had completed the Team Performance Scale as a component of the course to assess quality of team interactions. The intervention group (answer-until-correct) consisted of students enrolled in the course in the 2013 fall semester when IF-AT forms were used to grade tRATs. Topics included in each course offering varied slightly from year to year based on instructor availability. Table 1 displays the topics included in each course offering. Topics were consistent in the course except for anticoagulation part 2, women’s health, and chronic noncancer pain. Students were divided into 5-6 member teams. At the start of each class session, students completed an iRAT, and immediately after, the tRAT. The topic instructor then led a discussion of misconceptions identified by the tRAT. Students completed the team application activity, which was followed by interteam discussions led by the topic instructor. Students completed one course examination at the conclusion of the course, first as individuals, then as teams. All assessments comparing individual and team performance used the same items.

All iRATs and tRATs consisted of 10 multiple-choice questions with 4 answer choices. The total possible points earned on tRATs changed when the answer-until-correct method was implemented. The full-credit group could earn up to 10 possible points on the tRAT, 1 point per question. The answer-until-correct group could earn up to 40 possible points on the tRAT. Partial credit was awarded based on the number of answer choices revealed on the IF-AT form. Full credit (4 points) was awarded when students revealed 1 answer choice. Half credit (2 points) was awarded when students revealed 2 answer choices. Students who revealed 3 and 4 answer choices were awarded 1 and 0 points, respectively. Grades from the answer-until-correct group’s tRATs were scaled to a 10-point scale for comparison.

Study outcomes included student performance on iRATs, tRATs, individual final examination, team final examination, and student-rated quality of team interactions using the Team Performance Scale, an 18-item survey that has evidence for reliability (Cronbach α=0.97) to assess the quality of team interactions.¹⁵ Each item was assessed on a 0-6 Likert scale with 0 representing none of the time and 6 representing all of the time (highest possible composite score was 108). Higher composite scores indicated higher performing teams.

Descriptive statistics were used to summarize student performance on the iRAT and tRAT by topic and overall and on the Team Performance Scale. For statistical comparison, number of correct items (raw score) was used as opposed to percent correct. Total possible points for the iRAT and tRAT were 110 points each (1 point per item, 10-item RATs, 11 RATs in each course). Total possible points earned for the individual and team final examinations were 50 points each (1 point per item, 50-item examination). Preliminary Pearson correlations were conducted to assess whether cumulative grade point average (GPA) was significantly associated to any of the outcome variables, and would therefore need to be controlled as a possible confounder. When such a confounding relationship was found, analysis of covariance (ANCOVA) was performed to compare the 2 study groups’ performance on each outcome while controlling for cumulative GPA. Independent t tests were used to compare the study groups when no confounding effect was discovered. All tests were 2-sided, and an alpha value below 0.05 was considered significant. Prior work established reliability of the Team Performance Scale in a sample of medical students.¹⁵ Cronbach alpha was calculated as a measure of reliability for the Team Performance Scale in a sample of student pharmacists to confirm it could be used in this population. Analyses were conducted using IBM SPSS v.21 (IBM Inc., Chicago, IL). The study was deemed exempt by the Institutional Review Board.

RESULTS

Eighty-four students enrolled in the courses were included in the analysis (full credit, n=54; answer-until-correct, n=30). Cumulative GPA prior to enrollment in the course was similar between groups (full-credit mean 3.1 (0.3), answer-until-correct mean 3.1 (0.4), p=0.693). However, GPA was significantly related to both iRAT and tRAT scores, so it was controlled for in these analyses. The average iRAT scores are listed in Table 2. Students enrolled in the full-credit course offering scored significantly higher on iRATs overall (full credit mean 88.7 (5.9), answer-until-correct mean 82.8 (10.7), p<0.001) with differences found on the following topics: heart failure, hypertension, anticoagulation part 2, women’s health, smoking cessation, and dyslipidemia. The average tRAT scores are listed in Table 3. There was no difference between study groups in overall performance on the tRAT (full-credit mean 105.0 (2.9), answer-until-correct mean 104.7 (3.9), p=0.635). However, there was a significant difference found in student performance on tRATs for heart failure, hypertension, anticoagulation part 2, asthma, women’s health, and dyslipidemia.

There was no statistical difference found in student performance on the individual final examination (full-credit mean 38.0 (3.7), answer-until-correct mean 39.4 (4.4), p=0.147). Students in the answer-until-correct group scored significantly higher on the team final examination (full-credit mean 45.8 (1.5), answer-until-correct 47.8 (1.4), p<0.001).

The reliability testing using Cronbach alpha indicated the team performance scale was a sufficiently reliable instrument (Cronbach α=0.94). Quality of team interactions differed significantly between study groups (full-credit 97.1 (9.1), answer-until-correct 103.0 (7.8), p=0.004). Table 4 summarizes the results for each item of the Team Performance Scale. Statistical comparison of each item was not conducted as the tool was not intended to be used in this manner. However, the mean score reported for each item was higher in the answer-until-correct group.

DISCUSSION

Student performance was mixed depending on type of assessment and method of grading. Students performed significantly better on iRATs when using traditional methods of grading for tRATs. This contradicted the hypothesis that different methods of grading team assessments would not impact performance on individual assessments. One theory why this may have occurred is that student preparation prior to class sessions may have declined when partial credit was available for team assessments. Part of the rationale behind using both an individual and team assessment prior to the application phase of TBL is that students are held accountable to themselves in the individual assessment and to the team in the team assessment. This process may entice students to prepare more thoroughly before class to not let themselves or the team down. Thus, when partial credit on team assessments is introduced, some pressure for team accountability is potentially alleviated, resulting in less optimal preclass preparation; however, this theory needs to be scientifically tested.

Overall student performance on tRATs was similar between grading methods used. However, students in the full-credit group performed significantly better on 4 of the 11 course topics, whereas the partial-credit group performed better on 2 of the 11 course topics. This outcome was unexpected, as one would assume the opportunity for partial credit would have potential to increase the grade for the assessment, and indicates that use of partial credit using an answer-until-correct method did not inappropriately inflate grades for the tRAT. There were, however, anecdotal observations made by course faculty members that students who used the answer-until-correct method spent more time in team discussion while completing the tRAT compared to students in prior course offerings. Persky and colleagues also observed that students required additional time to complete assessments when using the answer-until-correct method for individual course examinations.¹² Prior to the implementation of IF-AT forms in the pharmacokinetics course, examinations consisted of open-ended problems, multiple-choice, true/false, and short-answer questions and were scheduled for 2 hours. Following implementation of the IF-AT forms, the entire examination was made up of multiple-choice questions, and its duration increased to 3 hours to allow sufficient time for completion.

Student performance on individual course examinations did not differ between groups as was expected. Performance on the team course examination was significantly higher in the answer-until-correct group. One possible reason for the observed but small difference in team final examination performance was the emphasis on team development throughout the course that led to improved team performance. The Team Performance Scale scores were higher for teams that used the answer-until-correct assessment method compared to the full-credit group. However, overall tRAT scores did not differ significantly between methods. Future research should be conducted to determine if either assessment method used in this study could help teams work through Tuckman’s stages of team development (forming, storming, norming, performing).¹⁶ It can be hypothesized that teams who reach the performing stage would perform better on team activities such as a tRAT or team final examination.

A potential limitation of the study was that topics and faculty facilitators changed slightly in the course each year, although 80% of the topics and faculty facilitators remained the same. Also, differences in RAT items among courses might have contributed to the differences observed in iRAT and final team examination scores. While the content was similar in each course offering, the RAT questions did not remain exactly the same, based on feedback from students during the appeal process that faculty members used to improve questions for future offerings. This may have limited the ability to compare findings from different offerings of the same course. However, constant revisions and updates to course content reflect the reality of teaching practice. Another limitation was that the sample size was relatively small, but the analysis was conducted in an elective course with a smaller enrollment than required courses in the curriculum. A similar comparison in a course with higher student enrollment should be conducted to confirm the results of this study.

CONCLUSION

The answer-until-correct assessment method compared to traditional methods using full-credit resulted in lower iRAT scores, similar tRAT and individual final examination scores, and improved team final examination scores and perceptions on team interaction quality. Additional research is needed to identify reasons for decreased performance on iRATs and to assess the impact answer-until-correct assessment methods have on time to complete the assessment. We also recommend that additional research focus on assessing progression through stages of team development to help identify activities that assist teams in reaching the performing stage.