Differences In Multiple-Choice Questions of Opposite Stem Orientations Based On A Novel Item Quality Measure

Abstract

Objective. To determine whether there are differences in the performance and quality of multiple-choice items with opposite stem orientations (positive or negative), based on a novel item quality measure and conventional psychometric parameters.

Methods. A retrospective study was conducted on multiple-choice assessment items in professional years 2 and 3 for Pharmacotherapy and related courses administered between August 2018 and December 2019. Conventional psychometric parameters (difficulty and discrimination indices), average response time, non-functional distractor percentage, and a novel measure of item quality of negatively worded items (NWIs) were compared with those of control, positively worded items (PWIs) (N=103 each). This novel measure uses difficulty and discrimination in tandem for the decision to reject, review, or retain items in an assessment. Continuous and categorical variables were analyzed with Mann-Whitney U and Chi-square tests, respectively. The relationship between difficulty and discrimination was tested with Spearman correlation, and differences in correlation coefficients between PWIs and NWIs were tested with Fisher’s r-to-z transformation.

Results. Stem orientation was not significantly associated with the novel measure of item quality. Also, there were no significant differences between PWIs and NWIs in any of the psychometric parameters. There were significant, negative correlations between difficulty and discrimination indices in both groups, and the correlation coefficients were significantly stronger in PWIs versus NWIs.

Conclusion. Items with opposite stem orientations are neither different in the novel item quality measure nor in conventional measures of performance and quality, except in difficulty-discrimination relationships. These suggest that NWIs should be used when necessary, but cautiously.