A Plea for Psychometric Rigor

To the Editor: We wish to commend Dr. Cowart and colleagues for their report of a multiple mini-interview (MMI) and its correlation with PharmD students’ grade-point averages.¹ Activities described in this investigation were considerable work to accomplish. Necessarily so, non-traditional (also known as non-cognitive²) factors are becoming increasingly relevant and being considered within pharmacy school admissions.³ An important facet overlooked in this MMI report is educational psychometrics, including reliability. Foundational to understanding rigor in this report’s numbers, authors must quantify measurement error (ie, reliability), so that readers can appreciate whether score variations are true differences or simply noise from variations secondary to measurement error. Measurement error can occur with variability among interviewers, inconsistency in rubric scoring and context specificity by using too few stations.⁴ Unfortunately, without reporting measurement error, readers cannot differentiate between true signal variation and noise. That is, inferential statistics in this report do not seem actionable for us.

Regarding reports of pharmacy education research, Peeters and colleagues have recommended that authors using educational measurement report their reliability.⁵ Subsequently, Peeters and colleagues⁶ as well as Cor and Peeters⁴ have shown how Generalizability (G) theory can be an excellent model to report entire process reliability with interviews (including the multiple mini-interview, and other objective structured clinical examinations). Notably, G theory is not new and was reviewed over a decade ago in medical education.^7,8 A recent step-by-step primer also is available.⁹

Knowing this, we urge pharmacy education stakeholders – authors, peer reviewers, and editors – to seek and insist on reporting this rigor in future study descriptions that used educational measurement. Furthermore, we caution generalizations based on any single performance-based assessment of a student. Psychometrics is a key science for educational assessments that educators should all understand and use adeptly.¹⁰

• © 2017 American Association of Colleges of Pharmacy