Regarding Use of Objective Structured Teaching Exercises (OSTE) for Summative Evaluations (Authors’ Response)

To the Editor: We read with interest the comments regarding use of objective structured teaching exercises (OSTE) for summative evaluations made by Drs. Peeters, Kelly, and Cor.¹ We hope that readers ultimately view OSTE as a tool best reserved for formative skill development as opposed to summative evaluation. However, we agree that should OSTE be used for a true “high stakes” summative evaluation of preceptor or faculty teaching performance, then careful attention during OSTE development to minimize construct underrepresentation and construct-irrelevant variance must be made.² It should be noted, however, that Quirk et al did demonstrate a generalizability coefficient nearing the ideal of 0.70-0.80 in a 5-station OSTE that utilized a single rater per station³—a number of stations far fewer than the 14-18 stations recommended for achieving similar generalizability coefficients in high-stakes objective structured clinical examinations (OSCE).⁴ This difference may perhaps be attributed to a narrower scope of focus of their OSTE compared to high-stakes OSCEs.

We feel it is unlikely that an OSTE would be used as the sole determinant of teaching effectiveness in decisions regarding promotion and tenure, the only true “high stakes” endpoint for most faculty members and preceptors. We intended to promote “summative” OSTE as a method for identifying those with weaker teaching skills who require further development and/or as an evaluation component of other faculty development efforts. In both situations, an OSTE with a small sampling of stations that focus on a single teaching skillset (eg, addressing lapses in student behavior) would likely suffice. We also encourage investigators with interest in the method to continue to grow the body of literature that identifies the optimal number of OSTE stations appropriate to its intended use.

• © 2015 American Association of Colleges of Pharmacy