A Primer for Objective Structured Teaching Exercises

INTRODUCTION

The objective structured teaching exercise (OSTE) is a high-fidelity training and assessment method for advancing the teaching and interpersonal communication skills of faculty members, preceptors, and residents.¹ First introduced in the medical literature in the early 1990s, OSTE engages learners in performance-based teaching activities with a standardized student. The OSTE method not only provides a unique and innovative way to teach, enhance, and evaluate educational skills, but also presents an ideal platform for future scholarship.

The body of OSTE literature is primarily descriptive in nature and not as robust as the literature regarding standardized patients in health profession curricula. However, interest in the OSTE technique stems from the postulation that standardized student interactions will provide benefits to teachers similar to those that standardized patient interactions provide to students. In the 1960s, neurologist Dr. Howard Barrows was seeking a better method than traditional clinical observations to assess the clinical skills of his medical residents. He developed the concept and first used standardized patients by tapping into an available pool of actors in Southern California.² Using cases of actual neurology patients, Barrows trained the actors to present the same signs, symptoms, history, emotional state, and, in some cases, physical examination findings to learners. Using the standardized patient, he could then educate, observe, and assess each learner in a safe and controlled environment. Standardized patients are now used extensively throughout health professional education and the technique has been adapted for use with individuals portraying students for faculty development. Standardized student encounters can be used for instruction and practice of skills, performance evaluation, program assessment, and research. These encounters allow for repeated experiences in which the learner can attend to the critical aspects of a situation and improve performance in response to feedback. The learner is an active participant in the experience rather than a passive observer. Standardized student encounters enable expert educators to observe and improve the interaction of the participant with the standardized student.

There is a paucity of published literature regarding OSTEs, particularly regarding effectiveness of the technique. Nevertheless, when OSTEs have been used as a teaching and learning tool, participants have been uniformly positive about the experience. Participants like the technique, feel the cases are realistic, and self-perceive that their teaching skills have improved as a result of participation.^1,3 In the only published study that has attempted to examine if participation in an OSTE changed teaching behaviors, no differences were found in student evaluations of preceptor performance 6 months prior to and 6 months after participating in an OSTE.⁴ However, student ratings of preceptor performance were high at baseline, thus, a ceiling effect may have limited the ability to note any meaningful change.

There is slightly stronger evidence that OSTEs can be an effective method for evaluating the benefits of faculty development efforts. Morrison et al created a 13-hour curriculum to help develop the teaching skills of second-year generalist medical residents.⁵ Residents were randomized into 3 study groups; 2 of these groups served as controls and 1 as the intervention group.⁶ The intervention group participated in a pre-curriculum, 8-station OSTE, completed the 13-hour teaching curriculum, and then repeated the OSTE 6 months after curriculum completion. The first control group only completed the 2 OSTEs, without participation in the 13-hour curriculum. The second control group only completed the final post-curriculum OSTE so that the investigators could determine if participation in the pre-curriculum OSTE “primed” the teaching behavior of the residents independent of curriculum participation. The curriculum itself consisted of a 3-hour mini-retreat followed by 10 hour-long noon conferences.⁵ Teaching methods included role-play using expert and peer feedback based on the Stanford Faculty Development Program 26 (SFDP-26) instrument. Results of the study revealed that intervention group residents performed significantly better on the post-curriculum OSTE as compared to both control groups, indicating that the 13-hour curriculum resulted in at least short-term improvement in teaching skills. One year later, most of the original participants agreed to participate in structured interviews about the process.⁵ During those interviews, intervention residents reported more enthusiasm for teaching, greater use of learner-centered approaches, and a better understanding of general teaching principles as compared to controls.

Incorporating OSTE into pharmacy faculty development programs may be beneficial, but requires significantly more effort than traditional lectures and workshops. This manuscript is intended to serve as a “how-to” primer for those interested in exploring the technique.

PROGRAM DESIGN CONSIDERATIONS

The OSTE is not a “one size fits all” process. Instead, OSTEs can be structured in a variety of ways based upon the purpose, type of learner, standardized student expertise, and available resources (time, personnel, financial). Critical questions include: (1) Will the OSTE be used as a training tool, or will the OSTE be an assessment tool for other teaching development efforts? (2) Will each participant work with a standardized student, or will a group OSTE be conducted in which one participant volunteers to run the scenario in front of others? (3) Does capability exist to video record standardized student encounters, and how will those videos then be used? (4) Who will be providing feedback to the learner? The standardized student, an expert educator, or both? (5) Will feedback be provided in a group or individual setting? (6) How long will each encounter last? Answering these questions in advance of selecting case content, writing cases, and developing assessment tools will help optimize the time and effort required to create the event. As a guide, the large majority of the literature describes OSTE encounters of 10 to 15 minutes’ duration, video recording of participant-standardized student interactions, and large group debriefings of the experience, which often include watching the video of a few volunteer participants. The literature describes greater variety in the purpose of the OSTE (learning vs assessment) and in the person who provides individual feedback to the participant (ie, the standardized student or the expert educator).

CONTENT SELECTION

Just as not all student learning outcomes are best taught and assessed using performance-based measures, not all teaching skills are best developed and tested using the OSTE format. The OSTE can be time consuming and resource intensive, so careful selection of content is critical. Most importantly, any skills taught and tested through an OSTE should be those that can be directly observed, otherwise the energy and time to conduct OSTE will not be worthwhile.⁷ As an example, OSTE would not be the best method for teaching faculty members how to write learning objectives or test questions. On the other hand, cases that focus on faculty-student communication and psychomotor skills are well suited for the OSTE format.

The literature most commonly describes OSTE cases that focus upon skills such as orienting learners to an introductory or advanced practice experience, questioning to elicit students’ clinical knowledge, providing feedback to learners regarding patient interviewing and case presentation skills, and teaching students how to conduct a physical examination or procedure.^8-11 The OSTE also provides an excellent venue for developing the competency to effectively deal with student’s professionalism and behavioral lapses. Cases regarding ability to engage in discussions regarding lack of student initiative, rude behavior towards other healthcare professionals and patients, not completing assignments and tasks on time, and argumentative behavior towards a faculty member have also been described.^3,9,11,12

Ultimately, the selection of skills and behaviors for OSTE depends upon the type of learner (faculty, resident, or preceptor) and the purpose of the OSTE (formative learning or summative evaluation). A needs analysis that includes review of the literature, examination of student teaching evaluations, and feedback from the target learning population will help to further refine OSTE goals. Programs may also find that developing a blueprint for OSTE cases will aid in focusing energy and efforts. Table 1 provides an example OSTE blueprint for non-discipline-specific teaching skills (ie, those that could engage clinical or pharmaceutical sciences faculty members).

CASE DEVELOPMENT

A carefully developed OSTE case is a critical step if there is a strong desire to maintain standardization of the experience. Three elements must be considered: (1) the information the participant will be told about the encounter prior to interacting with the standardized student; (2) the manner in which participant performance will be evaluated; and (3) the statements and responses that will be made by the standardized student. We suggest that a group of experienced educators develop each case, ideally using information from evaluations from students.

The pre-encounter information provided to the OSTE participant should be complete enough to enable the participant to successfully accomplish the scenario’s objectives. For example, pre-encounter instructions for a case in which the participant will listen to a student present a patient case may be as simple as “Sally just saw Mr. Smith for his initial anticoagulation visit in your clinic. Ask Sally to present the patient to you and precept her as you normally would in practice.” Alternatively, the instructor/coordinator/administrator may find that a more complex pre-encounter scenario is needed. Three examples from our own experiences include provision of test-item analysis to the OSTE participant prior to asking him/her to speak with a student challenging a question on that test, pre-recording mock student-patient interactions so that the OSTE participant can watch that encounter and then provide feedback to the student as she/he would in a precepting situtation, and providing a completed final student rotation evaluation to the OSTE participant prior to him/her having to converse with a student about failing performance. In general, any advanced knowledge that would be available in the real world should be available to the participant as part of the pre-encounter OSTE materials.

Once each OSTE case’s goals, objectives, and pre-encounter materials and instructions are developed, the performance metrics for the case should be created. Usually, metrics take the form of checklists, rating scales, or rubrics. For clinical teaching scenarios, valid and reliable methods to determine the quality of the OSTE performance have been developed by Morrison et al,¹⁰ with confirmation of reliability by others.^5,13 In these studies, the OSTE rubric is based upon the Stanford Faculty Development Program (SFDP-26) questionnaire.^10,14 The SFDP-26 tool contains numerous statements describing clinical teaching performance and asks the evaluator to rate agreement with each statement on a 5-point Likert scale.¹⁴ In order to adapt the SFDP-26 to OSTE, Morrison et al selected statements from the SFDP-26 relevant to each OSTE case’s objectives and created case-specific anchors for each Likert rating. For example, the SFDP-26 item “evaluated learners’ knowledge of factual medication information” was anchored by statements of “did not ask learner helpful questions to probe what learner recalled…” at the lowest end of the Likert scale and “asked learner appropriate recall questions to probe his/her knowledge base about asthma” at the highest end of the Likert scale.¹⁰ Although this process will not work for every OSTE case, the general concept can be applied by those struggling to develop evaluation metrics.

Finally, the case must be populated with statements and responses to be made by the standardized student. Case writers must visualize the scenario and anticipate any and all statements and behaviors that might be displayed by the participant so that a scripted standardized student response to those statements and behaviors can be developed. It is also helpful to separate the concept of statements or questions that the standardized student will make to each participant under every circumstance vs those that will only be provided when triggered by something the participant does or says. We have found it helpful to create a full student persona for each standardized student, even if elements of that persona are not anticipated to be directly relevant to the case, so that the standardized student can better visualize the person he/she is portraying. A template for drafting cases is provided in Appendix 1.

FINDING AND TRAINING STANDARDIZED STUDENTS

Collaboration and creativity can make what appears to be a daunting task of hiring and training standardized students a manageable process. Accreditation standards require every medical school in the United States to use standardized patients. This can be an advantage for pharmacy programs that are able to collaborate with established standardized patient centers. Often actors are hired to portray the standardized student, but this is usually not necessary. The key characteristics needed of a potential standardized student are flexibility, excellent communication skills, strong work ethic, and the ability to memorize complex scenario information. Partnering with an established standardized patient program helps ensure that the best people are hired and an experienced standardized patient educator is available to help with project development. For those pharmacy programs without access to or resources to engage with an established standardized patient program, pharmacy and medical students have been successfully used in the OSTE process.^1,3 However, if a program chooses to use actual students, careful planning is necessary to ensure that no bias is introduced into the process from familiarity of the faculty member or preceptors with the students.

Regardless of who is portraying the student, vigilant and consistent training is essential to guarantee the reliability and accuracy of the portrayal of the situation and the assessment of participants. Educating the standardized student on the role of pharmacy faculty/preceptors if there is no prior experience working with pharmacy cases. Standardized students should be provided with a case or scenario to memorize prior to training, after which rehearsal is required prior to portraying the case to participants. Ideally, training should be done by a subject matter expert and/or experienced standardized patient program trainer. Under the direction of the subject matter expert or standardized patient program trainer, the standardized students will review the key learning objectives, presenting situation, history, psycho-social background, and any communication challenges the standardized student will have. During training, attention should be given to: (1) specific verbal and nonverbal behaviors appropriate for the role; (2) a detailed review of case information; (3) directives regarding how to answer participant questions and what information can or cannot be provided; and (4) specific participant behaviors to remember for assessment or feedback. Group training allows each standardized student to see the performance of his/her peers and enables the subject matter expert to give performance direction that all will follow. Such training is critical because keeping the encounter standardized allows every participant an equal opportunity to demonstrate his or her skills in key areas.

Training should include a pilot phase. The subject matter expert, standardized program trainer, and standardized students run the event in real time with a few volunteer participants to ensure accuracy and reliability. Information obtained from this process helps to ensure that both the content and the portrayal of the standardized student is realistic. It is during the development process of the scenario, and most importantly the training and quality assurance check of the standardized student, when reliability measures can be implemented to ensure an equitable experience for all participants. During the training and rehearsal phase, the subject matter expert can assess case validity by participating in the OSTE and completing post-encounter assessments.

Standardized students can also be trained to give participants immediate, one-on-one feedback regarding participant performance. Standardized students will often complete the checklists that assess communications and interpersonal relationship skills and behaviors based on the perception of the character as defined in the case and script. The standardized student can also be trained to give this feedback verbally. Whether feedback is verbal or written, training standardized students ensures they understand the criteria and benchmarks they should follow when answering questions to determine the effectiveness of participant communication and interaction.

PROVIDING PARTICIPANT FEEDBACK

Subject matter experts conducting the OSTE often have a different perspective than the standardized student regarding the success and challenges of each participant. When it comes to the identification of interpersonal skills that need refinement, standardized students are equally reliable as subject matter experts.¹ Consequently, it makes sense for the participant to receive that information from the standardized student. While the standardized student can be trained to assess teaching skills, the expert educators who set the learning objectives are often in a better position to evaluate each participants’ performance against those standards. Thus, the development of the case’s learning objectives and evaluation rubric should include what and how each skill will be assessed. Even if the subject matter expert will be the final evaluator, he/she will need some training to assure consistency of performance evaluation, but typically this does not require the extensive training required of the standardized student for scoring encounters and providing feedback.

Debriefing standardized student encounters can be conducted in a small group session or on a one-to-one basis. Based on the structure of the standardized student encounter (group or individual), standardized students often give feedback to the participant immediately following the encounter. Consideration must be given regarding the impact the feedback will have on the participant in a group encounter. The advantage is that all participants learn from the success and failures of their peers. The disadvantage is the risk of the participant feeling as if he/she are being critiqued in a public forum. Regardless of where or how the feedback is delivered, the manner in which the verbal feedback is provided must always be structured. This feedback should be nonjudgmental and based on participant behavior. In a group setting, there should never be comparison of one participant with another.

CONCLUSION

Objective structured teaching exercises are a high-fidelity training method for advancing the teaching and interpersonal communication skills of faculty members and preceptors. Working with standardized students provides immersive experiences that can optimize learning using consistent and objective methods while facilitating improvement of knowledge and skills leading to competent and confident preceptors and faculty members. Implementing an OSTE program for faculty members and preceptors also requires implementing continuous quality improvement, conducting program evaluations, and keeping up with the latest research on the topic. Those that use the OSTE method should implement program evaluation techniques to determine if the training is achieving stated goals and objectives and to contribute to the growing body of literature regarding the technique. Creating accountabilities, expectations, and roles and responsibilities for the project will ensure ongoing success.

Although OSTEs can be time and resource intensive to implement, the consistently favorable reactions by participants reported in the literature endorses the utility of this method. When used in conjunction with other educational strategies, faculty development using standardized students can substantially improve the quality of the learning environment and lead to health professions faculty members who are better equipped to educate and precept students.