Protecting the Integrity of the Virtual Objective Structured Clinical Examination

Maintaining OSCE Integrity

Academic integrity remains a cornerstone of higher education.² According to the Accreditation Council for Pharmacy Education, student assessment should take place under circumstances which minimize violations of academic integrity.³ Several mitigation strategies have been used to detect and deter academic dishonesty for conventional examinations, such as the use of browser lockdown tools, plagiarism detection tools, time restrictions, and question randomization.⁴ However, strategies to maintain the academic integrity of virtual OSCEs have not been well studied or reported in the literature,⁵ despite the protection of OSCE integrity being noted as a best practice.⁶ Guidance for maintaining OSCE integrity may be translated from what is known about online courses and from other health disciplines. Although virtual OSCEs may be proctored, feedback should be withheld until all students have completed the examination. Additionally, generating a variety of case scenarios to reduce discussions among students and limiting the examination administration time may help maintain the integrity of the OSCE. As a performance-based summative assessment and gold-standard of clinical skills assessment, learners often find their knowledge of the subject matter within OSCEs meaningful as demonstration of their progress towards professional goals.^7,8 This stance, coupled with an academic honesty statement prior to OSCE administration, may set the tone for protecting the integrity of the examination.

Approach to Protecting OSCE Integrity

A summary of the characteristics of three Doctor of Pharmacy programs, as well as the OSCE characteristics of each program pre- and post-COVID-19 pandemic can be found in Table 1. At the Purdue University College of Pharmacy, OSCEs are high-stakes in that students are required to pass each station in order to progress in the PharmD curriculum. Prior to COVID-19, each OSCE consisted of two to three stations, one of which involved an active encounter with one student and one standardized participant. Multiple faculty members served as evaluators for all stations, and active encounters were recorded for purposes of reflection, instruction, and reevaluation where applicable. Passive stations were monitored by faculty and staff to aid in maintaining the integrity of the OSCE. Due to class size and course schedules, OSCEs were administered Monday through Friday for each class. Because of the opportunity for students to share information about the OSCE as the week progressed, different cases were used each day.

Because of COVID-19, all six OSCE stations planned for April 2020 were transitioned to remote platforms. Active encounters were conducted using Cisco Webex (Cisco Systems, Inc.) and were recorded. As a result of limited time to recruit virtual standardized participants, faculty served as both patients and evaluators. Some active stations were reformatted to follow the structure of telehealth encounters, and supplies used in the encounters (eg, inhalers) were mailed to faculty for use during the encounter. Participation in a live practice opportunity for each active station was required of all students prior to completing the summative OSCE. While the intent of the practice opportunity was to provide students with formative feedback on the skills they would be assessed on during the summative OSCE, the opportunity also served as a trial run in using the remote platform technology. Faculty served as standardized participants for the practice OSCE just as they did in the summative remote OSCEs. Cases different from the summative OSCE but similar in structure and objectives were used. Passive stations were implemented online using ExamSoft and the Blackboard Learn (Blackboard Inc) learning management system (LMS). To comply with university regulations, windows of time in which students could complete the passive stations were specified. To protect the integrity of “banked” cases for future use, fewer cases were used throughout the week for both active and passive stations.

At the University of Florida College of Pharmacy, OSCEs conducted prior to the COVID-19 pandemic consisted of 12 stations. All OSCEs were high stakes in that students were required to pass them in order to continue progression through the PharmD curriculum. Six of the stations involved active encounters between a student and a standardized participant, and six stations involved passive post-encounters where the student completed activities at a computer station. A checklist was used to assess the student’s performance on the active stations, which consisted of observable tasks and skills the assessment team felt the student should complete during the station. Global checklist items were used to assess the student's communication skills, organization of thought, and empathy during the interaction. The items for each active station remained consistent when the college began conducting the OSCE remotely.

The school’s SPs are actors trained to play the role assigned to them and also to assess each student. In terms of maintaining integrity, each station was actively monitored and recorded by a video team. Faculty also viewed and listened to encounters in real time. Based on class size, it took up to one week for all students to complete the OSCE. Students signed an honor code prior to participation in the virtual OSCE, and faculty reviewed students’ overall performance from day to day and did not notice any significant change in grades throughout the week. Post-OSCE assessments included analyzing grades by campus as well as by day of the examination. However, there was the opportunity for students to share information as the stations were not changed through the week.

Because of the pandemic, the first year OSCE, which had been scheduled for May 2020, was modified to ensure that students were able to complete the required course. Each student completed four active encounters with a standardized participant using Zoom (Zoom Video Communications) breakout rooms. The encounters were completed sequentially and recorded for quality assurance. After all of the students completed the active encounters, they were given a break. Then all students completed the passive encounters together at the same time, and each station was completed sequentially one after the other. The passive encounters were built as multiple-choice quizzes in the e-learning LMS using Honorlock (Honorlock Inc).

At the University of South Florida Taneja College of Pharmacy, OSCEs are high-stakes examinations and require students to pass in order to progress to the next professional year in the curriculum. Prior to COVID-19, the OSCEs took place in a simulation center on campus with standardized participants. Each OSCE typically consisted of one case with multiple components to be addressed within two or three stations. The encounters were recorded to allow opportunity for self-reflection. Faculty graded live during the examination via a rubric built into ExamSoft, and course coordinators could review and reevaluate the recording if needed. For each didactic year, the OSCE was administered within one day, and students were assigned a specific time to complete the examination. This allowed for some protection of examination integrity, although students who completed the OSCE earlier in the day may have been able to discuss the case with their peers scheduled later in the day. However, cases were also cycled throughout the day, so students did not know which case they would have until arriving at the simulation center.

Due to COVID-19, the OSCEs administered during the 2019-2020 academic year were converted to remote platforms. Faculty used Blackboard Collaborate Ultra (Blackboard Inc) in the first and third years, and Microsoft Teams in the second year to successfully deliver the OSCEs. Standardized participants were still used, but sessions were not recorded. To make the transition to delivering virtual OSCEs as smooth as possible, only one case was used in each year of the OSCE. Because some of the case documents had to be posted or shared with students electronically, there was the potential of students taking a screenshot or photo of the screen. Integrity may have been compromised and, therefore, the same cases will not be used in the future.

Best Practices for Overcoming Barriers/Challenges with Virtual OSCEs

Several challenges to protecting the integrity of the virtual OSCE have been identified. Although time is often a limiting factor, purposeful planning early in the development stage may help to create a seamless process. Those interested in implementing virtual OSCEs within a pharmacy curriculum should consider the overall advantages and disadvantages (Table 2). The following considerations may be helpful.

Allow time to train faculty, students, and standardized participants in how to use new technology and examination structure. Faculty are becoming increasingly creative with the use of various LMSs and meeting platforms. However, these platforms frequently change their technology requirements, functionality, capabilities, and overall design. Thus, multiple issues may arise that lead to confusion or uncertainty, thereby compromising examination integrity as students and instructors scramble to rectify these problems. Issues such as selecting a compatible and supported internet browser, etiquette regarding the use of cameras and microphones, and contingency planning in the event of an unstable internet connection must also be considered. For these reasons, it would be prudent to conduct a practice session prior to the examination to address concerns regarding access and troubleshoot any problems in advance of the virtual OSCE. Additionally, development of a resource document, revised rubrics, and a plan to access materials on the day of the examination may be helpful to alleviate stress associated with conducting virtual OSCEs.

Consider involving instructional designers in developing virtual OSCEs. The expertise of instructional designers is helpful in evaluating the structure of an OSCE and transitioning the examination for remote delivery by highlighting underutilized aspects of learning management systems. For example, they may offer practical suggestions to sequentially roll out case scenarios to groups of students based on the examination schedule to maintain the integrity of the examination. Additionally, because it may not be feasible to assess students’ performance of certain physical assessment techniques in a virtual format, alternative approaches, such as virtual reality technology, can be used. Instructional designers may also be involved with integration of technology, revision of rubrics, and alignment of learning outcomes with the adapted examination.

Exercise caution when using breakout rooms. Multiple programs exist for delivering virtual OSCEs, and each has various capabilities to ensure students complete the examination individually and do not share information with other students. Some programs may feature large classroom settings as well as smaller “breakout rooms” for small group discussions or individual examinations. These can be used by either faculty or examination proctors to move students to their designated rooms. Alternatively, students can be assigned a specific time to log in, allowing them to move themselves to various breakout rooms. While the former requires more time and strain on the faculty, the latter has led to overlap among students as they may enter their breakout room before their designated time, while the previous student is still in the room. This can compromise examination integrity if the student overhears a portion of the examination. Other programs allow for the scheduling of simultaneous individual meetings, with invitations sent only to those involved in the session. Doing this eliminates the issue of students entering the breakout room before their designated time and is one way to maintain integrity; however, it is more time intensive to set up.

Use video monitoring of examinations in place of traditional proctoring. The need for proctoring OSCEs may be addressed by faculty monitoring student videos during the session. While OSCEs assess students’ verbal and nonverbal cues, these may be difficult to observe through video conferencing depending on how the camera and microphone are set up for the encounter. For this reason, setting clear expectations for both passive and active stations, including visual frame and volume, can preemptively address this challenge. Additionally, repeated observations of multiple students taking examinations can be taxing. To avoid missed information, video recording of the session is available in many programs and may be advantageous. The recording option may also be useful for remediation purposes, self-reflection, or to review grading. The recording feature may not be available in the individual breakout rooms depending on the program. This should be considered in the planning phase to protect the integrity of the virtual OSCE.

Continued use of mock devices in OSCEs is important. Objective structured clinical examinations often involve the student using medical devices (ie, nasal sprays, inhalers, glucometers, blood pressure cuffs, or insulin pens) while completing the required tasks at active stations. When feasible, it may be acceptable for the school to lend these devices to students through an organized process. Alternatively, if supplies are limited, mock devices can be lent to standardized participants for use during active stations. Although the use of mock devices may not affect the integrity of the OSCE, it is still an important component to consider when planning the examination.