Abstract
Objective. The objectives of this study were to develop situational judgment test (SJT) scenarios for use in teaching empathy, assertiveness, and interprofessional communication in second-year pharmacy (P2) students and ethics in third-year pharmacy (P3) students, determine whether the SJTs developed were effective at measuring students’ communication skills and ethical judgment, compare the performance of individual third-year students to the performance of teams of third-year students on the ethics SJT, and evaluate student feedback about SJTs as a teaching tool.
Methods. Pharmacy faculty developed five SJT scenarios related to communications and five SJT scenarios related to ethics and piloted the scenarios with P2 and P3 students, respectively. Second-year students completed SJTs individually, while P3 students completed SJTs individually and in teams. Scenarios and responses were discussed with faculty after completion of the SJTs, and students completed a questionnaire to provide feedback on the SJTs.
Results. The communications SJT was completed by 59 P2 students with a mean score of 67.5%. The ethics SJT was completed by 57 P3 students with a mean score of 80.1%. The ethics SJT was also completed by 10 teams of P3 students resulting in a mean score of 93.2%. Students indicated the SJT content was realistic and the tests provided the opportunity to reflect on how to approach challenging situations.
Conclusion. Situational judgment tests were useful for teaching empathy, assertiveness, interprofessional communication, and ethics in pharmacy students. Future research should focus on predictive validity of SJTs for these content areas.
INTRODUCTION
Situational judgment tests (SJTs) are an emerging method of teaching and assessing noncognitive attributes in pharmacy education. 1-5 Previous studies have described SJTs for use during the pharmacy curriculum to develop or assess empathy, teamwork, ethics, self-directed learning, and problem-solving. 1-3 Situational judgment tests consist of written or video scenario-based tests that assess judgments common in the test takers’ typical work environment that should not require clinical or procedural knowledge to complete. 5 Test takers are presented with a scenario, then asked to rate or rank the appropriateness of various responses to the scenario. Test takers’ answers are compared to the answers of subject matter experts (SMEs) and can be scored in a variety of ways, including giving higher scores for answers that more closely align with the SMEs’ answers. 6, 7 SJTs have been shown to be a valid and reliable way to assess attributes in the affective domain such as professionalism and leadership. 8, 9 They have also been shown to predict future behavior of medical and dental students and medical residents. 10, 11
The current study sought to understand the difference between individual performance and team performance on the same SJT. This question is important because SJTs are usually completed independently, so it is not well understood how peer teaching affects SJT performance. Evidence from team-based learning suggests that students would perform better on an SJT completed with a group of peers rather than individually. 12 Therefore, the objectives of this study were to develop SJT scenarios for use in teaching empathy, assertiveness, and interprofessional (IP) communication in second-year pharmacy (P2) students and ethics in third-year pharmacy (P3) students, determine whether the SJTs developed were effective at measuring communication skills (empathy, assertiveness, IP communication) in P2 students and ethical judgment (ethics) in P3 students, compare the performance of individual P3 students to the performance of teams of P3 students on the ethics SJT, and evaluate student feedback about SJTs as a teaching tool.
METHODS
The development of the two SJTs in this study began by identifying desired professional attributes to assess in a second-year (P2) clinical communications course and a third-year (P3) pharmacy law and ethics course (Figure 1). These courses were selected because the content areas relate to noncognitive attributes desired of pharmacy graduates. For the clinical communication SJT, the selected professional attributes were empathy, assertiveness, and IP teamwork, and for the pharmacy law and ethics SJT, the selected professional attribute was ethics. An author with previous training in SJT development and delivery interviewed the faculty teaching clinical communication and pharmacy law and ethics courses. These faculty were selected as SMEs because they hold advanced degrees in the area, teach the topic areas in the curriculum, and work closely with pharmacy students on a regular basis.
The process of creating and analyzing the situational judgement tests used in the study. Figure adapted from content in articles by Smith and colleagues 2 and Patterson and Driver. 13 Abbreviations: SJT, situational judgment test; SME, subject matter expert; LMS, learning management system; IP, interprofessional.
The SMEs were interviewed to gather examples of common scenarios from classroom or pharmacy practice settings in which they have observed pharmacy students act appropriately or inappropriately. These examples were used to develop SJT scenarios with eight to 11 possible responses. Situational judgement tests were written using a rating-type question format, wherein test takers are asked to rate the appropriateness of each possible response to the scenario presented. An example SJT can be found in Appendix 1. The SMEs reviewed the scenarios for their respective courses to ensure SJT best practices. 2 The response key was finalized based on consensus among the SMEs. The items were uploaded to the learning management system (LMS) Desire2Learn (D2L Corporation) and reviewed to ensure consistency with visual representation and content. Items were scored using a previously published matrix, based on a near-miss system that allowed for awarding partial credit (Table 1). 2 Higher scores were given when ratings of appropriateness more closely aligned with those of SMEs.
Scoring Matrix Detailing Points Awarded Based on Alignment with Subject Matter Experts’ Answers for the Situational Judgment Tests
A pilot study was conducted to assess reliability and impact of these SJTs. Second-year students completed the clinical communications SJT in two sessions, one related to empathy and the other related to assertiveness and IP teamwork. Second-year students were not preassigned to groups for other team-based activities, so they were not asked to complete the SJTs as a team. Third-year students completed the ethics SJT with five scenarios related to ethics during a single class session, first independently and then again as a preassigned team of four or five P3 students with whom they had been working during the course. All SJTs were administered in fall 2020 when classes were being offered through synchronous video conferencing. Students were instructed to complete a questionnaire in the LMS after their final SJT to evaluate their perceptions of the SJT as a teaching tool. The questionnaire was similar to test-taker perception questionnaires from other SJTs in the literature and included up to seven items on a four-point Likert scale (strongly agree, agree, disagree, strongly disagree). 2 This study was deemed exempt by the institutional review board at the University of Oklahoma Health Sciences Center.
Internal consistency of the SJTs was measured using Cronbach alpha. Missing data were imputed using mean scores rather than deleting individuals listwise. To improve the internal consistency of each SJT, scenario items that reduced Cronbach alpha by .005 or more were removed. 2 Only the retained items were used in the final psychometric analysis to evaluate the internal consistency, test difficulty, fairness, and face validity. Item quality was determined via item discrimination index (DI), using quality delineations of good (DI > 0.25), satisfactory (DI 0.24-0.17), moderate (DI 0.16-0.13), or limited (DI < 0.13). 13 Test difficulty was determined using mean scores, standard deviation (SD), and score distribution. Face validity was measured using survey results, which were summarized using frequency (percentage). Data analysis was conducted using SAS software version 9.4 for Windows and JMP Pro version 15.2.1 (SAS Institute Inc).
RESULTS
Four scenarios with eight to 11 possible response items were included for the pilot communication SJT completed by 59 P2 students. Four percent of 1550 item responses were missing and imputed with mean score rather than dropping individual rows. Out of the 37 response items, 25 (67.6%) performed well psychometrically (based on impact on Cronbach alpha) and were included in the final analysis. Item quality distribution determined by DI included seven good items (28%), six satisfactory items (24%), four moderate items (16%), and eight limited items (32%), which is consistent with other SJTs published in the pharmacy education literature. 1, 2 Scenarios retained between 50% and 88% of items. The overall mean score for the communication SJT was 67.4% (raw mean score 58, SD = 6.4, range 42 to 70, maximum possible score 86). The communication SJT demonstrated good reliability with regards to internal consistency (α=0.59).
Five scenarios with six to 10 possible responses each were included on the pilot ethics SJT completed by 57 P3 students. One percent of 1,710 item responses were imputed. Out of the 38 response items, 30 (78.9%) items performed well psychometrically (based on impact on Cronbach alpha) and were included in the final analysis. Item quality distribution determined by DI included 14 good items (47%), three satisfactory items (10%), four moderate items (13%), and nine limited items (30%), which is also consistent with other SJTs published in the pharmacy education literature. 1, 2 Scenarios retained between 71% and 88% of items. The overall mean score for the ethics SJT was 80.2% (raw mean score 87.4, SD = 8.7, range 51 to 100, maximum possible score 109). The ethics SJT demonstrated good reliability with regards to internal consistency (α = 0.79).
Third-year students were divided into 10 teams to complete the same ethics SJT. The overall mean score for the team ethics SJT was 93.2% (raw mean score 101.6, SD = 6.9, range 86 to 109, maximum possible score 109). Individuals increased their scores on 36% of scenario items when answering as a team and decreased their scores on only 5% of items. Differences in individual vs team performance on the ethics SJT were observed for item responses where students were asked to report unethical behavior to a preceptor or faculty member.
The response rate for the student evaluation questionnaire was 100%. Students were asked to indicate their level of agreement about the SJT regarding its relevance, difficulty, clarity, and ability to differentiate responses (Table 2). Eighty-eight percent of P3s (43 out of 49) and 60% of P2s (34 out of 57) said they would like to see more scenarios like the SJT in the curriculum. Examples of open-ended feedback from students included statements such as “Good exercise to promote conversation,” “The scenarios were helpful to get me thinking about these types of situations, but the debrief portion was when I really felt like I got the full benefit of the exercise,” and “This would be a great thing to add and ask about as we go through pharmacy school so it gets us more use to being active thinkers.”
Pharmacy Students’ Evaluation of Situational Judgment Tests
DISCUSSION
The SJTs developed in this study were designed to teach empathy, assertiveness, communication, and ethical decision-making. The Cronbach alpha for the communication SJT (α = 0.59) and ethics SJT (α = 0.79) suggest scores accurately reflect the test takers’ skills in these domains. While the minimum desired level of reliability for a summative assessment is α = 0.70, SJTs used for formative assessments may serve as a useful learning tool even if they do not reach this level. 14, 15 The difficulty of the items in this SJT, as measured by DI, were similar to other SJTs reported in pharmacy education. 1, 2 Furthermore, these SJTs were developed with fewer items than other SJTs reported in pharmacy education but still reached desired levels of reliability. 1, 2 Development and piloting of an SJT can be a significant time and/or financial commitment, but this study demonstrates the potential value of smaller, more focused SJTs to develop these skills in pharmacy students.
In the scoring method selected for these SJTs, students received higher scores for ratings that more closely aligned with faculty SME ratings. Therefore, it may be helpful to understand where students differ from SMEs in their ratings of appropriateness to adjust or augment content in the curriculum related to areas where the greatest differences are seen. Two themes arose in responses where students differed from SME ratings of appropriateness: potential confrontation and timing of intervention. Student responses reflected a higher fear of conflict, especially with peers. This tendency may reflect hesitancy to act, even when inaction may not uphold professional standards or obligations. It is also possible that students are not willing to invest the time and energy into reporting unethical or dangerous behavior that does not directly affect their own learning or seem to lead to self-harm, though corresponding behaviors in practice could have broader implications. SME ratings of appropriateness of responses to these complex situations seem to show that SME faculty are willing to intervene sooner than students. Students were more likely to rate “waiting” as appropriate or very appropriate, regardless of the situation. This tendency may represent an underdeveloped sense of responsibility to others (students, patients, the public) and/or an unawareness of risk management concerns, both of which continue developing after graduation. These results suggest that activities related to difficult conversations should be included in personal and professional development curricula throughout pharmacy school. 16, 17
As expected, the performance on the ethics SJT improved when students completed the test with a team of peers (individual performance mean score 80.1% vs team performance mean score 93.2%). Team answers aligned with SME answers more frequently (range: 60% to 100%) than individuals (range: 5.6% to 98.2%). Change in agreement with SME answers ranged from 1.8% to 64.2% when completing the SJT as a team vs individually. When completing the SJT as a team, 38% of students increased their score (positive influence of team), 6% decreased their score (negative influence of team), and 56% had the same score. One scenario answer that had different ratings of appropriateness between the individual SJT and the team SJT can be found in Appendix 1. The SMEs rated the response, “Ignore the text from Kendra and continue studying on his own” as “Inappropriate, but not awful” because, while it was appropriate that Adam did not join the group in looking at the old exam, it would have been best if he had alerted a faculty member to the potential academic misconduct. When completing the SJT as individuals, 16% of students agreed with the SMEs, but when completing the SJT as a team, 80% of teams agreed with the SMEs. We hypothesize that students were avoiding conflict or confrontation when asked to respond as an individual, but when among a group of their peers they were emboldened to report the unethical behavior observed. This speaks to the significant positive or negative influence one may experience from a group of peers. Further, team-based learning is known to have a moderating effect on individuals within teams. 12 Team participants who are naturally more assertive may become more cautious once they discover their teammates disagree with them, while participants who are less likely to speak up may be put on the spot and asked to defend their choice. While this activity was low stakes, we hypothesized that team dynamics would remain the same for this activity and the rest of the course activities as these student teams would work together for the duration of the course.
Results of the student evaluation questionnaire indicated students found the SJT scenarios were realistic and allowed them to better understand the complexity and nuances associated with decision-making in practice. While not all students felt they had all information necessary to make the most appropriate decision related to the scenarios, this mirrors practice, where practitioners are often required to make decisions without having all information desired. Students would benefit from additional modeling of decision-making in the face of uncertainty and complexity. 18
Limitations of this study include SJTs being only in written format. Emotions and tone would likely be better communicated through video SJTs, which could have led to different ratings of appropriateness on the same scenarios. Additionally, there is inherent human error when one team member is entering an answer into a learning management system for a team of students, although we assumed that the answer put into the LMS did indeed reflect the opinion of the entire team. Because SJTs are measuring more than one characteristic, it is possible that a test taker’s answers may be confounded by their preferred conflict or assertiveness styles. For example, a student could be aware that an observed behavior is unethical, but they may lack the assertiveness needed to report it. Future studies could evaluate whether other personal attributes impact responses to SJT items. Further research may also include the use of SJTs to assess additional attributes (leadership, emotional intelligence) or assess professional identity development longitudinally throughout the curriculum, especially a sense of responsibility to others. Wolcott and colleagues have begun to study how prior knowledge and experience affect decision-making on SJTs. 5 As we improve our understanding of these potential influences, refinement of SJTs may improve their utility. Other areas of further inquiry could be determining the predictive validity of SJTs or assessing the correlation between SJT performance and future performance on professionalism assessments (for example, by preceptors on clinical learning experiences, or by a state board of pharmacy).
CONCLUSION
Situational judgment tests are an emerging method of teaching and assessing essential professional attributes in pharmacy education. In this study, SJTs were developed to teach communication and ethical decision-making to P2 and P3 pharmacy students. When P3 students took the SJT with peers, they influenced each other to respond more like the faculty SMEs, demonstrating the positive impact peers can have on each other. Not only were the developed SJTs found to be valid and reliable, but students found the SJTs realistic. After completing the SJTs, students and faculty discussed their thought processes on the SJT scenarios, which improved student understanding of decision-making in pharmacy practice.
Appendix 1. Example of a situational judgement test scenario with items presented to pharmacy students.
Scenario
Adam is a third-year pharmacy student. While studying for an upcoming exam, Adam receives a group text message from his classmate Kendra which includes about 10 members of the third-year pharmacy class. The text message says “This material is hard! I’m so glad I got a copy of last year’s exam. Who wants to meet up later to go over it together?”
Response Instructions:
How appropriate are each of the following responses by Adam in this situation?
Rating Options:
1 = A very appropriate thing to do
2 = Appropriate, but not ideal
3 = Inappropriate, but not awful
4 = A very inappropriate thing to do
Possible Responses:
Ask his classmate, Roy, who was not on the group text, if he wants to go to the meet up with him.
Reply to the group text saying he does not think looking at last year’s exam is a good idea.
Ignore the text from Kendra and continue studying on his own.
Ask his classmate, Roy, who was not on the group text, what Roy thinks he should do.
Reply to the group text saying he would like to meet up to talk about last year’s exam.
Ask his faculty advisor what he should do if he suspects cheating might be occurring.
Send an email to the course coordinator stating Kendra has a copy of last year’s exam.
Send an email to the course coordinator stating he has heard there may be cheating occurring.
- Received June 8, 2021.
- Accepted October 8, 2021.
- © 2022 American Association of Colleges of Pharmacy