Use of Situational Judgment Tests to Teach Empathy, Assertiveness, Communication, and Ethics

DISCUSSION

The SJTs developed in this study were designed to teach empathy, assertiveness, communication, and ethical decision-making. The Cronbach alpha for the communication SJT (α = 0.59) and ethics SJT (α = 0.79) suggest scores accurately reflect the test takers’ skills in these domains. While the minimum desired level of reliability for a summative assessment is α = 0.70, SJTs used for formative assessments may serve as a useful learning tool even if they do not reach this level. ^{14, 15} The difficulty of the items in this SJT, as measured by DI, were similar to other SJTs reported in pharmacy education. ^{1, 2} Furthermore, these SJTs were developed with fewer items than other SJTs reported in pharmacy education but still reached desired levels of reliability. ^{1, 2} Development and piloting of an SJT can be a significant time and/or financial commitment, but this study demonstrates the potential value of smaller, more focused SJTs to develop these skills in pharmacy students.

In the scoring method selected for these SJTs, students received higher scores for ratings that more closely aligned with faculty SME ratings. Therefore, it may be helpful to understand where students differ from SMEs in their ratings of appropriateness to adjust or augment content in the curriculum related to areas where the greatest differences are seen. Two themes arose in responses where students differed from SME ratings of appropriateness: potential confrontation and timing of intervention. Student responses reflected a higher fear of conflict, especially with peers. This tendency may reflect hesitancy to act, even when inaction may not uphold professional standards or obligations. It is also possible that students are not willing to invest the time and energy into reporting unethical or dangerous behavior that does not directly affect their own learning or seem to lead to self-harm, though corresponding behaviors in practice could have broader implications. SME ratings of appropriateness of responses to these complex situations seem to show that SME faculty are willing to intervene sooner than students. Students were more likely to rate “waiting” as appropriate or very appropriate, regardless of the situation. This tendency may represent an underdeveloped sense of responsibility to others (students, patients, the public) and/or an unawareness of risk management concerns, both of which continue developing after graduation. These results suggest that activities related to difficult conversations should be included in personal and professional development curricula throughout pharmacy school. ^{16, 17}

As expected, the performance on the ethics SJT improved when students completed the test with a team of peers (individual performance mean score 80.1% vs team performance mean score 93.2%). Team answers aligned with SME answers more frequently (range: 60% to 100%) than individuals (range: 5.6% to 98.2%). Change in agreement with SME answers ranged from 1.8% to 64.2% when completing the SJT as a team vs individually. When completing the SJT as a team, 38% of students increased their score (positive influence of team), 6% decreased their score (negative influence of team), and 56% had the same score. One scenario answer that had different ratings of appropriateness between the individual SJT and the team SJT can be found in Appendix 1. The SMEs rated the response, “Ignore the text from Kendra and continue studying on his own” as “Inappropriate, but not awful” because, while it was appropriate that Adam did not join the group in looking at the old exam, it would have been best if he had alerted a faculty member to the potential academic misconduct. When completing the SJT as individuals, 16% of students agreed with the SMEs, but when completing the SJT as a team, 80% of teams agreed with the SMEs. We hypothesize that students were avoiding conflict or confrontation when asked to respond as an individual, but when among a group of their peers they were emboldened to report the unethical behavior observed. This speaks to the significant positive or negative influence one may experience from a group of peers. Further, team-based learning is known to have a moderating effect on individuals within teams. ¹² Team participants who are naturally more assertive may become more cautious once they discover their teammates disagree with them, while participants who are less likely to speak up may be put on the spot and asked to defend their choice. While this activity was low stakes, we hypothesized that team dynamics would remain the same for this activity and the rest of the course activities as these student teams would work together for the duration of the course.

Results of the student evaluation questionnaire indicated students found the SJT scenarios were realistic and allowed them to better understand the complexity and nuances associated with decision-making in practice. While not all students felt they had all information necessary to make the most appropriate decision related to the scenarios, this mirrors practice, where practitioners are often required to make decisions without having all information desired. Students would benefit from additional modeling of decision-making in the face of uncertainty and complexity. ¹⁸

Limitations of this study include SJTs being only in written format. Emotions and tone would likely be better communicated through video SJTs, which could have led to different ratings of appropriateness on the same scenarios. Additionally, there is inherent human error when one team member is entering an answer into a learning management system for a team of students, although we assumed that the answer put into the LMS did indeed reflect the opinion of the entire team. Because SJTs are measuring more than one characteristic, it is possible that a test taker’s answers may be confounded by their preferred conflict or assertiveness styles. For example, a student could be aware that an observed behavior is unethical, but they may lack the assertiveness needed to report it. Future studies could evaluate whether other personal attributes impact responses to SJT items. Further research may also include the use of SJTs to assess additional attributes (leadership, emotional intelligence) or assess professional identity development longitudinally throughout the curriculum, especially a sense of responsibility to others. Wolcott and colleagues have begun to study how prior knowledge and experience affect decision-making on SJTs. ⁵ As we improve our understanding of these potential influences, refinement of SJTs may improve their utility. Other areas of further inquiry could be determining the predictive validity of SJTs or assessing the correlation between SJT performance and future performance on professionalism assessments (for example, by preceptors on clinical learning experiences, or by a state board of pharmacy).