Role of Knowledge and Experience in Situational Judgment Test Responses of Pharmacists and Pharmacy Students

INTRODUCTION

Situational judgment tests (SJTs) have been used in the health professions as a strategy to evaluate social and behavioral attributes across multiple disciplines, including pharmacy.^1,2 Situational judgment tests include written or video-based assessments that present hypothetical scenarios intended to measure a specific construct. The examinee is asked how they should respond to the scenario, which is expected to be indicative of their behavior in practice.^3-5 Unfortunately, escalating interest in SJTs has eclipsed efforts to generate sufficient validity evidence to support the use of SJTs in the health sciences.

The proliferation of SJTsine presents an opportunity for more investigations on the theoretical underpinnings of the SJT assessment methodology.⁶ Lievens and Motowidlo present the most contemporary model of the knowledge determinants and antecedents that influence the SJT response process.⁶ Their theory posits that multiple antecedents influence SJT performance regardless of the construct being evaluated or the context of the test. In their model, attributes such as emotional intelligence, interests, values, personality traits, cognitive ability, and experiences (both general and specific to the job) are described as critical precursors to informing decision-making processes on an SJT. An important question is the extent to which general or job-specific knowledge and experiences have an impact on these response processes. In other words, researchers debate how much these elements are utilized in the SJT response process.⁷

From a broad perspective, the consensus has been that SJTs measure procedural knowledge, which refers to knowledge about how to respond to a scenario rather than whether individuals possess an inclination to carry out the response in person.^8,9 In addition, procedural knowledge could include when or how to apply that knowledge based on the presented scenario. Motowidlo and Beier advanced this concept based on the understanding of knowledge acquisition.^10-12 They suggested that procedural knowledge includes two types of knowledge: general domain knowledge and specific job knowledge. Each of these knowledge antecedents have been shown to impact SJT performance and uniquely contribute to the response process.¹⁰

General domain knowledge reflects fundamental socialization processes and personal dispositions in life that are not obtained through job-specific experiences. General domain knowledge is also referred to as implicit trait policies, which refers to the appreciation of the costs and benefits of expressing a trait (ie, following a certain action) in response to a given situation and are often informed by general experiences.^13,14 Specific job knowledge, conversely, can be learned only through that particular job or a similar job.⁶ With respect to the health professions, this could include other service-oriented jobs such as human resources, teaching, social services, or public safety. Specific job knowledge is informed by specific or related job experiences and practice.

There is limited understanding about how both types of knowledge and experience, ie, general and job-specific, may present in the decision-making process, which could inform SJT item design and score interpretation. Situational judgment tests are assumed to measure complex decision-making processes that integrate knowledge and experiences that would be predictive of job performance; however, how these factors may manifest in the response process is unclear. The purpose of this study was to investigate the role of general and job-specific knowledge and experiences in the SJT response process and how the type of person completing the test (ie, student or experienced pharmacist) and item context (ie, health care or non-health care setting) relate to these factors.

METHODS

This study used a convenience sample of 30 participants: student pharmacists (ie, individuals completing their Doctor of Pharmacy (PharmD) degree; n=15) and experienced pharmacists (ie, those with more than five years of experience as a licensed pharmacist; n=15). The two levels of experience were critical to identify how general and job-specific knowledge and accompanying experience may influence the SJT response process. The small sample size was selected because of the exploratory nature of the study and has been justified by a previous work suggesting that 11 participants were sufficient to achieve coding saturation when evaluating SJT response processes.¹⁵ Participants were provided an alphanumeric identifier to designate whether they were a student (indicated by the label “S” followed by a number from 1 to 15) or a pharmacist (indicated by the label “P” followed by a number from 1 to 15). In references to specific participants, we used these alphanumeric identifiers to ensure their anonymity.

The SJT was designed to measure empathy, which is a critical construct of professional competence.¹⁶ Empathy is considered a multidimensional construct that includes two factors: cognitive empathy, which is the ability to understand another person’s perspective, and affective empathy, which is the ability to understand and internalize the feelings experienced by others.^17-21

The SJT used a construct-driven approach that was designed in collaboration with 11 faculty pharmacists to ensure it included relevant scenarios.²² The SJT included 12 items equally distributed based on the empathy component (ie, six items each on affective and cognitive empathy labeled A and C, respectively) and also setting (ie, six items each referring to a health care and non-health care setting labeled H and N, respectively). Items with different settings were important as we hypothesized that a health care setting would elicit a participant’s job-specific knowledge and experiences, whereas items in non-health care settings would elicit a participant’s general knowledge and experiences. Each item included a brief scenario followed by five plausible response options.

Situational judgment test items used a knowledge-based format where examinees were asked how they should respond to a given scenario. Examinees then ranked the response options from most appropriate to least appropriate. The ranking-response format required participants to be more explicit in their decision-making process and offered an advantage compared to other response formats.²³ Examinees had approximately two minutes per question to create their response. Additional details about the design and psychometric analysis of SJTs are provided elsewhere.^24,25

The study was approved by the University of North Carolina Institutional Review Board. Participants met with the researcher for a 90-minute one-on-one interview that included a think-aloud interview, cognitive interview, and demographic survey. The interviews were audio recorded for transcription purposes.

During the think-aloud interview, participants completed the full 12-item SJT and were instructed to verbalize their thoughts as they worked through the items.^25,26 Participants then progressed to the cognitive interview where they were asked about their understanding and thought process on eight pre-selected SJT items. Items were randomized and stratified so that each participant reviewed four items related to non-health care settings, health care settings, affective empathy, and cognitive empathy, respectively. Participants had the opportunity to review each item during the cognitive interview; however, they could not change their responses. The cognitive interview included questions about the experiences and memories, if any, that were considered while responding to the item. In summary, each SJT item had 30 think-aloud interviews (ie, 15 from students and 15 from pharmacists) and 20 cognitive interviews (ie, 10 from students and 10 from pharmacists).

Situational judgment test response data and demographic survey responses were compiled into an electronic database and labeled using the participant’s unique identifier. Deidentified transcripts were segmented to optimize data analysis. Think-aloud interviews were often short, and the entire interview was maintained in order and grouped by participant type (ie, student or pharmacist). Cognitive interviews were organized by item so features could be identified specific to the type of item. Interviews were coded by two independent reviewers using an a priori codebook developed from existing theoretical and empirical understanding of the SJT response process.^6,28-31 The supplemental appendix contains the codes, descriptions, and examples related to knowledge and experiences.

The final set of coded interview transcripts were analyzed for the prevalence and context of the utterances; patterns and relationships among the codes were discussed among the researchers to establish consensus. The Pearson chi-square test and Fisher exact test were used to make comparisons between groups where appropriate. Descriptive and statistical analyses were conducted using Stata, version 15 (StataCorp, LLC, College Station, TX). A p value <.05 was considered statistically significant.^32,33

RESULTS

Student participants were predominantly female (n=11, 73.3%) with a median age of 24 years (range, 22-45 years). Most of the students were entering their third or fourth year of pharmacy school (n=11, 73%), and 13 of the students (87%) worked in a health care-related job outside of school. Eight of the students (53%) reported working in a non-health care human services field with one year of experience being the median (range, 0-10 years). The pharmacist participants were also predominantly female (n=13, 86.6%) with a median age of 36 years (range, 29-51 years). Participating pharmacists had a median of eight years of experience as a licensed pharmacist (range=6-23 years), and all were employed in a university hospital setting. Eleven pharmacists (73%) reported working in a non-health care human services field and had a median of four years of experience (range, 0-10 years).

Analysis of the performance data on and psychometric qualities of the SJT instrument used are reported elsewhere.^24,25 Of note, performance was moderately correlated (r_s=.34) with another instrument used to measure empathy, and most items had a high level of concordance in response selections among participants.^24,25 The SJT provided sufficiently reliable and valid data assessing empathy considering the exploratory aims of the study, small sample size, and limited number of items for convenience.

A total of 7,252 coded statements were made during the cognitive and think-aloud interviews. This included 480 (6.6%) utterances that referred to knowledge and experiences: 45.2% related to job-specific knowledge or experience, 27.5% related to general knowledge or experience, 17.9% related to a lack of experience, and 9.4% were nondescript experiences that could not be distinguished as either general or job-specific (the frequency and examples of these are presented in Appendix 1). The distribution of comments was sufficiently equal between students and pharmacists (51.0% and 49.0%, respectively).

As presented in Table 1, Pearson’s chi-square test revealed a significant difference in the reference to job-specific and general knowledge or experiences based on whether the item was in a health care or non-health care setting (χ²=73.6, p=<.001). In this case, job-specific knowledge or experiences were referenced more often when the setting was health care related, whereas general knowledge and experiences were more commonly cited in non-health care setting items. A significant difference was observed based on whether the item measured affective or cognitive empathy (χ²=14.5, p=<.001); job-specific knowledge and experiences were referenced more frequently during questions related to cognitive empathy compared to those of affective empathy. Conversely, there was no statistical difference in the job-specific and general knowledge and experiences reported by students compared to that reported by pharmacists.

Knowledge references included factors such as: information (eg, factors or observations pertinent to the situation), strategy (eg, a plan or approach to achieve an objective), or skill (eg, ability or set of strategies to achieve an objective). References to general and job-specific experiences often included features related to the location (eg, setting of the experience), actors (eg, individuals included in the experience), and task or topic (eg, challenge or goal of the experience). In addition, the experiences could be classified according to three factors: similarity to the presented scenario, specificity of the details provided, and recency of the memory to the present moment.

The references to job-specific knowledge were also consistent between students and pharmacists. The majority of job-specific knowledge references related to information, which included details about disease state management, specific medications, legality of certain actions, and hierarchical structures in health care. Pharmacists and students often referred to different types of skills. For example, many pharmacists referred to service recovery training they had received, which one pharmacist described as “when you have a situation that has escalated and how it is best to handle it.” In this study, there were no student references to similar skill training.

Conversely, few pharmacists referred to mental health training, which was discussed by several students. For example, one student described how mental health first aid training “explicitly emphasized that you shouldn’t talk about yourself in mental health crises and you should really be focused on addressing that person’s need and affirming them.” In this study, we found evidence that suggests there are minimal differences in the types of knowledge participants use to answer SJT items, regardless of their level of experience. However, there may be some nuances in participants’ responses based on workplace requirements.

Compared to job-specific knowledge references, the discussion of general knowledge was very limited. When mentioned, general knowledge often referred to information such as social norms. This could be seen in an utterance by one student who stated, “just thinking about social norms, you wouldn’t confront somebody in the grocery store.” Overall, there were few conclusions that could be drawn regarding the use of general knowledge as it appeared infrequently in participant transcripts. The scant presence suggests general knowledge may not have been a substantial component in SJT response processes.

With respect to job-specific experiences, pharmacists and students referenced these elements in similar ways; however, there were some notable exceptions. The most substantial difference was the location of the referenced experiences. Most pharmacist examples of job-specific experiences were explicitly connected to their work; very few references included examples from school. Conversely, students’ “job-specific” experiences were more diverse and included experiences from school, clinical rotations, and some work experiences. The greater distribution was likely because of the recency of these experiences for students compared to pharmacists. Thus, pharmacists with more experience were more likely to rely on their work-based experiences than experiences from earlier years when they were in training.

In addition, student experiences more often included observations of indirect interactions in which they were not an active participant. For example, one student discussed how they had seen “some pharmacists delivering sensitive information about what could happen with certain drugs”; conversely, one pharmacist, when discussing the same test item, shared “about a situation when I was practicing in the HIV clinic.” Other examples from two students who stated, “I know we talked about a lot of different scenarios in class… especially diabetes patients,” and, who shared, “we’ve talked about medication errors in class a lot and I’ve talked about it on some of my rotations.” We found that when describing job-specific experiences, students more often than pharmacists integrated experiences that related to or they had observed during their education and training, which often did not include their direct involvement in the scenario described.

Moreover, when pharmacists discussed job-specific experiences, they often included a greater amount of detail about the scenario than did students, who tended to provide more generic descriptions. A pharmacist, for example, shared a story that “two days ago, that patient we had that was on [drug], it was documented in the clinic notes” and then continued to describe in detail the experience of identifying a medication error. Students, in contrast, were less descriptive when describing similar scenarios. For example, a student talked about an experience that included “going into the patient’s room when the patient’s family is upset at something” and had difficulty recalling details about the event. In general, the data suggest that when pharmacists did provide an example, they often included more details and elements compared to students.

The use and quality of general experiences was not significantly different between pharmacists and students. The experiences tended to be vague but still closely related to the SJT scenarios presented. The actors in these scenarios were often friends and family members, and the mention of these experiences occurred mostly when discussing items referring to non-health care settings. One notable feature was that examples from televisions shows were sometimes referenced as viable experiences. For example, when a pharmacist was discussing the item related to a friend taking a medication to help them study, their immediate response led them to reference “Jesse Spano from Saved by the Bell.” A student also stated, “I think of experiences that a lot of times I watch on TV shows like Dateline.” Overall, there is minimal evidence to suggest that general experiences contributed to SJT response processes differently for student pharmacists and pharmacists.

Nondescript experiences were not analyzed extensively because of the ambiguity of these references. Examples of this include where a pharmacist stated, “This [question] is a tough one because I feel like this is a reality every day” and one student shared that “this one felt familiar to me.” References to a lack of experience, however, were reviewed to determine if they were more prevalent in specific scenarios. There were no differences in the number of references in health care and non-health care settings (41 to 42 references, respectively), but there were more references to a lack of experience when completing questions intended to measure affective empathy (48 statements) compared to those measuring cognitive empathy (35 statements).

Overall, there were minimal differences in how participants referred to or how they perceived their lack of experience. Most participants, specifically students stated “I don’t really have very much to draw on” or simply “this has never happened” as shared by one of the pharmacists. One difference, however, was that pharmacists tended to be more specific when they considered whether they had relatable experiences. For example, a pharmacist stated, “I haven’t had a particular scenario with regards to chemotherapy,” whereas students discussing the same question would state more generally that they hadn’t been “in a situation where a family member is that upset.” The data suggest that pharmacists may be more attentive to granular details compared to students when searching for similar experiences.

There were few references to knowledge and experiences during the think-aloud interviews (ie, five or less participants made a reference to any component). Job-specific knowledge was referenced by 12 pharmacists and nine students. A majority (91%) of the references to job-specific knowledge were related to information, such as disease state management, the legality of response options, and their responsibilities as a health care provider. The remaining references were regarding strategies for engaging with patients, such as apologizing and sharing experiences in times of emotional stress. Overall, there was scant evidence to verify the role of knowledge and experiences in the think-aloud interviews, except that job-specific knowledge may be explicitly involved.

DISCUSSION

Lievens and Motowidlo presented a model of the theoretical underpinnings of SJTs that posits that general and job-specific knowledge contribute to the procedural knowledge assessed by these instruments; however, the extent to which these factors are incorporated into the participant’s response process has not been studied.⁶ In addition, it is unclear whether there is variation in the extent to which these factors are incorporated as it relates differences in the item setting or participant type.³⁴ The aim of this research was to describe the role of knowledge and experience in the SJT response process.

In this study, job-specific knowledge and experiences were referenced more often by participants than were general knowledge and experiences during both the cognitive and think-aloud interviews. Overall, there was limited evidence in the sample studied that general knowledge contributed to the response process. This observation may have been influenced by the health care setting in which participants took the assessment or the expectation to think about health care settings for the study. Therefore, we propose that general knowledge may not be a significant feature to include in future studies about SJT response processes. This distribution of knowledge and experiences, however, may differ when testing other health professions or different constructs. Additional research is needed to confirm this finding.

Of note, the test incorporated an equal number of items that were in health care and non-health care settings to elicit knowledge and experiences that were not exclusively related to a health care job. The higher prevalence of references to job-specific knowledge and experience suggests that participants may use varying degrees of general and job-specific knowledge and experiences when responding to SJT items. Conversely, participants may simply integrate job-specific knowledge even in non-clinical scenarios. For example, much of the individual’s identity may be connected to their work as a clinician and those experiences may be more readily accessible because of the substantial amount of time spent in those settings. As a result, participants may readily integrate those features into their collective decision-making processes. The study did not include questions or analyses that could explicitly identify why this observation was present; thus, this topic warrants further investigation.

In addition, the results also contributed evidence regarding two previously unreported observations: participant awareness of a lack of experience or knowledge and participant descriptions of the types of knowledge and experiences retrieved. The findings demonstrated that participants often identified times in which they had little to no experience or knowledge about a particular topic, which was unexpected. The study did not investigate explicitly how this awareness contributed to SJT performance, and it is unclear whether this is a significant feature that should be considered in validity studies or when interpreting SJT scores. The findings also contributed to the literature as this was the first study to describe qualities of the experiences and knowledge that were referenced by participants during this SJT. The elements, such as the location, actors, and tasks, should be considered during SJT design processes as they were frequently identified by participants to be relevant components of SJT scenarios. These features may also be particularly relevant if there is a certain type of knowledge or skill that is to be evaluated by the test item.

Finally, the study evaluated whether there were differences in the types of knowledge and experiences that were recalled based on individual items, their characteristics, and participant type. As expected, job-specific knowledge and experiences were more often referenced in health care setting questions compared to non-health care setting questions, which suggested these items were capable of targeting a participant’s job-specific knowledge and experiences. Interestingly, there was no evidence to suggest that students and pharmacists differed in the prevalence of job-specific knowledge and experiences that were recalled; however, there was evidence to suggest that pharmacists more often recalled work-based experiences while students recalled classroom-based or learning experiences. In addition, there were few cases where there were differences in the knowledge or experiences retrieved for an individual item. This finding suggests that response process validity data are not necessarily generalizable and that the response processes being elicited can be sample dependent. Currently, there is a lack of research to corroborate these findings as often the research is focused exclusively on the sample of interest. Overall, this highlights that response process validity data should be interpreted with caution and be discussed in reference to the sample being evaluated.

The results of this study should not be interpreted without consideration of several limitations. First, the results are based on a small sample of participants from a specific region in the United States with their own unique experiences and perspectives. We designed this study to engage learners and practitioners from a variety of backgrounds to offer a more comprehensive approach; however, we do not assume that our findings are generalizable to the response process of participants from other regions. The study did not evaluate how other variables, such as years of experience, may relate to the frequency of certain knowledge or experience utterances or SJT performance. In addition, the study findings were predicated on the SJT design, which followed a rigorous process to ensure alignment with the intended construct. The results are limited to understanding the response process for a specific SJT model (ie, ranked response) and a specific construct (ie, empathy); therefore, further exploration to validate the findings in broader contexts is warranted.

In summary, findings from this study were the first to confirm the types of knowledge and experiences that are most often retrieved during SJT response processes involving pharmacists and pharmacy students. Job-specific knowledge and experiences remain the most salient features retrieved, which was expected considering SJTs are grounded in human resources research and were designed as an instrument to predict job performance.^3,4 The implications of this research inform the design and implementation of SJTs in pharmacy education. For example, design of SJT scenarios should include broadly applicable situations that most participants would have exposure to as this could avoid cases where a lack of knowledge or experience may disadvantage an examinee. In addition, the design process should be attentive as to why certain situations are selected, how they most relate to the knowledges and experiences that are most relevant.

CONCLUSION

Results from this study demonstrate how general and job-specific knowledge and experiences are a critical component of SJT response processes. Data from cognitive interviews show that job-specific references are more prevalent (regardless of the setting of the test item) and there can be significant variation in the job-specific experiences retrieved by participants. There is minimal evidence, however, that experience and knowledge are explicitly referenced during the response process according to think-aloud interview data. Overall, these findings contribute to SJT research in that this was the first attempt at generating evidence about the types and features of experiences and knowledge recalled during SJT response processes.