Abstract
Objective. For pharmacy students to successfully meet competencies related to Accreditation Council for Pharmacy Education Standards 3 and 4, it is essential for pharmacy programs to assess student progression in the affective domain. The purpose of this study was to develop and assess the validity of a situational judgment test (SJT) for evaluating student mastery of Standards 3 and 4.
Methods. A multi-institutional faculty team developed an 18-item SJT that consisted of scenarios asking the respondent to rank the effectiveness of four response options mapped to Standards 3 and 4. The research team systematically reviewed the literature, created items, and deliberated until consensus was achieved. Subject matter experts (SMEs) reviewed and provided feedback on the instrument. Students from two institutions were recruited to participate in cognitive interviews about the finalized instrument. Cognitive interview data were analyzed to identify themes.
Results. After edits were made to the instrument based on SME feedback, students (n=18) in the cognitive interviews identified item length as a concern and commented on item/response clarity and comprehensiveness. Data from the cognitive interviews were used to modify the SJT to reduce the length and clarify items. The result was two shorter versions of the instrument, both with similar mapping to all elements in Standards 3 and 4.
Conclusion. Early steps in validating the SJT suggested that the instrument may be a promising tool to assess student progression in the affective domain. The SJT instrument is intended to provide evidence of student pharmacist development that occurs in the didactic, experiential, and co-curricular portions of pharmacy education. The instrument can serve as one part of a comprehensive assessment plan.
- co-curriculum
- situational judgment test
- validity evidence
- ACPE standards
- affective domain
- cognitive interviewing
INTRODUCTION
The goal of pharmacy education is to produce well-rounded, competent pharmacists ready to face the challenges of an ever-evolving health care field and to demonstrate practice- and team-readiness. Standards 2016 issued by the Accreditation Council for Pharmacy Education (ACPE) include an intentional focus on the curriculum, co-curriculum, and experiential curriculum working in harmony to develop team-ready, practice-ready pharmacists. Compared to past standards, Standards 2016 specifies the provision and the assessment of co-curricular activities as a necessary component in pharmacy programs, in addition to demonstration of students’ clinical and practice knowledge.1 Standards 3 and 4 focus on affective domain skills and are adopted from the American Association of Colleges of Pharmacy (AACP)’s Center for the Advancement of Pharmacy Education (CAPE) Educational Outcomes 2013.2 Standard 3 focuses on skills necessary for pharmacy practice and patient care, including problem solving, education, patient advocacy, interprofessional collaboration, cultural sensitivity, and communication. Standard 4 focuses on essential components of personal and professional development, including self-awareness, leadership, innovation and entrepreneurship, and professionalism.1 Together, Standards 3 and 4 are typically reinforced within the co-curriculum of pharmacy programs, and Appendix 3 notes using co-curricular experiences as part of the required documentation, along with curricular components, to develop competence in the affective domain.1
Since the adoption of the Standards, pharmacy programs have developed a variety of ways to provide, track, and assess elements of their co-curriculum. A national survey of pharmacy programs indicated that the most common method for documenting completion of co-curricular requirements was student reflections.3 While this assessment method is valuable and allows students to practice self-awareness and critically self-assess their performance, there are challenges with quantifying and evaluating data from these assignments. Furthermore, evaluating these reflections in a systematic and timely manner often crates an added workload for pharmacy programs. For many of the outcomes identified in Standards 3 and 4, there are no readily available comprehensive assessment instruments. This makes it challenging to determine whether graduates are adequately equipped with those affective-domain related competencies. In the absence of an assessment with evidence of validity, it is not only difficult to assess key elements of Standards 3 and 4, but also to track students’ growth and advancement in these areas throughout their time in a pharmacy program. Thus, there is a need to create comprehensive instruments and associated validity evidence that can supplement self-assessment measures.
One potentially valuable approach to measuring affective skills is using situational judgement tests (SJTs).4,5 These tests are low fidelity simulations designed to assess pharmacy students’ judgement in situations that emphasize non-academic skills. Scenarios are constructed to reflect a situation that pharmacists are likely to encounter in practice, and then students are asked to respond to this hypothetical scenario. With SJTs, there are often no “correct” responses.5,6 Instead, for each scenario, participants are given a list of possible responses and asked to identify the best course of action. Each response can be individually rated for appropriateness, or the responses can be ranked in according to their appropriateness.4⇓-6 Students’ responses are then compared with those provided by content experts, which are provided a priori. Situational judgement tests are based on the expectation that past behavior or knowledge is a good predictor of future behavior.4⇓-6 Historically, SJTs have been used in a variety of professional disciplines as part of the hiring process to identify the most qualified applicants.5 More recently, their use has been seen in admissions screening for medicine where they have demonstrated predictive validity superior to knowledge- and cognitive ability-based tests.7 In these approaches, SJTs are more commonly used to identify student understanding of what they should do in a scenario. Less has been published on the use of SJTs in pharmacy programs.5,9 Situational judgement tests also tend to perform well across demographic groups.8,10 The few SJTs that have been developed and evaluated have covered a few domains, such as empathy and professionalism. These SJTs have demonstrated evidence of validity and potential value as measurable standardized assessments or formative assessments.5,11,12
Demonstrating evidence of validity for any instrument is an important component of the instrument development and refinement process.13 The cognitive interview method has increasingly been employed during the survey development process to ensure the overall quality of an instrument and to identify factors that attribute to incorrect responses. In addition, it is one method for providing evidence of content validity and is used to determine what and how survey items should be revised.14 The main purpose of the cognitive interview is to determine whether study participants interpret the survey items in the way the researcher intended,15 so that misalignments in interpretation between the researcher and the respondent can be elucidated.14 Thus, the focus of this interview method is not on the person providing a response to the question, but on the survey question.14,16 Cognitive interviews have been successfully used in other pharmacy education studies.17
Given the lack of readily available standardized assessments and the potential utility of SJTs, the purpose of this study was to develop and demonstrate initial validity evidence for a SJT instrument designed to assess competence relating to key elements in ACPE Standards 3 and 4. Such an instrument can be used to measure student pharmacists’ growth in competencies over time (ie, student progression and competency level), with reduced impact on program resources.
METHODS
A three-phase process was used to develop and demonstrate face and content validity evidence of an SJT instrument for assessing pharmacy students’ mastery of the 10 key elements (hereafter referred to as domains) of Standards 3 and 4. These elements include problem solving, education, patient advocacy, interprofessional collaboration, cultural sensitivity, communication, self-awareness, leadership, innovation and entrepreneurship, and professionalism. For an overview of the methodology, please see Figure 1.
Overview of the development process of an situational judgement test (SJT) to assess ACPE Standards 3 and 4 Student Competencies.
Prior to phase one, a group of 20 faculty who were members of the AACP Assessment Special Interest Group conducted a comprehensive literature review to identify any previously developed instruments for each domain in ACPE Standards 3 and 4. The individuals had expertise in assessment, along with additional expertise in social and administrative sciences, drug information, and clinical sciences. Some of the members had experience in instrument development. The instruments found were student self-assessment scales and surveys; however, a comprehensive assessment tool encompassing all the Standard 3 and 4 domains was lacking. The research team determined that an SJT was the best approach for assessing the affective domains based on prior studies.5,11 Thus, the faculty workgroup initiated the three-phase process to develop such an instrument, with guidance from scales found in the literature review as background for item development.
In Phase One, the research team divided up into 10 small working groups to create SJT items for each domain of Standards 3 and 4. The goal was to construct two or three scenarios per domain that would contain four response options and be ranked. While each scenario was designed to primarily address one domain, because of the nature of the scenarios, they often addressed multiple domains. When dealing with patient or professional interactions, several domains will often overlap because of the nature of these interactions and required skillsets. For example, many items mapped to communication in addition to a primary domain, such as professionalism.
Four responses to each SJT scenario item were devised to illustrate different actions that could be taken, displaying skills in the corresponding domain. The instruments identified during the literature review, as mentioned earlier, were used as a guide during the idea generation phase of the responses to provide ideas for the research subgroup. The subgroup members created a set of responses that could be represented clearly in a written SJT format. The appropriateness of each response was determined via a modified Delphi method that consisted of a consensus-building, iterative discussion process among the research team. The responses were subsequently rank ordered to determine the extent to which each response met the domain outcome, with the more appropriate response having the higher score. Currently, with SJTs, there is no one scoring approach that is considered better than others.6 The scoring algorithm was as follows: four points if all options were ranked correctly, three points if three options were ranked correctly, two points if two options were ranked correctly, one point if one option was ranked correctly, zero points if zero options were ranked correctly. The team then verified the mapping of each SJT item to Standards 3 and 4, with the option to map scenarios that functionally assessed more than one domain to multiple domains, working until consensus was met. Initial item generation resulted in the development of 22 items. Items that could not achieve 100% consensus were eliminated (n=4), resulting in 18 items. For the four eliminated items, consensus could not be achieved on stem and option clarity or ideal ordering of answer choices. The entire working group reviewed the scenarios and finalized the SJT (Appendix 1).
In Phase Two, eight additional experts in both assessment and affective domain subject matter from outside the working group reviewed and provided feedback on the instrument. The subject matter experts (SMEs) were selected based on either formal training in or significant scholarly contributions to their respective fields. The SMEs were asked to suggest improvements, verify mapping, examine the rank-order of responses, and provide any areas that needed clarification. Three assessed the entire instrument (SJT and affective domain assessment SMEs), and five assessed the scenarios designed to address affective domains within their realm of expertise (education, interprofessional education, leadership, and professionalism). The SMEs were sought for these four domains specifically based on recommendations of the working groups that created questions to assess those domains. After each SME provided comments, members of the research team met to discuss feedback and implement changes before disseminating to the next SME. The SJT was considered appropriately vetted after two consecutive SMEs returned the instrument without suggested changes. Wording changes made from these reviews resulted in the version used for the cognitive interview phase of pilot testing. Comments provided by the SMEs primarily consisted of suggestions for shortening the content in each scenario and shortening the assessment overall. The SMEs also suggested changes to some answer choices with subject matter within their areas of expertise to clearly delineate differences in competency reflected by the answer choices and more closely align the content of the answer choices with real-life practice situations. A summary of the changes made to the items in each SME domain are provided in Table 1. No items were removed as a result of Phase Two processes. In Phase Three, cognitive interview was conducted with current student pharmacists in four-year Doctor of Pharmacy (PharmD) programs to elicit student understanding of each item for clarity, consistency in interpretation, and appropriateness of items. Students were recruited from two programs, Cedarville University and Manchester University, via emails sent out to all students in these professional programs, with a goal of obtaining up to 10 interviews per institution. These institutions were chosen because of their extensive involvement in the project and having researchers with experience in cognitive interviewing. Fifteen students (eight from Cedarville University and seven from Manchester University) volunteered and were interviewed. The students represented all four years of the PharmD program at each institution. The cognitive interview method can be divided into two techniques: think-aloud and verbal probing.14 The method of verbal probing can be concurrent or retrospective. This study used concurrent verbal probing, which requires asking additional specific questions intended to elicit more information after the study participant answered a question. The goal of this method is to learn more about the cognitive process followed in answering a question.17
Changes Made to a Situational Judgment Test Based on Suggestions From Subject Matter Experts
With each student who agreed to participate, one-hour sessions were scheduled to complete the cognitive interview. At each interview, students were told the purpose of the study, informed that the session would be audio-recorded for data analysis purposes. All students were asked for and provided consent to participate as designated in the study IRB approval. Since the full SJT instrument was long, each student was assigned a random list of questions from the full instrument to answer. Each student was asked to read an SJT item and answer the question. Students were encouraged to read and think out loud if it did not interfere with their understanding of the material. After providing their ranked responses, a research team member asked additional probing questions to gain additional insight into how students interpreted the question, their judgment related to the items and responses, what issues they faced when determining their rankings, and address any clarity or inconsistency with the item or the responses. Last, students were asked what skills each SJT item addressed as an open-ended question. While this was not a primary goal of the development of the SJT and was minimally discussed during the cognitive interviews, the goal of asking this question was to determine if students understood the underpinning concepts in the questions related to affective domain skills. The interviewers took detailed notes during each session. The interviews also were audio-recorded and transcribed verbatim by a trained research assistant. Students completed questions until the one-hour time slot had passed, they had finished all of the questions, or they decided to stop answering questions.
Based on transcribed interviews and notes taken, three researchers met to discuss common themes with regards to item clarity and length, instrument length, and any other issues that arose during the sessions. Preliminary themes were disseminated to the full research group, and they iteratively revised the SJT until consensus was achieved. Two researchers then reviewed the interview transcriptions to confirm themes and that all necessary changes had been discussed.
RESULTS
For phase three, of the 15 PharmD students who were recruited and participated in the cognitive interviews, eight were from Cedarville University (three first year students, one second year student, three third year students, and one fourth year student) and seven were from Manchester University (one first year student, two second year students, two third year students, and two fourth year students). At least three students from each year in the professional program (P1-P4) participated. Students identified scenario and response length as a concern. Major themes that emerged were: length of the scenarios and response options, clarity of the scenarios and responses, and suggestions of more suitable responses than presented to the students. Most students were not able to complete more than four to eight scenarios in a 45- to 60-minute cognitive interview session. For example, one student stated, “I think that if a little more information was in the prompt and not in the answer choice, it could maybe cut down on the wording because all the answers were very long.” Another student reported, “Some of the questions are just kind of wordy, which is kind of required to get a good understanding of some of the questions, but just like this one in particular was just like extremely long.”
Students also felt some scenarios were clear, but some questions could be rephrased to be more concise and straightforward. For example, one student found a scenario very clear, “I liked the question; it was well worded. The answers were easy enough to understand, and it does show an issue that is had in the workplace.” Another student found a different scenario more challenging, “I wasn't sure like what B was implying… So that was a little unclear so I picked that as one. But I might have picked A as my first choice if I knew that he was doing that in English.”
In some scenarios, students proposed alternative responses to be used in place of the options listed. For example, one student noted on a scenario in which an inappropriate prescription was called in to a pharmacy that they would want “to ask to speak to the physician directly and just have a chat.” They did not want to go through another individual to resolve a patient problem. Another student, responding to a scenario where a potential pattern of inappropriate prescribing was identified, wanted to expand the approach to include an “evidence-based solution of trends by other pharmacists in the department, so it's not just you recognizing the errors because it may not be an error on other people's ends.” When presented with a listing of the elements from ACPE Standards 3 and 4, students appropriately identified many of the domains to the scenarios as originally mapped by the research team or similar skills, even if they did not know which answer was best in the scenario. For example, students often indicated that the scenarios were addressing “interprofessional communication,” “problem-solving,” “communication,” or “making good judgments.”
The cognitive interview process identified that students had to really think through several of the answers, weighing which ones were optimal. For example, one student noted, “It was hard to choose the answers in some sort of order just because there were so many competing good ideas.” Another noted, “I definitely think I have a response that's the most effective, but I'm not sure if I can decide between a couple of the others.” That student then discussed their thought process regarding how they ranked further questions.
Data from the CIs were used to modify the SJT in the following ways. First, items were modified to enhance clarity according to student feedback on the responses and reduce both item and response length. Second, it was determined that the 18-item instrument needed to be reduced in size to ease the cognitive burden on students who complete the instrument. As such, two versions of the instrument with nine items each and comparable in terms of the domains assessed were created. (Table 2) In the original instrument, nine of the 10 domains had two questions written for each, with communication integrated in multiple items. After dividing the SJT into two versions based on primary domain assessed, the item mapping across all 10 domains in both versions remained substantially similar. Thus, the split between the two versions was performed to ensure at least one question per primary domain was included in each version. (Table 2).
Mapping Situational Judgment Test Items to ACPE Standards 3 and 4 Domain
DISCUSSION
The SJT was developed using an approach similar to that used by others who have created SJTs for use in pharmacy education.5,11 Patterson and colleagues developed an SJT to assess integrity, empathy, team involvement, and critical thinking/problem-solving, while Smith and colleagues developed an SJT to assess professionalism.5,11 In both studies, investigators developed scenarios and used subject matter experts (SMEs) to refine their instrument over several rounds before moving to psychometric evaluation.5,11 In the present study, we also used two approaches to establish content validity. First, SMEs were recruited to participate in a multiple iterative process until consensus was reached. Second, cognitive interviewing was performed with a group of student pharmacists, which enabled further refinement of the scenarios and responses. Based on the findings from cognitive interviews, the SJT was split into two parallel versions that each addresses all domains while reducing the cognitive burden on students taking the assessment. Additional psychometric testing will be conducted to further refine the SJT.
As mentioned, cognitive interviews were used as a method of instrument refinement and to provide further evidence of content validity prior to psychometric testing. Cognitive interviews provide an opportunity to determine whether study participants interpreted the survey items in the way that authors intended15 and to address any differences in interpretation.14 Students highlighted several aspects of the SJP that were not interpreted as intended in addition to determining that the test was too long. Thus, the information from the cognitive interview resulted in more clear and concise scenarios and response items, as well as dividing the instrument into two shorter versions of the SJT. Reducing the length also allows the instrument to be completed in a shorter time, while still mapping to all 10 elements of Standards 3 and 4. In contrast, increasing length of scenarios in SJTs can increase the cognitive load of the instrument.18 Institutions could use either version in assessing students and even rotate versions to decrease test-retest limitations.
While further evidence of the validity of this instrument is needed and is currently being conducted, an SJT designed to assess ACPE Standards 3 and 4 could be useful for institutions as part of the broader assessment of affective domain components.1 Because the written SJT can only be designed to assess certain skills within each affective domain, it should be used as part of a comprehensive assessment plan that includes other assessment techniques such as direct observation, self-assessment reflections, and assignments that are scored using standardized rubrics.19 The SJT could be used to benchmark student progress throughout the curriculum and provide additional data within the assessment plan regarding student competency attainment.
This study had several limitations, including conducting the cognitive interview at only two institutions, both of which were private institutions located in the Midwest, which may impact the generalizability of our findings. Future iterations of the SJT should assess additional student pharmacist populations (eg, institution types, geographic regions, racial/ethnic groups) to increase generalizability. In addition, the rankings of the response items may be subjective and can vary between practicing pharmacists. The multiple iterative process with several SMEs helps ensure that the scenarios, items, and response options have been peer reviewed and examined for intended outcomes. Other limitations include potential variations in the delivery of cognitive interview and data collection among research team members, even though the researchers met beforehand to ensure consistency. Despite the limitations, this is the first study to address all the domains within ACPE Standards 3 and 4 using a single instrument, an SJT. Second, the use of cognitive interviewing of students as an approach to providing further evidence of the content validity allowed for refinement of the instrument and ensured that feedback was received from the intended audience for the SJT.
To further demonstrate the validity of the SJT, the authors are piloting the two versions of the test at multiple institutions, administering the test to preceptors to identify benchmarks for minimum competency, and conducting psychometric testing. The authors’ goal is that the final version of the instrument will provide a more objective, less faculty time-intensive assessment approach that could be used as part of a larger holistic assessment of ACPE Standards 3 and 4.
CONCLUSION
This study describes the development and initial testing of an SJT instrument aimed at addressing the affective domains associated with ACPE Standards 3 and 4. The results provide early evidence of the content validity of the instrument as demonstrated by consensus reached by subject matter experts and cognitive interviews conducted with student pharmacists. Such instruments may be a promising tool to assess student competency and progression in the affective domain during their pharmacy education. This can provide pharmacy programs with valuable information to aid in modification of curricular and/or co-curricular activities. Further studies are underway including psychometric analyses to inform further refinement and subsequent pilot testing of the SJT instrument.
ACKNOWLEDGEMENTS
The authors thank subject matter experts Sarah Gordon, Melissa Rasmussen, Kathryn Smith, Michelle Farland, Kristin Janke, Katie Axford, and Russell Attridge for contributing to creation and revision of the SJT instrument, and Robert D. Beckett for assisting with cognitive interviewing.
- Received December 10, 2020.
- Accepted June 16, 2021.
- © 2022 American Association of Colleges of Pharmacy