A Situational Judgment Test to Assess Students’ Achievement of Accreditation Council for Pharmacy Education Standards 3 and 4

METHODS

A three-phase process was used to develop and demonstrate face and content validity evidence of an SJT instrument for assessing pharmacy students’ mastery of the 10 key elements (hereafter referred to as domains) of Standards 3 and 4. These elements include problem solving, education, patient advocacy, interprofessional collaboration, cultural sensitivity, communication, self-awareness, leadership, innovation and entrepreneurship, and professionalism. For an overview of the methodology, please see Figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Overview of the development process of an situational judgement test (SJT) to assess ACPE Standards 3 and 4 Student Competencies.

Prior to phase one, a group of 20 faculty who were members of the AACP Assessment Special Interest Group conducted a comprehensive literature review to identify any previously developed instruments for each domain in ACPE Standards 3 and 4. The individuals had expertise in assessment, along with additional expertise in social and administrative sciences, drug information, and clinical sciences. Some of the members had experience in instrument development. The instruments found were student self-assessment scales and surveys; however, a comprehensive assessment tool encompassing all the Standard 3 and 4 domains was lacking. The research team determined that an SJT was the best approach for assessing the affective domains based on prior studies.^5,11 Thus, the faculty workgroup initiated the three-phase process to develop such an instrument, with guidance from scales found in the literature review as background for item development.

In Phase One, the research team divided up into 10 small working groups to create SJT items for each domain of Standards 3 and 4. The goal was to construct two or three scenarios per domain that would contain four response options and be ranked. While each scenario was designed to primarily address one domain, because of the nature of the scenarios, they often addressed multiple domains. When dealing with patient or professional interactions, several domains will often overlap because of the nature of these interactions and required skillsets. For example, many items mapped to communication in addition to a primary domain, such as professionalism.

Four responses to each SJT scenario item were devised to illustrate different actions that could be taken, displaying skills in the corresponding domain. The instruments identified during the literature review, as mentioned earlier, were used as a guide during the idea generation phase of the responses to provide ideas for the research subgroup. The subgroup members created a set of responses that could be represented clearly in a written SJT format. The appropriateness of each response was determined via a modified Delphi method that consisted of a consensus-building, iterative discussion process among the research team. The responses were subsequently rank ordered to determine the extent to which each response met the domain outcome, with the more appropriate response having the higher score. Currently, with SJTs, there is no one scoring approach that is considered better than others.⁶ The scoring algorithm was as follows: four points if all options were ranked correctly, three points if three options were ranked correctly, two points if two options were ranked correctly, one point if one option was ranked correctly, zero points if zero options were ranked correctly. The team then verified the mapping of each SJT item to Standards 3 and 4, with the option to map scenarios that functionally assessed more than one domain to multiple domains, working until consensus was met. Initial item generation resulted in the development of 22 items. Items that could not achieve 100% consensus were eliminated (n=4), resulting in 18 items. For the four eliminated items, consensus could not be achieved on stem and option clarity or ideal ordering of answer choices. The entire working group reviewed the scenarios and finalized the SJT (Appendix 1).

In Phase Two, eight additional experts in both assessment and affective domain subject matter from outside the working group reviewed and provided feedback on the instrument. The subject matter experts (SMEs) were selected based on either formal training in or significant scholarly contributions to their respective fields. The SMEs were asked to suggest improvements, verify mapping, examine the rank-order of responses, and provide any areas that needed clarification. Three assessed the entire instrument (SJT and affective domain assessment SMEs), and five assessed the scenarios designed to address affective domains within their realm of expertise (education, interprofessional education, leadership, and professionalism). The SMEs were sought for these four domains specifically based on recommendations of the working groups that created questions to assess those domains. After each SME provided comments, members of the research team met to discuss feedback and implement changes before disseminating to the next SME. The SJT was considered appropriately vetted after two consecutive SMEs returned the instrument without suggested changes. Wording changes made from these reviews resulted in the version used for the cognitive interview phase of pilot testing. Comments provided by the SMEs primarily consisted of suggestions for shortening the content in each scenario and shortening the assessment overall. The SMEs also suggested changes to some answer choices with subject matter within their areas of expertise to clearly delineate differences in competency reflected by the answer choices and more closely align the content of the answer choices with real-life practice situations. A summary of the changes made to the items in each SME domain are provided in Table 1. No items were removed as a result of Phase Two processes. In Phase Three, cognitive interview was conducted with current student pharmacists in four-year Doctor of Pharmacy (PharmD) programs to elicit student understanding of each item for clarity, consistency in interpretation, and appropriateness of items. Students were recruited from two programs, Cedarville University and Manchester University, via emails sent out to all students in these professional programs, with a goal of obtaining up to 10 interviews per institution. These institutions were chosen because of their extensive involvement in the project and having researchers with experience in cognitive interviewing. Fifteen students (eight from Cedarville University and seven from Manchester University) volunteered and were interviewed. The students represented all four years of the PharmD program at each institution. The cognitive interview method can be divided into two techniques: think-aloud and verbal probing.¹⁴ The method of verbal probing can be concurrent or retrospective. This study used concurrent verbal probing, which requires asking additional specific questions intended to elicit more information after the study participant answered a question. The goal of this method is to learn more about the cognitive process followed in answering a question.¹⁷

With each student who agreed to participate, one-hour sessions were scheduled to complete the cognitive interview. At each interview, students were told the purpose of the study, informed that the session would be audio-recorded for data analysis purposes. All students were asked for and provided consent to participate as designated in the study IRB approval. Since the full SJT instrument was long, each student was assigned a random list of questions from the full instrument to answer. Each student was asked to read an SJT item and answer the question. Students were encouraged to read and think out loud if it did not interfere with their understanding of the material. After providing their ranked responses, a research team member asked additional probing questions to gain additional insight into how students interpreted the question, their judgment related to the items and responses, what issues they faced when determining their rankings, and address any clarity or inconsistency with the item or the responses. Last, students were asked what skills each SJT item addressed as an open-ended question. While this was not a primary goal of the development of the SJT and was minimally discussed during the cognitive interviews, the goal of asking this question was to determine if students understood the underpinning concepts in the questions related to affective domain skills. The interviewers took detailed notes during each session. The interviews also were audio-recorded and transcribed verbatim by a trained research assistant. Students completed questions until the one-hour time slot had passed, they had finished all of the questions, or they decided to stop answering questions.

Based on transcribed interviews and notes taken, three researchers met to discuss common themes with regards to item clarity and length, instrument length, and any other issues that arose during the sessions. Preliminary themes were disseminated to the full research group, and they iteratively revised the SJT until consensus was achieved. Two researchers then reviewed the interview transcriptions to confirm themes and that all necessary changes had been discussed.