Pharmacy Students’ Standardized Self-Assessment of Interprofessional Skills During an Objective Structured Clinical Examination

INTRODUCTION

Interprofessional collaborative practice occurs when multiple health workers from different professional backgrounds provide comprehensive services to deliver care across different settings.¹ Graduates of health and non-health professional programs are increasingly required to work collaboratively as part of interprofessional health care teams. As such, programs must train and assess their students’ competence in the areas of interprofessional team dynamics, communication, and education.²

The objective structured clinical examination (OSCE) has been used widely in healthcare education as it provides a structured, observed assessment of student performance.^3,4 More recently, interprofessional OSCEs have been used to develop critical skills necessary for successful interprofessional collaboration including abilities outlined in the Interprofessional Education Collaborative (IPEC) Core Competencies.^4,5 However, assessment of interprofessional skills, even during interprofessional or team OSCEs, is often accomplished via student self-report rather than independent observation and evaluation.^6-8 Additionally, most evaluations include measures that address student attitudes and perceptions, without an assessment of actual interprofessional skills and behaviors.⁶

The purpose of this study was to assess a newly developed OSCE that included three cases designed to evaluate pharmacy students’ interprofessional collaborative practice skills while interacting with a simulated health care professional. The OSCE occurred during the final didactic semester, prior to students beginning advanced pharmacy practice experiences (APPEs). Throughout the previous three professional years, students participated in various interprofessional experiences. These experiences included collaboration with other students and were developed to help them understand the roles of other professionals and the importance of teamwork and communication for improving health outcomes. Although students did receive some feedback regarding their interprofessional interactions during these experiences, no formal grade was assigned. The assessment measures used in these previous experiences have examined reaction to the experience and/or change in perceptions or attitudes.^9-11 This study contributes to the literature by examining a method to measure students’ interprofessional abilities by including both self-assessment and evaluator data in the analyses. These data are important to understanding the pharmacy students’ ability to participate in interprofessional collaborative practice prior to entering APPEs.

METHODS

This study was conducted during the spring 2018 semester and was determined to be exempt by the University at Buffalo Institutional Review Board. In the semester prior to the APPE year, students were enrolled in a capstone assessment course during which they completed the Pharmacy Curriculum Outcomes Assessment (PCOA) and 10 required cases (two practice and eight graded) as part of a capstone OSCE.¹² Students were expected to provide objective evidence that they were able to successfully implement the Pharmacists’ Patient Care Process (collect, assess, plan, implement, follow up) in simulated activities while interacting with patients and health care professionals.¹³ The OSCE was conducted across three class sessions during the spring semester. During the first session, students completed two practice cases where they self-evaluated their performance and receive feedback, but no grades were provided. During the second and third session the students completed eight graded cases. Students had 10 minutes to complete each case. After completing a case, the standardized actor evaluated the student using a tool which include a five-point Global Rating Scale (GRS).^14,15

Teams of pharmacy faculty members developed cases for the OSCE. A nursing faculty member was part of the team that developed the cases evaluating skills needed for effective interprofessional collaborative practice (cases 1-3). Cases 1 and 2 were similarly structured, in that the standardized actor assumed the role of a prescriber. These two cases were designed to evaluate student skills related to interprofessional collaboration in the context of an interaction with a prescriber seeking the pharmacist’s input regarding pharmacotherapy decisions. In case 3, which was also designed to evaluate student skills related to interprofessional collaboration, the standardized actor assumed the role of a nurse. This case involved a nurse seeking to confirm compatibility information for an intravenous antibiotic; however, the information the nurse had conflicted with the information found in tertiary drug references. Therefore, the student was required to both make an accurate recommendation and navigate a potentially confrontational situation. The remaining seven cases that made up the OSCE involved pharmacist-patient interactions and therefore did not specifically evaluate skills needed for effective interprofessional collaborative practice.

Three sets of data were collected and analyzed: student responses to seven questions assessing the quality of the OSCE cases, results from the Interprofessional Collaborative Competency Attainment Survey (ICCAS), and student performance on each of the three cases designed to evaluate skills needed for effective interprofessional collaborative practice as measured by the GRS.^10,14,15 All statistical analyses were conducted using SPSS Statistics 24 (IBM Corp., Armonk, NY).¹⁷ When tests for significance were conducted, p values less than .05 were considered significant.

To assesses the quality of the OSCE, students were asked a series of questions as part of the survey after completing the cases. The items used the same seven-point Likert-type scale as used in the ICCAS. Data were presented as means and standard deviations.

The ICCAS is an instrument developed to self-assess interprofessional collaborative behaviors based on established interprofessional collaboration competencies.^10,16 The reliability and validity of the instrument has been examined with participants from a variety of health profession programs with evidence in support of using the instrument for measuring self-report retrospective pre- and post- interprofessional education intervention competency attainment.^10,16 The ICCAS consists of 20 items using a seven-point Likert-type scale (1=strongly disagree, 2=moderately disagree, 3=slightly disagree, 4=neutral, 5=slightly agree, 6=moderately agree, 7=strongly agree), with research suggesting that in addition to examining the individual items, computing an average overall score for individuals is justifiable.^10,16 The reliability of the pre- and post-OSCE scores on the ICCAS were estimated using the Cronbach alpha. In addition, each item was examined by estimating the reliability of the scale score if the item was removed from the scale. The pre- and post-OSCE ICCAS score for each student was created by calculating the mean score for the 20 items on the scale. Use of the mean of the item responses, as opposed to the summated score, makes the scores easier to interpret. The mean and standard deviation (SD) for each item were presented, along with the pre- and post-OSCE ICCAS scores, and difference scores for the ICCAS scale. In order to examine differences between pre and post, paired sample t tests were conducted; p values <.05 were considered significant. In addition, the Cohen’s d effect size was computed for each comparison, which indicates the number of standard deviations by which the pre- and post-examination differ. Cohen’s d values less than 0.2 were considered very small and meaningless, greater than 0.2 and up to 0.5 were considered small but meaningful, greater than 0.5 and up to 0.8 were considered medium, and greater than 0.8 were considered large.¹⁸

The GRS has been used to assess pharmacist interactions and uses a 5-point Likert-type scale and assesses performance in four process-related domains: verbal communication, non-verbal communication, ability to respond to feelings and needs, and degree of focus, logic, and coherence.^14,15 While the GRS assesses interaction and has been used to examine interprofessional interactions, it does not differentiate between interprofessional communication and general communication skills.¹⁴ The instrument has demonstrated acceptable psychometric properties.¹⁵ The reliability of the items on the GRS were estimated using Cronbach α, the most common measure of internal consistency. Ideally, Cronbach α reliabilities of 0.90 or above are desired but the Cronbach α should not drop below 0.70.¹⁹ In addition, each item was examined by estimating the reliability of the scale score if the item was removed from the scale. The GRS score for each case was created by calculating the mean score for the five items on the scale. Use of the mean of the item responses, as opposed to the summated score, makes the scores easier to interpret. The means and standard deviations were presented for each of the three cases.

Finally, Pearson correlations were conducted to examine the relationships between student performance as measured by the GRS and ICCAS pre- and post-OSCE scores. Both the correlation coefficient (r) and the significance level were reported. The correlation coefficients provide information on the magnitude of the effect or effect size. Correlation coefficients less than 0.30 are considered small, between 0.30 and 0.50 moderate, and greater than 0.50 strong.¹⁸

RESULTS

There were 127 students enrolled in the capstone assessment course, and 124 (97.6%) consented to be part of the study. Two students had missing responses on the pre-OSCE ICCAS. Therefore, the total valid sample size for the ICCAS was 122, while the sample size for the other variables was 124.

Students agreed with most of the quality assessment items as demonstrated by the means of their responses: “The OSCE scenarios today were realistic” (M=5.8, SD=1.1); “The OSCE measured application of skills and abilities required in pharmacy practice” (M=5.8, SD=1.1); “The level of difficulty of the OSCE cases today were appropriate” (M=5.3, SD=1.5); “The topics covered in the session today were relevant to me” (M=5.6, SD=1.3); “The OSCE is a good way to learn about interprofessional case management” (M=5.8, SD=1.2); “The OSCE cases today were a fair measure of my communication skills” (M=5.7, SD=1.3); and, “I prefer the OSCE over examinations” (M=4.9, SD=2.0).

The results of the ICCAS are displayed in Table 1. The Cronbach α for the ICCAS was 0.96 pre-OSCE and 0.94 post-OSCE, indicating high reliability. The corrected item-total correlation, which denotes the correlation between each item and the total reliability score, was acceptable for all items. Therefore, no items were removed from the scale. For each item on the ICCAS there was a significant increase in score from pre- to post-OSCE, with the effect size ranging from medium to large. The average total ICCAS scale score prior to the OSCE was 5.1 (SD=0.8) and after the OSCE was 5.9 (SD=0.6); this difference was significant and resulted in a large effect (Cohen’s d=1.2).

The Cronbach alpha for the GRS scores was 0.70 for case 1, 0.90 for case 2, 0.85 for case 3, and 0.74 for the three cases combined. All were 0.70 or above, indicating high reliabilities. The corrected item-total correlation, which denotes the correlation between each item and the total reliability score was acceptable for all items. Therefore, no items were removed from the scale. Table 2 presents the student performance on each of the interprofessional cases and the overall score for all cases combined, as measured by the GRS. In general, the GRS item means for case 1 were lower than the means for case 2 and case 3. The GRS item means of case 2 and case 3 were very similar. Most of the GRS item means were above 4.0 with the exception of the item, “Degree of focus, logic, and coherence,” for case 1.

Table 3 displays the bivariate correlations between the ICCAS pre- and post-OSCE scores and student performance on each case as measured by the GRS. The ICCAS pre-OSCE score presents a strong and positive correlation with the ICCAS post-OSCE score (r=.63, p<.01), and a strongly negative correlation with the ICCAS change score (r=-.68, p<.01). For the OSCE cases, the GRS scores for case 1 and case 2 were significantly correlated with a weak strength (r=.26, p<.05). Significant correlations were also found between the ICCAS post score and the OSCE GRS score for case 2 (r=s.22, p<.05), as well as between the ICCAS post-OSCE score and the OSCE all cases (r=.17, p<.05).

DISCUSSION

After completing an OSCE that included three cases evaluating skills needed for effective interprofessional collaborative practice, students self-reported competence toward interprofessional collaborative practice increased. Students generally performed well on the interprofessional cases, as indicated by GRS item and total means greater than four on a five-point scale. Additionally, students agreed to the quality assessment items reflecting that the cases included in the OSCE were of good quality.

A goal of OSCEs is to provide a reasonably accurate real-world simulation of situations that students may face in order to understand how they will perform in similar circumstances. While faculty members attempt to design the cases to evaluate important skills and to be as true-to-life as possible, student perception and “buy-in” is critical. To assess students’ perception of the OSCE, students provided feedback regarding the quality of the OSCE. All items except, “I prefer the OSCE over examinations,” had higher than 70% positive agreement. Students generally agreed that the cases designed to evaluate skills related to interprofessional collaborative practice were realistic, that they measured application of skills and abilities required in pharmacy practice, and that the topics covered were relevant. These items are designed to investigate the OSCEs ability to achieve the goal of having a high-fidelity simulation, which matches well with practice expectations. Students also generally agreed that OSCEs are a good way to learn about interprofessional case management, that they serve as a fair measure of communication skills, and that the difficulty of the OSCE cases were appropriate. These items are designed to evaluate the OSCE as an educational and performance assessment tool.

The ICCAS is a validated instrument that is reliable in predicting meaningful outcomes with regard to attitudes toward interprofessional competency attainment.^10,16 While many IPE assessment tools exist, the ICCAS tool was chosen for this study because it is a recommended tool for student self-assessment of attainment of IPE competencies.⁸ Student self-assessment is a common method of evaluation within IPE literature, and inclusion of this method of evaluation in the present study allows for our data to be compared to that of other studies. Additionally, self-awareness is a key element of students personal and professional development.² By incorporating both student self-assessment and evaluator assessment of student performance into the OSCE, students have the opportunity to practice self-assessment, improve self-awareness of the interprofessional competencies, identify areas for improvement and, understand how self-assessment compares to an evaluator’s assessment of their own performance.^2,8 The instrument can be analyzed by each individual item as well as by the cumulative total score to determine the impact the interprofessional activity had on students’ self-assessed behaviors.^10,16 In our study, scores for each item and the overall total score on the ICCAS instrument increased from pre to post. The magnitude of effect for the overall score was large, and ranged from a medium to large effect for each of the individual items. These results indicate that the cases designed to evaluate skills related to interprofessional collaborative practice were able to positively affect student self-reported competence toward interprofessional collaborative practice. While these were significant changes, the educational significance of these findings should be considered. Prior to taking the OSCE, students overall slightly agreed that they were able to demonstrate the interprofessional collaborative behaviors. Large increases in self-reported competence were unlikely, given the already positive self-assessment students made prior to the OSCE. After completing the OSCE, students were closer to moderate agreement that they were able to demonstrate the interprofessional collaborative behaviors.

The effect size for most of the individual items on the ICCAS were large. Five of the items demonstrated a moderate effect size. This change in effect size likely can be explained by the nature of the OSCE exercise and the interprofessional behavior that was being assessed by the ICCAS instrument. For example, “Actively listen to IP team members’ ideas and concerns” demonstrated a moderate effect size. During the OSCE exercise, individual students were not expected nor required to elicit the standardized individuals’ ideas related to the issues presented in the OSCE. Therefore, we did not expect a large change in this item. Likewise, “Include the patient/family in decision-making” was not a part of the OSCE and therefore a large effect size was not expected. Still, a medium effect size was achieved for items on the ICCAS instrument that were not directly embedded within the cases. This is likely a limitation of the retrospective pre-post design of the ICCAS instrument. The retrospective pre-post approach may introduce post-score inflation (ie, positive response bias) because of respondents’ desire to reflect positively on their own abilities and may help to explain the positive changes in the ICCAS items not directly embedded within the cases.²¹

Correlation was found between student performance as measured by the GRS on case 1 and case 2, but not case 3. Cases 1 and 2 were similarly structured (ie, pharmacist-prescriber interaction regarding pharmacotherapy decision), while case 3 involved compatibility information in a potentially confrontational conversation with a nurse. Throughout the required curriculum, students have opportunities to participate in interprofessional case-based learning and discussion, and complete a simulation activity with students from another health profession, yet these are not meant to invoke confrontation. Generally, students who performed well on case 1 performed similarly on case 2. While the performance of case 3 was not correlated with the other cases which may be attributed to this case assessing a different type of skill. These data highlight the importance of having a variety of scenarios and of not relying on student self-assessment alone to evaluate interprofessional collaboration skills as some skills may be difficult to measure in this way.

This study contributes to the literature as it provides information regarding an innovative way to measure students’ interprofessional abilities by including both self-assessment and evaluator data in the analyses. These data are important to understanding and assessing pharmacy students’ ability to participate in interprofessional collaborative practice prior to entering APPEs. Cullen and colleagues described the implementation of an interprofessional team objective structured clinical examination for medical and midwifery students.⁷ Student team performance was not formally assessed, nor were interprofessional attitudes or behaviors. Feedback regarding the experience was gathered from subjective student and teacher evaluations of the experience. Overall, students and teachers involved in the sessions felt that students learned best about other’s roles while working in small interprofessional teams.⁷ Our study was similar in that we included quality assessment items as a way to collect student feedback and understand the quality of the cases included in the OSCE. However, by including the ICCAS and the GRS, we attempted to capture the impact the cases evaluating skills needed for effective interprofessional collaborative practice had on student self-assessment of competence toward interprofessional collaborative practice and overall evaluator assessment of student performance.

Similarly, Symonds and colleagues describe a formative OSCE for medical and midwifery students that was designed to promote teamwork and enhance knowledge of other practitioners’ roles and responsibilities.⁸ After completing the OSCE student teams were given feedback by a facilitator, which included feedback on overall teamwork. At the end of the session students were asked to complete a questionnaire which asked them to rate their response to a series of questions on a five-point Likert scale (1=strongly disagree, 5=strongly agree). Two items related to interprofessionalism: “This session enhances my awareness of the knowledge of other professionals” (mean response∼4); and “This session helped promote a team attitude towards the management of labor situations” (mean response∼4). The authors concluded that the team OSCE supported positive attitudes toward team working and was an effective tool for interprofessional education in the maternity services.⁸ Of our seven quality assessment items, one item related directly to interprofessional collaboration: “The OSCE is a good way to learn about interprofessional case management.” Scores for this item were favorable (M=5.8, SD=1.2), indicating that students overall agreed with the statement.

This study has several strengths. This is the first study to report changes in students’ self-assessed competence toward interprofessional collaborative practice after completing three cases describe changes in pharmacy students’ self-assessed interprofessional collaborative behaviors after participation in three OSCE cases focused on interprofessional behaviors. The study employed data collection tools (eg, ICCAS and GRS) that had been validated.^10,14-16 While the cases designed to evaluate skills related to interprofessional collaborative practice focused on pharmacist interactions with either a physician or nurse, the cases covered a variety of different interactions which tested different skills. The cases were also designed so that they could be easily adapted for use by the other profession (eg, nursing faculty could use the cases with the pharmacist being the standardized professional). Lastly, the study involved nursing faculty in the creation of the OSCE cases, to ensure the simulated interactions were authentic.

There are several limitations to this study. First, the study was designed to include simulated interprofessional interactions rather than interactions with actual health professionals. Simulated health professionals are likely to act differently than other health professionals despite the preparation of scripts and the provision of training. For example, a true health professional would be more likely to pronounce medication and pharmacy terminology correctly. Additionally, a practicing physician served the role of a physician for approximately 12.5% of the students for one of the interprofessional cases. Second, students are aware that the cases were simulated rather than real. Students are instructed to treat the situations as real; however, one cannot be certain that they will approach these simulations the same as they would a real-life case. The individual cases have relatively high stakes, with each case contributing 10% to the student’s final grade in the course, so it is expected that students will put significant effort into each case. Nevertheless, students’ behavior may not have truly matched their “real world” behavior. This is especially true in situations where students were aware that they were being graded and videotaped.

An area for improvement within the cases was identified based on student feedback. Students were asked a question which was expected to necessitate the use of a particular tool in the provided electronic drug reference during case 3, which involved an interaction with a nurse. This was the first time that many students had used this tool within this drug reference. Students were expected to have experience with a similar tool in general, but may not have used the specific software provided during the OSCE. Students were given the opportunity to practice with the provided electronic drug reference prior to their first scheduled OSCE session; however, attendance at these practice sessions was low and students were not told what tools within the software they would need to use during the graded cases.

CONCLUSION

Within an OSCE, cases focused on skills important to effective interprofessional collaborative practice were effective at assessing students’ skills and positively impacting students’ self-assessment of interprofessional collaborative behaviors. Future investigations should examine the impact of these or similar cases in other health care professions, including nursing. Additionally, future research should examine if it is possible to effectively and accurately evaluate individual student performance on an interprofessional group OSCE. While this study contributes to the body of knowledge related to teaching and assessing interprofessional competencies, more work is needed to determine the best approach to measuring changes in interprofessional skills and behaviors in health professions students.