Abstract
Objective. To determine the appropriateness and feasibility of implementing the Individual Teamwork Observation and Feedback Tool (iTOFT) in advanced pharmacy practice experiences (APPEs) to allow direct observation and rating of students’ interprofessional teamwork skills.
Methods. In the academic year 2018-2019, preceptors administered iTOFT at least once during required acute care and ambulatory care APPEs. After administration, which involved direct observation of students’ behavior on an interprofessional team, preceptors evaluated iTOFT results and offered feedback to students. To determine implementation success of the iTOFT activity, an implementation satisfaction survey was administered to preceptors and students, which included questions on appropriateness and feasibility.
Results. The iTOFT activity was completed 149 times each during required acute care and ambulatory care APPEs. Results demonstrated positive teamwork behaviors, eg, no students received an “inappropriate” rating on any iTOFT item. The implementation survey response rate was 33% (34/102) for preceptors and 40% (50/125) for students. Both groups reported that the iTOFT activity was feasible. Respondents also reported that completing the iTOFT activity did not interfere with workflow and that it was not difficult to find opportunities to complete it. Per preceptor report, mean time to complete the iTOFT activity was 37.7 minutes, but there were concerns identified regarding fidelity with all iTOFT steps.
Conclusion. Incorporation of the iTOFT activity into the required acute and ambulatory care APPEs at a school of pharmacy was feasible. Future directions include determining ways to increase iTOFT fidelity.
INTRODUCTION
Improvements in interprofessional practice can decrease patient mortality,1,2 rehospitalization,3 and health care costs4 and increase patient satisfaction.5 Therefore, it is important for Doctor of Pharmacy (PharmD) students to learn interprofessional teamwork and communication prior to graduation. This is reflected in the Accreditation Council for Pharmacy Education (ACPE) Standard 11 regarding interprofessional education (IPE).6 By graduation, pharmacy students need to be able to contribute to an interprofessional team.
To ensure that all PharmD graduates are “team ready,” a direct assessment of interprofessional teamwork and communication in a real health care setting is ideal.7 The Individual Teamwork Observation and Feedback Tool (iTOFT) is a validated tool that is designed to assess an individual’s performance on a team rather than the entire team’s performance; uses a consistent evaluation scale; and is relatively short (11 items).8 ⇓-10 Other available validated tools measure student perceptions of interprofessional teamwork, evaluate the performance of a team, or are much longer and use a different scale to assess each skill.10
Given the wide range of practice settings and health systems where pharmacy students complete APPEs, it is important for schools to ensure that these experiential sites are able to find student opportunities for interprofessional teamwork and that preceptors find the activity acceptable. Implementation research determines the success of the implementation process and can offer an experiential education office a framework to evaluate the extent students and preceptors are able to complete required activities in a satisfactory manner.11 By evaluating implementation success, the outcomes of a new activity are measured; and if the outcomes are not as expected, further investigation can determine whether the intervention or activity is ineffective or the implementation of it was poor.12 Common implementation outcomes include adoption, acceptability, feasibility, fidelity, and sustainability.13,14 Results from implementation research can be used to learn how to better introduce and sustain new activities across multiple practice sites.
A purposeful evaluation of students’ interprofessional teamwork was added to the University of Wisconsin-Madison School of Pharmacy advanced pharmacy practice experience (APPE) curriculum for required acute and ambulatory care experiences during the 2018-2019 experiential year. The iTOFT was adopted for preceptors to use in evaluating and offering feedback to APPE students following a direct observation of their performance on an interprofessional team. As this assessment was a new requirement across a variety of APPE settings, and individual sites and preceptors were tasked with incorporating the iTOFT evaluation into the APPE, an assessment of implementation was conducted. The objective of this evaluation was to determine the appropriateness and feasibility of implementing the iTOFT in APPEs to allow direct observation and rating of students’ interprofessional teamwork skills.
METHODS
During the 2018-2019 experiential year, the iTOFT was implemented at the University of Wisconsin-Madison School of Pharmacy’s experiential sites for preceptors to document APPE students’ interprofessional teamwork behaviors and give them formative feedback. The iTOFT offers a validated basic and advanced version.8,9 Following a pilot of the tool with five purposefully selected practice sites and thorough discussion with preceptors, the 11-item basic version of the tool was selected to use. Following a request from an APPE coordinator, the “discusses safety issues” item from the advanced tool was added to the assessment. Thus, the version of iTOFT that was administered in this evaluation included 12 items (Table 1). The evaluation for each skill had four assessment options: not applicable, inappropriate, appropriate, and responsive. Appropriate indicates, “The student is engaged with the team in the activity. However, does not take the opportunity to further develop teamwork [behaviors].”9 Responsive indicates, “The student is actively engaged with the team in the activity and demonstrates commitment in learning about treamwork.”9 A tool to document the iTOFT evaluation was built by the school’s Instructional Information Technology Department and integrated into the already established online clerkship database. In addition to collecting the items on the iTOFT tool, the other members of the health care team, which students interacted with during the observation, and the mode of team interaction (ie, face to face, telemedicine, or telephone) were collected. Other health care providers were designated as practitioners if they held an entry-level practice degree or as students if they did not hold an entry-level practice degree.
Preceptor iTOFT Evaluations of APPE Student Interprofessional Teamwork (n=149)
Observation of student interprofessional communication and teamwork with feedback and online documentation of the iTOFT was required once during both the acute care and ambulatory care required APPEs. Preceptors could complete the tool multiple times but documented it once during the rotation. Completion of the activity was required, but the evaluation by the preceptor did not contribute to the students’ final letter grade. However, preceptors were encouraged to consider the students’ performance on the iTOFT when completing an IPE item on the final student performance evaluation. A 23-minute online presentation explaining the tool and expectations was available for both students and preceptors to review the purpose, expectation, and items on the tool. The iTOFT activity was also introduced at the 2018 Annual Clinical Instructor meeting and described in the General Experiential Manual and course orientations.
To assess launch of the iTOFT activity, the implementation outcomes of appropriateness and feasibility were adopted from the Conceptual Model of Implementation Research.11,13 Appropriateness was operationalized as satisfaction with the iTOFT activity and feasibility was operationalized as fit or suitability of the iTOFT activity in practice. This was measured as an implementation satisfaction survey to preceptors and students (items assessed in Table 2). The surveys contained seven and six questions for preceptors and students, respectively, to assess the implementation outcomes using a four-point response scale ranging from “do not agree” to “completely agree.” The questions were adopted from an implementation research question bank and previous implementation research.15 Students and preceptors were also asked two additional feasibility questions regarding which steps of the iTOFT activity they completed and how many minutes the activity took to complete. Students and preceptors who completed the iTOFT evaluation by the third rotation after the introduction of the activity were invited to complete the implementation satisfaction survey. The survey was administered online using Qualtrics (Qualtrics Labs, Inc).
Satisfaction Survey Results Regarding Implementation of the Individual Teamwork Observation and Feedback Tool
To triangulate the iTOFT scores and demonstrate improvement in student IPE performance, the revised Interprofessional Collaborative Competency Attainment Scale (ICCAS-R) was administered to students on required acute and ambulatory care APPEs. The ICCAS-R is a validated post-retrospective self-evaluation of interprofessional skills.16,17 It uses the following scale: 1= poor; 2=fair; 3=good; 4=very good; 5=excellent. The ICCAS-R survey was administered to students online via Qualtrics during the last week of the APPE .
A single reminder email was sent to students three to seven days following the initial invitation to complete the survey. At the end of the academic year, a report was generated using all iTOFT evaluation results. Descriptive statistics were used to characterize the results. Reliability tests and paired t tests were performed on ICCAS-R for acute and ambulatory care settings separately. This way, ICCAS-R student self-evaluation of IPE performance prior to rotation was compared to end of rotation. An alpha level of .05 was set to determine statistical significance. This evaluation was certified as a quality assurance project by the University of Wisconsin-Madison Education and Social/Behavioral Science Institutional Review Board.
RESULTS
There were 149 iTOFT evaluations completed in both the acute care and ambulatory care rotations (100% of the graduating class). On the acute care rotation, 125 (84%) of the observations were made face to face, 13 (9%) were made over the telephone, and 11 were made under other circumstances or left unanswered (7%). On the ambulatory care rotation, 90 (60%) of the observations were made face to face, 39 (26%) were made over the telephone, and 20 were made under other circumstances or left unanswered (13%). Most observed interactions involved a prescriber and a health care provider holding at least an entry-level degree.
Students were rated well and no major deficiencies were noted on the iTOFT evaluation (Table 1). No students received an inappropriate rating on any iTOFT item. Students were often rated as responsive (range 54%-86% in acute care and 46-79% in ambulatory care). In general, preceptors did not use the “not applicable” rating when completing the iTOFT evaluations.
Of the 102 preceptors who submitted an iTOFT evaluation, 34 (33%) completed the implementation satisfaction survey (20 acute care, 10 ambulatory care, four community practice preceptors). All preceptors were volunteer preceptors, which is the case for the majority of UW-Madison SOP acute and ambulatory care APPEs. Of the 125 students who had completed an iTOFT evaluation at the time of the survey, 50 students (40%) responded. Of those who completed the survey question, 75% of preceptors and 52% of students reported viewing the iTOFT training. Results of the student and preceptor implementation surveys had similar distributions (Table 2). There were mixed results for the appropriateness implementation outcome, but both students and preceptors reported the iTOFT activity was feasible. Respondents reported completing the iTOFT activity did not interfere with workflow, and it was not difficult to find an opportunity to complete the activity. Per preceptor report, mean time to complete the iTOFT activity was 37.7 minutes. In the acute care setting, the mean time was 50 minutes, and in the ambulatory care setting, the mean time was 20 minutes.
While not measured a priori, lower than expected fidelity to the steps of the iTOFT activity was observed (Table 3). While initially intended to be a feasibility measurement, we noted that some required steps were not completed and thus concern was raised that the activity was not being completed as intended.13 Of particular concern were the areas of direct observation and feedback within the iTOFT activity where 20% of preceptors and 40% of students did not report the preceptor completing direct observation, and 30% of preceptors and 40% of students did not report giving or receiving feedback.
Steps Completed Using the Individual Teamwork Observation and Feedback Tool
Response rates for the ICCAS-R included 121 students (81%) and 130 students (87%), in the acute care and ambulatory care settings, respectively. When comparing pre-rotation to end of rotation mean scores for all 20 ICCAS-R items, we found that students in the acute care setting self-described a significant mean increase in IPE performance scores from 3.3 (SD=0.67) at baseline to 4.1 (SD=0.57; p<.001) at the end of the rotation. Results were similar in the ambulatory care APPE, with a baseline mean score of 3.2 (SD=0.66) and an end of rotation mean score of 4.1 (SD=0.62; p<.001). There was also a significant increase in all ICCAS-R items for both the acute care and ambulatory care APPEs when assessed individually. At the end of the acute care APPE, 58% of students described their overall ability to collaborate interprofessionally as “much better” and 35.5% described it as “somewhat better.” This was lower in the ambulatory care APPE, with 48.2% selecting “much better” and 40.7% selecting “somewhat better.” The baseline ICCAS-R had high internal consistency with a Cronbach alpha of 0.96 in the acute care APPE and 0.97 for the ambulatory care APPE.
DISCUSSION
The inclusion of the iTOFT activity in both required acute care and ambulatory care APPEs appears feasible. All students and preceptors on acute care and ambulatory care submitted the iTOFT activity, suggesting the activity is feasible to incorporate and use in clinical settings.8,10 Both preceptors and students reported that there were opportunities in practice to complete interprofessional interactions and the activity did not interfere with workflow. While the length of time to complete iTOFT activity in the acute care setting may seem high, this often included the time to observe the student on rounds and was acceptable to the preceptors completing the survey. This is consistent with the work of Crowl and colleagues who introduced the advanced version of the iTOFT to pharmacy preceptors.18 After they received training and piloted the iTOFT, the preceptors reported that it was easy to use, relevant to practice, and took a reasonable amount of time to complete.18
It was not surprising to find that the introduction of the iTOFT activity did not change precepting behaviors or improve student learning as the goal of the activity was to document student interprofessional teamwork behaviors. This was inconsistent with the findings by Crowl and colleagues where preceptors reported the introduction of interprofessional assessment tools increased the frequency of feedback they shared, which then improved student interprofessional behaviors.18 This may be because preceptors in that study used the interprofessional assessments several times over the course of an APPE, whereas the UW-Madison School of Pharmacy only required the iTOFT activity be documented once during the required acute and ambulatory care rotations. While the implementation satisfaction survey suggests some preceptors were not satisfied with the additional assessment activity, the iTOFT activity can document student-readiness for practice, thus meeting the needs of the pharmacy school. Anecdotally, the preceptors who were less satisfied with the additional assessment tended to be those who had already incorporated IPE activities into their rotation.
An open-ended question was included on the implementation satisfaction survey and may elucidate why some preceptors appreciated the addition of the iTOFT activity. Some preceptors described increased emphasis on the value of interprofessional collaboration and providing structure to their student feedback on interprofessional teamwork. One preceptor wrote, “Students were completing interprofessional activities prior to the iTOFT, but now this gives a formal structure for feedback.” This was consistent with the findings of Crowl and colleagues, who also described pharmacy preceptors reporting an increase in the structure of their feedback when they started using validated interprofessional assessment tools.18
There are several limitations to this evaluation. This evaluation was observational and only assessed after the iTOFT activity had started. While the implementation satisfaction survey was reviewed for understanding by the investigators, it was not a validated survey. Finally, while there was a high rate of completion for the iTOFT activity and the ICCAS-R, there was a low preceptor and student response rate to the implementation satisfaction survey portion of the evaluation. This may have been because of the lack of requirement or incentive associated with completion of this survey.
The low fidelity to some iTOFT activity steps that both students and preceptors reported as not completed was concerning. Most notably a specific direct observation or discussion of the iTOFT results had a low rate of reported completion. The low rate of training completion may have contributed to lower than anticipated fidelity, especially for students, and the training was made a required part of APPE orientation in the subsequent year. Fidelity is also an ongoing future direction to identify why steps are being omitted and how a school can better promote the activity or make it easier to complete as intended.
CONCLUSION
Incorporation of the iTOFT activity into required acute and ambulatory care APPEs was feasible and achieved the programmatic goal of documenting the pre-graduation readiness of students to perform on interprofessional teams. Future directions include determining ways to increase the fidelity of the program.
ACKNOWLEDGMENTS
The authors acknowledge and thank Kate Hartkopf, PharmD, for her review of the assessment plan; Mike Pitterle, MS, RPh, and Kathy Chylla, BS, for building the electronic iTOFT tool; and Robert Breslow, RPh, and Denise Pigarelli, PharmD, for their support in the implementation of the iTOFT activity.
- Received February 6, 2021.
- Accepted June 28, 2021.
- © 2022 American Association of Colleges of Pharmacy
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.