The Importance of Written Feedback on the Individual and Team Performance of Student Pharmacists

INTRODUCTION

Teaching teamwork and team roles and responsibilities is essential to pharmacy curricula to prepare student pharmacists for future practice. A related skill, the ability to give and receive effective feedback through peer and self-evaluation, is an important aspect of working on a team. Tracking and evaluating team function, team satisfaction, student self-assessment, and peer-evaluation within a team can be a tedious task from an instructional standpoint.

The Comprehensive Assessment of Team-Member Effectiveness (CATME) SMARTER Teamwork system (https://info.catme.org) is an electronic, web-based system that gathers robust data regarding the overall team experience.¹ This assessment tool allows the instructors to collect data on the quality of how well the team is functioning and how well students self- and peer assess. The data collected in the CATME system is based primarily on a quantitative approach using Likert-style rating scales, with the option to provide open-ended self- and peer feedback, including discursive comments to peer team members. The system allows the instructor to hide peer comments (ie, make them visible only to instructors) or release the feedback (ie, make comments visible to student team members and instructors). Students can view feedback provided by peers with or without the author of the feedback being identified. This unique and robust assessment tool has been studied in non-medical, educational settings.^2,3

While studies on the impact of self- and peer feedback in pharmacy curricula have been published, there is a paucity of data assessing the impact of written peer and self-feedback in a team setting. One study published in the pharmacy literature discusses the impact of peer and self-assessment on small group work in a foundational pharmacy practice course.⁴ The study describes the effectiveness of peer and self-assessment towards performance during a group project. Students were asked to rate team member performance using a Likert scale at two time points during the semester (midpoint and final). Students were also given the opportunity to provide optional written comments to individuals or the group. The authors did not formally assess the written comments, but noted the written comments were complementary. The authors measured student perceptions about the utility of peer and self-assessment using an end of semester course evaluation. The results indicated that students felt peer feedback was important to professional growth and development (64% agreement) and that constructive feedback aided in identification of individual strengths and weaknesses (80% agreement) and assisted with development of skills for working in a group (68% agreement).

The academic medical literature includes a manuscript describing student perceptions of peer assessment in an interprofessional setting.⁵ The population of the study included health professional students from seven different fields (including pharmacy). After an interprofessional activity, students were asked to rate the feedback received from team members on a five-point Likert scale for usefulness and positivity. Results showed that students felt the peer feedback was useful (means across professions ranged from 3.84 to 4.27) and positive (means ranged from 4.17 to 4.86).

Exposing student pharmacists to teamwork and evaluation of performance using an assessment tool early in the Doctor of Pharmacy (PharmD) curriculum has the potential to enhance their future competency in the team setting. This study explored the impact of written feedback on individual and team performance using an online, peer-to-peer evaluation system (CATME) in a course that used a team-based, project-based learning model.

METHODS

This mixed-method study took place in Foundations of Pharmacy Practice, a two-semester course required of all first-year student pharmacists at the Wilkes University Nesbitt School of Pharmacy. This course provided students with the foundational knowledge, skills, and attitudes needed to enter pharmacy practice. A key component of the course was a project-based assignment in which students worked collaboratively to solve healthcare-related problem. In teams of six, students researched the problem and then created and implemented a deliverable to solve, publicize, or further investigate the issue. The project was designed to foster teamwork, stimulate creative thinking, enhance written and verbal communication, and improve student pharmacists’ ability to give and receive feedback. The project was divided into three phases. In phase 1, student-teams defined the problem based on a literature review and produced a written report. In phase 2, students drafted a potential deliverable. Examples of potential deliverables for this project included public health initiatives, research, or presentations. For example, one team was given the topic of combatting substance use disorder. The group researched the topic and chose to educate the university’s public safety department about Naloxone use during an interactive session. Phase 3 included implementing the deliverable, writing a final report, and giving an in-class presentation. At each phase of the project, students were required to provide self- and peer evaluations for their teammates. In past years, instructors used non-standardized, paper peer-evaluation forms to monitor team function and progress. During the 2016-2017 academic year, the instructors adopted the CATME online system for peer and self-evaluations in place of previously used nonstandardized paper forms.

At the completion of each phase of the project, students completed peer and self-evaluations of their team members using the Likert-based peer assessment system within CATME. After completing the quantitative assessment within CATME, students were required to provide written self- and peer evaluation comments. Students had to complete the assessments within 48 hours of notification for phases 1 and 2 and within five days of notification for phase 3. After each CATME assessment was completed, the instructors reviewed the quantitative data as well as the peer comments. If a student’s written peer feedback was not adequate (eg, the evaluation was missing detail, corrective feedback, or positive comments) the instructors sent a standardized coaching email to the student, encouraging them to provide more thoughtful feedback in subsequent evaluations. After the evaluations were reviewed by the faculty member, they were released for students to view.

To assess the impact of giving and receiving written peer feedback on student pharmacists’ individual and team performance, the two sections of the course were handled differently. Students in both sections of the course used the CATME system to provide feedback to teammates using numerical ratings and written comments. The CATME data that included the quantitative rankings were made available to students in both sections; however, the anonymous peer-to-peer comments were only released to students in section A. Students were not informed that the feedback process was being handled differently between the two sections, although blinding of the process was not possible.

For the first part of the study, upon completion of the health care-related project, an electronic survey developed by the research team was sent out to all students enrolled in the course. Student pharmacists were asked to rate their agreement with survey items using a five-point scale ranging from strongly disagree to strongly agree. Data collected included students’ ratings of their performance, the performance of their teammates, and their receptiveness to receiving feedback. Quantitative data were analyzed using SPSS (SPSS Inc, IBM, Chicago, IL) to describe the sample and draw comparisons between the two groups, section A (peer comments included) vs section B (peer ratings only). Inferential statistics, specifically independent t tests, were used to determine whether there was a significant difference between the means of the two groups. The survey asked questions related to demographic and background items (Table 1) prior to the questions about their perceptions of receiving peer-to-peer feedback. Additionally, grades for section A and section B were compared.

In addition to the survey data, a series of focus groups were conducted. To create focus groups, each student on each six-member team was assigned a number from one to six. A random number generator was used to identify a student from each group whose assigned number was the same as the number randomly generated, and that student was invited to participate in the focus group. If the invited student was unable to participate, the process was repeated with each of the remaining students until four groups of four to six students each were created. Not all six-member teams were represented in each focus group because not all students agreed to participate.

In addition, participants in section A who had received the discursive comments, described the impact of receiving the peer-to-peer comments and their perceptions of these comments on the effectiveness of the group. They also shared if, how, and to what extent the individual peer-to-peer feedback was different in nature and how it impacted and influenced them compared to the numerical ratings they received. Finally, participants from both class sections provided recommendations that they thought might improve the feedback process or improve group effectiveness. Through focus groups, participants shared the experience of rating their teammates, described the experience of providing individual written feedback to each of their teammates, described the experience of receiving feedback and how they responded to both constructive and positive feedback, and discussed the influence of the feedback they received on their perception of the effectiveness of the group.

RESULTS

The sample consisted of 61 participants, with equivalent demographic and background characteristics (Table 1). Of the participants, 47.5% were enrolled in section A (comments included with survey ratings) (n=29) and 52.5% in section B (survey ratings only) (n=32) section of this course. For 14 of the 15 items, the mean scores were higher for section A than section B, indicating generally higher agreement with and affirmation of positive statements related to giving and receiving peer-to-peer feedback. The survey results can be found in Table 2. The quantitative analysis included results from an electronic survey about giving and receiving feedback. For three of the 15 questions, there was a significant difference in responses between students in the comments included section (n=29) and those in the ratings only (n=32) section of the course. The three items for which responses were significantly different were “the feedback helped me to better understand my performance on the team,” “the feedback received by the team improved group/team performance,” and “this experience has prepared me to provide feedback to others in the future.” A review of the significant survey items is provided in Table 3. The distribution of the respondent’s ratings of agreement with the electronic survey items are presented in Table 4. Additionally, an analysis of project-related grades found no difference between the groups.

The overall theme of the qualitative data indicated that students perceived that giving and receiving feedback on self- and team performance improved functioning and that feedback was especially affirming and valuable when accompanying comments were shared.

To begin the focus group, students were asked about the experience of providing quantitative and qualitative feedback to their teammates. Students in the focus groups reported that the peer-to-peer comments were more involved than the numerical ratings and that giving negative or constructive feedback was more difficult than composing positive comments. Despite the additional time, the peer-to-peer comments were preferred by many students. The students who received the peer-to-peer comments had an overall positive perception of the survey and indicated that the feedback was well-received and thought to be accurate and fair. Conversely, students who were not able to view the peer-to-peer comments described a general disinterest in, and even disregard for, using CATME. The specific comments made by the students in the focus groups about their perception of using CATME are provided in Table 5.

A similar phenomenon occurred relative to the perceived influence of the feedback on group effectiveness. Teams that received peer-to-peer comments reported a positive effect. Teams that did not receive peer-to-peer comments indicated that the feedback had little or no impact, and in some cases, a negative impact on perceived group effectiveness. The comments made by the students in the focus groups regarding perceived influence of the feedback are presented in Table 6.

The focus groups concluded by asking the participants for recommendations that they thought could improve the feedback process and improve team effectiveness. Suggestions included, “Having a team meeting and talk about going forward as a team and how you could all improve together.” The final, and nearly unanimous, recommendation was to release the peer-to-peer comments to everyone.

DISCUSSION

This study aimed to analyze the usefulness of an online peer and self-evaluation system (CATME) for individual and team performance and the impact of peer-to-peer written feedback for those who received it. The type of feedback received differed between the two groups of participants, with one group receiving CATME ratings and written comments and the other group receiving CATME ratings only. The results of this study indicate positive student perceptions about using the CATME system. Data revealed that the feedback had a more significant and meaningful impact when both written comments and ratings were shared with individuals.

Three items on the electronic survey revealed a significant difference between the groups, indicating that feedback helped students to understand individual team performance, improve group/team performance, and be prepared to provide feedback to others in the future. Students in group A (comments included) perceived the experience more positively than students in group B (numerical ratings only). For 14 of the 15 items on the survey, the mean scores of students in section A were slightly higher than those of students in section B, indicating generally higher agreement with and affirmation of positive statements related to giving and receiving peer-to-peer feedback. For the 15 questions on the electronic survey, the means were low, indicating a general lack of comfort with giving feedback, which may be consistent with the general inexperience of first-year students.

The focus group participants had different perceptions about the usefulness and effectiveness of the CATME peer-to-peer rating system on their individual performance and that of their teammates, with the group that received ratings and comments having had a much more positive experience than the group that only received ratings. An overwhelmingly majority of students from both groups recommended the use of both ratings and comments in the future.

Limitations of this study relate to the setting and sample. The findings of this study may not be generalizable to other schools because of the unique nature of the project. Additionally, the sample was relatively small and came from just one school of pharmacy. In addition, no psychometric testing was done on the survey instrument used to collect student perceptions of the CATME process. Another limitation is that data were self-reported by students enrolled in a class without blinding. While anonymity and confidentiality were ensured, the lack of blinding may have influenced the accuracy of students’ self-reported data. The lack of blinding could have also impacted the study results. While students were not informed that the sections were being handled differently, there was no way to account for the potential discussion about the written feedback among students in the different sections. Future studies with a similar focus could address these limitations to improve the generalizability of the quantitative data. A larger sample that included students from diverse institutions would improve the generalizability of the quantitative results. Additionally, the dynamics of the team-based project itself may have been a confounder in terms of how students viewed the feedback related to the assignment.

The results of this study are similar to those in a study published by Popovich⁴ where students felt peer assessment helped them to better understand individual team performance. The results of our study also were consistent with the conclusions of a study by Van Schaik⁵ in which students in an interprofessional setting found peer feedback to be useful and positive.

CONCLUSION

This mixed-methods study investigated student perceptions of giving and receiving feedback in a team setting. The results indicate positive student perceptions of receiving written feedback via the CATME system as it relates to performance on a team. Students indicated that written comments and ratings had a more significant and meaningful impact when both were shared with individuals. The findings of this study support the use of an online peer and self-evaluation system for student pharmacists working collaboratively on a group assignment to solve an authentic health care problem. The process of giving and receiving written feedback in the team setting in the classroom has the potential to enhance future teamwork in pharmacy practice.