Peer- and Self-Grading Compared to Faculty Grading

INTRODUCTION

Performance assessments provided by supervisors or peers in conjunction with self-evaluations are an integral part of a pharmacist's professional development. Self-reflection is an essential part of this process, as it provides insight about how people view themselves and their behaviors relative to peers and supervisors. It also reveals self-assessed strengths and weaknesses, identifies the barriers that must be overcome to surmount limitations, and expresses future opportunities for growth.^1-3 Likewise, peer reflection demonstrates strong or poor performance relative to someone on the same level.⁴ Formal self-assessments in pharmacy school curricula, which is mandated by the Accreditation Council of Pharmacy Education, provides consistent and reliable assessments among faculty members and preceptors.⁵ Pharmacy educators are in a position to provide consistent and high-quality feedback so that students are able to develop more accurate self-assessments as part of their pharmacy education.⁶ Previous studies report inconsistent results regarding the value of self-grading and there are a limited number of reports about peer-grading. There are no reports regarding pharmacy alumni perspectives on the value of peer and self-assessment as part of the pharmacy curricula. The objective of the study was to evaluate the reliability and value of peer- and self-reported student evaluations in the grading of pharmacy students by assessing the differences in peer, self, and faculty grades in 2 courses offered at the Ernest Mario School of Pharmacy at Rutgers University.

METHODS

The reliability of peer- and self-evaluations were assessed by measuring the differences in grades assigned to students in faculty evaluations, student-peer evaluations, and self-evaluations for seminar presentation and advanced pharmacy practice experience (APPE) courses. The course objectives and study procedures were reviewed with faculty members and students at the beginning of each semester during the seminar presentations course and reinforced by the individual student's preassigned faculty mentor during each of the 5-week APPEs. Students were told that the self- and peer-evaluations would not be included in their grades and that faculty members would not see the peer- and self-evaluations prior to assigning student grades. Thus, both faculty members and students would be blinded until evaluations were completed.

The value of including peer- and self-evaluations in the school curriculum was evaluated by surveying alumni of Ernest Mario School of Pharmacy (EMSOP) about perceived and real benefits of self- and peer-evaluations in current professional practice and in the pharmacy school curriculum.

RESULTS

Two hundred thirty-four students and 31 faculty members participated in the 1-year grading project for the seminar and APPE courses. The majority of students were either Asian (43.2%) or white (38.9%), and there were more female than male student participants (68.1% and 31.9%, respectively). Of the 31 participating faculty members, 22 were full-time and 9 were adjunct, and most were white and female (90.3% and 61.3%, respectively). Thus, the faculty members and students were about equal with respect to ratio of gender, but the percent of Asian faculty members was less than that in the cohort student ratio. The mean duration of faculty member teaching experience at the time of the study was 5.3 ± 3.3 years. The mean grade point average (GPA) of the students was 3.1 ± 0.4 for undergraduate courses and 3.4 ± 0.4 for graduate level courses.

The faculty members assigned 217 seminar grades. The mean final seminar grades from peer evaluators (3.8 ± 0.2) were significantly higher than those assigned by both faculty members (3.7 ± 0.4) and student self-assessors (3.7 ± 0.5) (P<0.001) (Table 1). Student-peers consistently graded higher on all 4 evaluation form components (content, presentation style, audio-visual aids, and handout). Student self-evaluations were not significantly different from faculty assessments for the final grade or for any of the 4 components of the evaluation form. The faculty members assigned 176 APPE grades. Peer-evaluators graded student overall performance higher (3.9 ± 0.2) than did faculty evaluators (3.8 ± 0.3) and self-evaluators (3.7 ± 0.3) (Table 2). The differences in overall final APPE grades were significant among peer, self, and faculty evaluations (P < 0.001). Peer-evaluator grades for all 5 components (general objective/professional performance, assessment of general knowledge, clinical skills, communication skills, and formal presentations) were significantly higher than faculty-assigned grades. Students graded themselves lower overall and for formal presentations than did faculty members, but the differences between self-assessed and faculty-assigned grades were not significant for the other components of the evaluation form. Of the 2,357 EMSOP alumni contacted by means of e-mail, 107 (4.5%) responded within the 2-week period. Most respondents (80.4%) had not performed peer- or self-evaluations while attending the EMSOP program. However, in their current professional practice, 64% of the respondents indicated using peer evaluations at least once each year and, 85% performing a self-evaluation at least annually. Of the 19.6% of respondents who had participated in peer- and/or self-grading during pharmacy school, the majority (76%) thought it was a helpful experience.

Most respondents agreed that peer- and self-evaluations should be used for group projects and oral presentations in didactic courses and clerkships at colleges and schools of pharmacy (Table 3). They also agreed that peer- and self-evaluations should be used to assess students’ general knowledge as well as their interactions with patients or medical team members in APPEs. Most respondents felt that peer- and self- assessment should not be part of the overall grade for clerkships or seminars. A large number of respondents (14% to 30%) remained neutral about peer- and self-grading.

DISCUSSION

In this study, the use of peer- and self-evaluations in seminar and APPE courses offered insight into the grading trends of students compared with those of faculty members and preceptors. Students tended to grade their peers more generously while grading themselves the same as faculty members on the evaluation form components but lower than faculty members for their overall grade. Even though the differences among peer, self, and faculty grades in this study were significant, they may not be academically significant because differences of 0.1 to 0.4 were unlikely to result in a student receiving a different letter grade (Tables 1 and 2). As students receive additional training and gain experience, these differences may diminish, especially considering that most of the students in the study likely had little prior experience in peer and self-grading.

With respect to seminar and APPE grades, peer evaluators tended to be more generous in assigning high grades than faculty members and self-evaluators. These findings suggest that peer grading may be less useful in overall seminar and APPE grading because it may falsely inflate the true academic merit of a student's performance. The tendency toward leniency with peers may result from students assessing their peers’ effort rather than their performance⁴ or not wanting to offend their classmates. Additionally, because students spend more time together, they may have an opportunity to observe a different set of skills in their peers than do faculty members. Despite concerns students may have about incorporating peer assessments into their grades, written feedback or critiques by peers may be useful to students based on studies showing that peer-assessment is valued and accepted by students and residents.⁴ Peers also can provide useful comments regarding content and suggestions for future improvement and comment on other students’ contributions to group projects. It is difficult to measure the true difference between faculty members and student grading when the students know that their evaluation will not have an impact on the their final grades. More education is needed to help students learn how to evaluate their peers before including peer evaluations other than comments in final grades.

Agreement between self- and faculty assessments demonstrates students’ ability to successfully recognize their own strengths and weaknesses. One study suggests that self- assessment may actually be more consistent than perceived, because the self-assessment grades did not change much from the pilot study to the follow-up study, whereas faculty and peer grades decreased over that same timeframe.⁴ This is supported by Fjortoft who suggested that self-assessment skills develop early in life and remain stable thereafter.⁶ However, the self-assessment process should be validated, students should receive adequate training on how to use these instruments, and students should be allowed to practice self-assessment frequently. A student's self-assessment can be validated by videotaping interactions with peers, faculty members, patients, and various hospital staff members in a range of skill-based assignments, such as patient interviews, physical examinations, and educational presentations. After reviewing the video, the student may be better able to perform self-assessments.

Self-assessment is believed to be an important lifelong skill. There could be dangerous consequences for practitioners with flawed self-assessment skills, especially if they overestimate their own abilities. This is an issue warranting concern based on study findings that pharmacy students are not accurate self-graders.^1,13 Quality of self-assessment may be influenced by a student's academic ability, as students who are in the lowest GPA quartiles tend to overestimate their skills while those in higher quartiles are more likely to underestimate them.^1,14 Possible explanations for the small differences between faculty and student grading in the current study may be that most of the participating students had a high GPA (3.4 of possible 4.0), and 4 of the 31 faculty members gave all their students during the study year a 4.0, which may have increased the mean faculty GPA.

Student portfolios are not adequate for self-assessment because they usually are an unguided personal reflection of a student's practice experience that neither provides enough information to improve student performance¹³ nor validates the student's ability to self-assess. While portfolios are tools students can use to analyze their experiences, they usually cannot confirm a student's self-perception of professional development. Adding peer- and self-evaluations of performance throughout the curriculum may enhance students’ overall experience and improve their self-assessment skills.

One study noted that pharmacy students were more likely to overestimate their professional skills than their drug knowledge during APPEs.¹⁵ It is probably more difficult to assess skills versus knowledge because skill assessment includes subjective evaluations. Students’ assessment of their own professional skills also may be influenced by their previous performance in didactic classes, which measure knowledge. Thus, self-assessment may be improved by creating better evaluation tools and spending more time teaching students how to use them. Several studies indicated that using video recordings as an assessment tool improved students’ self-confidence, interviewing, and self-assessment skills.^13,14,16 As in our study, using a detailed rubric-based evaluation form that includes assessment of professional behavior, performance on various skills, and knowledge also may facilitate the process.⁴

Limitations of the study include that it was confined to only 1 academic year and there was a low response to the alumni survey. In assigning a grade for a student's seminar, the evaluating faculty member could be influenced by the other 2 faculty members in the room, based on tone or type of questions asked of the student. Additionally, studies evaluating student self- and peer-assessments assume that the faculty evaluation of the student is most accurate, which may not be true in some cases. Grading differences between student and faculty members may be difficult to ascertain correctly if faculty members are inconsistent or inexperienced graders. For reasons related to confidentiality and the study's short duration, the faculty grading profile could not be evaluated in this study. However, pharmacy administrators could use peer- and self-evaluations of students to help compare grading trends between faculty members and adjunct faculty preceptors. This information could be helpful in faculty or preceptor development designed to improve grading skills and may be a valuable addition to student evaluations of faculty members and preceptors. Improving the grading skills of faculty members may, in turn, help students become better peer- and self-evaluators.

CONCLUSION

Because the majority of pharmacists are expected to perform peer- and self-evaluations in the workplace, we propose that pharmacy schools expand the evaluation process to include these types of assessments. Current problems in the validity of peer- and self-evaluations call for curricular changes to better train students to meet these expectations after graduation. While it is not necessary to incorporate peer- or self-evaluations into a student's final grade, feedback from peers and self is valuable. If given at the end of each APPE, feedback can be used to improve knowledge and/or skills before moving on to the next APPE. This process encourages students to be accountable for their learning and progression throughout their pharmacy education. The faculty member/preceptor should use discretion when using student-peer evaluations and avoid inappropriate disclosure of peer comments to the student. It is the preceptor's responsibility to find validity in peer comments and mix that feedback with his/her own to ensure that peer feedback remains anonymous. If used correctly, the combination of self-, peer-, and faculty assessments can provide an opportunity for pharmacy students to grow and develop and become better prepared to meet the evaluation requirements of future employers.