Abstract
Objective. To evaluate the modified peer-grading process incorporated into the SOAP (subjective, objective, assessment, plan) note sessions in a skills-based pharmacy course sequence.
Design. Students assessed a de-identified peer’s SOAP note in a faculty-led peer-grading session followed by an optional grade challenge opportunity. Using paired t tests, final session grades (peer-graded with challenge opportunity) were compared with the retrospective faculty-assigned grades. Additionally, students responded to a survey using 4-point Likert scale and open-answer items to assess their perceptions of the process.
Assessment. No significant difference was found between mean scores assigned by faculty members vs those made by student peers after participation in 3 SOAP note sessions, which included a SOAP note-writing workshop, a peer-grading workshop, and a grade challenge opportunity. The survey data indicated that students generally were satisfied with the process.
Conclusion. This study provides insight into the peer-grading process used to evaluate SOAP notes. The findings support the continued use of this assessment format in a skills-based course.
INTRODUCTION
The formulation and documentation of patient care plans is a skill central to the provision of patient-centered care, supported by the Center for Advancement of Pharmacy Education (CAPE) 2013 Educational Outcomes.1 To teach and evaluate this skill, Midwestern University College of Pharmacy – Glendale (MWU-CPG) incorporated a series of active-learning experiences relevant to pharmacy practice using the “subjective, objective, assessment, plan” (SOAP) written documentation format. Supplementing each SOAP note-writing workshop, students assessed and scored an anonymous peer’s written SOAP assignment using a faculty-developed rubric in a required faculty-led workshop, followed by a self-reflection opportunity.
This assessment method was designed to meet the Accreditation Council for Pharmacy Education (ACPE) Draft Standards 2016 recommendation that schools and colleges should “utilize teaching/learning methods that facilitate achievement of learning outcomes, actively engage learners, promote self-directed learning, and foster collaborative learning”.2 In addition, this method helps address Domain 4 of the CAPE 2013 Outcomes, which focuses on educating future pharmacists to be self-aware by providing a structured opportunity to develop the skills of peer- and self-assessment.1
Peer assessment not only serves as a grading procedure, but a method to practice self-evaluation skills. Peer assessment is a valuable form of assessment in higher education for a variety of assignments, and students generally perceive it as beneficial.3 Peer assessment is used in pharmacy education for student case presentations, laboratory courses, medication management assignments, patient interviews, and advanced practice experiences.4-9
A proposed benefit of the peer assessment strategy is providing an enhanced learning environment for students in which a high order of thinking is utilized and guided by evaluation. Other benefits include self-reflection, peer interaction, and a decreased faculty workload.4 Finally, peer assessment can help to foster higher levels of responsibility among students by requiring them to be fair and accurate with the feedback and evaluations they make of their peers.3 Limitations identified in previous research include a lack of student understanding of the purpose of the peer-grading session, lack of student experience in grading, and lack of confidentiality. In these studies, peer grades were generally higher than faculty grades for the assignments.4-9
A previous study comparing student and faculty scores for 3 peer-graded assignments at the college resulted in the students assigning significantly lower scores than faculty members to student peers, which was inconsistent with other research.10 One goal of using peer grading was to replace the need for traditional faculty SOAP note grading, so several changes were made to improve the consistency of faculty and peer-assigned scores. A standardized format, titled the Comprehensive Medication Management Plan (CMMP), was adopted for guiding SOAP-note writing, which detailed the expectations for each section and served as the basis for the grading checklists used in the peer-grading sessions. This approach attempted to provide the students clear and consistent expectations among different faculty members when formulating their written SOAP note.
Additionally, a “challenge” opportunity was implemented to allow each student the chance to self-evaluate their peer-graded note prior to the final grade assignment and submit a grade challenge if appropriate.10 Outcomes of a “challenge” strategy, including the accuracy of final grades after this opportunity, has not been formally evaluated in the literature. The intent of these changes was to improve the scoring process so the peer-assigned scores would be similar to those assigned by a faculty grader, thus validating the peer-grading process used for the SOAP notes sessions as a stand-alone assessment.
The purpose of this study was to evaluate the modified peer-grading process incorporated into the SOAP note sessions. The specific aims were to compare the final recorded student grade (given after the SOAP note session and “challenge” opportunity) with a retrospective traditional faculty-assigned grade, and to gather the student’s perceptions about the process.
DESIGN
Professional Skills Development (PSD) is a required 8-quarter, workshop-based course sequence in a 3-year didactic curriculum (PSD 1-8). In the second year of the sequence in 2012-2013, there are 3 SOAP note sessions. Each session consists of 3 parts: a writing workshop, a peer-grading workshop, and a challenge opportunity that build on the foundational knowledge taught in the therapeutic course sequence (PSD 5, 6, and 8). The final score received after the 3 sessions is considered a summative assessment of this knowledge.
This study involved a single cohort of students completing the second year of their PSD sequence. All students enrolled in PSD were required to participate in the 3 SOAP note sessions. The case content for each session corresponded to the material taught by the subject matter expert in the concurrent therapeutics course during each of the 3 quarters (hyperlipidemia, psychiatry, and cardiovascular). The subject matter expert also designed the case and grading checklist, led the peer-grading workshop, was present at the challenge opportunity, and graded the challenge submissions.
Organization of Sessions
For each SOAP note writing workshop, students were given a template prior to writing their SOAP note, which included a list of medical problems students were required to address for the patient case. Given the subjective and objective data, students were asked to individually write the assessment and plan for the patient. The first SOAP note-writing workshop was completed outside of class and submitted via Safe Assign, a program found in the Blackboard learning system (Blackboard Inc., Washington, DC), to check for plagiarism. The other 2 SOAP note-writing workshops were completed within a 2-hour time block in the university’s testing center. The subject matter expert was not present at the writing workshop. To aid in developing their assessment and plan, students were given a CMMP grid (Appendix I). Completion of the CMMP was optional during the SOAP note-writing workshops, although highly encouraged, as the grading checklists for each note were based on the same categories.
Peer-grading workshops occurred during class time within one week of completing the writing workshop. Each of the 3 peer-grading workshops were held in 2 sections (2 hours each) on the same day to accommodate one-half of the class at a time. This ensured that students who wrote the SOAP notes being evaluated were not present in that particular section of the peer-grading workshop. In an effort to hold the students accountable for attending the session, for grading the SOAP note to the best of their ability, and for providing quality feedback to their peers, students wrote their names on the cover sheet of the evaluation form. The course coordinator kept track of who graded each SOAP note, but students and the subject matter expert were blinded throughout the entire process. The grading checklist was distributed first, prior to the cases to be graded. The subject matter expert directed the students to review the grading checklist point by point. This was an opportunity for students to reflect on what they had individually written on their own note and ask questions for clarification, prior to being held responsible for grading a peer’s note. Peer notes then were distributed, and students were given approximately 20 minutes to read through their assigned note and grade based off of the supplied grading checklist. Once students had completed grading, they again reviewed the rubric. At this time, students could request whether an alternative written response not included within the grading checklist be considered for points by the subject matter expert facilitating the session. Once the grading was complete, students were instructed to provide anonymous constructive written feedback to their peers.
A 1-hour challenge opportunity was offered within 24 hours after each peer-grading workshop for students to review their peer-graded note and to challenge their peer-assigned grade. Only students who attended the challenge opportunity had the ability to submit a challenge form. If students wanted to proceed with a challenge, they completed a preprinted form specifying which point(s) they wanted to challenge and providing justification for each. The subject matter expert was present at the challenge opportunity to answer additional student questions regarding the case or their responses. After the challenge opportunity, the subject matter expert reviewed the challenge forms and addressed only the inaccuracies noted by the student. However, the subject matter expert reserved the right to regrade the assignment if necessary. Grade challenges were not accepted after this session was completed.
Evaluation of Sessions
To evaluate the scores assigned during each of the 3 SOAP note sessions, the subject matter expert retrospectively graded each assignment using the standard grading checklist used in the peer-grading workshop which modeled the traditional grading method used at the college. This traditional method did not include a peer-grading workshop or challenge opportunity and resulted only in a scored rubric with written comments. The subject matter expert was blinded to the student names and all results of the peer-grading workshop and/or challenge opportunity.
The final session grades (peer-graded with challenge opportunity) were compared to the retrospective faculty-assigned grades for each session. Evaluations were made using paired t tests with a p value of ≤0.05 being considered significant. An anonymous survey was also administered to all students enrolled in the final quarter of the PSD course sequence. The survey was voluntarily completed during a course period after the last of the three SOAP note sessions. The survey assessed the students’ opinions of the SOAP note sessions utilizing 4-point Likert scale statements, multiple-choice questions, and open-ended questions. Descriptive statistics were utilized to analyze the findings of the survey. Approval for the study was obtained from the Institutional Review Board (IRB) at Midwestern University.
EVALUATION AND ASSESSMENT
Scores for 453 peer-graded assignments (151 for session 1, 152 for session 2, and 150 for session 3) were analyzed. No difference was found between the mean scores for final recorded grades and retrospective faculty assigned grades (Table 1). Differences between the mean scores for each of the 3 individual sessions were 0.8, 0.9, and 1.3, respectively. A subgroup analysis was performed to compare the final recorded grades and the retrospective faculty-assigned grades based on participation in the challenge opportunity (Table 2).
Final Recorded Grades Using Peer-Grading Method vs Retrospective, Traditional Faculty Grades for 3 Sessions
Final Recorded Grades Using Peer-Grading Method vs Retrospective, Traditional Faculty Grades for 3 Sessions Based on Participation in the Challenge Opportunity
We found that for all 3 sessions combined, 158 students (35%) challenged their peer-graded SOAP note for faculty review (93% of which resulted in a score change), 243 (54%) reviewed their peer-graded note but did not challenge for faculty review, and 52 (11%) did not review their assignment. No difference was found if the student reviewed their submission and either opted to challenge or not to challenge. However, there was a significantly lower score assigned for students who did not review their assignments in the challenge opportunity.
One hundred twenty students (80%) participated in the survey assessing their perspectives on the SOAP note sessions. (Table 3) Responses from 2 surveys were not included because the respondents did not meet the criteria of having attended at least two challenge opportunities. Regarding the SOAP note-writing workshop and subsequent peer-grading workshop, the majority of students either “agreed” or “strongly agreed” that the CMMP helped them structure their SOAP notes (89%), the instructions for participating in the sessions were clear (96%), the grading checklists were easy to follow (95%), the peer grading sessions enhanced learning (70%), and faculty guidance during the peer-grading workshops allowed them to effectively grade a peer’s note (90%). Regarding the challenge opportunity, the majority of students either “agreed” or “strongly agreed” that the challenge opportunity allowed them to self-reflect on their work (73%), that the challenge sessions are necessary to receive a fair score on peer-graded assignments (89%), and that faculty members awarded points fairly after the grade challenge opportunity (77%). The lowest score was in response to whether peer-provided comments were constructive and useful in improving SOAP note skills. However, 54% of the students“agreed” or “strongly” agreed with this statement as well.
Students’ Opinions Regarding SOAP Note Sessions in the Professional Skills Development Course Sequence
The open-ended questions asked what the students liked most and least about the SOAP note sessions and suggestions for improvement. (Table 4) Students most commonly stated that their learning was enhanced by exposure to the various ways their peers approached the case. Other commonly reported responses included enhancement of disease state knowledge through immediate faculty feedback, fairness in grading, and improvement of general SOAP note-writing skills.
Student Comments Regarding Peer Grading Process in the Professional Skills Development Course Sequence
The majority of the responses regarding what students liked least included the process taking too long or classmates asking too many questions. Other common responses included inconsistency in points awarded by peer graders, rubrics being too strict, and reviewing the rubric prior to starting taking too much time or being too repetitive. The most frequent responses for how to improve the process included structuring the sessions to take less time and/or limiting the number of questions asked by other students. Other suggestions included revising the rubrics to be less stringent, holding the graders accountable for inaccuracy in grading, and allowing the notes to be typed instead of handwritten.
DISCUSSION
The revised peer-grading process for SOAP note sessions in the PSD course sequence resulted in the same overall score when compared to traditional faculty grading. This contrasts to a previous study, which showed higher traditional faculty-assigned scores than student peer-assigned scores.8 This result strengthens our confidence in the use of the process in place of traditional faculty grading. The most likely contributor to this result was the institution of the challenge opportunity; allowing students to self-assess their work, review peer comments, and submit a challenge form if inaccuracies were noted. In the subgroup analysis, we noted a difference in the final recorded grade vs the traditional faculty-assigned grades for those students who did not participate in the challenge session.
The use of the CMMP document was another possible contributor to the equalized scores, since it served as a structure by which all grading checklists were formatted. This may have streamlined the grading process by improving the clarity of the grading checklists used in the peer grading sessions.
Advantages and disadvantages of the peer-grading process from student and faculty perspectives were noted. Student noted that learning from their peers’ approach to the case, obtaining immediate faculty feedback that enhanced disease state knowledge, and fair grading were advantages. From the faculty perspective, an advantage to this process was the reduced faculty grading workload. Scoring student-submitted challenge forms (n=158) took a total of approximately 3 hours for the subject matter experts to complete. In contrast, scoring each submission separately (n=453) for the purposes of this research approximated 10 minutes per submission.
This process is not without disadvantages. Some students said the peer-grading workshop took too long because other students asked too many questions even though the workshop was limited to 2 hours and was conducted during normal course time. Some also indicated an inconsistency among peer graders and may have viewed the challenge as an opportunity to double-check their peers rather than an opportunity for self-reflection. Finally, some students indicated a lack of constructive feedback from peers despite receiving examples from instructors of appropriate vs inappropriate feedback. One disadvantage from a faculty perspective was the negative connotation implied by the terms “peer-grading” and “challenge opportunity.” Despite an formal orientation on the benefits of peer and self-assessment, some students perceived this process as a way for faculty members to decrease their workload and transfer the burden to students.
There are several limitations to this research. The 3 SOAP note sessions were each designed, conducted, and graded by 3 different subject matter experts and focused on a wide array of therapeutic topics. Given this, the students’ perceptions about the process could be influenced by the therapeutic topic or faculty member. Additionally, the SOAP note sessions were conducted in the fall, winter, and summer quarters, while the survey was administered at the end of the final session, possibly reflecting the students’ opinions of the most recent session. Finally, this study is reflective of a specific cohort of pharmacy students enrolled in a workshop-based course sequence that may not be applicable to other schools or other disciplines.
The SOAP note sessions in PSD have continued with subsequent student cohorts, without the use of traditional faculty grading. Several adjustments have been instituted based on the findings of this study. For example, in addition to the formal orientation, a more formal method for providing peer feedback has been instituted. After the peer-grading workshop, students are asked to state 3 things the author did well and 3 areas for improvement. Additionally, the procedure during the peer-grading workshop has been streamlined to allow students to grade peers’ submissions at the beginning of each session in pencil, eliminating the initial faculty review of the rubric with the class. Once all individual grading is complete, the subject matter expert reviews the rubric and entertains student questions and feedback. This change was based on feedback that students found it redundant to review the rubric twice and that it lengthened the overall session time. Finally, the description and name of this process has changed from “peer grading” to “peer assessment” to further convey the positive academic intent of the process. In addition, the term “challenge” opportunity was changed to “self-reflection and review workshop” to promote the individual benefits of participating in the process.
CONCLUSION
This study provides insight into the peer-grading process used in the PSD course sequence at the college. Our findings indicated no difference between grades assigned to students using the peer-grading process and traditional faculty grades for all 3 sessions. Additionally, students perceived that the process was beneficial to their learning, supporting the continued use of this assessment format in our curriculum. We feel the process has the potential to be successfully used in other professional programs as well.
Appendix
Comprehensive Medication Management Plan (CMMP)
- Received September 11, 2014.
- Accepted March 18, 2015.
- © 2015 American Association of Colleges of Pharmacy
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.