Abstract
Objectives. To determine the ability of first-year students to self-assess communication skills and measure the impact of video review on students' self-assessment.
Design. Students participated in a digital video-recorded, counseling-simulation exercise and completed self-assessment before and after viewing their video. A faculty member evaluated the students using the same counseling assessment tool.
Assessment. Correlation between the students' self-assessment scores and the faculty member's scores were poor (pre-video: r = 0.38, post-video: r = 0.46). The largest portion of the students overestimated their skills in comparison to the faculty member's evaluations (47.1% pre-video and 67.9% post-video). Those in the lowest quartile overestimated their skills, while those in the upper quartile underestimated their skills (pre-video). Video review brought about an increase in the self-assessment scores for nearly two-thirds (62.1%) of the students.
Conclusion. First-year pharmacy students had difficulty self-assessing, and video review increased their perception of skill achievement. A curriculum should include opportunities for students to develop self-assessment skills early in the program, and this should be reinforced throughout the curriculum.
INTRODUCTION
Students' self-assessment has become an important element in the evaluation of pharmacy curricula and individual student's professional development.1–6 This is due in part to the recent revision of the Accreditation Council for Pharmacy Education's Standards 2007, which specifically emphasize self-assessment, development of a “personal learning plan,” and lifelong learning.6–9 Students also need self-assessment skills to participate effectively in the educational process.6 As Fjortoft stated, “students are expected to be able to analyze their own learning needs, allocate time appropriately, and then accurately self-assess to determine if those learning needs were met.”6 These factors have led schools and colleges of pharmacy to re-evaluate their assessment process and include or expand self-assessment activities.
To use self-assessment data to evaluate a curriculum or guide a student's development, it is important to understand students' ability to self-assess. The quality of self-assessment has been examined in a large number of studies in the general and professional education literature; yet the accuracy of self-assessment has been studied only recently among pharmacy students, and has involved only upper level students.1, 10–13
Austin et al evaluated pharmacy, bachelor of science students' self-assessment ranking compared to the same assessment by peers, a standardized patient, and a pharmacist.12 The findings indicated students' self-assessment skills were not at the level expected. Students in the lowest 2 quartiles overestimated their abilities to the greatest degree, while students at the top of the class ranking underestimated their abilities. Similarly, comparison of fourth-year pharmacy (P4) students' self-assessment results to preceptors' ratings at the end of the advanced pharmacy practice experiences (APPEs), students also showed a frequent overestimation of skills.13 Specifically, students overestimated performance on a larger percentage of the pharmacy skill items than drug knowledge items (94% and 29%, respectively), suggesting a greater difficulty in evaluating skills versus knowledge. Abate et al evaluated P4 students' self-assessment in a drug information APPE.1 Although students' and faculty members' mean scores were similar, disagreement existed when the individual student's results were compared with the faculty members' evaluation (ie, large percentage of over and underestimation, although differences were small). Pharmacy students in their last year of the curriculum appear to be limited in their ability to self-assess, which has also been reported for upper level medical students and residents.1,14–18 Whether first-year pharmacy (P1) students demonstrate similar tendencies is of critical importance because limitations in self-assessment influence students' academic development throughout the curriculum.
Video review has been used to facilitate skill development and examine self-assessment.19–23 Research studying a videotaping program for third-year medical students found that 74% of students improved in areas of deficient skill performance.21 The impact on skill development has also been studied among nursing students.22 In this study, improvement in skills was measured by comparing test scores for nursing students completing a self-assessment following video review with a control group who self-assessed after describing the steps in the process.22 Test scores were higher for the students who self-assessed following video review, which support the use of video recording as a training modality.22
The quality of self-assessment following video review has been examined. Third-year medical student video reviews showed that student self-assessment skills improved with practice.21 This was demonstrated by a decrease in the number of students who under or overestimated their abilities after 1 review. Alternative approaches to self-assessment with video review have involved open-ended questions.20,23 In a study of first-year medical students' ability to self-assess communication skills following video review, students on average identified 5 strengths and 2.8 opportunities to improve, along with examples to support their conclusions.23 This study did not compare results to an external evaluator but recognized the positive impact of identifying strengths and weaknesses based on evidence.23
Research has not examined the accuracy of P1 students' ability to self-assess or the effect of video review on self-assessment. Therefore, the objectives of this study were to determine the ability of P1 students to self-assess communication skills in a capstone patient counseling exercise, and to measure the impact of video review on P1 students' self-assessment. This information is critical to help students develop effective self-assessment abilities, to guide appropriate use of video reviews, and to correctly interpret self-assessment data for curricular evaluation.
DESIGN
This P1 capstone counseling assessment was part of a pharmacy practice class in spring 2008 and 2009. Students engaged in 5 hours of lecture focusing on patient counseling and participated in 2 laboratory sessions, 2 hours each, where groups practiced counseling skills on mock patients, using the study tool to evaluate themselves and each other.
The counseling assessment tool, designed by the course coordinator, contained 25 equally weighted items evaluated on a 3-point Likert scale (0 = did not complete; 1 = partially completed; 2 = fully completed) for a total of 50 points. Included were communication of general information (introduction, patient name, patient allergies, etc) for 18 points; communication of technical information (indication, side effects, storage, etc) for 18 points; interpersonal communication aspects for 12 points; and cultural competency skills for 2 points. The tool is available upon request from the authors.
Face validity was established by review and input of 3 practice faculty members. The pilot test of the tool involved 2 faculty members completing the assessment tool on 4 video-taped student counseling sessions from 2007. The 2 faculty members' scores were highly correlated including the total scores (r = 0.98, p < 0.01) and the 3 skill categories (general information r = 0.99, p < 0.005; technical information r = 0.98, p = 0.01, interpersonal r = 0.97, p = 0.02).
Students were given the name of the drug involved in the exercise prior to the counseling exercise and allowed to bring resources to the consultation. As part of the counseling session, the student was given the patient's information, the prescription, and a labeled vial containing the patient's medication. The student counseled 1 of 3 college support personnel members who had been selected to serve as standardized patients based on required attributes (eg, similar characteristics to the patient, ability, suitability, credibility), and trained in accordance with published guidelines (eg, scripted statements and responses to questions).24
The session was digitally recorded and the faculty member evaluated the student on closed circuit television, using the assessment tool. The student completed the assessment tool immediately after the counseling session (pre-video) and again within 1 week of the exercise after privately reviewing the video (post-video). The video and student's written reflection were posted in the student's e-portfolio.
Seventy students participated in the counseling session each year, which involved 24 hours of 1 faculty member's time (ie, set up, viewing, assessing), 2 hours for each of the 3 standardized patients, and 1 hour for each student (ie, counseling, reviewing the video, self-assessing).
A Pearson correlation coefficient for the total scores was examined for the 3 groups. The total scores from the faculty member's evaluations were used to create students' percentile ranking, allowing the students to be divided into quartiles based on their total score. The faculty member's scores within each of the 3 categories (eg, interpersonal skills) were used to create student rankings, but because of the smaller scale, students were divided into those above and below the 50th percentile. A paired student t test was used to test significance for total scores and skill category scores for all students and within the ranked groups. Statistical significance was set at p < 0.05.
The analytical approach was based on the research methods review for measuring self-assessment written by Ward et al9 and on a modified approach to quartile data used by Austin et al.12 Approval for the study was obtained from the South Dakota State University Institutional Review Board.
One hundred forty P1 students participated in the counseling exercise over 2 years, and 1 faculty member completed all faculty evaluations. The correlation between the faculty member's and students' pre-video scores was 0.38. A larger percentage of students scored themselves' higher than the faculty member (47.1%), compared to the percentage of students who agreed with the faculty member's or those who scored themselves lower than the faculty member, suggesting that students often overestimate their abilities (Table 1). However, no significant difference was found in the average faculty member's score and pre-video value (Table 2).
Student's Pre- and Post-Video Total Scores Compared to Faculty Total Scores on the Counseling Assessment Tool
Average Total Scores on the Counseling Assessment Tool for All Students and Students Divided into Quartiles Based on Faculty Ranking
Results were examined based on the faculty member's ranking of the students. Average self-assessment scores for students in the highest and lowest quartiles differed significantly from the faculty member's scores (Figure 1, Table 2). Students in the lower quartile significantly overestimated their scores (43.0 versus 41.5), and those in the highest quartile significantly underestimated their scores (44.7 versus 47.2) as shown in Table 2.
Average total scores on the counseling assessment tool based on faculty ranking.
Based on average scores from analysis of the 3 skill categories, students initially overestimated their scores in the communication areas of general (16.5 and 15.7) and technical information (16.0 and 15.3), but significantly underestimated their interpersonal scores (10.0 and 11.1) (Table 3). Differences were noted in the students' skill level among these 3 categories (Table 3). Based on average scores prior to video review, students below the 50th percentile significantly overestimated their skills in conveying general (14.9 and 16.1) and technical information (14.1 and 15.5), while there was no difference for students above the 50th percentile in these 2 areas (Figure 2 and Figure 3, Table 3). Conversely, prior to video review, both those students above and below the 50th percentile significantly underestimated their interpersonal skills according to average scores (Figure 4, Table 3).
Skill Category Scores on Counseling Assessment Tool
Average scores for communication of general information on the counseling assessment tool based on faculty ranking.
Average scores for communication of technical information on the counseling assessment tool based on faculty ranking.
Average scores for interpersonal communication on the counseling assessment tool based on faculty ranking.
The correlation between the faculty member's and students' post-video scores was 0.46, and the post-review average score was significantly higher than the faculty member's average score (44.0 versus 45.7) (Table 2). To further explain the results, after video review, over two-thirds (67.9%) of the students scored themselves' higher than the faculty member (Table 1). In addition, students in all but the highest quartile scored themselves significantly higher than the faculty member based on average scores (Figure 1, Table 2). Analysis of the tool's 3 categories yielded similar results for the post-video review group as were noted for the pre-video review group (Table 3).
The correlation between students' pre- and post-video scores was 0.68. The average score for students' pre-video self-assessment was significantly lower than the average score post-video (44.2 versus 45.7) (Table 2). Consistent with these findings, 62.1% (n = 87) scored themselves higher after watching the video, compared to 18.6% (n = 26) who lowered their score, and 19.3% (n = 27) who did not change their scores. These results were examined further based on faculty member's ranking of the students. In all 4 quartiles, the average student score was higher after watching the video compared to pre-video scores (Figure 1, Table 2).
Based on analysis of the tool's 3 categories, students' average post-video score was higher than the pre-video score in all 3 areas, but results were significant only for interpersonal communication skills (10.8 and 10.0) and communication of technical information (16.5 and 16.0) (Table 3). While interpersonal scores increased after video review, the average score remained lower than the faculty member's average score (10.8 and 11.1).
Examination of the 3 skill categories based on student proficiency levels showed a significant increase in average scores from pre- to post-video review for both those above and below the 50th percentile in the areas of communicating technical information and interpersonal skills, while no significant change was noted for communicating general information (Figure 2, Figure 3, and 4; Table 3).
DISCUSSION
This study sought to determine whether P1 pharmacy students were able to accurately self-assess their counseling communication skills. P1 students struggled with self-assessment in both the pre- and post-video setting, evidenced by the poor correlation of the students' scores with the faculty member's scores (r = 0.38 and 0.46, pre- and post-video, respectively). Students tended to overestimate their skills as shown by the largest percentage of students in both pre- and post-video groups overestimating their scores (Table 1), and the significantly higher mean for the post-video group (Table 2). These difficulties in self-assessment are consistent with literature examining more advanced pharmacy students1,12,13 and students in other health professions.11,14–16,18
Research suggests that overestimation may be due to a variety of causes. One theory is termed the “above-average effect,” based on the idea that a person knows himself best (eg, talents, skills), and understands others less.12 Therefore, the person assumes he is more talented than others. This is more common for people who have had high levels of success in the past, such as pharmacy students. An additional theory relates to the feedback provided to students, in what has been referred to as the “applause society.”12 Feedback is often tempered, blunt negative assessments are avoided, and positive aspects are overemphasized. This approach gives students the perception that they are performing at a high level, when improvement may be needed.
Longitudinal studies have examined self-assessment skills. Arnold et al found that the correlation of self-assessment to external evaluation decreased over time,25 while Fitzgerald et al showed self-assessment to be a stable characteristic that was less accurate in a new environment (eg, from classroom to clinical setting), where performance was poorer in comparison with a familiar activity.26 This may be relevant to our research because students had limited experience with the task of counseling in their first year. Self-assessment results may have been different if the assessment had involved a more familiar task.
Students who are less skilled typically overestimate their skills, and those near the top of the class tend to underestimate their abilities.12,14,16,18 Studies have examined medical students, pharmacy students, and residents who are farther along in the educational process compared to the sample in this study. However, the P1 students' self-assessment results in both the pre- and post-video groups are similar to these findings, with students in the highest quartile underestimating their work, and students in the lower quartile overestimating their work based on average scores (Figure 1, Table 2).
One theory relating to why less-skilled students overestimate their abilities involves the concept that the knowledge necessary to perform an activity effectively is the same knowledge needed to self-assess.12 Therefore, a student performs an activity poorly because of a lack of knowledge to do the task properly, and the student does not recognize the deficiency because of the same lack of knowledge.
In this study, students primarily below the 50th percentile significantly overestimated communication of general and technical information based on average scores (Figure 2 and 3, Table 3). Conversely, students underestimated interpersonal communications based on average scores, except for students below the 50th percentile in the post-video group (Figure 4, Table 3). Austin et al also examined specific skills and knowledge (clinical knowledge, communication skills, interview nature, and empathy) of senior students in a clinical simulation laboratory session and found that excluding the very highest performers, students overestimated their abilities in comparison to their peers in all subcategories.12 Conversely, Abate et al examined students' self-assessment on a drug-information APPE and found variations in the percentage of students who over- and underestimated their abilities, depending on the aspect of drug information in question.1
The difference found between our study and that of Abate's study is in line with Fitzgerald's premise that the task and situation impact students' ability to self-assess.1,26 Students often have the greatest difficulty evaluating the skill that is most challenging to them.21 Therefore, it is important to avoid making generalizations beyond the setting used in the research.
Video review has been identified as an effective tool to improve practice skills.19,21–22 This has been found to be true for communication skills and practice activities,19,21–23 and the improvement is brought about by students recognizing their own strengths and weaknesses.23 Not only did students' skills improve from utilizing video review, but the students' ability to self-assess also became more accurate on subsequent video review.21
In this study, the post-video evaluation results correlated fairly well with the pre-video scores (r = 0.68), which was expected because the same evaluator (the student) completed both the pre- and post-video evaluations. In addition, we had hypothesized that students would be more critical of themselves after watching the video, and their scores would be lower in the post-video evaluation. This did not occur. Sixty-two percent of the students scored themselves higher after video review. This was true across all 4 quartiles, and was significant for communication of technical information and interpersonal communication aspects (Table 2 and Table 3). Further research is needed to verify these findings in other settings and explore causes.
Our study suggests that video review can build confidence in interpersonal skills, which may be beneficial as scores were below the evaluator's scores, even with the increase in values following video review. In addition, this information may be used to guide faculty members' feedback. Students should receive more critical feedback on the communication of general and technical information, while receiving encouragement in the area of interpersonal communication.
Self-assessment is critical to many educational processes, including life-long learning and continuous professional development.6,8–9 However, false assessments limit the educational process. For example, overestimation of skills may produce misdirection of educational efforts and persistent incompetent practice. Of interest, in a study involving third-year medical students and the relationship between perceived weaknesses/strengths and time allocation, some students did not devote more time to areas of weaknesses.27 This may have been a result of the areas emphasized within the program (eg, assigned clinical service, patient cases) that influenced students' time allocation more than the students' own self-assessment.27 Similarly, overestimation of abilities may limit skill development, making the student less likely to accept feedback.28 Faculty members should be aware of this frequent overestimation and the potential for a student who overestimates abilities to become disgruntled.28 Feedback may be more effective if presented in a discussion rather than an accusatory manner.28
From a practice standpoint, faulty self-assessment may create an environment predisposed to practice pitfalls where deficiencies are not recognized and corrective actions are not taken. A practitioner should have confidence in actual skills, but it may be of concern when the perception of skills exceeds abilities.12 The educational system should be designed to prepare students to identify and address these blind spots.
A suggested approach to improve self-assessment may include frequent practice with self-assessment activities in a variety of settings throughout the curriculum because practicing self-assessment has been shown to improve skills.21,28 In addition, students should receive not only timely feedback on their skill development, but also input on how they have evaluated themselves.6 This will help students realize whether they are over or underestimating their performance. Self-assessment may also be improved by requiring students to provide evidence to support their evaluation.6 Use of video has also been suggested as a way to improve self-assessment.21 Finally, students may need formal training in self-assessment techniques.28
This research of P1 students' ability to self-assess took place in a counseling session, and the results cannot be extrapolated to self-assessment of other activities (eg, non-counseling practice activities such as intravenous product preparation). In addition, the tool was specific in its design (eg, asked patient if they had questions), and students' ability to use a more open-ended tool for self-assessment was not evaluated. This was a new activity for students, and research has shown that familiarity with the activity influences the self-assessment process. However, the focus of the study was to examine students in their first year of the pharmacy program when nearly all activities are new experiences. Therefore, this is more the point of the study rather than a limitation. Finally, one faculty member's evaluation served as the standard against which the students' evaluations were compared, which may limit the reproducibility of the results. However, evaluation of rater reliability in a pilot study found a very high correlation among faculty evaluators' scores.
CONCLUSIONS
First-year pharmacy students had difficulties performing accurate self-assessment of counseling skills, with the largest percentage of students overestimating their performance. Students in the lower quartiles tended to overestimate while those in the upper quartile underestimated their skills. Among the skill categories on the evaluation tool, students tended to overestimate communication of general and technical information, and underestimate interpersonal communications. Video review resulted in an increase in the students' total scores (average score and percent of students overestimating performance) and the average score in all quartiles. Video review significantly increased mean scores for students in the areas of communication of technical information and interpersonal communication. These results can help guide the feedback provided to students using video review. In addition, curricula should contain frequent self-assessment opportunities, along with feedback on skill development and self-assessment accuracy. Finally, the importance of accurate self-assessment in the continuous professional development process should be stressed to students.
- Received August 31, 2009.
- Accepted November 15, 2009.
- © 2010 American Journal of Pharmaceutical Education
REFERENCES
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.