Abstract
Objective. The ability of pharmacy students to self-regulate and reflect on knowledge and skills is important as proficient self-assessment skills guide learning strategies and prompt students to bridge their knowledge gaps. The objectives for this study were to determine how well third-year pharmacy students self-assess, explore the rationales behind their self-assessments and determine whether there is a correlation between self-assessment accuracy and academic performance.
Methods. A quasi-experimental one-group pre-/post-test design was conducted with third-year pharmacy students. Examiner grades, student self-assessment grades, comparative reports, and end-of-semester grades were collected. Students were categorized into tertiles based on academic performance for data analysis. Paired t tests, Pearson r and percentage agreements were conducted to investigate self-assessment accuracy. Correlational statistical tests were implemented to examine the relationships between self-assessment accuracy and academic performance.
Results. One hundred sixty-two third-year pharmacy students were included. On average, students demonstrated poor self-evaluation skills and underestimated themselves by 4.9%. Lower performing students were generally overconfident in evaluating their performance. There was no significant correlation between students’ self-assessment accuracy and academic performance on the subsequent end-of-semester examination questions.
Conclusion. Overall, students tended to underestimate their academic performance. Further research on self-assessment is needed to better understand how students think about their performance, which may help to improve education methods, such as inclusion of reflective practices after case-based activities.
INTRODUCTION
Metacognition refers to “one’s knowledge concerning one’s own cognitive process,”1 and is an integral skill for health care professionals. For pharmacy students, the ability to self-regulate and reflect on their own knowledge and skills is essential because proficient metacognition guides learning strategies and prompts students to bridge their knowledge gaps.2,3 Good awareness of metacognitive skills in the classroom has been correlated with higher achievement, possibly due to students having higher levels of problem-solving skills and critical self-reflection which facilitate their academic journey.4
One way to evaluate metacognitive skills in pharmacy students is to determine how accurately they self-assess their performance.5 Self-assessment accuracy is frequently demonstrated by an acceptable variation between self and an expert’s ratings of academic performance based on a standardized grading scheme. A meta-analysis showed that students lack self-assessment abilities and, as a result, misjudge their performance, with lower-performing students overestimating their capabilities.6 Inflated self-assessment is a well-documented phenomenon in undergraduate pharmacy students.7-9 However, a more recent study reported that most pharmacy students underpredicted their grades and high-performing students demonstrated more accuracy in error identification.10 This shows that students with varying performance levels possess different patterns of self-assessment accuracy.
Methodological designs involving self-assessments after completing a course activity have shown that students’ self-assessment accuracy may change over time. Some researchers have reported overconfidence in students and no improvement in self-assessment accuracy,11-13 but results from another study showed improvement in self-awareness over three semesters.14 After the introduction of “benchmark performance” (defined in this context as a self-assessment of performance compared with an expert response), self-ratings became more aligned with expert ratings.13 Because of these varying results, self-assessment accuracy in pharmacy students warrants further investigation.
Given that the utilization of “benchmark performance” has been shown to improve the accuracy of self-assessment, a primary aim of this study was to determine how well students evaluate their action plan using the integrating Science and Practice (iSAP) format after viewing expert practitioner responses.15 The iSAP format is an active-learning method that incorporates metacognitive strategies to support self-regulated learning. The iSAP format involves a case/scenario presented in realistic fashion, a student written action plan, an expert response, and a student comparative report or reflective analysis (Figure 1).15
Summary of the sequence of events for pharmacy students in following the iSAP format.
The iSAP format was introduced into the new Monash University pharmacy degree in Australia to promote a patient-centered action plan and reflective practice. One key feature of the degree is a student-directed learning approach and a focus on student self-development of essential skills for a practicing pharmacist, such as therapeutic problem-solving and communication. Through the iSAP process, students develop skills via the use of expert and peer feedback, real-life case-based studies, and student reflection.
In addition to investigating self-assessment accuracy of third-year pharmacy students in the Monash pharmacy degree, another aim was to determine whether there was a correlation between self-assessment accuracy and academic performance. Additionally, we sought to determine if there was a relationship between self-assessment accuracy and subsequent performance on an examination question that assessed similar content. Another aim of this study was to understand the rationales that students provided for their performance and gain insight into how they self-assess their performance by analyzing self-reflections. Based on previous studies on self-assessment, we hypothesized that students would display low to moderate ability to self-assess, with low-performing students being more likely to overestimate their performance. This study addressed the following research questions (RQs): RQ1: How well do pharmacy students self-assess their action plan performance after viewing an expert response? RQ2: How does self-assessment accuracy relate to academic performance on an iSAP assessment completed by pharmacy students? RQ3: How does self-assessment accuracy relate to the performance of pharmacy students on a similar end-of-semester examination question? RQ4: How does self-assessment accuracy relate to changes in pharmacy students’ performance between the iSAP assessment and a similar end-of-semester examination question? RQ5: What types of reasons do pharmacy students provide for their performance on the iSAP assessment compared with expert responses?
METHODS
A quasi-experimental, one-group pretest/posttest research design was used to measure the impact of an intervention in which pharmacy students viewed an expert response and conducted self-assessments and reflections by comparing students’ scores before and after the intervention, ie, iSAP assessment grades and end-of-semester examination grades. The subjects were third-year pharmacy students enrolled in a therapeutics course. Excluded from the study were students who were absent from the assessment, provided an incomplete or no self-assessment, opted out of a research data registry, or were involved in the data analysis process.
Students were given 100 minutes to review an oncological case presented in the iSAP format and create their action plan using a standard model of patient care to identify medication-related problems (MRPs) and provide therapeutic reasoning for their recommendations. After they submitted the management plan to be assessed by a course examiner, the students viewed an expert response. Two days later students were asked to self-assess their management plan using a grading rubric similar to that used by the examiner and complete a comparative report (using the “What? So what? Now what?” format16). One month later, students completed the end-of-semester examination for this course. A single examiner graded both the iSAP management plan and the end-of-semester examination question and evaluated the same content on the iSAP case for consistency (Figure 1).
As part of the standard quality improvement processes for the pharmacy degree and consistent with the Student Privacy Collection Statement, student performance data were collected, de-identified, summarized, and shared with the cohort. This study was approved by the Monash University Human Research Subjects Ethics Committee. The key data points collected were management plan rubric grades completed by the course examiner and students (their self-assessment), grades for an end-of-semester examination question that evaluated similar content to that of the iSAP case, and student comparative reports. The data points were then used to calculate the iSAP course examiner and self-assessment grades and determine the self-awareness score of each student.
The grading rubric included eight criteria (Appendix 1). To calculate the iSAP course examiner grade and self-assessment grade, the following criteria were excluded: academic integrity, rationale for recommendation, and overall evaluation. This was to enable a more meaningful comparison between the iSAP grade and the end-of-semester examination question grade, as students were not assessed on their rationale for recommendation, use of references, or overall competency in the end-of-semester examination question. The final calculation for the iSAP course examiner grade and self-assessment grade was the sum of the grades awarded for the remaining criteria on a scale of 100%.
To determine the self-awareness score, the difference between iSAP examiner grade and self-assessment grade for each rubric criterion was calculated. All criteria except for academic integrity were considered in the self-awareness score. Absolute values (used to avoid negative and positive values cancelling each other out) of these differences were summed, with 100 being the highest possible value. With values calculated in this manner, there is an inverse correlation, ie, a value closer to 0 signifies higher self-assessment accuracy. To produce a direct correlation, the value was taken away from 100, meaning that a value closer to 0 signifies lower self-assessment accuracy.
To determine the level of self-awareness for each student, a deviation of one point on the scale has been deemed acceptable in prior research.17,18 Since the majority of the rubric criteria had a four-point scale, a deviation of one point in the scale is a difference of, on average, 25%. Thus, an awareness score of 75% or greater was defined as high awareness, and a score of less than 75% was defined as low awareness.
Data analysis undertaken for RQ1 and RQ4 was in groups according to student academic performance. Given the Dunning-Kruger effect19 if analyzed as a whole, there is a probability that the overestimation in lower-performing students could negate the underestimation in higher-performing students, resulting in a misleading high level of accuracy. Students were categorized into tertiles, meaning there was an even distribution of students per group.
For research questions 2, 3, and 4, the association between self-awareness and academic performance was investigated using Pearson r and regression analysis. For research question 4, a one-way ANOVA was conducted to better visualize the data and obtain more information about the difference between each group of students. Following this, a linear regression test was performed to investigate the relationship between self-assessment accuracy and changes in performance of students in the iSAP assessment compared to a similar end-of-semester examination question.
The fifth research question (RQ5) involved analyzing reflective reports of students by thematic analysis. Because of the large quantities of reflections and the repetitive themes found in the pilot study, we reported the qualitative results of the 21 participants. This number was chosen based on a preliminary scan of reflective reports that estimated saturation of responses would be reached after approximately 20 reports. Thus, a random sample of seven reports from each tertile was selected, totaling 21 reports. An iterative and inductive coding process was adopted to derive themes of reasoning rationales and reasoning patterns. First, one coder read all the included reflections to create preliminary codes and develop a codebook, which was then reviewed by two other coders. Each reflection was coded with codes that described reasons given by students for their errors and the observed pattern of their self-assessment. Then, raw code patterns and themes across different comparative reports were combined and refined, and themes were identified and defined.
RESULTS
Of the 215 students enrolled in the therapeutics course in 2019, 162 (75%) were included in this study. Three students opted out of the study, one student was absent, 40 students did not provide a self-assessment, four students did not fill out the self-assessment rubric completely, and five student researchers were excluded. A summary of the statistical analysis of research questions 1, 2, 3, and 4 are included in Table 1.
Statistical Analysis of Research Questions Posed in a Study of Pharmacy Students’ Ability to Self-Assess Their Therapeutic Decision-Making Skills
For research question 1 regarding self-assessment accuracy, students were categorized into tertiles based on their performance in the iSAP assessment. The highest tertile (N = 55) had a self-assessed iSAP grade that was on average 19% below the grade given by the course examiner, suggesting a considerable underestimation of their ability. The mean difference appeared to be the lowest in the middle tertile (N = 53), comprised of students who underestimated their performance by an average of 7.7% compared with the grade given by the course examiner. The lowest tertile (N = 54) rated their performance an average of 12% higher than the course examiner grade, suggesting an overestimation of their ability. The self-assessment grades were widely distributed on the scale of 100, irrespective of the students’ academic performance, which indicates that students in each tertile had varying degrees of self-awareness (Figure 2).
Comparison of iSAP assessment marks from examiners and self-assessment of pharmacy students categorized into tertiles.
A Pearson correlation test showed that the correlation between self-assessment and course examiner grade for all students was poor (r = .27, p < .001). The lowest and middle tertiles demonstrated the same correlation coefficient (r = .18, p = .10). However, the highest tertile demonstrated a negative but insignificant correlation coefficient (r = −.08, p = .28).
The mean of course examiner and student self-assessment grades for each criterion in the iSAP assessment was calculated. There was a significant difference of 23% (p < .001), 9.3% (p = .002), and 11.3% (p < .001) between the means for Overall Evaluation of Clinical Decisions, Priority Identification, and MRP Identification, respectively, which indicated that students underestimated themselves in these criteria.
Research question 2 explored the relationship between self-assessment accuracy and academic performance on the iSAP case. A linear regression test showed a significant, weak positive correlation between awareness and academic performance (r = .22, p = .01). As the iSAP grade increased by 1.0%, awareness score increased by .15%. However, the R-squared value of .048 suggested a poor fit of the regression model, as only 4.8% of the variance in awareness was accounted for by the iSAP grade.
A linear regression test showed no relationship between awareness scores and academic performance on the subsequent end-of-semester examination question for research question 3 (r = .13, p = .11). Research question 4 explored the relationship between self-assessment accuracy and changes in students’ performance between the iSAP assessment and a similar end-of-semester examination question. The regression test showed no relationship (r = −.04, p value = .58). A one-way ANOVA was conducted to determine if there were any changes between the iSAP assessment grades and end-of-semester examination grades of students with low and high awareness. On average, the grades for the end-of-semester examination question were lower than those on the iSAP assessment by an average of 15%. In the lowest tertile, students with low awareness had a mean of +1.1, suggesting an improvement of 1.1% in grades; whereas those with high awareness had a decrease of 3.7% in grades. On the other hand, for the middle tertile, both the high awareness and low awareness groups had a decrease in grades of 25% and 16%, respectively. Similarly, for the highest tertile, the grades of students with high and low awareness were lower by 23% and 25%, respectively. However, an ANOVA showed no significant differences, indicating that the two self-awareness groups from all tertiles had similar changes in performance between the iSAP assessment and the end-of-semester examination.
For research question 5, students’ self-identified reasons for their performance on the iSAP, 21 self-reflections were included. Students reflected mostly on how they performed on medical problem list and patient management plan criteria of the grading rubric (Appendix 1). Seven students stated that their MRPs were correct, but almost all of them (n = 20) were aware of their clinical errors regarding the MRPs, dose and duration of therapy, and monitoring points. Upon analyzing student reflections, some patterns of self-assessment were identified. A proportion of students only reflected on their errors and did not comment on their success. In contrast, one student who grossly overestimated their abilities was unable to contrast their clinical plan with the expert response and only provided positive self-ratings. Various answers were given in the “So what” section, which prompted students to consider the potential patient outcomes of their recommendations. Most students only commented on the negative consequences, but one student, who had a self-awareness score of 91%, stated “we can expect good patient outcomes.” Interestingly, some students chose to justify their incorrect treatment options instead of commenting on the possible negative outcomes of their recommendations.
The themes relating to the iSAP performance are shown in Table 2. Five major themes and their corresponding subthemes were identified. Most themes related to their weaknesses, but the aforementioned student with a self-awareness score of 91% stated that it was a confidence boost. Furthermore, most reasons for iSAP performance were attributed to their own shortcomings. However, two external reasons that emerged were “time constraint” and “resources are not clear.”
Pharmacy Students’ Reflections on Their Performance on the Integrating Science and Practice (iSAP) Assessment
Grading Rubric for the iSAP Assessment and Student Self-Assessment
DISCUSSION
Using the innovative approach of the iSAP format, the self-assessment accuracy of students and the rationales behind their evaluations was explored, and the association between self-awareness and future examinations was investigated. In agreement with previous studies,9,12,20,21 self-assessment abilities of students in this study were indicative of the Dunning-Kruger effect19 where lower-performing students tended to overestimate themselves while higher-performing students tended to underestimate themselves. Additionally, our results demonstrated that overall there was a low correlation between the self-assessed and examiner grade.12,21 Self-assessment of clinical knowledge and skills is a progressive process that requires practice to develop.9
Despite the similarities mentioned, our findings did not entirely align with previous literature. For example studies conducted on third-year pharmacy students found that they were generally overconfident in self-evaluating their grades.8,11 Substantial research conducted on other populations, including pharmacy students from different year levels and medical students, also documented inflated self-assessment.9,22,23 However, overall, participants in the current study underestimated themselves. Furthermore, past studies indicated that more academically inclined students exhibit better self-awareness,9,11 while our findings showed a very weak and nonsignificant correlation between self-assessment accuracy and academic performance.
Based on previous research conducted on meta-awareness,24 we hypothesized that proficient self-assessment abilities may improve pharmacy students’ subsequent academic performance. However, our findings showed no significant differences in students’ grades pre- and post-test among all tertiles, irrespective of the students’ self-assessment accuracy. This is potentially because critiquing oneself is a skill that is developed over time. Given the limited follow-up opportunities in our study and a time of only three weeks between the self-assessment and the end of semester examination, a significant improvement in metacognitive ability is unlikely. Additionally, the average grade on the end-of-semester examination question was lower than the average iSAP grade, perhaps reflecting the more difficult nature of an end-of-semester examination worth a larger proportion of the course grade.
The strengths of this study include that it explored the relationship between self-assessment accuracy and future academic performance, of which there is currently minimal published data in the literature. Furthermore, it may be the first study to test if the iSAP format has an impact on student self-assessment abilities and future performance. Contrary to the commonly evident phenomenon of self-assessment skill overestimation in pharmacy students, this study showed that on average, pharmacy students in this study tended to underestimate their academic performance. Future studies can implement strategies such as engaging in reflective practice after workshops and soliciting feedback from peers or preceptors to boost their self-confidence and enable development of student self-assessment skills over time.
Limitations of this study include the lack of a control group. This was a consequence of the self-assessment and comparative report being compulsory as part of the course syllabus, and because this was the first time this course was offered. Another limitation was that study participants only accounted for approximately 75% of the third-year students at one pharmacy school, and only one follow-up assessment was used to compare academic performance. Therefore, not only larger scale studies, but also inclusion of more follow-up assessments and other pharmacy cohorts are needed to confirm the findings and improve their generalizability and reliability. Another limitation is that the relevance and reasoning of including the iSAP format was not outlined to students; therefore, students may not have had a vested interest in completing the intervention. Thus, the full potential of the study may not have been realized. The end-of-semester examination question was situated at the end of the paper, which may have influenced the ability for students to answer the question due to potential time constraints.
Future studies should investigate the relationship between self-assessment abilities and subsequent academic performance. Self-awareness and metacognition require continual practice with feedback in order to develop.11 Tracking of students throughout a PharmD program may determine how they develop self-awareness and improve their clinical decision making over time, or even after they complete their studies and are practicing pharmacists.
CONCLUSION
The pharmacy students in this study tended to underestimate their academic performance. Countermeasures to reduce self-assessment inaccuracies are essential to promote self-confidence. With further research on self-assessment, an understanding of student perceptions about their performance may improve education methods, such as inclusion of reflective practices after case-based activities, to enable students to practice self-reflection and thereby continue to improve their skills throughout their pharmacy career.
ACKNOWLEDGEMENTS
The authors acknowledge Professor Marilyn Baird and Dr. Allie Ford for their assistance in the creation of iSAP assessments, and the contributions of the following who assisted with this project as part of their pharmacy course requirements: Aisha Imam, Gia Pham, To Nhu Pham, Wan Zainina, and Xin Hui Tan.
- Received April 19, 2021.
- Accepted August 13, 2021.
- © 2022 American Association of Colleges of Pharmacy