Abstract
Objective. To review the literature related to potential associations between self-assessed perceptions of knowledge and/or confidence with actual competence.
Findings. Twenty-two articles involving a variety of disciplines, including undergraduate studies, dentistry, medicine, and pharmacy were included following the screening process. Most studies focused on student self-reported confidence rather than competency assessed by a formative examination. Only a handful of studies were centered on pharmacy education.
Summary. Educational research that evaluates student learning should employ measures of competency as the primary outcome rather than student perceptions. Using student perceptions as the primary measure of student learning should be avoided, but student perceptions may have some utility as an adjunct to competency data.
INTRODUCTION
Competency-based outcomes have long been the standard measure used to evaluate pharmacy and other curricular programs. As students progress through a Doctor of Pharmacy (PharmD) degree curricula, the goal is for them to attain, at a minimum, baseline knowledge and skills that will allow them to be effective professionals upon graduation. The Accreditation Council for Pharmacy Education (ACPE) publishes standards that must be met in order for schools and colleges of pharmacy to achieve and maintain accreditation status. These standards include key elements ranging from formative and summative assessments to outcomes for student achievement.1 Schools must be able to demonstrate that students have achieved a certain level of competency as they progress through PharmD programs.
Researchers have long employed measures of competence, such as pre- and posttests and examination scores, as endpoints in their evaluation of educational interventions.2 Some have begun to increasingly concentrate on students’ perceptions as a surrogate endpoint for educational research rather than on more traditional competence metrics. There may be several reasons for this phenomenon, including the relative ease with which perceptions data can be collected as well as an increasing emphasis within academia on student satisfaction. These investigations often focus on changes related to self-perceived knowledge of or confidence in a particular subject or task. In order to gain confidence, a learner must accumulate a certain level of presumed competence. Educators may feel more comfortable using traditional evaluations of student learning when determining the impact of a course. However, some may argue that student perceptions are important to consider when evaluating coursework. Many studies in higher education now consider both perceptions and more tangible data points (eg, pre- and post-intervention testing, examination scores, final course grades) when evaluating an intervention within a curricula.3-5 This has led some to question the reliability of student confidence and perceptions data when considered in isolation.6 We reviewed the available literature with regards to the potential pros and cons of using student perceptions as a surrogate endpoint for competence within education research.
METHODS
Studies were identified using the search databases PubMed and Google Scholar. Four hundred and eighty-four articles from 2000-2020 containing the terms of “student,” “competence,” “confidence,” and “correlation” were captured.
Screening involved identifying abstracts and articles that attempted to correlate competence and confidence. Articles were included if they involved a measure of graduate learner confidence compared against measured competence. Articles were excluded if they did not attempt to test correlation between these two variables in health professions students. A summary of the articles reviewed is provided in Table 1.
Summary of Published Papers Examining Correlations Between Confidence and Competence in Health Professions Students and Residents
FINDINGS
Evaluating Confidence
Some may contend that confidence is a valuable and meritorious trait that health care practitioners should possess. However, a unilateral relationship between confidence and competency is not well established. As studies appear to increasingly use perceptions data, it is fair to question the reliability of self-assessments of confidence. Dunning and Kruger observed that individuals with less understanding of a particular topic were more likely to overestimate their own abilities, while proficient individuals were able to more accurately discern their own competencies.7 Overconfidence is a bias that can be especially disruptive in health care. Hypothetically, an overconfident practitioner might deliver suboptimal patient care more often than not as they fail to recognize their incompetencies.
A 2016 study of 251 medical students enrolled in the Buenos Aires University School of Medicine evaluated student performance using a multiple-choice examination.8 Researchers assessed performance on the examination using objective outcomes data along with measures of student confidence. They found that 12% of students demonstrated overconfidence in their abilities. Furthermore, those that performed poorly on the examination displayed a higher incidence of overconfidence than those who performed well, while under-confidence was more prevalent in high-performing students.
Moreover, people often tend to attribute success to internal factors while failures are often attributed to external factors. This bias was detailed by Karpen in 2017 when he observed that people tend to overestimate their abilities when asked to conduct a self-assessment.9 If most populations tend to overestimate their ability to perform, one might reason that pharmacy students may also be biased in the perceptions of their own learning and knowledge.
Accreditation Council for Pharmacy Education (ACPE) Standard 4.1 focuses on self-evaluation, stating “the graduate is able to examine and reflect on personal knowledge, skills, abilities beliefs, biases, motivation, and emotions that could enhance or limit personal and professional growth.”1 As students’ progress through a PharmD curriculum they must learn to accurately reflect and self-assess. This is highlighted by the aforementioned Dunning-Kruger effect, which acknowledges that individuals who have yet to master a subject, such as matriculating pharmacy students, are not likely to be accurate self-evaluators.7 While this phenomenon has not been extensively examined in pharmacy students, the results are not likely unique to any particular cohort of learners. A 2017 meta-analysis reviewed studies that attempted to evaluate the possible relationship between physician self-assessments and their performance on an external measure (eg, an objective structured clinical examination [OSCE]).10 Thirteen of the 20 studies included in the analysis demonstrated that physicians were unable to reliably self-assess their knowledge, skills, or performances when compared to an objective evaluation of these performance measures. However, the analysis included only 20 studies and the studies used different comparisons and measures, which might have led to inconsistent conclusions when analyzed together. Correlations between students’ competence and confidence have not been extensively studied at the undergraduate level, however slightly more data are available with regards to professional programs.
Studies of Dental Students
A common goal of professional health care programs is for students to achieve a certain level of competence and self-confidence before entering practice. Because of this, professional schools have begun to evaluate potential links between the two; however, data for students in undergraduate health professions programs are lacking. A 2004 study conducted at the University of Peradeniya in Sri Lanka that involved 212 dental students found that students’ self-assessed confidence correlated well (p<.001, Kendall Tau rank correlation test) with exposure time to specific skills.11 The study followed five different student cohorts (n=212) and had students report their confidence on a three-point scale across 46 skills related to dental practice. As exposure to particular skills increased, so did students’ confidence ratings. The authors did not specifically evaluate relationships between student competence and confidence, but examined exposure time instead. A study by Stacey and colleagues observed that clinical teachers scored final-year dental students differently in regards to clinical competency than students scored themselves via self-assessment.12 Students and their clinical teachers completed a questionnaire that evaluated six aspects of clinical competency for each student from semester one to semester two.12 Student results were then compared directly to those from clinical instructors to evaluate any possible differences in each groups assessment of student competency. The study found that clinical instructors assigned much lower scores than the students gave themselves, suggesting that students were often overconfident about their abilities. This finding was corroborated by Emam and colleagues who followed 111 dental students at The Ohio State University.4 Students were enrolled in the second year of the dental curriculum within an internal medicine course that was directed by the Division of Oral and Maxillofacial Surgery (OMFS). The authors found that students, on average, overestimated their ability by 12.5% as compared to instructor feedback on a medical competency examination. A 2015 study from a dental school in Turkey found that the ability of second-year dental students to self-assess closely mirrored the findings reported by Dunning and Kruger.7,13 Students were asked to self-evaluate their performances as well as that of a classmates following preclinical examinations on tooth restorations.13 The student self-evaluation scores, classmate evaluation scores, and graded examination scores were then compared. Overconfidence was more prevalent in poor-performing students while under-confidence plagued higher-performing students. Interestingly, this study also evaluated third-year students and found that self-assessment scores were more consistent with examination scores. Nevertheless, this was a small cohort of 75 dental students and involved international students, making generalizations to US cohorts difficult.
Studies of Medical Students
Studies in medicine have not only involved medical students but also postgraduate trainees and practitioners. A retrospective study conducted in 2004 at Monash University evaluated 352 medical students in their final academic year. The researcher examined student attitudes towards clinical consultations and their confidence in performing them as well as 28 core skills associated with general practice consultation.14 Students were assigned a composite score that included their fifth- and sixth-year individual examination scores, case scores, and sixth year OSCE score. Students were also asked to complete pre- and post-assessment self-evaluations to compare to their objective scores. While attitudes toward a particular subject seemed to play a role, the study found that feeling confident was not associated with student performance. Research conducted by Brinkman and colleagues in 2015 at the Vrije Universiteit (VU) Medical Center in the Netherlands reinforced these findings.15 The study followed 403 fourth-year medical students who were given a formative examination covering several prescribing skills and also asked to self-assess their abilities. Self-reported confidence did not correlate well with individual results (r2= 0.2). This study’s response rate was a limiting factor in interpreting results as only 61% of eligible students participated.
Postgraduate trainees are expected to have a clearer understanding of content as they progress to the intern and residency phases of their medical training. Considering the Dunning-Kruger effect, one might assume that as students’ knowledge continues to grow, they would become more reliable evaluators in terms of their own self-efficacy and competence. The Department of Medical Education at University of Sydney first attempted to confirm this hypothesis in 2004.16 They evaluated 30 first year medical interns (PGY1) to assess their self-reported confidence and their observed competence across seven clinical skills.16 Barnsley and colleagues found that there was no correlation between self-reported confidence and competency assessment scores.16 Some may argue that physicians in postgraduate training programs may not yet qualify as “experts” and therefore may not be as well equipped to self-evaluate. This premise is supported by the findings of Ganni and colleagues who compared the self-assessments of 150 medical residents to expert evaluations of laparoscopic cholecystectomy procedures.17 The researchers found a strong correlation (r2> 0.5; p<.001) in all categories of assessment with the exception of “tissue handling.” These results seem to further confirm Dunning and Kruger’s findings.7 However, a more recent 2018 study from East Carolina University that followed 21 medical residents from PGY1 to PGY5 found no correlation between confidence and quality of anastomoses procedures (p=.9).19 While these results differ from those of Ganni and colleagues, the reliability of the study was limited by a very small sample size.17,18
Studies of Pharmacy Students
The earliest study investigating correlations between competence and confidence among pharmacy students was conducted in 2006 at the University of Colorado School of Pharmacy where 108 second year (P2) pharmacy students were queried regarding their knowledge of various topical areas.19 Specifically, the study assessed student confidence and competence regarding patients with hypertension and dyslipidemia and found that student knowledge decreased from pre-testing to post-testing, while self-confidence was consistently rated as moderate-high. This study was limited by its focus on a single cohort of students across a narrow topic area. A more recent study conducted in 2014 examined a cohort of students within an elective class in toxicology.20 The authors from South University observed 45 pharmacy students across two classes that enrolled in the elective and noted that, while there was no correlation between knowledge and confidence at baseline, there was moderate correlation (r2=0.63) upon completion of the course. Notably, this study only enrolled 45 participants. As mentioned previously, researchers have suggested that as students become more educated about a particular subject, their self-awareness of their knowledge and skills will likely increase and their perceptions may become more reliable. With this in mind, one could reason that senior pharmacy students would be better equipped to accurately self-assess their abilities than first year pharmacy students. A 2016 study conducted at South University investigated the experiences of 177 third year pharmacy students in an Advanced Cardiovascular Life Support (ACLS) simulation laboratory and found no improvements in students’ knowledge or confidence, and there was no correlation, pre- or post-experience (r2=−0.031 and r2=0.13, respectively), between knowledge and confidence.21
Persky and colleagues designed and conducted a study that compared pharmacy students’ self-knowledge against that of other college-aged students. They studied 277 student pharmacists who were between 18 and 25 years of age and from one of four pharmacy programs.22 The comparator group of college-aged students was recruited via Amazon’s Mechanical Turk (a crowdsourcing marketplace that allows tasks to be outsourced by businesses and conducted virtually while being compensated for their participation). Students were first asked to provide their perceptions of self-knowledge regarding four topics (bats, prions, ozone, and “sweet tracks”) and their confidence to perform well on a possible quiz on these topics. Students from both cohorts completed a pre-quiz prior to an instructional intervention and then a final post-instruction quiz to assess their learning. The researchers then analyzed the data to determine whether there were any correlations between student learning and self-perceptions of their learning. The majority of correlations between student learning and self-perceptions of learning were not significant. Post-quiz scores did have a weak but significant correlation (rs<0.4) to student’s self-assessed perceptions of their learning. The topics assessed here were generalized and not particular to pharmacy in order to improve generalizability. However, consideration might be given to a study design that incorporates topics with which pharmacy students are more familiar, such as pharmacotherapy and pharmacology, to more reliably measure pharmacy student learning and metacognition.
DISCUSSION
In a time of curricular shift in pharmacy education, educators can not afford to undertake pedagogic interventions that fail to significantly impact student knowledge. Perceptions data appear to be limited in terms of providing significant correlations regarding a student’s ability to meet competency expectations, a crucial component of ACPE’s first four standards for pharmacy programs.1 Improvements in students’ confidence likely do not reflect the quality of most interventions. If students are not learning but feel as though they are learning, the result might lead them as well as educators to reach unfounded conclusions with regards to the quality and effectiveness of various interventions. Therefore, researchers should avoid relying on perceptions data alone as a surrogate for more reliable measures of student competency. Researchers intent on including perceptions data should make an effort to triangulate their findings with outcomes data to provide a more comprehensive picture. When designing studies aimed at assessing the impact of educational interventions, researchers should aspire to include competency or knowledge-centric data as the primary outcome used to evaluate the value of such projects. If students and professionals struggle to accurately evaluate themselves, questions may be raised as to the validity of using these measures as markers of competence or knowledge attainment.
This review has several limitations, most of which are related to the lack of health professions education articles in the literature that fit our search criteria. There were limited data available directly examining correlations between confidence and competence. Without a plethora of data, it was not possible to arrive at definitive conclusions. Additionally, many studies were from international sources or from health care professions not specific to pharmacy. While this improves the generalizability of the data to students across higher education, these programs may be inherently different from pharmacy education programs. Finally, many of the studies included in this review had fairly small sample sizes and/or limited diversity within the sample in terms of experience. The majority of studies focused on one class or one specific cohort of students in a given program and often lacked comparator groups to determine whether the interventions made had significantly different results from before its inclusion in a given curriculum. Future studies should include a preintervention comparator groups to give the results more context.
CONCLUSION
Student competence remains a critical element in evaluating educational interventions. This holds true across many types of professional education. While student confidence should continue to be evaluated, it should be accompanied by more traditional learning outcomes data that allow interventions to be placed into perspective. As students progress through curricula, their knowledge base and ability to self-assess should naturally improve. As they improve as self-evaluators, the reliability of the data generated should improve with it. Professional students may be at a crossroads where they are not competent enough at baseline (and even postgraduation in some fields) to accurately and reliably self-evaluate. Because of this, educators should strongly reconsider the use of confidence data as the primary measure when evaluating learning outcomes.
- Received September 22, 2020.
- Accepted January 5, 2021.
- © 2021 American Association of Colleges of Pharmacy
REFERENCES
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.