Confirmation Bias as a Factor in Pharmacy Student Assessment of Research Study Quality

INTRODUCTION

Teaching students to evaluate drug literature is a fundamental component of pharmacy education programs. The Accreditation Council for Pharmacy Education requires that pharmacy programs demonstrate that their graduates can analyze and evaluate professional and scientific literature.¹ New pharmacy graduates should be able to evaluate research evidence and draw conclusions from the evidence to change or influence their clinical decision-making, a practice known as evidence-based medicine.

Studies have shown that reading medical literature can appropriately change the beliefs of expert readers based on the quality of the empirical evidence presented to them.² Second-year Doctor of Pharmacy (PharmD) students enrolled in a drug literature course, however, were not able to critically evaluate literature as well as expert researchers could, nor were their opinions as strongly changed by what they read in the research.³ The reason for this disparity was not clear as critical thinking skills, as assessed by the California Critical Thinking Skills Test, were not correlated with the students’ inability to evaluate literature like the experts. Medical students had a change in their beliefs about the use of a diagnostic cervical spine radiograph after reading an abstract explaining the importance of the results, but they were not influenced by the validity of the study.⁴

One reason people are reluctant to change opinions is confirmation bias, which is the proclivity to search out and favor information that aligns with one’s personal beliefs and theories. This bias has been shown to influence data collection and decision-making in many disciplines, including animal behavior research,^5,6 psychiatric diagnoses,⁷ criminal investigation,^8,9 and middle-school level science education.¹⁰

The phenomenon of confirmation bias has been seen in psychology literature.^11,12 Peer reviewers were more likely to accept research that aligned with their beliefs than that which differed, even when the methodology was exactly the same.¹¹ A similar study by Hergovich and colleagues noted that academic psychologists rated evidence qualitatively higher when the information presented aligned with their beliefs.¹²

Confirmation bias in pharmacy has been implicated in medication errors such as similarities in medication packaging^13-16 and look-alike/sound-alike drug names.^17,18 The influence of confirmation bias on the ability of pharmacists and pharmacy students to evaluate drug literature accurately is unknown. The thought processes of pharmacists and pharmacy students in evaluating drug literature are also largely unknown.

Because of the large body of evidence describing confirmation bias, we proposed that confirmation bias would interfere with the ability of pharmacy students to assess drug literature abstracts accurately. We approached this issue by examining if prior beliefs about commonly used drug products approved by the US Food and Drug Administration (FDA) and dietary supplements affected the ability of pharmacy students enrolled in a drug literature course to accurately analyze the quality of drug literature abstracts and use them for decisions. While confirmation bias can be tested in multiple ways, we felt this was a straightforward and relevant approach. The hypotheses tested were that prior student beliefs favor FDA-approved drug products over dietary supplements, and thus students will be more willing to recommend FDA-approved drug products than dietary supplements; students will rate and rank the same study design quantitatively and qualitatively higher if the subject studied is a product they had high prior belief in despite recognizing the quality of the study; and confirmation bias will make students more likely to recommend a drug product than a dietary supplement after reading an abstract, regardless of the quality of the study design.

METHODS

Seventy-six second-year PharmD students enrolled in a required three-credit drug literature evaluation course in fall 2016 were eligible for this study, which was administered during a regular 50-minute class period about two-thirds of the way through the course. At this point in the course, students were familiar with study design and had practiced evaluation of it though active-learning exercises. Students were unaware of the hypotheses being studied. The North Dakota State University Institutional Review Board approved the project, and students gave informed consent to participate in the study. Participation was voluntary and no course points were awarded to students for their contribution.

After signing a consent form to participate, all students completed an initial survey to determine their views on the four products that were studied in the abstracts they would be evaluating. We asked students if they would recommend each of the products to a patient based on their previous knowledge of and experience with the listed indications: butterbur for migraine prophylaxis, propranolol for migraine prophylaxis, elderberry for influenza treatment, and oseltamivir for influenza treatment. The response options on the survey instrument were “yes,” “no,” and “I don’t know.” This portion of the study was to determine whether students had preexisting impressions about the products.

Next, students were randomly assigned to receive one of two forms to evaluate four drug literature abstracts of varying quality and study design (Appendices 1 and 2). Students were instructed to answer the questions in order and not to write their names on any of the pages to maintain anonymity.

The abstracts used were about a small, placebo-controlled trial evaluating elderberry syrup for influenza treatment¹⁹ (abstract 1), a large placebo-controlled trial evaluating the use of oseltamivir for influenza treatment²⁰ (abstract 2), a small trial evaluating the use of propranolol with nortriptyline for migraine prevention²¹ (abstract 3), and a larger placebo-controlled randomized trial evaluating the use of butterbur for migraine prevention²² (abstract 4). The studies described in abstracts 2 and 4 were considered by the investigators to be of higher quality than those described in abstracts 1 and 3 because they involved larger numbers of patients, provided detailed results, and were randomized and double-blind. The study reported in abstract 1 was randomized but had fewer patients. Also, fewer details of the results of the study were given in that abstract than in abstract 2. Abstract 3 was considered the weakest as the study it described had the smallest sample and no placebo control, and it included the fewest details about the results.

Students randomized to group 1 were given exact versions of abstracts 1 and 2. Abstract 3 had the same wording as the original propranolol study except the word “propranolol” was replaced with “butterbur.” Abstract 4 on form 1 was the original butterbur study but the term “butterbur” was replaced with “propranolol.”

Students randomized to group 2 were given exact copies of abstracts 3 and 4. Abstract 1 had the term “elderberry syrup” replaced with “oseltamivir.” Likewise, abstract 2 had the word “oseltamivir” replaced with “elderberry syrup” (Table 1).

After reading each abstract, the students were instructed to list two strengths and two weaknesses of the study, rate the absolute quality of each study on a scale from 1 to 10 with 1 being the very worst quality to 10 being the very best quality. Next, the form asked students whether or not they would recommend the product for the indication evaluated in the abstract based on what they had read. Finally, the form asked students to rank the four studies in terms of relative quality from best (1) to worst (4). The students had the full class period to complete the study.

The chi-square test was used to evaluate response to initial bias. A Mann-Whitney U test was used for comparing ranks for difference in ratings between the groups. The Fisher exact test was used for proportions. Finally, the McNemar test for proportions was used for change in recommending a product after reading the abstracts. A p less than .05 was considered statistically significant in all cases. Responses from the strengths and weaknesses questions were compiled and analyzed by the investigators to identify major themes and keywords.

RESULTS

Fifty-nine second-year PharmD students (78% of the class) were present in class to complete the study questionnaire. The student demographics were as follows: 59% were female, 90% were white, mean age was 23 years, and 10% had a prior bachelor’s degree or higher. Thirty were assigned to group 1 and 29 to group 2.

There was no significant difference between the two groups on initial recommendations regarding butterbur, propranolol, elderberry syrup, or oseltamivir. Overall, students were most likely to initially recommend oseltamivir for its indication (53/59 would recommend), followed by propranolol (26/59). Only two of the students recommended use of elderberry initially, and no students were willing to recommend butterbur. The overall difference in students recommending an FDA-approved product vs a dietary supplement was significant (p<.0001), indicating an expected initial bias in favor of recommending FDA-approved drug products.

Addressing our second research question, the abstracts with the same study design and results but different products were compared by the 1 to 10 absolute quality rating given by the students (Table 2). Students consistently rated the same abstract higher if it described an evaluation of a drug product vs a dietary supplement by anywhere from a 0.2 to 0.6 difference on the 10-point scale. The contrast was greater for oseltamivir vs elderberry than for propranolol vs butterbur. However, when analyzed using a nonparametric test for comparison of medians, the differences were not significant. Thus, the results were not able to confirm hypothesis 2, that confirmation bias influenced the absolute quality rating of the study.

When qualitatively analyzing the abstracts, the students in each group picked out similar strengths and weaknesses. Students frequently used terms such as “randomized,” “placebo-controlled,” and “double-blind” to describe the strengths of the studies reported in abstracts 2 and 4. The size of the study was considered a weakness of the studies reported in abstracts 1 and 3. Obvious weaknesses included lack of blinding in the study reported in abstract 1 and no placebo control in the study reported in abstract 3. The students’ qualitative analysis was similar to what would be expected by knowledgeable readers, and was not affected by the type of product being studied.

Table 3 compared relative quality rankings for the same abstract when the identity of the product was switched. One student in group 1 and one student in group 2 did not provide answers for the quality ratings. The median rank (with 1 being best and 4 being worst) for each of the study designs is shown in Table 3. Students in groups 1 and 2 ranked the studies similarly. The studies reported in abstracts 2 and 4 were ranked the strongest, regardless of whether they discussed an FDA-approved drug or a dietary supplement, and the weakest study (abstract 3) was ranked as the least strong. However, those trends were stronger when students believed an FDA-approved drug had been studied. Although the data seem to indicate some confirmation bias, significant differences were present only in the elderberry vs oseltamivir comparison for abstract 1.

Abstracts 2 and 4 were rated and ranked higher than the weaker studies regardless of the product studied. Abstract 3, which reported on the study with small numbers, no placebo control, and vague results, was rated and ranked lowest.

To test hypothesis 3, we asked students if they would recommend the product they had just read about. Table 4 shows that for each individual abstract, the study was more likely to induce a positive recommendation from the student when the abstract discussed a drug product. However, the difference between the two versions of the abstract was only significant for the oseltamivir and elderberry pairing in abstracts 1 and 2.

Changes in students’ willingness to recommend a product before and after reading the abstract are reported in Table 5. This analysis represented how the quality of a study influenced students’ opinion of a product. In five of eight cases, student opinion about the product changed significantly.

Although students had a strongly favorable prior opinion of oseltamivir, the strong study described in abstract 2 did not change their opinion of the product, but the intermediate-quality study described in abstract 1 tended to make students’ opinions less favorable (ie, more switched from a yes to a no opinion than vice versa). For butterbur, students already had a strong negative bias, so the poor-quality of the study reported in abstract 3 did not change that opinion. However, the stronger study reported in abstract 4 strongly shifted their opinion to favorable. Students’ prior opinion about propranolol was moderately favorable, and the strong study described in the abstract significantly shifted students’ opinion to more favorable while the weak study shifted their opinions to more negative. Finally, students’ prior opinion regarding elderberry was negative and reading both abstracts shifted their opinion to favorable, but the stronger study reported in abstract 2 shifted it more strongly.

DISCUSSION

The study confirmed our first hypothesis that the students would be more willing to recommend the FDA-approved drug products than the dietary supplements described in the selected research abstracts. As for our second hypothesis, the study provided weak evidence that confirmation bias related to prior belief was at work when students evaluated intervention studies. Students ranked the quality of study abstracts a little higher if they described FDA-approved products rather than a supplement. However, students were more likely to recommend drug products than dietary supplements after reading an identical abstract regarding the product, which confirmed hypothesis 3.

Students gave consistently higher, but statistically insignificant, ratings to abstracts that described drugs versus supplements. The contrast was greater for oseltamivir vs elderberry than for propranolol vs butterbur, which is not surprising, as oseltamivir was the product they felt most confident about in initial recommendations.

Students gave correct weight to the quality of the study design, showing their ability to assess studies qualitatively and to rank the studies by quality regardless of the product studied. Thus, when analyzing abstracts, our students were influenced by some degree of confirmation bias, but this had less influence than their ability to appropriately analyze literature for application to practice.

Ranking of abstract quality was also correct. This is what should be expected from qualified evaluators. Thus, some confirmation bias may have modified the evaluation of the studies but did not change the students’ ability to distinguish between the strengths of studies.

Confirmation bias is common among both health professionals and the public, and therefore important for educators to address. Confirmation bias can be caused by numerous factors including anchoring bias and social conformity. We know that physicians who are given abstracts of drug studies interpret the studies’ value differently depending on whether they believe a study was supported by a drug company vs the National Institutes of Health.²³ Prior belief about a subject is a powerful influence on our processing of factual information.²⁴ While it is natural for people to try to make new information conform to their prior beliefs, health care professionals in particular need to be aware of this aspect of human nature at all times. Because confirmation bias occurs unconsciously, curricular approaches to addressing this concern need to focus on making students more aware of how they process information.

Our study helps to quantify the degree to which confirmation bias occurs and the circumstances in which it matters, but this topic should be further explored within literature evaluation courses and the broader curriculum. The Accreditation Council for Pharmacy Education’s accreditation standard 25.7 addresses clinical reasoning, and we believe confirmation bias deserves attention when this standard is addressed in the curriculum.

There are several limitations to this study. First, we studied second-year pharmacy students rather than practicing pharmacists. The pharmacy students may not have been familiar with certain products as neither propranolol nor the two herbal supplements had been covered in the curriculum up to that point. Also, the study on propranolol mentioned concurrent use with nortriptyline, which may have caused confusion. On the other hand, our students were concurrently enrolled in a course on self-care that covered dietary supplements and were familiar with oseltamivir from another concurrent course. For fourth-year pharmacy students or practicing pharmacists who are further removed from training and practice in study evaluation, the influence of confirmation bias might have a greater effect.

We only chose two drugs and two dietary supplement products for the study, and student opinions about these products may not be typical of drugs or dietary supplements in general. However, by asking students’ opinions of the products at the beginning of the study we determined bias in favor of the drug products as real. The specific products used in the study abstracts we selected should have been irrelevant as confirmation bias would be expected to occur for any product.

Finally, the study was carried out at a single school and the demographics and cultural attitudes toward supplements may differ from those of students at other schools. Another limitation was not having adequate numbers of students to give sufficient power to conduct some analyses. Additional studies with more participants and several schools of pharmacy are needed to confirm and expand our findings.

CONCLUSION

Confirmation bias had a minor influence on the way pharmacy students reacted to reading abstracts of clinical studies. The students correctly identified the superior quality studies after reading and comparing four abstracts, so while their assessment of the studies was tempered by confirmation bias, the bias did not overwhelm their assessment of study quality. Understanding and being aware of confirmation bias is critical for all practitioners, and explaining confirmation bias should be an important part of teaching literature evaluation in the pharmacy curricula.