Skip to main content

Main menu

  • Articles
    • Current
    • Early Release
    • Archive
    • Rufus A. Lyman Award
    • Theme Issues
    • Special Collections
  • Authors
    • Author Instructions
    • Submission Process
    • Submit a Manuscript
    • Call for Papers - Intersectionality of Pharmacists’ Professional and Personal Identity
  • Reviewers
    • Reviewer Instructions
    • Call for Mentees
    • Reviewer Recognition
    • Frequently Asked Questions (FAQ)
  • About
    • About AJPE
    • Editorial Team
    • Editorial Board
    • History
  • More
    • Meet the Editors
    • Webinars
    • Contact AJPE
  • Other Publications

User menu

Search

  • Advanced search
American Journal of Pharmaceutical Education
  • Other Publications
American Journal of Pharmaceutical Education

Advanced Search

  • Articles
    • Current
    • Early Release
    • Archive
    • Rufus A. Lyman Award
    • Theme Issues
    • Special Collections
  • Authors
    • Author Instructions
    • Submission Process
    • Submit a Manuscript
    • Call for Papers - Intersectionality of Pharmacists’ Professional and Personal Identity
  • Reviewers
    • Reviewer Instructions
    • Call for Mentees
    • Reviewer Recognition
    • Frequently Asked Questions (FAQ)
  • About
    • About AJPE
    • Editorial Team
    • Editorial Board
    • History
  • More
    • Meet the Editors
    • Webinars
    • Contact AJPE
  • Follow AJPE on Twitter
  • LinkedIn
Research ArticleResearch Articles

Pharmacy Students' Knowledge of Black Box Warnings

Karen E. Moeller, Theresa I. Shireman, Joyce Generali, Sally Rigler and Angela Mayorga
American Journal of Pharmaceutical Education February 2010, 74 (1) 5; DOI: https://doi.org/10.5688/aj740105
Karen E. Moeller
aDepartment of Pharmacy Practice, University of Kansas Medical Center
bDepartment of Psychiatry and Behavioral Sciences, University of Kansas Medical Center
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Theresa I. Shireman
cDepartment of Preventive Medicine & Public Health, University of Kansas Medical Center
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Joyce Generali
aDepartment of Pharmacy Practice, University of Kansas Medical Center
dDrug Information Center, University of Kansas Medical Center
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sally Rigler
eDepartment of General and Geriatric Medicine, University of Kansas Medical Center
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Angela Mayorga
bDepartment of Psychiatry and Behavioral Sciences, University of Kansas Medical Center
eDepartment of General and Geriatric Medicine, University of Kansas Medical Center
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • PDF
Loading

Abstract

Objective. To evaluate the progression of pharmacy students’ knowledge of black box warnings across 3 years of didactic training, and to determine how they stay current with new warnings.

Methods. A cross-sectional survey instrument was administered to pharmacy students in their first (P1), second (P2), and third (P3) professional years. The survey assessed student awareness of medications possessing a black box warning and familiarity with the warning content for 20 medications (15 with and 5 without warnings).

Results. Mean number of correct responses identifying the presence or absence of a black box warning among the 20 medications were 5.8 ± 3.3, 9.6 ± 4.0, and 14.8 ± 2.8 for the P1, P2, and P3 students, respectively. Knowledge of black box warning content was variable. Students were least aware of the warning content for stavudine and enoxaparin. Students were most familiar with the warning content for paroxetine and estrogen.

Conclusion. Students' awareness and understanding of black box warnings was proportional to their educational progression, but their knowledge level was inconsistent across drug groups.

Keywords:
  • black box warning
  • pharmacy students
  • adverse drug effects
  • product label
  • package insert

INTRODUCTION

Black box warnings are the most serious warnings imposed by the Food and Drug Administration (FDA) for prescription medications and highlight potentially fatal, life-threatening, or disabling adverse effects for prescription drugs.1 In addition, black box warnings may include information regarding restriction of use and/or distribution of medications. The FDA mandates that black box warnings be separated and highlighted from the other text in the package insert and typically are characterized with a black box border.

Most black box warnings are not the result of controlled clinical trials but often the result of postmarketing surveillance.2 Lasser and colleagues found that most warnings were imposed within 7 years of a drug's introduction to the market.3 Their study also found of the 548 new medications introduced from 1975–1999, 10% received a new black box warning or were withdrawn from the market. They projected that the rate of warnings for new medications in the next 25 years would approach 20%.

Wagner et al conducted a retrospective study using claims data from 10 different health plans in the United States to assess the prescribing frequency for 216 medications with black box warnings, and to evaluate adherence with the warning's recommendations.4 Medications with a black box warning had been prescribed to more than 40% of the patients. Additionally, prescribers' adherence rate to the warning criteria ranged from < 1% to 50% depending on the medication.

Currently there are approximately 350 drug entities with a black box warning.5 With this large number of medications, it is difficult for clinicians to stay knowledgeable regarding the recommendations given in warnings. How much pharmacy students learn about black box warnings and how they stay current with new warnings once they are in practice is not known. The goals of this study were to evaluate the progression of pharmacy students' knowledge of black box warnings across 3 years of didactic training, and to inquire how they stay current with new warnings.

METHODS

A cross-sectional survey instrument was administered to pharmacy students in their first (P1), second (P2), and third (P3) professional years at the end of the spring 2007 semester. The survey instrument assessed students' awareness of medications possessing a black box warning and familiarity with the warning content for 20 medications (15 with and 5 without).

The survey instrument was developed as a collaborative effort by faculty members in the schools of pharmacy and medicine. Medications were chosen to represent a wide range of indications and organ systems, low and high frequency of use in generalist practice, brand and generic medications, prescription-only and nonprescription medications, number of years on the market, and medications with and without recent publicity about the addition of black box warnings or serious adverse effects. Eighty percent of the medications on the survey originated from the top 200 branded or generic drug lists.6,7 The primary outcomes of the survey were students' awareness of medications possessing a black box warning (yes or no response) and their familiarity with the warning content (using free-text response). Respondents were asked their current year of training, and to self-report how they stay current with black box information. Students were allowed 10 to 15 minutes to complete the survey instrument at the end of class and were not allowed to use any reference materials or discuss the survey instrument with other students. This study was approved by the University of Kansas Medical Center Human Subjects Committee.

Free-text responses for the reason for the black box warning were judged by 2 independent raters, each of whom classified the responses as correct or incorrect. Kappa scores for interrater reliability were accessed for each medication and ranged from 0.8–1.0, with most drugs demonstrating complete agreement regarding correctness of the reported reason for the black box warning. Afterwards, any remaining areas of discrepancy were adjudicated and consensus was reached. Descriptive statistics were obtained, including percentage of correct responses for each individual drug, the mean percent correct for presence/absence of a black box warning for all 20 drugs, and the mean percent correct for the reasons cited for each of the 15 drugs carrying warnings. Analysis of variance was used to compare total scores across the P1, P2, and P3 students. Statistical significance was defined at a p < 0.05 level. Data was analyzed using Excel and SPSS (SPSS, Inc, Chicago, Illinois) for Windows.

RESULTS

An 82% survey response rate was observed (230 of 278 students; P1 = 50/72, P2 = 77/103, P3 = 103/103). The remainder of the students opted not to complete the survey form or returned it without answers. The ability to identify correctly the presence or absence of a black box warning on the list of 20 medications increased in association with years of education. Mean (± SD) number of correct responses identifying the presence or absence of a black box warning was 5.8 ± 3.3, 9.6 ± 4.0, and 14.8 ± 2.8 for the P1, P2, and P3 students, respectively (p < 0.05). Correct responses by professional year and specific medications are detailed in Figure 1. More than 90% of the P3 students correctly identified amiodarone, estrogen, infliximab, and paroxetine as medications with a black box warning. The only medication with less than a 50% accuracy rate for the presence of a warning in the P3 population was metformin, with a correct response rate of 36%.

Figure 1
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 1

Percentage of pharmacy students by curricula year correctly stating whether the labeling of specific medications contains a black box warning.

For the 15 medications carrying black box warnings, students' knowledge of the reason for the warning varied across drugs but correct responses increased with advancing year of education (Figure 2). Students were least aware of the black box warning content for stavudine (P1 = 0%, P2 = 4%, P3 = 5%), enoxaparin (P1 = 0%, P2 = 1%, P3 = 8%), and olanzapine (P1 = 0%, P2 = 0%, P3 = 12%). In contrast, students were most familiar with the warning content for estrogen (P1 = 0%, P2 = 33%, P3 = 71%), paroxetine (P1 = 0%, P2 = 46%, and P3 = 78%), and warfarin (P1 = 6%, P2 = 35%, and P3 = 71%). Additionally, many free-text answers which were scored as incorrect did mention an important drug safety concern, but not necessarily one specifically provided in the black box warning.

Figure 2
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 2

Percentage of pharmacy students by curricula year correctly identifying the reason for specific medication's black box warnings.

Self-reports about methods for staying current with black box warning information were diverse. Unfortunately, most students responded that they did not keep up with the literature (P1 = 92%, P2 = 62%, P3 = 77%). Web sites (eg, lexi-comp, FDA), e-mails from organizations, and professional journals or magazines were the most commonly identified responses for staying current. Media (eg, television or news shows), work, and school were also identified as sources of information.

DISCUSSION

Pharmacy students' knowledge of black box warnings improved significantly as they progressed through the curriculum. At our institution, the classroom instruction on black box warnings currently occurs in the P3 year. It consists of one 2-hour lecture in the drug information/biostatistics class, accompanied by 2 examinations in the pharmacy skills laboratory class. The lecture consists of reviewing the history, definition, FDA criteria, and types of information included in black box warning. Additionally, the lecture focuses on limitations of black box warnings and the use of warnings as a communication tool. Last, medications with black box warnings are discussed in the lecture.

As expected, low knowledge levels about warnings were found among P1s. P2 students, however, showed knowledge gains compared to P1s, even though formal classroom introduction on this topic did not take place until the P3 year. This suggests that informal exposure is occurring as students progress toward their P3 year, and that some learning on this topic is already taking place. Earlier formal classroom instruction on this topic might enhance learning outside the classroom. In addition, our students are now being exposed early in the curriculum to pharmacy practice (eg, introductory pharmacy practice experiences), thus some knowledge of black box warning issues prior to initiating pharmacy practice experiences would be beneficial.

Recent revisions or additions of black box warnings significantly impacted students' knowledge of them. Paroxetine, warfarin, and estrogen were recognized most consistently as drugs that possessed black box warnings and were recognized most accurately for warning content. These prescription medications all had black box warnings introduced or modified within 5 years prior to the survey. Recent publicity (eg, news coverage) about warning additions may have been the reason why students were able to identify the warnings. For example, not long before the survey was administered, outcomes of hormone replacement therapy from the Women's Health Initiative had been widely discussed in the media, as had studies demonstrating an association between antidepressants and suicide in young people. Previous studies had shown that highly publicized warnings make an impact on prescriber's behavior as opposed to less publicized material.8

The scoring system required that students give the correct reason for the black box warning, and citation of other important safety concerns were not credited with a correct score even if they were important information for pharmacists. For example, olanzapine's warning about the risk of diabetes, weight gain, and hyperlipidemia had been publicized in the media near the time that this survey was administered, and that media coverage may have influenced learners. Only 12% of the P3 students correctly identified the reason for olanzapine's black box warning (increased mortality in older patients with dementia-related psychosis); many students indicated olanzapine's warning as new-onset diabetes or weight gain. The scoring system underrepresented students' overall knowledge of drug safety issues for various medications because they were not given credit for knowledge of legitimate non-black box warnings.

The study also found that a majority of the pharmacy students reported not staying current with new black box warning releases or additions. To what extent this reflects registered pharmacists' awareness of black box warnings is not known. Mailings to prescribers have little impact on prescribers' behavior. 9–11 Several venues for black box warning information include the FDA Web site and a free Web site dedicated to providing a list of medications with warnings.5 The identification of effective methods to disseminate information regarding black box warnings requires further study.

A major limitation of this study was that it was conducted at only 1 institution. While this institution has a faculty member who was a recognized expert in black box warning-related issues, and who releases hospital-wide notifications on new warnings, additional studies should be done at other institutions to determine whether these results are generalizable. Other schools of pharmacy may introduce black box warnings at different points in the curriculum, potentially producing different results. Another limitation is that this survey instrument was not validated.

Currently the study of black box warnings is not mentioned in the Accreditation Council for Pharmacy Education (ACPE) Accreditation Standards and Procedures; however, this may be assumed under the umbrella topic of medication safety.12 All pharmacy schools should offer formal training to their students in the area of black box warnings. Surveying registered pharmacists would help to determine how practicing pharmacists stay current with new black box warning information.

CONCLUSION

Our cross-sectional survey of 20 medications (15 with and 5 without black box warnings) administered to P1, P2, and P3 students determined that pharmacy students' awareness and understanding of black box warnings was proportional to their educational progression; however, knowledge deficits remain. Early training in black box warnings is warranted in pharmacy schools' curricula. Discovering effective ways to educate pharmacy students and other health care professionals, and helping them stay current with new black box warnings once in practice, is vital to improving patient quality of care and prescription medication safety.

  • Received June 5, 2009.
  • Accepted August 2, 2009.
  • © 2010 American Journal of Pharmaceutical Education

REFERENCES

  1. 1.↵
    21 Code of Federal Regulations paragraph 201.57(e) (2000).
  2. 2.↵
    1. Beach JE,
    2. Faich GA,
    3. Bormel FG,
    4. Sasinowski FJ
    Black box warnings in prescription drug labeling: results of a survey of 206 drugs Food Drug Law J. 1998 53 (3) 403 411
    OpenUrlPubMed
  3. 3.↵
    1. Lasser KE,
    2. Allen PD,
    3. Woolhandler SJ,
    4. Himmelstein DU,
    5. Wolfe SM,
    6. Bor DH
    Timing of new black box warnings and withdrawals for prescription medications JAMA. 2002 287 (17) 2215 2220
    OpenUrlCrossRefPubMed
  4. 4.↵
    1. Wagner AK,
    2. Chan KA,
    3. Dashevsky I,
    4. et al.
    FDA drug prescribing warnings: is the black box half empty or half full? Pharmacoepidemiol Drug Safety. 2006 15 369 386
    OpenUrlCrossRefPubMed
  5. 5.↵
    Generali JA. Black box drug list. www.formularyproductions.com/blackbox. Accessed January 12, 2010.
  6. 6.↵
    Top 200 brand drugs by retail dollars in 2007. Drug Topics. March 10, 2008. http://drugtopics.modernmedicine.com/drugtopics/data/articlestandard//drugtopics/102008/500221/article.pdf. Accessed January 12, 2010.
  7. 7.↵
    Top 200 generic drugs by retail dollars in 2007. Drug Topics. March 10, 2008. http://drugtopics.modernmedicine.com/drugtopics/data/articlestandard//drugtopics/102008/500218/article.pdf. Accessed January 12, 2010.
  8. 8.↵
    1. Weatherby LB,
    2. Nordstrom BL,
    3. Fife D,
    4. Walker AM
    The impact of wording in “Dear doctor” letters and in black box labels Clin Pharmacol Ther. 2002 72 (6) 735 742
    OpenUrlCrossRefPubMed
  9. 9.↵
    1. Cluxton RJ Jr.,
    2. Li Z,
    3. Heaton PC,
    4. et al.
    Impact of regulatory labeling for troglitazone and rosiglitazone on hepatic enzyme monitoring compliance: findings from the state of Ohio medicaid program Pharmacoepidemiology Drug Safety. 2005 14 (1) 1 9
    OpenUrlPubMed
  10. 10.↵
    1. Guo JJ,
    2. Curkendall S,
    3. Jones JK,
    4. Fife D,
    5. Goehring E,
    6. She D
    Impact of cisapride label changes on codispensing of contraindicated medications Pharmacoepidemiology Drug Safety. 2003 12 (4) 295 3
    OpenUrlPubMed
  11. 11.↵
    1. Smalley W,
    2. Shatin D,
    3. Wysowski DK,
    4. et al.
    Contraindicated use of cisapride: impact of food and drug administration regulatory action JAMA. 2000 284 (23) 3036 3039
    OpenUrlCrossRefPubMed
  12. 12.↵
    Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree, Accreditation Council for Pharmacy Education. www.acpe-accredit.org/pdf/ACPE_Revised_PharmD_Standards_Adopted_Jan152006.pdf. Accessed January 12, 2010.
PreviousNext
Back to top

In this issue

American Journal of Pharmaceutical Education
Vol. 74, Issue 1
1 Feb 2010
  • Table of Contents
  • Index by author
Print
Download PDF
Email Article

Thank you for your interest in spreading the word on American Journal of Pharmaceutical Education.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Pharmacy Students' Knowledge of Black Box Warnings
(Your Name) has sent you a message from American Journal of Pharmaceutical Education
(Your Name) thought you would like to see the American Journal of Pharmaceutical Education web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
7 + 6 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Pharmacy Students' Knowledge of Black Box Warnings
Karen E. Moeller, Theresa I. Shireman, Joyce Generali, Sally Rigler, Angela Mayorga
American Journal of Pharmaceutical Education Feb 2010, 74 (1) 5; DOI: 10.5688/aj740105

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Pharmacy Students' Knowledge of Black Box Warnings
Karen E. Moeller, Theresa I. Shireman, Joyce Generali, Sally Rigler, Angela Mayorga
American Journal of Pharmaceutical Education Feb 2010, 74 (1) 5; DOI: 10.5688/aj740105
Reddit logo Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • INTRODUCTION
    • METHODS
    • RESULTS
    • DISCUSSION
    • CONCLUSION
    • REFERENCES
  • Figures & Data
  • Info & Metrics
  • PDF

Similar AJPE Articles

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Science of Safety Topic Coverage in Experiential Education in US and Taiwan Colleges and Schools of Pharmacy
  • The Capacity Ratio as a Measure of Solvency in Experiential Education
  • Reliability and Credibility of Progress Test Criteria Developed by Alumni, Faculty, and Mixed Alumni-Faculty Judge Panels
Show more Research Articles

Related Articles

  • No related articles found.
  • Google Scholar

Keywords

  • black box warning
  • pharmacy students
  • adverse drug effects
  • product label
  • package insert

Home

  • AACP
  • AJPE

Articles

  • Current Issue
  • Early Release
  • Archive

Instructions

  • Author Instructions
  • Submission Process
  • Submit a Manuscript
  • Reviewer Instructions

About

  • AJPE
  • Editorial Team
  • Editorial Board
  • History
  • Contact

© 2023 American Journal of Pharmaceutical Education

Powered by HighWire