Abstract
Objective. To develop a brief instrument for academic pharmacists or physicians to use in assessing postgraduate residents’ knowledge of polypharmacy.
Methods. Five clinicians used a modified Delphi process to create a 26-item multiple-choice test to assess knowledge of polypharmacy in geriatric primary care. The test was distributed to 74 participants: 37 internal medicine (MD) residents, six nurse practitioner (NP) residents, nine primary care attendings, 12 pharmacists and pharmacy residents, and 10 geriatrics attendings and fellows. Construct validity was assessed using factor analysis and item response theory. Overall group differences were examined using a Kruskal-Wallis test, and between group differences were assessed using the Wilcoxon rank sum test.
Results. The response rate for the survey was 89%. Factor analysis resulted in a one factor solution. Item response theory modeling yielded a 12-item and six-item test. For the 12-item test, the mean scores of geriatricians and pharmacists (88%) were higher than those of MD and NP residents (58%) and primary care attendings (61%). No differences were found between MD and NP residents and primary care attendings. Findings for the six-item test were similar.
Conclusion. Both the 12-item and six-item versions of this polypharmacy test showed acceptable internal consistency and known groups validity and could be used in other academic settings. The similar scores between MD and NP residents and primary care attendings, which were significantly lower than scores for pharmacists and geriatricians, support the need for increased educational interventions.
INTRODUCTION
Polypharmacy, defined as the use of multiple medications or inappropriate medications, is common in older adults and is associated with adverse health outcomes, even after adjusting for comorbid illness.1-3 In postgraduate education, dedicated outpatient training in addressing polypharmacy is limited. However, as the need for enhanced training in geriatrics for future primary care providers is increasingly recognized,4,5 there is opportunity for greater educational emphasis on addressing polypharmacy.
Deprescribing, the process of safely reducing medications that may be causing harm or no longer providing benefit, can be quite complex in older adults with polypharmacy. Therefore, an interprofessional approach to education and clinical practice is needed. As is occurring in all health professions, academic pharmacists are increasingly being called on to educate both within and across disciplines and in interprofessional settings. Pharmacists are likely to be leaders or expert collaborators in educational interventions related to polypharmacy in any setting. However, academic physicians must also recognize polypharmacy and address the problem with trainees at the point of care, as well as model collaborative practice with pharmacists in efforts to improve medication safety in geriatric primary care. An instrument designed to assess polypharmacy knowledge may aid in defining knowledge gaps and measuring improvements with such educational interventions in various settings including resident training programs in pharmacy, medicine, and nursing, graduating students in these fields preparing for residency, and continuing education for providers involved in medication management in older adults. While several validated tests to assess general geriatrics knowledge in medical residents exist,6-8 to our knowledge, there is no validated test for assessing knowledge specific to polypharmacy and deprescribing in geriatric primary care.
In this study, we aimed to develop a brief instrument that could assess internal medicine (MD) residents’, nurse practitioner (NP) residents’, and primary care attendings’ knowledge of polypharmacy, particularly that pertaining to optimal medication management for community-dwelling older adults. We performed validity assessments, with pharmacists and geriatricians viewed as experts, to determine how effectively the instrument could be used to identify educational needs.
METHODS
A polypharmacy clinic has been in operation since 2013 in the Veterans Affairs Connecticut Center of Excellence in Primary Care Education (COEPCE). The COEPCE is an innovative, interprofessional, team-training program that provides the setting and postgraduate trainees for the clinic. The emphasis for the trainees, which include MD residents, NP residents, pharmacy residents, and health psychology fellows, is on learning about medication safety in older adults, eliciting patient perspectives and values related to their medical management, systematically reviewing a medications list with a patient, generating a safe deprescribing plan through shared decision-making, and counseling patients with a goal of optimizing medication safety and adherence. Following a successful one-year pilot of this polypharmacy clinic, the authors realized that an instrument designed to assess trainee knowledge of polypharmacy would be a helpful way of assessing trainees who participate in the clinic, and could be used similarly in other settings of polypharmacy education.
Approval from the Veterans Affairs Connecticut Research and Development Committee was obtained for the study protocol and was considered exempt by the Yale University Human Investigation Committee. To create this instrument, a team of five clinicians experienced in addressing polypharmacy in the primary care setting, including two internists, one geriatrician, and two geriatric pharmacists, created a multiple-choice knowledge test comprised of 26 questions. The test was intended to cover only one factor, knowledge of polypharmacy, with an emphasis on optimal medication management for older adults in the primary care setting. To ensure completeness in the range of medication classes represented in the instrument, the American Geriatrics Society Beers Criteria9 and STOPP/START criteria10 were reviewed and questions were added as appropriate. A consensus for the pre-validation set of 26 questions was arrived at through a modified Delphi process11,12 that included iterative rounds of discussion. Team members reviewed primary and secondary literature and discussed disagreements until consensus was achieved. Subsequently, one item was eliminated because it differed from new recommendations from a federal agency related to safe medication disposal.13 The objective for each question is listed in Table 1.
Objectives for Each Test Item
The pre-validation test was given to MD and NP residents and faculty members in primary care, attendings and fellows in geriatric medicine, and preceptors and residents in pharmacy. The heterogeneous sampling was intended to capture participants with a wide spectrum of knowledge related to polypharmacy, and to allow for known groups comparisons among the following four groups: MD and NP residents, primary care attendings, pharmacists, and geriatricians. For the purposes of this study, pharmacists and pharmacy residents were combined into one group and geriatricians and geriatrics fellows were combined into one group because they were all considered content experts.
To accommodate the schedules of test takers from multiple professions, one of three authors administered the test during asynchronous, planned learning sessions, or meetings. The test was anonymous, optional, and untimed. There were no incentives for taking the test or consequences for not taking it. Test takers were asked to remain quiet during the test and not consult references or discuss test items with other individuals.
Construct validity was assessed using both factor analysis and item response theory (IRT) modeling. Factor analysis is a statistical method that allows the determination of how many domains (factors) are covered by a set of test items. This is done by designating one or more factors to describe the source of covariation among test items. The authors hypothesized that items developed would test a participant’s knowledge of polypharmacy. In this example, knowledge of polypharmacy was considered a factor, and this factor was thought to describe most of the covariation among test items. Factor analysis was used to confirm this hypothesis by assigning an increasing number of factors and determining whether adding each factor explains enough additional covariance to warrant retaining that factor in the model. Parallel analysis was used to confirm statistically by comparing randomly generated data averaged over 1,000 iterations. For a thorough guide to factor analytic methods, the reader is referred to Chapter 6 of the text by DeVellis and colleagues.14
Applying IRT then allowed for an examination of how well each test item distinguished participants with respect to their ability level. This information was then used as described below to guide test item selection. The use of IRT to aid in instrument validation in this study differs from the approach of prior studies that used classical test theory to validate instruments pertaining to geriatrics knowledge.6 There are key advantages to using IRT over the alternative of a simple measure of internal consistency such as Cronbach coefficient alpha. Applying IRT modeling entails analyzing each test taker and each test item individually, not assuming uniformity with respect to either component. This allows for the selection of test items generating the highest amounts of information over the desired ability range. In contrast, Cronbach alpha measures intercorrelations among test items, not accounting for variety of test item difficulty or test-taker ability that might influence internal consistency. Our use of item information integrals to rank test items and visualization of effect sizes of tests of varying lengths allowed for an objective approach in selecting the number and identity of test items for the validated survey instrument.
Applying classical test theory, we additionally provided measures of Cronbach alpha to allow for comparison with prior studies. Notably, Cronbach alpha is known to underestimate reliability when the number of test items is low.15 Given our aim to design a test that was both relatively brief and able to optimally distinguish between ability levels, classical test theory methods of scale development like Cronbach alpha would have likely been insufficient and warranted the additional use of factor analysis and IRT as outlined.
Specifically, factor analysis was performed for the 25 items using SPSS Statistics, Version 21.0 (IBM Corporation, Armonk, NY). Retained factors were determined using parallel analysis with a custom script written in MATLAB (The MathWorks, Inc, Natick, MA), where magnitude of the eigenvalue for the last retained factor exceeded an eigenvalue averaged from random data generated 1,000 times for an equal number of test items and participants.14 A factor’s eigenvalue represents the amount of information (in this case, number of test items) accounted for by that factor.
The remainder of the statistical analyses was conducted using MATLAB and Statistics Toolbox, Release 2015a. IRT modeling was performed with MATLAB IRTm toolbox to assist in test item selection for the same 25 items.16 Test data were fit using a two parameter logistic model consisting of item difficulty and discrimination. Item information curves (IICs) were derived from the item difficulty and discrimination parameters, effectively showing the capability of each test item to distinguish test takers at each ability level.
Additional steps were taken to determine which items and how many items to retain in the validated test. First, the area under the curve for each IIC was calculated by integrating each IIC over a range of ability levels from -1 to 1, which covered the majority of participants’ ability ranges, particularly that of the MD and NP residents. Next, the 25 test items were ranked from highest to lowest quality using the item information integrals, with a greater integral value indicating a higher quality test item. Sequentially longer test versions were constructed by starting with the highest ranked item and then adding the next highest ranked item for each consecutive test version. The ability of each test version to distinguish between the four testing groups was assessed by performing a nonparametric ANOVA (Kruskal-Wallis)17 for all 25 test lengths. In addition, items were ranked randomly 1,000 times, and the above process was repeated to calculate the average ability of each test length (with items chosen at random) to distinguish between groups. The effect sizes for all statistical comparisons were plotted to visualize the results and choose appropriate test items and lengths. In addition, Cronbach’s coefficient alpha was calculated for each chosen test length.18
Criterion validity for the 12-item and six-item tests was assessed primarily by looking for group differences between clinicians with different levels and types of training: MD and NP residents, primary care attendings, and pharmacists and geriatricians, with pharmacists and geriatricians viewed as experts. A secondary analysis was also performed looking for group differences between trainees by profession and postgraduate year (PGY): NP residents, PGY-1, PGY-2, and PGY-3. Scores of the participants were calculated using simple frequencies. Floor and ceiling effects were also calculated. Because of the non-Gaussian distribution of the scores, nonparametric methods were used to compare group differences. Overall group differences were calculated using the Kruskal-Wallis test,17 and between groups differences using the Wilcoxon rank sum test with Benjamini-Hochberg procedure to control the false discovery rate for multiple comparisons.19 Results were considered significant at a threshold of p<0.05.
RESULTS
The overall response rate was 89%. The 74 participants included 37 MD residents (16 PGY-1s, 11 PGY-2s, nine PGY-3s, and one unreported), six NP residents, 10 geriatrics attendings and fellows, 12 pharmacy preceptors and residents, and nine primary care attendings. Of the MD and NP residents who participated, 53% (23/43) were women and 36% (15/42) were nonwhite. Detailed demographics were not obtained for the other participants who requested that their confidentiality be protected.
Factor analysis resulted in a one factor solution, with the scree plot of test data showing a distinct “elbow” after the first factor (Figure 1). This was also confirmed statistically using parallel analysis, as the eigenvalue is greater than that of randomly generated data. Thus, no rotation was possible. After calculating and ranking item information integrals for each test item, sequentially longer test versions were constructed, starting with the highest ranked item and then adding the next highest ranked item for each consecutive test version. The ability of each test version to distinguish between the four testing groups was assessed using a Kruskal-Wallis test. Additionally, items were ranked randomly, and the process was repeated. The effect sizes for all statistical comparisons were plotted to visualize the results and choose appropriate test items and lengths (Figure 2). The effect size peaked at 12 items and was higher than a full 25-item test. When items were chosen at random to create consecutively longer test lengths, the ability of the test to distinguish between groups peaked at 25 questions with an effect size that is similar to the six-item ranked test, and lower than a 12-item ranked test. Based on these analyses, a 12-item test was chosen from the 25 questions. A six-item test was also chosen as a shorter alternative version. See Appendix 1 for included test questions for the six- and 12-item tests.
Factor Analysis. The scree plot of test data (•) shows a distinct “elbow” after the first factor and indicates that a one factor solution is appropriate. This was confirmed using parallel analysis.
Effect Sizes for Different Test Lengths. The 25 test items were ranked from highest to lowest quality using item information curves. The ability of each test version to distinguish between the four testing groups was assessed using a Kruskal-Wallis test (•). Additionally, items were ranked randomly (
). A 6-item (dashed line) and 12-item (dotted line) test demonstrated similar ability to distinguish test-takers by profession and training level.
For the 12-item test, seven participants answered all items correctly (one MD resident, two geriatricians, four pharmacists), and no participants answered all items incorrectly. There was a significant overall effect of training on test performance (Figure 3A, Kruskal-Wallis chi square = 32.8, p<.001). The mean scores of the pharmacists (88%) and the geriatricians (88%) were higher than those of MD and NP residents (58%) (p<.001 for both comparisons). The mean scores of both pharmacists and geriatricians were also higher than the primary care attendings (61%) (p=.003). However, no differences were found between MD and NP residents and primary care attendings (p=.71). In addition, there was no difference between pharmacists and geriatricians (p=.71). The mean scores of MD residents increased stepwise with year of training (51% to 68%, p=.26) (Figure 3B), but this trend was not significant. The Cronbach coefficient alpha was 0.72 for the 12-item test.
Test Performance by Training Level. Panels A and B show the average test performance by profession [Mean (SD)] and training level for the 6- and 12-item tests. MD=Medical Doctorate, NP=Nurse Practitioner, PGY=post-graduate year after medical school.
For the six-item test, 15 participants answered all items correctly (one MD resident, two primary care attendings, five geriatricians, seven pharmacists), and no participants answered all items incorrectly. There was a significant overall effect of training on test performance (Figure 3A, Kruskal-Wallis chi square = 26.8, p<.001). The mean scores of both pharmacists (86%) and geriatricians (92%) were higher than those of MD and NP residents (54%) (p<.001 for both comparisons). The mean scores of pharmacists and geriatricians were also higher than the primary care attendings (59%) (p=.056 and p=.025, respectively). However, no differences were found between MD and NP residents and primary care attendings (p=.767). In addition, there was no difference between pharmacists and geriatricians (p=.88). The mean scores of MD residents increased stepwise with year of training (47% to 65%, p=.16) (Figure 3B), but this trend was not significant. Cronbach coefficient alpha was 0.62 for the six-item test.
DISCUSSION
The ability to assess learner knowledge of addressing polypharmacy in older adults is essential for implementing and monitoring polypharmacy-related educational programs. Such assessments require the presence of valid and reliable instruments. The 12-item and six-item knowledge tests derived in this study were selected through factor analysis and IRT methods to optimize the information obtained about test-takers across the ability range most typical for our sample of MD and NP residents, and achieved effect sizes equal to or greater than the 25-item pre-validation test. The Cronbach alpha for the 12-item test was comparable to that demonstrated in prior studies,6,20,21 while for the six-item test it was somewhat lower. Both the 12-item and six-item tests also showed acceptable known groups validity, distinguishing between postgraduate MD and NP residents and the pharmacists and geriatricians who served as experts. There were no floor effects, and ceiling effects applied mainly to the geriatricians and pharmacists, with only one MD resident achieving a perfect score in either test. The scores of MD residents increased stepwise with training level without reaching statistical significance. There was no difference between the scores of primary care attendings and MD residents; however, the difference in scores between these two groups and the geriatricians and pharmacists was substantial.
The scores of the MD and NP residents on the test might be explained by less curricular emphasis on the topic of polypharmacy during the undergraduate and postgraduate training of these health providers.4,5 The average scores among MD and NP residents for both the six-item (54%) and 12-item (58%) test indicate that substantial improvements could be made. Furthermore, the similarity in scores between MD and NP residents and primary care attendings supports the need for educational interventions for both residents and attendings in primary care. Thus, curricula related to addressing polypharmacy could be a valuable addition to residency training as well as to continuing medical education. In addition, these findings also support the interprofessional approach to polypharmacy education and practice in primary care, which is to include pharmacists and geriatricians in the design and implementation of all interventions.
Practicing clinical pharmacists performed well on both the six- and 12-item tests. These brief instruments may be useful for pharmacy educators in assessing senior pharmacy students in advanced pharmacy practice experiences, particularly in the outpatient setting. In addition, pharmacy residents requiring remediation or pharmacists requiring continuing education in polypharmacy may be identified with this tool. Using the polypharmacy test items paired with individual test-item objectives, educators in any discipline can use test results and performance on individual questions to identify knowledge gaps and curricular needs among health care providers. Test takers may also use their results to tailor their independent study in outpatient medication management of older adults with polypharmacy.
There were some limitations to our study. The sample size is relatively low for the use of IRT, and this could affect the reliability of information obtained from each item. However, the heterogeneity of the participants allowed for meaningful known-groups comparisons that were sufficiently powered to detect differences between the groups. Second, the generalizability of our findings is limited because the survey was validated in a single academic setting. Additional validation studies in other academic settings could prove useful in characterizing the transportability of the test. Finally, given the nature of multiple-choice testing, the content of the test was not able to capture the nuances of the shared decision-making process in addressing polypharmacy or the skills involved in performing comprehensive medication reconciliations.
Both the six- and 12-item tests may be used in future educational research. Through serial assessments, the instruments could be used to determine the impact of educational interventions designed to advance polypharmacy knowledge.
CONCLUSION
In conclusion, our 12-item and six-item tests showed acceptable known-groups validity and achieved effect sizes equal to or greater than the pre-validation test. While the total effect size was less for the six-item test compared to the 12-item test, the two test versions demonstrated similar ability to distinguish test takers by profession and training level. Specifically, both versions could differentiate between experts (pharmacists and geriatricians) and non-experts (MD and NP residents and primary care attendings), but not between various levels of MD residents. Decisions about which test length to use might take into account the higher effect size of the 12-item test versus the time advantage of the six-item test. The instrument designed and validated in this study could be used to assess baseline polypharmacy knowledge among healthcare providers and trainees and identify knowledge gaps, which may reflect curricular needs in their continuing education or training programs.
ACKNOWLEDGMENTS
The authors wish to thank the Office of Academic Affiliations of the Veterans Health Administration for their encouragement and guidance. This work was supported in part by the Office of Academic Affiliations, Veterans Health Administration, United States Department of Veterans Affairs, the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (#P30AG021342 NIH/NIA) and an award from the Health Resources and Services Administration Geriatric Workforce Enhancement Program to Yale University School of Medicine (#U1QHP28745). The sponsors had no role in the design, methods, subject recruitment, data collections, analysis or preparation of the manuscript. This material is the result of work supported with resources and the use of facilities at the Veterans Affairs Connecticut Healthcare System. The contents do not represent the views of the US Department of Veterans Affairs or the United States Government.
Disclosures: S. Jeffery is a consultant for CVS/Caremark. The other authors have no disclosures.
Appendix 1. Polypharmacy Knowledge Test. This test can be given using all 12 items or the first six items only.
Test Instructions: Please circle the best answer for each question.
1. Which of the following statements about sulfonylureas is true?
a) Their effects diminish over time
b) Glipizide has active metabolites and should be avoided in kidney disease
c) They do not increase the risk of hypoglycemic events
d) They decrease carbohydrate breakdown
2. A 75-year old man takes warfarin for atrial fibrillation and wishes to reconsider the risks and benefits of anticoagulation. Which of the following statements is true?
a) The CHA2DS2-VASc and HAS BLED scores both estimate the risk of stroke
b) Gait impairment resulting in a fall is a contraindication to warfarin use
c) Bleeding events are generally more devastating than strokes
d) Poor nutrition may increase the risk of adverse effects from warfarin
3. Which of the following is correct about age-related changes?
a) The total body water increases and fat content decreases with age
b) Decreases in lean muscle mass result in decreased creatinine production
c) Medications are absorbed at the same rate, but to a lesser extent
d) CYP 450 metabolism decreases predictably by 5% each year.
4. Which of the following interactions correctly describes why a patient taking warfarin may have difficulty maintaining INR levels within therapeutic range?
a) St. John’s wort decreases sensitivity to warfarin
b) Ciprofloxacin decreases sensitivity to warfarin
c) Green, leafy salads increase sensitivity to warfarin
d) Vitamin K increases sensitivity to warfarin
5. A 70-year old woman has been taking omeprazole for years. She denies any heartburn in the past 6 months or any history of gastrointestinal bleeding. In your conversation with her, what concern related to adverse effects of proton pump inhibitors might you mention?
a) Increased risk of clostridium difficile-associated diarrhea
b) Increased risk of esophageal cancer
c) Increased risk of osteonecrosis
d) Increased risk of impaired glucose tolerance
6. A 70-year old woman takes temazepam every night for insomnia, but she heard about safety issues surrounding the drug and wants to discontinue it. She states she has tried sleep hygiene counseling in the past without much success. Anticipating that tapering off temazepam will be difficult for her, what would you recommend as adjunctive therapy during the taper?
a) Prescribe lorazepam because it is safer than temazepam
b) Advise her to repeat sleep hygiene counseling
c) Refer her to a health psychologist for cognitive behavioral therapy
d) Prescribe zolpidem because it is safer than temazepam
7. Which of the following anti-depressants has the strongest anti-cholinergic properties?
a) Sertraline
b) Nortriptyline
c) Bupropion
d) Venlafaxine
8. A patient with diabetes and a hemoglobin A1C of 12% reports frequent hypoglycemic episodes. She takes metformin, pioglitazone, NPH, and aspart sliding scale with meals. Which of the following is true?
a) Pioglitazone may increase her risk of heart failure
b) Metformin promotes gluconeogenesis in the liver
c) Her frequent hypoglycemic episodes are a contraindication to insulin use
d) Stopping NPH and increasing aspart sliding scale will improve her glycemic control
9. Which of the following statements regarding the safety of statin therapy is true?
a) Simvastatin may be a safer alternative to atorvastatin because it avoids the CYP3A4 pathway
b) Concomitant use of gemfibrozil and atorvastatin should be avoided
c) Use of a CYP3A4-inducing medication increases the risk of atorvastatin toxicity
d) Consumption of grapefruit juice accelerates the metabolism of atorvastatin
10. Which of the following statements about digoxin has the strongest supporting evidence?
a) When used adjunctively for heart failure, digoxin reduces mortality
b) When used adjunctively for heart failure, digoxin reduces hospitalizations
c) When used adjunctively for atrial fibrillation, digoxin reduces mortality
d) When used adjunctively for atrial fibrillation, digoxin reduces hospitalizations
11. Which of the following tools for approaching polypharmacy asks a series of questions about each medication to guide evaluation of the medication’s relative risks and benefits?
a) Beers List
b) STOPP
c) Medication Appropriateness Index
d) START
12. In carefully reviewing a patient’s medications list, you become concerned about risk for serotonin toxicity. Which of the following medications, in conjunction with sertraline, is most clearly associated with serotonin syndrome?
a) Tramadol
b) Lorazepam
c) Oxybutynin
d) Donepezil
- Received August 31, 2017.
- Accepted September 8, 2017.
- © 2019 American Association of Colleges of Pharmacy
REFERENCES
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.