Using Grey Literature to Prepare Pharmacy Students for an Evolving Healthcare Delivery System

INTRODUCTION

A healthcare delivery system course is a curricular requirement at most pharmacy colleges and schools in the United States. However, there are inherent challenges to teaching this course in the ever-changing healthcare delivery system, which is wrought with unanswered questions, such as: Will the planned health insurance exchanges work? Is the tax penalty substantial enough to incentivize everyone to buy health insurance? Will employers continue to offer insurance or opt to pay fines? Will Medicaid be expanded and in which states? Will health insurance premiums rise for the privately insured? The answers to many of these questions lie in the future of the Patient Protection and Affordable Care Act (PPACA).

The PPACA has numerous implications for the profession of pharmacy. Pharmacies can expect a higher volume of Medicaid patients in states where the Medicaid program is expanded and there is a shift from patients paying cash to patients being insured. Financial constraints on patients with Medicare Part D should decrease with the closing of the “donut hole” (ie, gap in coverage). Opportunities exist for pharmacists to play a greater role in managing patients’ drug therapies and thereby reducing hospital readmissions. Widespread implications on pharmacy may be less predictable as the financial implications of the law trickle through the drug supply chain.

Pharmacy educators are challenged to prepare students to practice pharmacy in this rapidly changing and uncertain healthcare environment. The Center for the Advancement of Pharmaceutical Education (CAPE) outcomes and Accreditation Council for Pharmacy Education (ACPE) competencies require that students be able to “manage and use resources” appropriately within the healthcare delivery system, a topic often covered in healthcare delivery system courses in pharmacy curricula.^1,2 However, textbooks covering the US healthcare delivery system are typically out of date by the time of publication. Even the most recent editions of these textbooks isolate healthcare reform in its own chapter at the end of the book.

Despite the shortcomings of textbooks for a healthcare delivery systems course, there are ample printed resources available for use in the classroom, including annual reports, white papers, and policy analyses. These types of publications are increasingly being described as “grey literature,” a library sciences term defined by the International Conference on Grey Literature as “information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing.”^3,4 Grey literature is distinguished from traditional sources in that it is not controlled by commercial publishers. Because of the delay between research and publication, grey literature can fill a void for access to more timely data in the literature.⁵ Common types of printed publications include annual reports, white papers, position papers, theses and dissertations, databases of ongoing research, conference proceedings and abstracts, and technical reports. Technology has expanded access to grey literature by introducing new electronic sources, such as Webcasts, press releases, blogs, and social media. Because grey literature is the most current and accessible source of information for many healthcare delivery system topics, it may be a useful curricular tool in teaching a US healthcare delivery systems class. The objective of this study was to assess how the use of grey literature in lieu of a traditional textbook in a healthcare delivery systems course would impact students’ perceptions of the relevance of healthcare delivery systems topics and their ability to identify credible sources of this information.

DESIGN

Health Care Systems and Socioeconomics is a required course offered in the second semester of the first year at Presbyterian College School of Pharmacy. This course is designed to help students develop an understanding of the characteristics, components, and structure of the US healthcare delivery system. The 3-credit-hour course, which meets 3 times per week for 50 minutes, is approximately 75% lecture and 25% project-based work.

Textbooks are commonly required in the pharmacy curriculum, despite their inherent limitations. For example, textbooks are stagnant, current only to the date that material was submitted to the publisher or last updated prior to publication. The time delay between submission of material and publication is of particular relevance in a healthcare delivery systems course, wherein the content should constantly change to keep pace with the implementation of healthcare reform and legislative changes. However, a review of current editions of healthcare systems textbooks demonstrated that recent innovations in healthcare delivery, such as emerging accountable care organizations and health insurance exchanges, were not included. Like most textbooks, almost all required pharmacy textbooks are available only in paper format and not in electronic format, as is common for other types of books. This is an important limitation for a generation of pharmacy students who rely heavily on electronic devices. A final concern is the trend among publishers of academic texts to have multiple authors contribute chapters to a book. The result is often a series of independent chapters lacking cohesion and containing duplicative or overlapping information. ⁶ These limitations may be responsible for students generally not finding textbooks useful.

Professors often use supplemental readings to compensate for the shortcomings associated with textbooks and other printed materials. In a healthcare delivery systems curriculum, there are ample resources available to aide in student learning. For example, professional pharmacy organizations publish white papers on topics relevant to their organization’s mission, such as the “Guide to Pharmaceutical Payment Methods,” published by the Academy of Managed Care Pharmacy.⁷ Government organizations have made their reports, policies, and most recent data more accessible through Internet resources, such as the Centers for Medicare and Medicaid Services Web site. Private companies, such as the Kaiser Family Foundation, a nonprofit health-policy analyst group, also track and publish trends in major health-related issues. Given the accessibility of these resources, which are both credible and regularly updated, this curricular innovation replaced the previously required textbook with required readings from the grey literature.

The syllabus from the previous academic year and the topic list from the previously required healthcare delivery systems textbook were used to develop the outline for this course. For each topic, the instructor reviewed the grey literature to identify required readings on each topic (Table 1). (Table 1 does not include course topics that were covered in lecture but did not require a corresponding required reading; it also does not include instructor-provided optional readings from both the grey literature and a textbook.) At the conclusion of each lecture, the instructor presented “My Favorite Sources,” which highlighted reliable Internet-based sources of information relevant to the content of the lecture.

EVALUATION AND ASSESSMENT

Following completion of this course, students were expected to be able to identify publicly available, current, and credible sources of information on relevant healthcare delivery systems topics. Achievement of this expectation was assessed by means of pre- and post-course survey instruments measuring students’ self-reported perceptions of their ability to identify sources as well as their unaided ability to list credible sources of information. Student-perceived relevance of healthcare delivery systems topics to the profession of pharmacy was also assessed.

Outcomes were measured using the lowest level of cognitive behavior from Bloom’s Taxonomy of Learning (knowledge), as students were simply asked to list a resource.⁸ However, by learning about the resources available in the grey literature, students were empowered with the ability to identify these resources after having completed the course. The expectation is that when students need to find the most current information on a healthcare systems topic in practice, they will be able to identify the most recent source of the materials used in class or identify other new and credible sources of information. Although assessing this skill was not within the scope of the study, it is classified as “learning how to learn” in Fink’s Taxonomy of Significant Learning, and is an important skill for pharmacists.⁹

The assessment design used was pre- and post-course testing within subjects. Students were surveyed on the first and last day of class during spring 2012, and unique identifiers were used to match students’ pre- and post-course responses. Students were not required to complete the survey instruments, and no grade or class credit was awarded for doing so. The pre-course survey instrument, which included demographic questions, had 32 items; the post-course survey instrument had 27 items.

The survey instruments were organized into 3 sections to capture student behaviors, perceived relevance of healthcare delivery systems information, and ability to identify a credible source for this information. On the student-behaviors section of the pre-course survey instrument, students were asked to rate their level of agreement with 4 statements about their purchasing and reading patterns for required and optional textbooks using a 5-point Likert scale ranging from 1=strongly disagree to 5=strongly agree. For example, students were asked to indicate their level of agreement with the statement, “I typically purchase a textbook when it is optional for the course.” On the same section of the post-course survey instrument, students were asked whether they had purchased and read the optional textbook for the course and whether they found it helpful. The same items were applied to assess student behaviors relevant to required readings from the grey literature. Students were asked on the post-course survey instrument whether they thought a textbook should be required for the course.

The next section in both the pre- and post-course survey instruments used repeated measures to evaluate student-perceived relevance of healthcare delivery systems topics to the profession of pharmacy. Topics included healthcare reform legislation, healthcare policy information, US population data, managed care information, Medicare and Medicaid policies, Veteran’s Administration and Tricare health insurance information, health insurance plan quality ratings, ongoing or completed clinical trials, brand-name drug information, US health data and priorities, and world health data and priorities. This section used a 5-point Likert scale ranging from 1=irrelevant to 5=very relevant.

The final section used repeated measures to assess student ability to identify credible sources of healthcare delivery systems topics (previously listed) based on students’ self-reports and their unaided ability to list a resource in both the pre- and post-course survey instruments. Students were asked to rate their level of agreement with the statement, “I know where to find current information on <topic>,” using a 5-point Likert scale ranging from 1=strongly disagree to 5=strongly agree. Students were then asked to write an example resource for each topic. The instructor evaluated each response and classified it as correct or incorrect. Generic responses, such as Google, TV, Internet, or class notes, were classified as incorrect. Correct responses varied by category. Some examples of correct responses included www.clinicaltrials.gov for ongoing or completed clinical trials; National Committee for Quality Assurance for health plan quality ratings; Centers for Medicare and Medicaid Services for Medicare and Medicaid policies; and Healthy People 2020 for US health data and priorities.

This study was approved by the Presbyterian College Institutional Review Board. Because it used a survey methodology, the study was approved through the expedited review process. Students were not required to complete the survey instrument and were not assigned a grade or credit for participating in the survey. Statistical comparisons were used to measure changes in outcomes from the pre- to the post-course survey instruments. Change scores for Likert scale responses were compared using a Wilcoxon signed rank test. Percentages of students able to list resources were compared using McNemar’s test. Alpha level was set a priori at 0.05. All analyses were conducted using Microsoft Excel.

All 76 students enrolled in the course completed both the pre- and post-course survey instruments, yielding a response rate of 100%. Four student pre-course survey instruments that could not be matched with their post-course instruments were excluded from the analyses, resulting in a usable study sample of 72 (94.7%). The student sample was 69% female, with a mean age of 24.5 years. Eighteen percent of students had completed a college-level course on healthcare delivery systems, and 56% had completed a college-level socioeconomics course.

On the pre-course survey instrument, students reported being more likely to purchase and read a textbook when it is required for the course than when it is optional (Table 2). According to the post-course survey, only 8% of students bought and read the optional textbook for the course. Of the remaining 92% of students who did not purchase and read the textbook, only 4 students responded on the post-course survey instrument that reading the textbook might have improved their understanding of the material presented in class or their performance on examinations.

On the post-course instrument, almost all students (96%) responded that a textbook should not be required for the course. Sixty-three percent of students indicated that they completed most of the required readings from the grey literature provided throughout the semester. Of the remaining 37% of students who did not read most of the required reading materials, 56% reported that doing so might have improved their understanding of the material, and 76% reported that it may have improved their performance on examinations.

Students perceived healthcare delivery systems topics to be moderately relevant to very relevant to the profession of pharmacy in both the pre- and post-course surveys. There were no significant differences in perceptions over the course of the semester (Figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Student Level of Agreement With Statements on Perceived Level of Relevance of Healthcare Delivery systems Topics to Pharmacy, N=72 (Rating scale: 1=irrelevant, 2=moderately irrelevant, 3=neither irrelevant/relevant, 4=moderately relevant, 5=very relevant)

Students knowledge of current and reliable sources of information on the listed topics improved significantly from the pre-course survey instrument to the post-course survey instrument, based on subjective and objective measures (p<0.05). The only exception was for brand-name drug information, as students were able to identify credible sources of brand-name drug information on the pre-course survey instrument (mean score, 4.5; 53%).

DISCUSSION

This study reports a curricular innovation of using required readings from the grey literature rather than a traditional textbook in a healthcare delivery systems course. There were 2 primary findings. After completing the course, pharmacy students were better able to identify related healthcare delivery systems information, and they perceived healthcare delivery systems topics to be relevant to the profession of pharmacy.

The ability to identify current and credible sources of healthcare delivery systems information is extremely important in today’s environment, in which change is constant. For drug information, pharmacy education consistently emphasizes the use of reliable sources. Although pharmacy students are to avoid general Internet sources and searches, they can be useful in identifying current Medicare policies, upcoming changes in healthcare policy, and trends in population-based health care, provided students are able to discern which sources are valid. Further, media sources often cover these topics but may present incomplete or misleading information. Thus, pharmacy educators must guide students on how to identify these resources and which sources are reliable and unbiased. This curricular innovation teaches students about credible resources for this type of information, with the intention that they will be able to identify these sources in the future as the information changes.

The second key finding identified high student-perceived relevance of healthcare delivery systems topics, which are required educational competencies in doctorate of pharmacy programs. Specifically, the ACPE Professional Competencies and Outcome Expectations state that graduates must have the ability to “manage and use resources of the health care system”² As delineated in the CAPE Educational Outcomes, these expectations include describing and demonstrating appropriate use of management principles and use of healthcare resources in the American healthcare delivery system by (1) identifying the key features of private and public payers of healthcare and (2) describing the objectives of insurance and managing healthcare.¹

This study found that students perceived the topics to be relevant to the profession at the start of the course and at the completion of the course, with no significant change. Although this study did not assess reasons for perceived relevance, the passing of the PPACA and subsequent debate, as well as exposure to insurance companies in introductory pharmacy practice experience settings, may have influenced this finding. Previous studies have identified strong student interest in managed care and Medicare Part D.^10,11 However, of the more than 600 students surveyed, 84% responded that the amount they had been taught about pharmacy benefit management in pharmacy school was inadequate or very inadequate.¹⁰ Given these findings, pharmacy colleges and schools should examine their approach to teaching healthcare delivery systems topics. For example, are faculty members with relevant healthcare systems expertise teaching these courses? Is the information current? Are students being adequately prepared to practice in today’s healthcare delivery system?

Students and faculty members have conflicting views about the use of textbooks.⁶ A study of pharmacy student and faculty focus groups revealed that students do not find most course textbooks helpful, whereas faculty members do perceive them as useful. On the contrary, both groups reported supplemental readings to be useful, though few students reported reading them. Although the use of supplemental reading materials is not new, this course replaced the required textbook with supplemental readings from the grey literature. Sixty-three percent of students reported that they read most of the required readings from the grey literature. The readings were identified in an intentional manner to familiarize students with authoritative organizations such as the Kaiser Family Foundation, the Centers for Medicare and Medicaid Services, the US Public Health Services, the Academy of Managed Care Pharmacy, and the American Society of Health-System Pharmacists. This approach will ensure that students are aware of and consider these organizations in the future when looking for the most recent information. Further, all resources used are also available online. Therefore, a student can use a search engine to locate a resource that was used in class whenever and wherever the information is needed without requiring access to a full textbook. Finally, this approach increases the use of these publicly available resources. These resources require funding and dedication from the organizations that produce them, and increasing student awareness is an important stewardship effort.

Implementing this curricular design requires additional effort from the faculty member teaching the course. At the beginning of each semester, the professor would need to review the assigned readings and determine whether they were current and whether additional resources were available. This activity, however, is something most professors routinely do as they update their class materials. Therefore, the increase in effort should be minimal. This approach may be useful in other courses, such as ethics or contemporary issues, in which content changes rapidly and for which authoritative content is available in the grey literature.

There are limitations to this study. Self-reported surveys are subject to certain biases, such as recall and response bias. The students may have artificially inflated the rating of their ability to identify sources in an effort to produce a response that was pleasing to the professor. Additionally, they may have chosen a high rating of their ability without actually being able to list a resource. This was true in many cases, given that the mean Likert scores on the post-course survey instrument for most items were above 4 and the percentages of students correctly listing a resource ranged from 21% to 68% on the post-course survey instrument. This disparity reflects an important limitation of this study, which relied upon open-ended, free-text student responses. Not surprisingly, many students left this response blank. However, they might have been able to provide an accurate response had they been given a multiple-choice question.

There are also limitations to the curricular design. This approach requires extra effort on the part of the faculty member, but the increase in workload, which should not be substantial, would likely be offset by the value to the student. Widespread adoption of this approach would have a negative impact on textbook sales, which is a limitation to certain stakeholders. Finally, students who learn through reading textbooks may be somewhat resistant to this approach. However, because the grey literature readings replace the textbook, resistance should be minimal. Although this study did not assess students’ attitudes toward the use of grey literature, it did find that 63% of students completed most of the required readings, and 96% responded that no textbook should be required.

SUMMARY

The use of “grey literature” provides an alternative approach to teaching required healthcare delivery systems topics through the use of publicly available resources. This approach can be used to ensure that the healthcare delivery systems information in the curriculum, which is required by accrediting bodies and is highly relevant to students, is the most current and credible information available. The use of grey literature might also be considered for other courses in which content changes rapidly.