Benchmarking in Academic Pharmacy Departments

Benchmarking in Academia

Universities and colleges are becoming more interested in benchmarking practices, as they are being asked more frequently to demonstrate the quality of their educational and research programs to the public and government stakeholders.¹⁵ The culture of collaboration, as well as the widespread use of analytical research methods, both fostered in academia, bode well for the acceptance of benchmarking in the academic environment.^9,16 A rigorous comparison among institutions, “rankings,” and professional colleagues is not a new concept in higher education. The novelty is in the formalization of these comparisons and the use of the term “benchmarking.” The ubiquity of information technology in academia can greatly aid the dissemination of benchmarking, as a wealth of metrics is readily available for use in decision analysis, including human resource management.

The goal of benchmarking in academia is to provide institutional leaders with reputable standards by which they can measure the quality and cost of administrative processes, instructional models, and research efforts, and to identify where opportunities for improvement reside. Leadership committed to improving the quality of offerings and activities can move forward by identifying a benchmark institution that shares a similar mission or structure. The choice of benchmark is often decided by reviewing data compiled by national education groups, including accreditation bodies.¹⁷ This external reference point can provide a standard by which to assess current programs, and it can also provide useful insights into problematic areas. The benchmarking of successes may help identify solutions to address noted weaknesses or to rectify identified deficits at the home institution.¹⁸ Thus, benchmarking can be prescriptive as well as diagnostic. Benchmarking also can be used to better inform extramural stakeholders as to the state of the institution and the need to expedite corrective actions.

A number of national professional organizations and private consulting firms provide benchmarking services for universities and colleges.¹⁹ One example of a focused academic benchmarking effort is the National Study of Instructional Costs and Productivity, (also known as “The Delaware Study”).²⁰ With decades of experience in academic benchmarking, this study provides comparative analyses of student credit hour production (credit value of a course multiplied by the student enrollment in that course), faculty members' teaching loads, and instructional, research, and service expenditures (direct expenditure data incurred for personnel compensation, supplies, and services used in the conduct of each of these functional areas) broken down by academic discipline at a departmental level.^21,22 At the time of this writing, over 500 universities and colleges participate in this longitudinal study, which allows not only point-in-time analysis but also permits data trending over time.

Benchmarking can play a significant part in the well-known 4-step approach to continuous quality improvement, the Plan-Do-Check-Act Cycle (Figure 1).²³ In academia, the first step would be to identify whom or what to benchmark by selecting the administrative, teaching, or research process to be studied. The second step would be to compile data from and about a benchmark institution, department, or program. The third step would be to analyze the compiled data and conduct comparative assessments to identify quality differences and to yield actionable recommendations for improvement. The final step, at least in the first iteration of the cycle, would be to implement facility, program, or personnel-specific changes. Then, the success of the intervention can be assessed against its ability to narrow the differences between the target and the benchmark. Thus, benchmarking formalizes the planning process to permit sound action to be taken to improve quality, and affords a standard by which the success of an intervention can be assessed.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

The Plan-Do-Check-Act Cycle

Benchmarking is most successful when a well-accepted standard is available. When an acknowledged best practice or leading institution cannot be identified clearly, process or industry composites may serve as a benchmark.³ Compiled reports can lead to rank ordering to identify a benchmark process or organization, but this data can also be aggregated to yield composite measures for a practice or discipline, providing a benchmark when a singular standard is not apparent. Academic institutions, especially professional schools, are accustomed to sharing aggregate student and faculty information, as they share an interest in producing competent graduates and advancing scientific and practical knowledge. Thus, the open nature of educational innovation, and the willingness of educators to work together to achieve coordinated goals, can provide readily available data that can be compiled to yield best practice and composite professional standards.

In 2006, the Secretary of Education's Commission on the Future of Higher Education report called for “a new consumer-oriented database and more and better information on the quality and cost of higher education.”²⁴ This data repository would be open for review by researchers, policymakers, and the general public. To adapt to an era of constrained budgets, higher education is being encouraged to shift from reputation-based organizations to performance-based institutions that cultivate a culture of transparency and responsibility. Universities and colleges will be asked to demonstrate the quality of their educational offerings and the productivity of their research enterprises to many constituencies, including students and faculty members, but also to professional and governmental bodies. As demand for accountability in academia increases, schools and departments will require data to demonstrate their contributions to an institution's mission, as they compete for increasingly limited resources. Benchmarking likely will play a vital role in this reporting and strategic planning.

Benchmarking in Academic Pharmacy Departments

The continuous use of benchmarking data to evaluate and support pharmacy programs is commonplace in health systems. Health systems use a variety of standards (eg, American Society of Health-System Pharmacists' Best Practices, Joint Commission's Comprehensive Accreditation Manual for Hospitals: The Official Handbook, Centers for Medicare & Medicaid Conditions of Participation Interpretive Guidelines and Conditions for Coverage, etc) to continuously benchmark their pharmacy processes, services, or personnel needs to identify and reduce variations in practice and any resulting outcomes (improved patient care, medication safety, etc). Benchmarking efforts in academic pharmacy may not be employed continuously, but nationally generated data could be used by academic pharmacy departments in a number of useful ways. Obviously, access to national norms would allow comparisons in areas of workload, resources (eg, number of faculty members), and productivity. Such considerations may form the basis for increased resource justification, or departmental planning and goal setting, provided that it is appropriately normalized or stratified to allow valid comparisons to be made. For example, institutions or departments with a given emphasis (research, education, practice, etc) should be compared to other similarly-focused institutions or departments.

The specific benchmark data to collect is an important consideration. Standards and guidelines can help facilitate benchmarking efforts by establishing best practices or setting accepted practice norms. For academic pharmacy, the Accreditation Council for Pharmacy Education (ACPE) seeks to assure and advance quality in pharmacy education through its accreditation standards and guidelines for professional programs in pharmacy. The ACPE standards (Standards 2007) serve to establish minimum standards in academic pharmacy. Specifically, standards 24 through 26 address quantitative and qualitative factors related to faculty members and seek to ensure that a given institution has “fair and equitable policies and procedures and capabilities to attract, develop, and retain an adequate and appropriate number of qualified faculty to contribute to and achieve its mission and goals”.²⁵ Descriptions of national benchmarking efforts in academic pharmacy literature are generally lacking. However, gathering and analyzing benchmarking data can help translate today's best practices into tomorrow's standards.²⁶ A criticism of the current ACPE standards is that they lack a guiding philosophy for pharmacy education and practice that translates into ambiguity, circular arguments, and non-nullifiable hypotheses.²⁷ Benchmarking departmental characteristics, responsibilities, and outputs within academic pharmacy could help address some of these concerns by establishing best practices or setting accepted norms for faculty members' efforts in the 3 academic missions (teaching, scholarship, and service), and perhaps clinical practice.

Benchmarking departmental characteristics, responsibilities, and outputs within academic pharmacy would require the development of an information framework to standardize data collection and submission. Such a framework would enable benchmarking processes like those performed annually by the American Association of Colleges of Pharmacy (AACP) to benchmark pharmacy faculty compensation. AACP collects and shares other descriptive data for each college/school of pharmacy that is relevant to benchmarking as well, such as the number of pharmacy faculty members in each discipline. While developing a system to benchmark best practices or accepted norms for the 3 traditional academic missions would be more challenging, we believe it is possible nonetheless. Data characterizing the academic missions can be tabulated readily and reported in aggregate or combined with departmental demographic data and stratified by academic rank, years of service, type of appointment, institution type, or full-time equivalents (FTE) allocated to the given mission. Of the 3 missions, scholarship might be the easiest to benchmark. Table 2 lists a variety of parameters that can be used to measure research and scholarly productivity.²⁸ Indeed, quantification of scholarly productivity, particularly authorship, is the area of academic pharmacy benchmarking evident in the literature and for which the quality of measure has been considered. Thompson and colleagues reviewed a variety of recently developed indices that measure the depth, breadth, and creativity in journal article publishing.²⁹ Each index has advantages and disadvantages, and several have been validated; and normative values in academic pharmacy have been determined, particularly among pharmacy practice faculty members and department chairs, and college/school of pharmacy deans.^29–35

• Download figure
• Open in new tab
• Download powerpoint

Table 2.

Representative Measures to Benchmark Academic Pharmacy

Benchmarking the efforts departments direct towards their teaching mission is more difficult due to a paucity of objective and validated tools for documentation and evaluation.³⁶ Examples of indices that could be measured are summarized in Table 2. The data could be useful to determine normative values for the percentage of the curriculum that departments usually support, experiential student-to-faculty ratios, typical teaching loads based upon academic rank, or FTE allocated to teaching. To ensure valid comparisons, the data should be normalized using common parameters. For example, separating the effort spent in providing clinical service and delivering experiential education is difficult. Faculty members may or may not provide clinical service 12 months a year, or there may be an inconsistent relationship between providing service to patients and precepting students (relative attention to each and/or degree of overlap of effort for each). Therefore, even though clinical practice and experiential education cannot be separated completely, presenting the number of clerkship students precepted per year by individual faculty member in the context of months of clinical service may provide a better estimate of the effort devoted to providing clinical education. Other more subjective or ambiguous measurements, such as student or peer evaluation scores, and institutional rewards for teaching excellence, are probably of limited value in benchmarking due to diverse methods/policies at different institutions.³⁶

Department effort in the academic service mission is perhaps the most challenging mission to benchmark because it is diverse in scope and therefore difficult to measure objectively.³⁶ Examples of common measures of this mission are summarized in Table 2. The data could be useful to determine normative values for the percentage of the committee loads, leadership roles, and contributions to the profession and society. As with benchmarking the teaching mission, separating provision of clinical service from clinical education is also difficult. Perhaps pharmacoeconomic metrics or patient-outcome measures (therapeutic endpoint targets, quality of life measures, etc) could be used to provide a better estimate of the effort devoted to providing clinical service. Deriving a national benchmark for this activity is particularly challenging as faculty members' practices (percent of effort or hours/day, scope of practice, etc) likely vary profoundly from 1 faculty member to another and from 1 practice site to another.

An Experience With National Benchmarking

To our knowledge, a national benchmarking study examining academic pharmacy departments has not been conducted previously. Because we believed that national data would be a powerful tool for department chairs, in late 2008 we developed and distributed an extensive survey instrument to all chairs of departments of pharmacy practice. A national group of chairs participated in the development of the survey instrument, and an online questionnaire was developed to capture data reflecting both departmental composition and performance. Sixty-one data/response categories were included.³⁷ The survey instrument was approved by the Institutional Review Board of the Medical University of South Carolina. We sought to measure department demographics (number of faculty members, ranks) and faculty performance in a number of areas (scholarship, teaching, and practice) but were unsuccessful in gaining broad input. Potentially useful results were evident although there was considerable variation in the responses in most categories.

Lessons Learned and Recommendations

Benchmarking is a process of assessment and innovation, but universities and colleges can be resistant to change. Therefore, it is advantageous for the advocates of benchmarking to employ reliable research techniques such as validated survey instruments, independent interviews, and confirmatory measures to insure the reliability of their data extraction and to bolster the credibility of their judgments and recommendations. While the data we gathered are of interest and probably have some use, the response rate to the survey instrument was disappointingly low. Although the “data” were in some cases not usable, our experience leads us to several recommendations for future attempts.

First, the survey instrument and the process to administer it should be simple. Perhaps because our survey instrument sought a great amount of detailed information, a number of respondents provided only partial information. This might be due to the chairs being deluged with requests to respond to/complete survey instruments that emanate from a variety of sources, both internal and external to a given institution. Thus, the phenomenon of “survey fatigue” is real. We now recognize that a 61-question survey instrument, requesting highly detailed information, was bound to have limited success.

Second, it is vital that the data collected allows comparison of “apples to apples.” As academic pharmacy departments vary significantly in size and composition, allowing for stratification is desirable to allow valid comparisons. Beyond demographic differences, there are variations in mission. For example, the ACPE expectation that all pharmacy faculty members be involved in scholarship is likely broadly interpreted and applied to varying extents at different institutions. If some departments have little or no expectations for scholarship, their data will skew the pooled results, and attempts to accurately determine national norms or benchmarks will fail. Additionally, a department that has little or no expectation for scholarship may have high expectations for teaching effort and/or professional practice, and again, their data could skew the resultant national average. Thus, we also must be able to stratify the results based upon school mission.

Third, it must be time-focused. If there is a real need for national benchmarking data and most chairs would like to use such information, motivation to participate should be high. However, the effort and time required to participate by providing information should be reasonable. Therefore, in addition to limiting the size of the survey instrument, presenting it in manageable, focused (ie, single mission) segments would be wise.

Fourth, an accurate survey population to sample is necessary. The Chairs of Departments of Pharmacy Practice list purchased from AACP contained over 130 listings, which exceeds the number of pharmacy schools in the country. This may reflect that some schools have geographically split campuses and/or departments and have more than 1 chair of the department of pharmacy practice on the AACP list. Deciding whether such data should be combined for a “school” response or whether the entities are large enough to justify separate consideration is another issue to resolve.

In conclusion, reliable national benchmarking data would be a powerful tool for academic pharmacy department chairs and deans, and further initiatives to gather, assess, and share such information are recommended. Whether such data should be collected by a group of motivated faculty members or chairs (as in our case), or by a pharmacy organization such as AACP, using its internal mechanisms to collect and disseminate these data on a regular basis remains to be determined. In any case, marshalling resources to perform benchmarking of academic pharmacy departments should be a priority for the academy.