Is It Time for a Quality Rating System for Colleges of Pharmacy?

Limitations of Existing Quality Indicators

The current quality-related metrics available to the public include the accreditation status of a school, the US News & World Report (USNWR) annual rankings, and the annual NAPLEX pass rates. However, each of these existing sources of quality-related data have their limitations. For example, the accreditation status of schools is of minimal use in differentiating between accredited institutions unless there is a significant deficit noted in the accreditation status (such as probation). It is our belief that that applicants will be unlikely to know that a school is on probation unless the school openly discloses this information to potential applicants.

The USNWR rankings of pharmacy school depends solely on the ratings of the deans of the schools of pharmacy.⁸ Only 40% of pharmacy deans participated in the 2015 survey by USNWR, and there is not a validated rubric for deans to use in assigning a rating to their school. Each respondent to the USNWR dean’s survey is asked to provide a rating for each pharmacy school on a scale of 1 to 5 wherein 5 is the highest rating. Thus, each dean uses whatever criteria he or she chooses and which may be based heavily on reputation of the institution for areas outside the educational quality of the PharmD program (eg, research).

In 2012, Ascione pointed out the “folly” of chasing the USNWR ranking because of the perceived lack of validity of the rankings.⁹ He noted that deans are unlikely to know much about the educational strategies and quality of education delivered by other schools, particularly newer schools. Thus, deans tend to rely on the prior rankings of schools, the cadre of academic leaders from each school, or the amount of federal research support received by the school. This may explain why the top-ranked schools tend to be those with the largest research programs and largest number of faculty members.

In 2016, NABP initiated changes in the NAPLEX and additional changes were made in 2017 and 2018. The new version of the NAPLEX is longer and uses more case-based assessments.⁶ The first-attempt pass rates for the new NAPLEX dropped significantly from the prior version and revealed substantial differences in pass rates between schools. The national average pass rates for first attempts over the three years of the new NAPLEX have been: 85.9% (2016), 88.0% (2017) and 89.5% (2018). For 2018 graduates, the first-attempt pass rates varied widely between schools, with a range of 57.8% to 100%. Nineteen schools had a first-attempt pass rate less than 80% for 2018 graduates.

Research by Mattingly and colleagues showed that the NAPLEX pass rate may be an acceptable metric for comparing schools of pharmacy.¹⁰ They used a Delphi method along with surveys to gather input from deans on the most preferable metrics for evaluating the quality of PharmD education. The deans did not prefer the USNWR rankings but highly ranked several other potential metrics including NAPLEX scores, student satisfaction, and job placement. They noted that job placement was not currently a metric that varied much between schools; however, this may change as more states experience an oversupply of pharmacists.

Because the NAPLEX pass rates vary between schools and are consistently tracked by NABP, these rates may be a reasonable starting point for a new rating methodology. However, pass rates are only useful if they are easily accessible to applicants or employers and can easily be compared to national or regional benchmarks. The ACPE requires schools to disclose their pass rates on the school website along with other selected metrics; however, our non-systematic review of websites revealed that quality information is often not easily found. An applicant may have to wade through several layers of webpages to find the information. Also, there is considerable variation in the timing of updates to the metrics and the way in which the metrics are reported. Understandably, schools attempt to “spin” the quality metrics in the most positive way by selecting timeframes or comparators that make the school appear as favorable as possible.

Some may question whether the USNWR rankings are already reflective of the quality of PharmD education as judged by the NAPLEX pass rates for schools. To investigate the relationship of NAPLEX pass rates and USNWR rank, we created a ranking system for schools based on NAPLEX pass rates. We calculated the three-year mean (2016-2018) for each school on the NAPLEX first-attempt pass rate. By using the three-year mean, we minimize the year-to-year variability in pass rates. We then ranked the schools in order of their mean pass rate.

As seen in Table 1, there are many schools that are ranked highly by USNWR but which rank much lower on NAPLEX pass rates. While all of the schools ranked in the top 10 by USNWR had mean NAPLEX pass rates that were above the national average, several of these top-ranked schools did not finish in the “top 50” for NAPLEX ranking. Conversely, Table 2 reveals that many of the schools with a high NAPLEX pass rate do not have a high ranking in USNWR. For example, South Dakota State University had the top-ranked NAPLEX pass rate for 2016-2018 but was ranked 70^th in USNWR.

Building a Better Method for Quality Ratings

The intent of this commentary is to note the shortcomings of current models for ranking pharmacy schools and to call for development of a meaningful new method for publicly reporting on the quality of pharmacy schools. While the details of a new quality reporting system will require the input of many stakeholders and rigorous modeling and testing work to be conducted by measurement experts, we have identified potentially relevant data for consideration in the new quality measures.

An ideal measurement system will include multiple measures of quality. The NAPLEX pass rates for schools of pharmacy may be one reasonable measure within a quality rating system. Additionally, the AACP has a robust and efficient system for collecting quality-related data about schools of pharmacy. The AACP Assessment and Accreditation Management System (AAMS), along with the AACP survey system, prompt schools to collect and report a consistent set of quality indicators along with other information that may support quality improvement and accreditation reports. These systems are designed for benchmarking and trending in the pursuit of quality improvement, but the data are not easily available to the public. In fact, there are restrictions on the ability of a school to identify the data for a specific competitor (ie, only comparisons to the average for a group of schools are permitted).

The data within the AAMS system, including survey results, could be used along with the NAPLEX pass rates to create a quality rating system for schools of pharmacy. As schools are already collecting a consistent set of data and have a mechanism to efficiently report the data, these operational hurdles for a quality rating system have already been overcome. There will be great debate over which data should be included in the technical specifications for the quality measures of a pharmacy school and how a composite rating should be generated. Some may argue that surveys of graduating students are irrelevant or lack validity, while others may assert that the experiences of graduates is highly relevant and reasonably valid as a measure of quality. Of note, methods for implementing the AACP surveys have varied widely between schools and there is not currently a sufficient level of consistency between schools to enable an “apples to apples” comparison of survey results. If one of the AACP surveys were to become a component of a quality rating system, we anticipate that schools will take steps to increase the response rate on the survey.

There are numerous experts within academia and outside of academia who can gather the input of all stakeholders and conduct the research necessary for building and testing a composite measure of quality derived from quality-related data collected by AACP along with the NAPLEX pass rates. This composite measure of quality could be translated into a rating system that could be easily interpreted by the public. While data from AACP and NABP may be useful in constructing the measures of quality, it may be appropriate for an independent organization to conduct the analyses and provide the public reports. This may avoid any perceived conflict of interest from AACP evaluating its member institutions.

The biggest obstacle to creating a publicly reported composite measure of quality for PharmD programs will be the fears of academic leaders. As physicians, hospitals, nursing homes, and health plans entered the realm of public reports on quality, each of these groups expressed fear that their organization might not score well on the rating of quality. In other words, the professionals being rated have often fought the implementation of quality ratings by criticizing the validity of the rating and the potential burden for data collection (in addition to their repugnance of the idea that their “art” or “professional expertise” could be quantified). We expect that we would see the same pushback if a serious effort to deploy quality ratings of pharmacy schools were instigated.

Given the evolution of the healthcare system and societal demands for quality-related information, we believe the time is right for the profession of pharmacy and pharmacy academia to seriously consider a means to provide the public, particularly prospective applicants to PharmD programs, with a simple method for identifying and comparing the relative quality of PharmD programs. It is important for leaders in academia to engage in the development and deployment of these programs, lest we allow external entities to continue driving a ranking system that is publicly reported but lacks validity. While USNWR is the best known entity for ranking pharmacy schools, other organizations have released their own reports on pharmacy schools, and some of these reports do not provide transparency in how the ratings were constructed.^11-13

The debate is likely to continue over the best methods for evaluating the quality of pharmacy schools.^13-16 We hope that academic pharmacy along with the profession of pharmacy are finally ready to work together to create a meaningful quality rating system for schools of pharmacy. The Academy’s desire for a perfect measure should not stop us from continually improving methods to evaluate the quality of pharmacy education and providing meaningful information to potential applicants and employers.