Association Between Accreditation Era, North American Pharmacist Licensure Examination Testing Changes, and First-Time Pass Rates

DISCUSSION

This study has five meaningful findings: First, the average NAPLEX pass rate declined as the number of programs increased; second, the annual NAPLEX pass rates of programs accredited before 2000 were higher than the average pass rate of programs accredited after 2000; third, the number of programs’ association with pass rate was confounded by other factors; fourth, pass rates were significantly associated with blueprint changes; and, fifth, pass rates were impacted even more by testing condition plus passing standard changes. Each of these findings will be discussed in turn.

Our first finding tentatively verified the veracity of critics’ concerns that the number of programs had a detrimental influence on pass rates; however, the number of applicants did not. From 2004-2010, the number of programs increased as did the number of applicants. However, after 2010, the number of programs continued to rise but the number of applicants steadily declined. One purported reason for the general pass rate decline is a lower-quality applicant pool because fewer students were choosing pharmacy as a career path. However, the period of the highest number of student applicants (2009-2011) coincided with the largest pass rate declines in 2015 and 2016. Yet, because data used for student admission decisions are not widely available, the student applicant quality assertion cannot be refuted or supported.

Our second meaningful result was that average pass rates of programs accredited before 2000 were significantly higher than pass rates of programs accredited after 2000 (Table 2, Figure 1b). Downward trajectories of pass rates for programs accredited before and after 2000 overlapped from 2008-2011 and began to diverge beginning in 2012. Most programs with pass rates above the national average for all five years between 2016-2020 were accredited before 2000. Conversely, most of the programs that never exceeded the national average were accredited after 2000. The differences in these two accreditation eras likely represent unmeasured characteristics of mature versus immature programs. The inability to adjust quickly to the unprecedented increase in the number of programs may have resulted in supply and demand imbalances across the Academy. At the beginning of this era, programs established before 2000 likely had optimal levels of resources required to operate an accredited program, as evidenced by the minimal decline in pass rates until 2015. However, programs accredited after 2000 needed to acquire sufficient resources to meet accreditation standards. Sometimes those resources were acquired at the expense of established programs. New and established programs competed for quality experiential programs, qualified faculty and administrators, and qualified students during this time.

At the core of this argument is the untested assumption that established schools would retain the same proportion of high-quality students, while newer programs would admit lower-quality students from the shrinking applicant pool. The statistical findings do not support this view for two reasons. If the statistical variability among program eras’ pass rates was due to the decline in the quality of students among the newer programs, it should have been visible beginning with the 2008 data. Rather, the statistical variation in NAPLEX pass rates for both accreditation eras was similar for the entire 2008 to 2020 timeframe. This finding is more in line with a hypothesis of the shrinkage of the whole applicant pool. The second reason is that when the 2015 and 2016 changes were examined as confounders, the pre- and post-2000 accreditation groups showed similar variability before both cut points, but not afterward. Notably, the applicant pool was at or near its highest point and coincided with declining pass rates represented by the 2015 and 2016 cut point variables. Thus, critics’ speculation about an influx of lower-caliber students being associated with lower pass rates during this temporal period was confounded by the blueprint and testing condition plus passing standard changes.³ Unfortunately, important information about proxy indicators for student caliber at admission is unavailable for this timeframe. Therefore, hypotheses of confounding between student quality, number of applicants, and NAPLEX changes can be neither refuted nor supported by this analysis. Questions about modifiable student and faculty academic elements associated with NAPLEX outcomes should be the subject of future research.

The remaining findings were unexpected, and our subsequent investigations were prompted by the finding of confounding factors. The NABP modifies the NAPLEX blueprint approximately every five years to reflect contemporary changes in pharmacy practice.¹⁷ The two accreditation eras’ pass rates overlapped until 2011, when NABP changed the blueprint.¹⁸ After 2011, the next meaningful drop in the NAPLEX pass rate coincided with the next blueprint change and occurred between the 2014 and 2015 testing cycles. Notably, the trend line for the pre-2000 programs was flat until 2014 and started its downward trajectory in 2015. The pre-2000 programs’ pass rates were unaffected until the blueprint changed, even though the post-2000 programs were impacted earlier. This finding is important because declines due to content changes are consistent with a validly constructed test,¹⁹ and it provides additional support for the hypothesis that it was not the number of programs but rather when programs were accredited that influenced the pass rates.

Conversely, after the testing condition plus passing standard changes in 2016, slope lines for programs from both eras reversed direction and trended upward. The average pass rates declined for both accreditation eras after the testing condition plus passing standard changes were implemented in 2016. The blueprint was the same in both 2015 and 2016, and the pass rates declined even further after the testing condition plus passing standard changes despite the blueprint being the same. Two explanations are plausible for the change in direction in 2016. First, 2016 was the nadir for the pass rates for both groups during the 13-year observation period. So, the slopes trended upward after the 2016 low point. This reversal of the trend somewhat mitigates critics’ concerns about the pass rate decline being due solely to the number of programs. Second, from a psychometric perspective, the downward trends attributable to the 2015 blueprint content changes are consistent with a reliable and valid test.¹⁹ In contrast, testing condition plus passing standard changes are not associated with content, and they likely introduced bias and error.¹⁹ Questions about the test’s validity after the testing condition plus passing standard changes should have introduced doubt as to whether the test accurately assesses licensure candidates.

Our final finding was that the number of programs and applicants were nonsignificant and had less than half of the impact of the 2015 and 2016 NAPLEX modifications. The modifications hypothesis provides a richer explanation for the precipitous decline and offers opportunities for constructive improvements rather than opining it was due solely to the increase in programs and decline in applicants. It is clear that the rapid rise in the demand and potential shortage of experienced faculty and administrators, qualified students, and capable experiential sites could have played a role in the decline. However, a modicum of culpability in the post-2015 and post-2016 trends must be attributed to the blueprint and testing condition plus passing standard changes implemented by NABP.

In their defense, NABP maintains that candidates’ scores on the NAPLEX are neither intended nor validated for assessing practice competency or judging the quality of pharmacy curricula.^20,21 Yet, despite these caveats, NAPLEX is still used in unintended ways. State boards of pharmacy make decisions about candidates’ readiness for licensure in part based on NAPLEX performance. Likewise, pass rates are a de facto measure of the quality of programs’ curricula because ACPE uses two standard deviations below the national average as a red flag to guide accreditation decisions.²² So, verifying the construct validity and controlling systematic biases are relevant.

Similar licensure testing experiences among other health disciplines have contemporary implications for pharmacy. For example, a blueprint change did not change examination performance for osteopathic licensure candidates,²³ testing condition changes irrelevant to content updates influenced United States Medical Licensing Examination results downward,^24,25 and results reported as pass-fail coincided with more failures by first-time dental licensure test takers.²⁶ NAPLEX blueprint changes foreshadowed significant and meaningful pass rate declines in 2011 and 2015. The NAPLEX blueprint changes were again implemented in 2021,^27,28 and consequences of those changes are imminent. Future NAPLEX results will be reported as pass-fail.²⁹ Collaborative research on the implications of these changes and other future modifications is warranted.

Our dependent variables were average pass rates. Use of a single numeric average to represent multiple years before and after relevant cut points limits interpretation, since a single numeric average disregards important fundamental information that is time based. We were unable to use better methods for time-dependent analyses because of the small number of observations (n=13). Future research should adjust for time dependency.

The second limitation is the relatively small number of NAPLEX testing years. The magnitude of relationships is associated with sample size. The small sample size could have attenuated or resulted in variability of the magnitude of the coefficients or decreased the study’s power. The difference between newer and mature programs was judged to be nonsignificant for 2004 to 2010.³ We found that the accreditation eras’ pass rates overlapped from 2008 until the 2011 blueprint change. These observations provide some credence to our assertion that the test modifications contributed in part to the steady decline in NAPLEX pass rates over the past two decades. While the statistical coefficients may have varied less with a larger sample size, the slopes’ patterns seem persuasive.

Despite these limitations, this study directly responds to critics’ speculation about the cause of decline in NAPLEX pass rates since 2008. Contrary to critics’ speculation, too many programs and too few applicants were not at the root of the largest single-year decline in pass rates since 2008³⁰ after acknowledging the effects of the accreditation era, blueprint changes, and testing condition plus passing standard changes. There are two implications from this work. One, additional research needs to be conducted examining factors measured at the individual program level that are amenable to modification. This work would provide guidance regarding programmatic factors that may reverse the downward trends. Two, given recent history, results of this research emphasize the need for internal and external stakeholders to collaborate on reducing bias and increasing reliability and validity introduced by extraneous factors, with the ultimate goal of optimizing students’ performance on the NAPLEX.

Programs accredited after 2000 generally had lower first-time NAPLEX pass rates. Even so, changes to the blueprint, testing conditions, and passing standards instituted by the NABP were more important predictors of the decline of first-time NAPLEX pass rates. Stakeholders should collaborate and embrace best practices for assessing practice-ready competency for licensure.