Impact of Intersession Course Remediation on NAPLEX/PCOA Scores in an Accelerated Doctor of Pharmacy Program

INTRODUCTION

Students completing a Doctor of Pharmacy (PharmD) program are expected to receive an education that produces a health care practitioner capable of providing competent and safe care to patients. Because of the rigorous coursework involved in PharmD programs, not all students who attempt to complete the degree requirements will succeed with ease. In situations involving academic difficulties, institution-specific progression policies delineate the appropriate steps students must take to remain on track for graduation or otherwise successfully complete the program. One particular type of progression policy involves “intersession” remediation (sometimes termed “summer restudy” for programs having summer semesters with little or no required coursework), which allows students who qualify to retake coursework in between academic terms and potentially remain with their peers in the entering cohort when required courses resume the following fall.¹ This type of remediation differs from a course repeat strategy, which requires students who fail to meet the appropriate academic standards for a course to take the course again the next time it is offered, often during the following academic year. Thus, a course repeat strategy, while simple and straightforward, may force the student out of sequence with their entering cohort and cause a type of “ripple” effect in delaying the student’s graduation and subsequent entry into practice.

An important concern related to any implemented remediation strategy is the impact it may have on successful licensure examination outcomes and whether abbreviated remediation strategies (such as intersession remediation) impact student success differently than repeating a course. Standard 17 from the 2016 Accreditation Council for Pharmacy Education (ACPE) Standards directs colleges and schools of pharmacy to address issues associated with academic progression and provide students with policies related to remediation and academic progression.² However, the process of how a pharmacy program should conduct coursework remediation is not specified in the standards or in corresponding ACPE policies. This results in heterogeneity of methods for addressing student success and a consequent lack of data concerning the efficacy of remediation strategies.^2,3 This data gap is particularly notable for accelerated (three-year) PharmD programs where implementing traditional remediation strategies (eg, summer restudy) may be more challenging because of the continuous nature of the curriculum.

At Sullivan University College of Pharmacy and Health Sciences (SU COPHS), students complete the PharmD requirements in three years vs the four years required to complete most traditional programs. This is achieved through 10 weeks of coursework, one week of finals, and a two-week break between quarters for the first two years (didactic coursework plus introductory pharmacy practice experiences) and 12 weeks of advanced pharmacy practice experiences (APPEs), with a one-week break between quarters during the third year (PY3). The program offers qualified students the opportunity to complete remediation coursework during the two-week break between didactic courses, known as the “intersession” period. Another relevant characteristic of our institution is the fact that the grading scale does not allow for D grades. Whereas in some schools a student may progress in the program despite having earned a few grades of D, at our institution a grade of F indicating failure is assigned for students whose course average is below 69.5%. Therefore, students whose grade in a course is within the range of 59.5% and 69.4% (the traditional D range) are given the opportunity to remediate the course within the two-week break between quarters. Any grade below 59.5% (a traditional F grade) necessitates retaking the course in its entirety. An analysis of the intersession remediation strategy and rates of attrition at SU COPHS has been previously published.⁴

The impact of intersession remediation as compared with course repeat or any other progression strategy on academic outcomes of interest, including professional licensure examinations, is not well-described in the literature. This may result from lack of standardization of remediation policies. A 2015 cross-sectional survey of 45 US pharmacy programs reported that repeating a course was the most common form of remediation (95.6% of responding programs), while intersession/summer restudy remediation was only reported by 42.2% of responding programs.⁵ This analysis, however, was limited by its low overall response rate, ie, 41.7% of US pharmacy programs at the time of the survey. Additionally, only 22 (48.9%) programs that responded to the survey provided information regarding their strategies to manage attrition and optimize student outcomes, none of which was a three-year accelerated program.

Available data appear to be in favor of a remediation model that offers individualized or focused support to students who are struggling, and this is included as a component of our intersession remediation strategy.⁴ In a 2019 study by Coelho and colleagues, struggling dental students who accepted optional academic remediation based on poor performance on a progress examination had improved academic profiles compared to students who declined remediation.⁶ A 2013 systematic review by Cleland and colleagues, which included 31 studies on remediation interventions for mostly medical students, noted the rarity of control/comparison groups, long-term follow-up measures, conceptual frameworks, and adequate sample sizes in the studies examined.⁷ The authors concluded that more rigorous approaches are needed for developing and evaluating remediation interventions.

A 2012 study by Madden and colleagues analyzed the first-time pass rates on the North American Pharmacist Licensure Examination (NAPLEX) for pharmacy students who did not require remediation vs those who did require it.⁸ The study found a significant difference in NAPLEX pass rates in those students who did not require remediation vs those who did (97% vs 70%, p<.001). However, the authors stated that the reason for the lower pass rates for students who remediated was unclear and that further studies were required to determine whether there was a deficiency in the students’ thinking process or if there was a deficiency in the remediation process itself.⁷

Overall, limited data are available regarding the impact of remediation models of any type on student outcomes at US schools and colleges of pharmacy. The primary objective of this study was to assess the impact of student remediation status on standardized outcomes on the Pharmacy Curriculum Outcomes Assessment (PCOA) and the NAPLEX.

METHODS

This study included data from three cohorts of students: the graduating classes of 2016, 2017, and 2018. Information collected included total numeric PCOA scaled scores, NAPLEX pass/fail status (as a binary outcome), NAPLEX total composite scores, number of course remediations, and number of course repeats. Students who voluntarily consented to release their NAPLEX score to the institution as a part of the college’s assessment plan were included in this study. Students were included in the analysis of PCOA scores if results were available. Students were included in the NAPLEX analysis if NAPLEX scores and pass/fail results were available. The total number of students from the three cohorts who reported their NAPLEX score and thus were included in this study were 91% from the class of 2016, 95% from the class of 2017, and 96.4% from the class of 2018. The Sullivan University Institutional Review Board approved this study via an exempt review.

Students were divided into four groups: those who did not remediate or repeat a didactic course (non-remediation), those who successfully remediated one course and did not repeat any courses, those who successfully remediated more than one course and did not repeat any courses, and those who repeated one or more courses. It is important to note for this analysis that students were only placed in one category regardless of whether they had successfully remediated one course but failed another. For example, if one student successfully remediated course X but then failed course Y and the subsequent remediation of course Y, they were placed into the repeated group for the purposes of this study. Thus, no student fell into multiple categories in the individual analyses. All data were examined using SPSS Statistics, version 22 (IBM, Armonk, NY).

Outcomes were evaluated using Pearson correlations, linear regression, and logistic regression. For comparison of PCOA and NAPLEX scores, Pearson correlations were applied to measure the strength of the relationship between the different groups. This allowed for comparison of all groups against each of the remediation/course repeat groups in addition to the non-remediation group. Linear regression with dummy coding was conducted to assess the relationship between remediation, course repeats, no remediation, and NAPLEX total composite and PCOA total scaled scores. The non-remediation group was the reference group for the dummy coding. For all models, F statistics were reviewed to inspect model fit (p<.05) and R² values were reviewed to determine best fit. Unstandardized and standardized regression coefficients were reviewed for all variables in the blocks. The T-statistic was used to determine whether results were significant (p<.05). Variance inflation factor (VIF) coefficients (VIF<10) were reviewed to ensure lack of multicollinearity among the variables. Graphs of the residuals were reviewed to ensure data properly fit the model.

Binary logistic regression was used to assess the relationship of no remediation, remediation, and course repeats, with NAPLEX pass/fail rates. Chi-square statistics using the Hosmer and Lemeshow test were reviewed to ensure good model fit. Unstandardized and standardized regression coefficients were reviewed for all variables in the model. The T-statistic was used to determine whether results were significant. An odds ratio of less than one implied a negative relationship. For all results, a p value of less than .05 was considered significant.

RESULTS

A total of 250 students were included in the PCOA analysis and 237 in the NAPLEX analysis. The average total PCOA scaled score was 354 (SD=46). A total of 17 students remediated one course, 17 remediated multiple courses, and 22 students had to repeat at least one course. The average total NAPLEX score was 90 (SD=16) with a total of 15 students who had to remediate one course, 17 students who had to remediate multiple courses, and 15 students who had to repeat at least one course. The NAPLEX pass rate for students in this analysis was 83.2%. There were not enough students in each individual graduation year cohort to analyze by cohort; thus, only aggregate data were reported.

The PCOA scores had a small negative yet significant correlation to remediation status (Table 1). Linear regression results for this outcome (Table 2) showed an R² of .16 (p<.001), suggesting that approximately 16% of variation in student PCOA scores may be attributed to remediation status. The linear regression equation for the relationship between PCOA scores and remediation was as follows: PCOA score=363.6-37.8 (one remediation and no course repeats) -51.0 (more than one course remediation and no course repeats) -42.4 (one or more course repeats). Students with no remediation were the reference group in the dummy coding. The regression equation showed that students who remediated or repeated a course had a decrease in their PCOA score. Numerical reduction in PCOA scores was estimated to be 37.8 points for the one remediation group (p<.001), 50.9 points for the multiple remediations group (p<.01), and 42.4 points for the course repeat group (p<.01). The reference score for the non-remediation group was 363.8 points.

When examining NAPLEX score data, no significant correlation was found between NAPLEX score and the numbers of students who remediated one course, as shown in Table 3 (p>.05). However, small but significant negative correlations were found between NAPLEX score and numbers of students who remediated more than one course as well as students who repeated a course. In the linear regression model for NAPLEX scores (Table 4), the R² value was .15, indicating that approximately 15% of the variation in NAPLEX scores can be explained by a student’s remediation status, which was significant (p<.001). The linear regression equation for the model was as follows: NAPLEX score=93.3-8.1 (1 remediation) -16.5 (>1 remediation) -20.2 (one or more course repeats). Like the previous analysis, students with no remediation were the reference group in the dummy coding. Significant differences were found in the group that completed more than one remediation and no course repeats as well as the one or more course repeats group. The unstandardized coefficient was found to be -16.5 for the greater than one remediation group (p<.05) and -20.2 for the repeat group (p<.05). The 8.1-point decrease in scores for students who remediated only one course was not significant (p=.05). These data demonstrate that students who repeated courses and/or completed multiple remediations had a 16- to 20-point reduction in NAPLEX scores on average.

The NAPLEX pass/fail rates were also reviewed to determine whether a students’ ability to pass the NAPLEX was influenced by their remediation status. The logistic regression model (results shown in Table 5) predicted pass/fail correctly 83.2% of the time with the Hosmer and Lemeshow Test indicating a good model fit (p>.05). All the odds ratios were <1, indicating that a student’s remediation status had a negative effect on their ability to pass the NAPLEX. The odds of passing the NAPLEX were reduced by 80.4% for students who remediated more than one course and by 84.4% for students who had to repeat an entire year. Results from students who only had to remediate one course were not significant.

DISCUSSION

There is not a national consensus on the optimal strategy for student remediation. A 2013 report of progression and retention policies available on US pharmacy program websites reported data from 98 programs (80% of the 122 pharmacy programs at the time of data collection).⁹ Only 38 programs (38.8%) listed any information about remediation on the their website, though that percentage was higher among private programs (50.0% of private programs vs 29.6% of public programs). Fifteen (39.5%) of the 38 programs that listed remediation information reported using a course repeat strategy, while eight (21.1%) reported a summer (or intersession) strategy. Six programs (15.8%) reported a different remediation strategy, though it is unclear whether that meant a hybrid of course remediation and repeat or an entirely separate remediation strategy.⁹

In addition to lack of consensus on the appropriate model for remediation, data regarding outcomes associated with remediation strategies in pharmacy programs are lacking. Our study contributes pertinent analyses regarding student outcomes and NAPLEX success in the event that remediation is required. What may be additionally noteworthy regarding this analysis is that it was conducted in an accelerated pharmacy program in which intersession remediation may be viewed as especially challenging to implement. We could not find any studies associating progression data with PCOA outcomes, and only a few associating progression data with pre-NAPLEX or NAPLEX results.

Madden and colleagues published an analysis of the impact of remediation status on NAPLEX first-time pass rates in three cohorts of students in a private pharmacy program.⁸ In their analysis, 5.0% of their students required remediation. The average passing rate of students in the non-remediation group in this published study was 97% (SD=0.85%), while students in the remediation group had a mean passing rate of 70% (SD=10.5%). The remediation program described in this article was given in the summer term between sessions, though with some key differences from our model. In the model described by Madden and colleagues, students who earned a final grade in a course of 60%-69.4% took a comprehensive remediation examination following the course. If the student earned a 70% or greater on that remediation examination, the student was allowed to continue to progress through the program. If a student earned less than a 60% in the course or failed the comprehensive remediation examination, the student would be enrolled in a remedial course during the summer. Our model includes a remediation examination, but students who are unsuccessful in remediation must repeat the course the following year rather than in a summer session, which is unavailable in our year-round program. Additionally, the type of analysis performed in our study (linear regression) is predictive in nature as compared to the analysis performed by Madden and colleagues.⁸

A 2014 study by Chisholm-Burns and colleagues assessed the impact of various student factors, including academic remediation, on pre-NAPLEX scores in a cohort of student pharmacists at a large public pharmacy program.¹⁰ The pre-NAPLEX is similar to the NAPLEX in structure and is intended to be used as a practice test for students who are candidates to take the NAPLEX. Both examinations are scored 0-150, and the pre-NAPLEX contains items that have previously appeared on the NAPLEX.¹¹ However, unlike the NAPLEX, which is administered under controlled conditions at a testing center, the pre-NAPLEX may be taken anywhere the test taker has an internet connection.¹¹ In the program described in the study, academic remediation was defined as repeating an academic year if the student had a cumulative grade point average (GPA) less than 2.3 (out of 4.0), a semester GPA less than 2.0 for a second time, or if the student earned a grade of less than a C- in two courses throughout the curriculum. The final stepwise linear regression model built from student data included pharmacy school cumulative GPA, academic remediation, academic probation status, and score on the Pharmacy College Admission Test (PCAT). Together, these variables were significant, and accounted for 22% of the variability in pre-NAPLEX scores. Academic remediation in the form of course repeat accounted for 3.5% of the overall variance in pre-NAPLEX scores, while academic probation accounted for 4.7% of the variance and PCAT accounted for only 2%.

In comparison to our report, the study by Chisholm-Burns and colleagues described repeating a course as the only remediation strategy for its data analysis. Therefore, it is impossible to know whether a different type of remediation strategy, such as an intersession model, would have similar results.¹⁰ Additionally, Chisholm-Burns and colleagues found that both academic probation and remediation contributed positively to a student’s pre-NAPLEX score, meaning improved student performance on this examination by those who had experienced academic difficulty during the program.¹⁰ The authors postulated the positive contribution of academic probation and remediation on pre-NAPLEX scores resulted from these students receiving enhanced preparation for the examination by repeated exposure to subject matter in their coursework, or an improved motivation due to their previous academic indiscretions. However, it is difficult to determine whether motivation for practice examination like the pre-NAPLEX differed or had an impact on outcomes as compared with motivation for the high-stakes NAPLEX, as neither the Chisholm-Burns study nor our study assessed test-taker motivation. As noted earlier, the pre-NAPLEX does not simulate the actual testing environment but is intended to help prepare students for the NAPLEX in terms of question type and overall examination construction.¹¹ Our study found overall decreased performance on both the NAPLEX and PCOA for students who remediated or repeated coursework in the program, though students who remediated only one course showed a nonsignificant decrease in NAPLEX scores. Further study is needed in this area to elucidate the overall impact of various remediation strategies on academic outcomes.

Another analysis from the same large public pharmacy program was published in 2019 by Spivey and colleagues.¹² In this study, data from three cohorts of students were analyzed using logistic regression in three different models: factors predictive of on-time graduation, factors predictive of dismissal from the pharmacy program, and factors predictive of passing the NAPLEX on the first attempt. In terms of the role of remediation/progression, the authors included a variable representing a student’s appearance before the program’s Academic Standing and Promotion Review (ASPR) committee. An appearance before the ASPR committee was necessitated by a student earning two grades of less than a C- in coursework, or if the overall GPA fell below a preset threshold. The ASPR committee’s recommendation could be for the student to repeat the course or repeat an entire semester or year of the program, or to be dismissed from the program. Thus, this dichotomous variable of appearance before the ASPR committee was more akin to a composite variable in this analysis as it included a wide range of progression outcomes. In terms of raw data, 25 students were identified as not passing the NAPLEX, of which 52% had appeared before the ASPR committee. This is compared to 8.5% of students who passed the NAPLEX on the first attempt and who had also appeared before the ASPR committee during their academic career. The final logistic regression model for factors predictive of passing the NAPLEX on the first attempt included appearance before ASPR committee among other variables and was significant. The model explained 37.7% of the variance in passing the NAPLEX. Individually, appearing before the ASPR Committee significantly decreased the likelihood of a student passing the NAPLEX. No other variables included in the model were significant. These data support our own study findings that remediation or course repeat was predictive of lower NAPLEX and PCOA scores.

There are limitations to this study. Overall, rates of didactic course failure were low, which may have resulted in a type II error for comparisons that were not significant. A power calculation for these comparisons was not performed because of the lack of similar studies from which to estimate significant values. Students who remediated multiple courses or who were required to repeat the course may have performed poorly regardless of the remediation efforts made and potentially confounded the results. Additional confounding factors were not evaluated because of the relatively small number of individuals included in the analysis. Also, NAPLEX results are voluntarily provided to institutions via student consent; thus, not all results were available to evaluate. Another limitation to consider is the generalizability of the data, as these data were derived from a single institution with a specific remediation policy and procedures. Also, three-year accelerated curriculum remediation outcomes may not mirror those of schools with other curricular models. Other institutions using different methods to remediate or advance students through the curriculum may not have similar outcomes.

CONCLUSION

Results of this analysis provide insight into the impact of an intersession remediation strategy on academic and licensure-related outcomes in an accelerated PharmD program. Our study showed that pharmacy students who had to complete a single course remediation had NAPLEX success rates similar to pharmacy students who completed all of their didactic coursework without repeating or remediating a course. However, a student having to complete multiple remediations and/or repeat a course contributed significantly to scoring lower on the PCOA and NAPLEX. Further research such as a multi-site analysis accelerated PharmD programs is needed to fully elucidate the effect of remediation on outcomes of interest and to identify best practices in progression strategies.