Use of Learning Management System Data to Predict Student Success in a Pharmacy Capstone Course

DISCUSSION

This exploratory analysis of LMS data indicated that students could be classified accurately into groups based on page view counts using GBTM. Page view counts alone, the only engagement metric available at the regular intervals required by GBTM, did not successfully predict student achievement until stratification by attendance at virtual synchronous learning sessions was performed. In this respect, we found an inconsistency between predictors of student performance and an assumption of GBTM, that cases do not vary within groups.¹⁹ This need to supplement the insights gained from GBTM with an additional measure of student engagement, ie, virtual synchronous learning session attendance, likely explains the effects of two factors.

The first factor is that, unlike medication adherence research using GBTM, the findings from our research suggested relatively stable patterns of student engagement.¹⁵ A possible explanation for this finding is that a student having an underlying predisposition to success prompts meaningful engagement. One path model analysis suggested that student grade on a prerequisite course was the strongest predictor of performance in a higher-level anatomy class, with LMS engagement measured throughout the anatomy class acting as a mediating factor.¹¹ This interpretation could explain why consistent, live interaction with instructors, which is a stable trait measured throughout the quarter, had value as a predictor of success beyond information provided by GBTM. Although that finding does indicate less than the anticipated value in using GBTM for this class, during the final didactic quarter of the curriculum, students’ patterns of engagement and predisposition for success were probably well established. Whether this finding would be similar in PharmD coursework delivered earlier in the curriculum or in different disciplines is unknown.

The second factor is the limitation of data available in the LMS. Exact time spent online might have provided a better weekly measure of active learning time than page view counts did. The post hoc finding of a negative association between total page views and total online time supports this interpretation by suggesting that some students with many page views clicked quickly through the pages, likely without reading them carefully. Our LMS made minutes of online time visible to instructors only for the entire course, rather than weekly as needed for GBTM. This constraint, along with the wide variety of LMS metrics reported in the literature, suggest a need to use LMS data that represents active learning, such as engagement in interactive tasks or counts of discussion postings.^10,11,14 Instructors interested in using this technique could request necessary system modifications from the LMS developer to obtain regular access to additional metrics already being collected.

To the authors’ knowledge, this is the first time GBTM has been used to correlate LMS data with student success.^15,16 Despite the limitations encountered in application of GBTM to this course, our success in classifying distinctive student groups with GBTM suggests the possibility of using this approach not only in pharmacy schools but also in other higher learning institutions to help identify at-risk students for early intervention to improve success rates. With the shift to more online courses, additional methods to quantify student engagement will continue to be important, despite LMS limitations.^10-14

Unlike blended courses, where in-person instruction makes LMS data incomplete, the abrupt shift to 100% online course delivery during the COVID-19 pandemic afforded instructors expanded opportunities to evaluate predictors of success. The finding of a potential association between course grade and virtual synchronous session attendance presented such an opportunity and has implications apart from use of GBTM. This finding may be explained by the unique benefits offered by synchronous and asynchronous teaching methods. While asynchronous instruction offers the student scheduling convenience, the opportunity to work at an independent pace, and time to process information, synchronous instruction can increase motivation because of the personal connections established with the instructor and student peers.²⁴ Additionally, compared with recorded lectures, synchronous sessions were more similar to the pre-pandemic learning environment, as our students were accustomed to in-person course delivery. Nonetheless, this research adds to a body of work suggesting better course performance when students engage with peers or the instructor, either online or in person.^8,10,23,25 Although this principle is seemingly obvious from the perspective of most faculty, our findings suggest that students working within asynchronous and anonymized didactic formats may be unaware of the need to engage with their instructors.

Several limitations should be noted. First, this evaluation represents the findings of one course offering at one program and is thus limited by small sample size and limited application outside of the context described. Second, the context for the movement of the course to 100% online administration was the global COVID-19 pandemic, when students may have experienced increased stress that could have affected their course engagement and/or performance. Studies done in different contexts may have produced different results. Third, students may have interacted to varying degrees with the course material outside of the LMS, such as by downloading handouts to view offline, making study materials, or engaging in other active-learning activities. Neither offline engagement nor the possibility of students viewing the material on a peer’s LMS could be measured in this research and remains an area for future inquiry. Finally, while attendance was taken twice at each virtual session, we did evaluate the degree of student engagement at these sessions.