Detection of Academic Early Warning Signs and Effective Intervention “Takes a Village”

EARLY WARNING SIGNS AND INTERVENTIONS

Since the mid-1990s, pharmacy education researchers have sought to identify variables that might help predict which students are at risk of encountering academic difficulty in pharmacy curricula. ^1-4 Early warning interventions proposed in the health sciences education literature involve admissions offices, which can identify warning signs preadmission, ^5-7 student services, which can help at-risk students build study skills, ^8-10 and administrators, who can develop suitable policies. ¹¹ Identifying key characteristics upon preadmission is encouraged by the Accreditation Council for Pharmacy Education (ACPE) Standard 17 (“Progression”), which calls for schools to connect admissions data as well as formative and summative assessment data with student academic performance. ¹² However, student academic difficulty may be a symptom of underlying challenges beyond intellectual capability or the inability to meet academic standards and, thus, may be detectable only after matriculation. Education researchers in other health professions have found that either forced or voluntary departure decisions result from a cumulative effect of multiple stressors in students’ academic, social, and/or external environments. ^{13, 14}

Much of the higher education literature addressing academic difficulty attempts to narrow down and isolate the cause of academic difficulty to a single source inherent within the student. ¹⁵ This approach minimizes the complexity of students as individuals and ignores the multifaceted skill set that pharmacy students are expected to develop to become competent and responsible practitioners. ¹⁶ Academic difficulty is a manifestation of a complex interplay among multiple factors. Complex problems require solutions that are flexible and, perhaps, complex in their own right. Furthermore, given that many of the stressors contributing to a student’s academic difficulty remain relevant after the student transitions to licensure and practice, those who learn to try multiple solutions, seek support services, and build resilience early in an academic program may find that the benefits last long after graduation.

Due to variations within and across pharmacy programs, individual programs should identify and select an optimal array of global and local risk factors that specifically challenge their students. While most programs have established intervention policies concerning subpar academic performance within a course, it is equally important for programs to consider which nonacademic accountability indicators may also warn of future academic difficulty and warrant attention or intervention. Table 1 includes potential academic, accountability, and behavioral factors that can trigger an intervention with a student. To systematically collect and organize these data, collaboration is required among multiple stakeholders involved in ensuring student success to access, report, and reconcile the inputs. When these factors trigger an intervention, these stakeholders may be able to identify and address additional personal factors contributing to the student’s behaviors, such as mental/behavioral health, food and other resource insecurities, confidence, or learning disabilities.

Treating these factors or indicators as data and incorporating them into one searchable database can help the student support team identify trends and provide a more holistic assessment of a student’s capability for success. Further, demonstrating that accountability and behavior metrics can be used to promote positive academic outcomes and successful professional behavior ¹⁶ can consistently communicate to students, faculty, and staff the importance of integrating attitudes and behaviors with knowledge and skills in pharmacy education and in the profession.

DATA ACCESS, SHARING, AND ANALYSIS

The next step toward identifying and supporting at-risk students is communication among faculty, course coordinators, student services staff, and assessment professionals. Multiple instances of the behaviors described in Table 1, in conjunction with poor grades on assignments or exams early in one or more courses, present a more complete picture regarding an individual student’s difficulties than will grades alone.

However, awareness of data markers is not enough; acting upon these data is essential. A professional program cannot be so risk averse or fearful of violating student privacy that they withhold information that, if shared, could lead to more engagement and intervention to benefit the student. The Family Educational Rights and Privacy Act (FERPA) should not prevent faculty and staff from communicating with one another to promote positive student outcomes. Instead of conceiving FERPA as a barrier, programs should feel an obligation to have such discussions early and often to help at-risk students thrive. A common misinterpretation of FERPA is that such conversations regarding disclosure of education records are prohibited when, in fact, they are not:One of the exceptions to the prior written consent requirement in FERPA allows “school officials,” including teachers, within a school to obtain access to personally identifiable information contained in education records provided the school has determined that they have “legitimate educational interest” in the information. ¹⁷

Formalizing the proposed student success team approach and transparently communicating to students what data may be examined for these purposes may also help reduce these concerns.

Systematically analyzing data and identifying factors that contribute to student difficulty can mean the difference between proactively supporting students at risk and responding reactively only after a student is in crisis. In this proposed systematic model, student performance data from either a “homegrown” or a commercially available data monitoring and analysis system could be reviewed monthly by a program’s assessment professional. Any concerns in a student’s data would be brought to the attention of a program’s academic progression or student affairs committee. The information regarding a specific student who was experiencing academic or behavioral challenges could then be freely shared with each member of that student’s support team, who would view concerns about a student through the lens of their own expertise. Instructors could focus on academic performance within a course; student services professionals could connect students with collegiate resources, such as tutors or institutional-level resources to assist with difficulties outside of class; and faculty mentors could evaluate accountability behaviors where intervention would be impactful. The student would then be contacted by an appropriate member of the program’s student support system for intervention (eg, student services, the dean of student affairs, the student’s academic advisor, etc).

Unfortunately, the team described above is missing an assessment professional; this is typical of current approaches to student support. Yet, assessment professionals can help bridge individual faculty/staff efforts and transform snapshots into an effective, collaborative system. By collecting, analyzing, and sharing student performance data, assessment professionals can assist with creating a data “dashboard” for each student and create a comparator group using warehouse data collected from a particular cohort, the entire student body, or beyond. Assessment professionals can also help to recognize trends within the data, refine the formula used to identify at-risk students, and measure the impact of interventions to determine which approaches positively and significantly influence outcomes.

Early warning systems are not foolproof; some medical education programs that have implemented systems using both academic and nonacademic warning flags have reported instances where students who had no flags still encountered academic difficulties. ¹⁸ Despite these shortcomings, including assessment professionals could become the key to refining the collaborative early warning and intervention model, as these professionals can use the identification and communication tools at their disposal. This model is appropriate in a culture that values information sharing and collaboration.

OVERCOMING CHALLENGES

Implementing an early warning and intervention approach takes time and concerted effort. A culture of assessment must be established that incorporates consideration of disparate data points (including time), facilitates communication, and closes the feedback loop. Regarding a culture that considers various data points and communication, a poor exam score is an obvious trigger to a course coordinator that a student is possibly at risk; however, communication across all stakeholders remains essential. An effective tracking system starts with systematic collation of cues (data points) from a centralized collections process using a learning management system, exam software, or other tools. These data must then be examined regularly for patterns. Assessment professionals are particularly well positioned to assist in this process, identifying signals that might indicate that a student’s academic difficulties transcend a single course. As noted above, these signals are then shared with the school’s student progression or student affairs committee and, subsequently, with an interdisciplinary group of faculty, staff, and advisors who can intervene with at-risk students early. This interdisciplinary group can help craft a written notification to the student (and to the student’s academic advisor) using compassionate verbiage so that the student feels encouraged and supported rather than targeted as a potential failure.

Closing the feedback loop for all team members is a frequently overlooked challenge in managing an early warning and intervention process. Feedback regarding the student’s subsequent performance, including sharing communications between a student’s advisor and faculty, must be purposefully provided to the entire team to assess an individual student’s progress. Additionally, the team needs to check with the faculty teaching the same cohort of students to monitor the performance trends of the class as a whole. Without this follow-through, stakeholders lack the necessary reinforcing knowledge that the monitoring/intervention process is important, feasible, sustainable, and is working.

A key to a successful system of monitoring and early intervention requires stakeholders to value and to commit to the process, and it requires diligence even when the student seems resistant to early intervention or avoids such overtures entirely. These behaviors can often be a sign of embarrassment or shame, particularly for a historically high-achieving student. Sometimes students need to be led to the discussion table. And while it can be frustrating and heartbreaking when a student ignores opportunities for support, such setbacks do not mean that all efforts to reach at-risk students are futile. Despite the challenges and potential barriers outlined above, a systematic, holistic approach for providing student support has the largest potential for identifying at-risk students early and, hopefully, assisting them before they encounter insurmountable academic difficulties.