Strategies, Ideas, and Lessons Learned While Engaging in SoTL Without Formal Training

Educational Research

Like any research project, educational scholarship begins with a question or idea. When scholars ask key questions about academic pharmacy and obvious answers cannot be found, an opportunity for scholarship presents itself. For example, the following questions all resulted in educational articles that were published in the American Journal of Pharmaceutical Education (AJPE): How do other programs run their student governments? What other degrees do pharmacy programs offer? How are introductory pharmacy practice experiences (IPPEs) delivered throughout the academy? What are residency directors looking for in applicants? What are the criteria used to offer graduation honors?^2-6 From all of these questions, the researcher can begin to stratify potential differences to develop a clear picture of how a program addresses the question. For example, when exploring IPPEs, questions that were asked included: What is the maximum distance a student would have to travel? Can a student complete a rotation at the same store, or the same chain of stores, that she works in? Careful consideration of a topic and perhaps a conversation with colleagues at other institutions can help ensure the questions being explored address all of the potential parameters that should be considered.

Before starting any project, a faculty member should consult with their institutional review board to determine what approvals, if any, they would require for the project. Surveys are one shot opportunities because once they are distributed, it is quite difficult to acquire any additional information from the respondents and, importantly, clarify any question that may be unintentionally vague. Surveys should be vetted through additional colleagues and preferably by faculty members at other institutions that have different educational models and realities (public vs private, three-year vs four-year, class size, etc). Finally, when designing surveys, and other instruments involved in SoTL, the scholar should carefully consider the validity (does it measure what it is intended to measure?) and reliability (is the grading consistent on the part of the grader or graders?).^7,8 Careful evaluation of response rate is essential when analyzing the generalizability of results.

Small Activities and Projects

Scholarship of teaching and learning projects can also be designed to address a discrete, focused learning objective within an existing course or laboratory sequence. Often, educators will create an activity in response to a gap in their students’ knowledge or to delve into greater depth on a topic than can be covered in the classroom. Educators can draw on his or her personal areas of clinical or research expertise as a way to interject new technology, information, or processes into a course. In other words, it can be helpful to “use what you know” as a starting point for designing an exercise. By planning an in-class or out-of-class activity that the students do on their own, the students are offered an opportunity to use the information learned during class in a different context, which can help the students to better apply the information in other settings.⁹

Once an educator has identified a learning gap or a general idea for the exercise, it is helpful to craft the learning objectives early in project development, as the learning objectives should serve as a guide to designing the assessments and activities. There are numerous resources online that can provide guidance in writing learning objectives, particularly through many university teaching centers. As a general guide, it is helpful to keep the objectives student-centered and focused on a specific, measurable outcome that is expected of the student. Additionally, to improve generalizability to other Doctor of Pharmacy (PharmD) programs, use of national guidelines such as the Center for the Advancement of Pharmacy Education (CAPE) Outcomes¹⁰ and the AACP Core Entrustable Professional Activities (EPAs)¹¹ should be strongly considered when developing and mapping the performance assessment. With learning objectives in hand, the instructor can plot how to incorporate assessment strategies that address each objective. Often, multiple pieces of overlapping assessment data can be included in the project in order to test each learning objective from multiple angles. By including assessment pieces within the exercise and in other settings in the course, one can assess the students’ competence in each learning goal as well as retention over time. For instance, questions within the project itself that directly address an objective as well as questions on examinations within the course that relate to the objective outside of the project can provide different perspectives and validation.

In addition to direct evidence of student learning (eg, essays, worksheets, examination questions), indirect evidence of student learning can be collected. Indirect evidence addresses the students’ perceptions of what they have learned. Often, perceived learning of material is assessed through pre- and post-exercise surveys that ask students to report their relative comfort with the material or skills in the exercise. Although most journals require direct evidence for publication, indirect evidence can be valuable for determining whether the students’ perceptions of learning match with their mastery of the material. While quasi-experimental studies can provide helpful insight and evidence of scholarly teaching, those seeking publication in AJPE and other major journals should also consider the value of using a true experimental control, where participants are compared to historical cohorts or even randomized to different interventions.¹

An important aspect of SoTL is the dissemination of findings for peer review, critique, and ultimately sharing with fellow educators. During the planning stages of a new project, it can be useful to search the literature for related content as well as to review submission requirements for relevant journals. If particular assessment data or study design is required for publication in a journal of choice, it is easier to incorporate that information into an assessment plan early in the process as opposed to when the data has already been collected. The Journal provides diverse examples of SoTL projects and activities on different topics, including point of care testing for infectious diseases, a project for medication error evaluation, and a flipped classroom approach to teaching cardiac arrhythmias.^12-14 For those educators who are interested in publishing a SoTL activity, these examples can provide an outline and perhaps inspiration for ways to approach assessment, data collection, and analyses of a SoTL project.

Redesign of an Existing Course

Revising a course to alter the content delivery method and (hopefully) improve student performance and retention is another way in which one can create SoTL that provides value to the academy. A course redesign may be warranted when traditional, lecture-based models are not consistent with evidence-based practices in teaching and learning or when student performance is not meeting the educator’s expectations. There are a myriad of methods, from case-based learning, problem-based learning, and discussions, to flipped classroom and blended learning, that can be used to incorporate active participation, more critical thinking, and higher levels of Bloom’s taxonomy. The scholar should consider the ultimate goal of the course and content, the methods or combination of methods that will best accomplish that goal, and how to address any curricular restraints put in place by the program. For example, it might be challenging to add a credit to the course to allow for more time for active content delivery, but optional recitation sessions or use of smaller sections of students may be one way to allow time for active learning.

When considering a redesign and hoping to publish findings, the point of comparison becomes the previous iteration of the course which can be used as a historical control group. Thus, the scholar should be mindful of the overall changes being made and not unnecessarily change content and topics. Data points, such as overall grade point average (GPA), grade in a prerequisite course, student work hours, age, and other factors are helpful in describing the samples as well as adjusting for potential confounding variables. Ideally, data from multiple offerings of the course in both the pre-design and post-redesign cohorts should be compared.

Numerous examples of course redesigns exist within the literature, ranging from a team-based learning model,¹⁵ a process-oriented guided-inquiry learning (POGIL) model,¹⁶ problem-based learning model¹⁷ or the creation of recitation sections¹⁸ that can be accessed for further information. Furthermore, faculty members can also publish the development, design, and implementation of a new elective course that demonstrates a unique topic.^19-21 These types of manuscripts describe the course design, course objectives, and course outcomes, generally in a detail that allows readers to design a similar educational opportunity at their institution.

Longitudinal Evaluation and Assessment

Pharmacy education should prepare students to apply the skills developed and practiced in their didactic coursework in more authentic settings, such as skills laboratory or case conferences settings, experiential settings, and ultimately in practice. Recent shifts in accreditation standards highlight the need for students to be prepared for advanced pharmacy practice experiences (“APPE ready”), interprofessional engagement (“team ready”), and to provide pharmacy patient care (“practice ready”).²² With this in mind, it has become increasingly vital to ensure students are able to not only recall knowledge and perform skills at the conclusion of a course, but also months or even years after the course was completed. This area of pharmacy education is ripe for scholarship.

Conducting longitudinal SoTL begins with a similar foundation as the course or activity level. The scholar should identify a teaching challenge or question with long-term relevance, examine current teaching pedagogy, and prepare an objective measurement to assess students’ ability to recall the knowledge, perform the skill, or exhibit the attitude of interest. In this type of scholarship, it is even more essential to consider each potential point of reinforcement, practice, or relevant integration (horizontal or vertical) as each of these points could impact students’ performance.

Scholars engaging in longitudinal SoTL need to take care in selecting the most appropriate design in order to ensure that they are best meeting their objective without introducing unnecessary confounders. A quasi-experimental method where student performance is linked at each measured time point can be appropriate for investigating whether students retain or even build upon very specific knowledge, skills, or attitudes after a course concludes.²³ It yields results with improved validity; however, the scholar must be purposeful in developing methods to link student performance results across multiple assessments (potentially in a blinded way, depending on local institutional review board requirements) as well as define completion thresholds for inclusion. Alternately, there may be longitudinal research questions that do not necessitate linking of multiple assessments on a student-by-student basis. For example, a cross-sectional “snapshot” evaluation may be appropriate if the measured skill is not repeated throughout the curriculum and logistically simpler if there is no pre-embedded assessment of this skill.²⁴

Like other types of SoTL, longitudinal evaluations should not simply assess student perceptions or self-efficacy but must also measure performance, outcomes, and/or level of entrustment. Care should be taken that each performance assessment included in the study is of comparable rigor, yet sufficiently distinct in content to avoid introducing repetition bias. The level of “stakes” should also be consistent to avoid introducing confounding variables related to student effort. For example, comparing an informal, in-class activity to a structured OSCE may be problematic. Embedded assessments within OSCEs, pharmacy practice skills laboratories, or capstone-style courses may represent ideal settings for longitudinal SoTL.