Determining Indicators of High-Quality Application Activities for Team-Based Learning

INTRODUCTION

Team-based learning (TBL) is intended to flip the classroom, freeing up class time for application activities in which students work in teams to solve the kinds of problems they will face in the future.¹ However, the design of application activities has not been a strong focus in pharmacy’s TBL literature. The use of TBL in US colleges and schools of pharmacy has been described, including factors affecting implementation² and perceptions of faculty members on educational outcomes.^2,3 Pharmacy TBL studies have focused on student knowledge attainment and progression,^4-12 student perceptions,^10,13-17 professionalism,^18,19 teamwork skills,^4,18-20 student performance on individual readiness assurance tests (iRATs) and team readiness assurance tests (tRATs),^21-23 student engagement and critical-thinking abilities,²⁴ individual variability in learning,²⁵ and use of feedback on TBL exercises and activities.²⁶

A primary goal in TBL is to move beyond content coverage and ensure that students have the opportunity to practice using course concepts to solve the kinds of problems they will likely face in the future.¹ One of the biggest difficulties for TBL instructors is creating effective group assignments that promote deep thinking and engaged, content-focused discussion.^3,27,28 In health professions education, instructors should use authentic problems that require students to make decisions like those encountered in the clinical arena.²⁹ Understandably, some application activities meet with more success than others. High-quality application activities are not easily constructed and often require repeated rounds of revision to achieve optimal learning outcomes.

In providing guidance for creating effective application activities, Michaelsen and Sweet describe four principles called the “4S rules”: significant problem, specific choice, same problem, and simultaneous report.¹ While this guidance focuses on the intent of application activities and elements of their structure and execution, instructors still face many decisions in the design of application activities. To optimize class time, more information is needed on the indicators of quality application activities. What makes a high-quality application activity that results in the learning outcomes we expect to achieve? Isolating and describing quality indicators is a logical step forward in improving this important component of TBL. To that end, the objective of this study was to determine the quality indicators for TBL application activities in pharmacy education. Once quality indicators are identified, interventions that aim to increase quality, such as training programs, checklists, and peer review, can be better examined to determine their effectiveness.

METHODS

This study used a two-round modified Delphi process to garner opinions from pharmacy TBL experts to determine the quality indicators of TBL application activities in pharmacy education. The Delphi process is a methodological technique that requests and refines the collective thoughts and opinions of a panel of experts.³⁰ Data are collected from participants and, through subsequent rounds, are summarized and presented back to them to obtain feedback and measure their level of agreement. The overarching goal is reaching a consensus on the topic being studied.³¹ Specifically, in pharmacy education the process has been used for creating consensus on guiding principles³² and competencies³³ for student leadership development, developing consensus on competencies for professional advocacy,³⁴ investigating students’ perceptions of professional engagement,³⁵ defining bullying behaviors in clinical training of pharmacy students,³⁶ creating consensus on criteria to assess communication skills of pharmacy students,³⁷ and defining the roles and education of veterinary pharmacists and veterinary pharmacy specialists.³⁸ In examining quality, the Delphi technique has been used to select quality indicators in healthcare,³⁹ develop quality criteria for patient decision support technologies,⁴⁰ and create quality indicators for general practice management.⁴¹

Experts have argued that the success of a Delphi study is determined by the expertise of the panel chosen.⁴² Recommended criteria for expert panel selection include competency within the specialized area⁴³ and credibility.⁴² To identify potential expert panelists for this study, a search of the pharmacy TBL peer-reviewed literature was conducted. Four specific journals were consulted and searched, including the American Journal of Pharmaceutical Education, Currents in Pharmacy Teaching and Learning, Innovations in Pharmacy, and Pharmacy Education, using each of the following key search terms individually and combined: team-based learning, team based learning (no hyphen), classroom, and education. All authors associated with the identified articles were emailed an online screening survey asking about their expertise and background in TBL. The responses to the screening survey were reviewed by members of the research team, and participants were selected to serve as the panel experts based on the following inclusion criteria: authorship of a peer-reviewed TBL pharmacy paper, at least four years of personal experience using TBL, teaching at least eight hours per year in the classroom, designing at least eight TBL sessions, and experience in training others to use TBL.

Agreement has not been established within the literature regarding the optimal number of experts needed for a Delphi study.^30,43,44 For a homogenous group, 10 to 15 panelists has been suggested as a sufficient number.⁴⁵ Generally, a panel is composed of under 50 people.⁴⁶ Considering the pool of available experts, a panel size of 20 was deemed appropriate for this study. Selected experts were invited via email to participate in the Delphi process.

In round 1, panelists were given a definition of application activities based on TBL literature and research team expertise. Application activities were defined as focused, in-class, team exercises, assignments, or tasks aimed at developing higher-level thinking skills in complex situations. In round 1, panelists were asked five open-ended questions about their most successful TBL application activities, including methods for creating positive student outcomes, satisfaction with and quality of the activity, and characteristics of strong activities. Panelist responses were reviewed and quality indicator statements were generated from themes in the panelists’ comments. Panelists made additional comments focused on the facilitation and outcomes of the application activities (vs design of application activities). For example, panelists commented on the importance of “keeping the emphasis on student-led discussion” after the team choices had been revealed. Comments focused on facilitation or outcomes were not studied further in subsequent rounds.

In round 2, a report from round 1 was returned to the panel containing the quality indicator statements (eg, the activity uses authentic pharmacy challenges or situations) and quotes from panelists that were used to create the summary statements (eg, “cases reflected important, practical scenarios that pharmacists commonly face” and “cases were authentic, meaning the students could see themselves doing the case in real life”). Panelists were asked to indicate their level of agreement with each of the quality indicator statements using a four-point Likert rating system (ie, strongly disagree, disagree, agree, and strongly agree). In addition to rating the quality indicator statements, panelists were specifically asked to comment, particularly where they disagreed with an indicator, to aid in refining the statements. In Delphi studies, there are a variety of recommendations on consensus levels⁴²; however, no definitive guidelines are available.^30,42 Consensus on agreement has generally been no lower than 55% and potentially up to 100%.⁴² Prior to initiating round 2, consensus was defined as 80% of panelists agreeing or strongly agreeing on the quality indicators in TBL application activities.

A comment period can provide additional context and background on the panelists’ perspectives. The quality indicator statements from round 2 were sent to the panel with the instructions: “If desired, please feel free to comment on this indicator. The comments will help with the descriptors/additional detail that accompany each indicator during reporting.” To give additional direction in understanding each indicator, descriptors were created cooperatively by two investigators using direct quotes and consolidating common phrases from the panelists’ comments in round 2 and the comment period. The descriptors were then reviewed by a third investigator after consulting original response data and approved by consensus among the three investigators. After approving the descriptors, the indicator statements were categorized by the authors into four groupings: structure; content; design and intent; learning actions and results.

For all rounds and the comment period, the web-based survey program Qualtrics (Qualtrics Labs Inc., Provo, UT) was used to collect panelist responses and comments. Both the University of Minnesota and Regis University Institutional Review Boards determined the study did not meet the regulatory definition of human subjects’ research.

As a means of verifying the results of the panelists’ work following completion of the Delphi process, the list of quality indicators was reviewed by two TBL practitioners and two faculty members trained and specializing in pharmacy education. Connections between the indicators and available TBL literature were identified by the TBL practitioners, along with applicable literature to aid in using the findings. Likewise, connections between the indicators and general educational concepts, theories, or philosophies were identified by the education specialists, along with applicable literature to aid in use.

RESULTS

One hundred three leads and coauthors were asked to complete an online screening survey regarding their expertise and background in TBL. Fifty-one authors’ responses were reviewed by the research team using the inclusion criteria. Twenty-three participants met the criteria to serve as expert panelists for the study. In round 1, 20 panelists (ie, 87% of those eligible) responded, and their responses were used to generate the quality indicator statements. In round 2, 17 panelists (85% participation) indicated their level of agreement with the quality indicators. In an open, optional comment period, 15 (75%) of the 17 panelists provided suggestions to help expand the descriptors used in the indicators.

Sixteen quality indicator statements were generated from the panelists’ responses in round 1. Of these, the panel approved 14 statements by the predefined 80% consensus in round 2 (see Table 1 and Table 2). The two indicator statements that did not meet consensus were “the activity has multiple right answers” (76% agreement) and “the activity incorporates elements from school-specific emphases (eg, faith, underserved)” (53% agreement). Ten of the indicators aligned with three general categories related to a constructivist approach: authentic learning, alignment of content with course objectives, and depth of thinking (Table 3).

DISCUSSION

The verification process identified several points of alignment between the indicators and contemporary higher-education thinking and approaches. Constructivism suggests the active involvement of the student is an essential component of the teaching/learning process. Indicator 14, “requires involvement/engagement of all students on the team,” speaks directly to this need for involvement. In constructivist theory, each student brings to the classroom a unique set of past experiences, assumptions, skills, and knowledge, and uses these to make meaning from the new information they are exposed to in the course.⁴⁷ When students are encouraged to use prior knowledge and experience as part of the learning process, they are likely to experience positive outcomes, including retention of information, motivation, persistence, creativity, and overall understanding of the course material.^48-50 Kang and colleagues have encouraged the incorporation of constructivism in pharmacy education.⁵¹

An authentic learning environment is one in which students are presented with problems or cases that closely mirror a real-world environment.⁵² The characteristics of an authentic learning activity include: real-world relevance, high level of complexity, opportunities to collaborate and reflect with peers, and an outcome that includes a finished product as a part of the assessment (as opposed to an examination being the only assessment of learning).⁵³ In a health professions program, students may find the content more accessible if the learning mirrors what they might see in future practice. This engagement causes deeper levels of understanding, critical thinking, and motivation to learn more.⁵⁴ Perhaps most importantly, an authentic environment is an active-learning space in which students are encouraged to contribute, as opposed to a passive environment in which memorization of information is the primary focus.

The theoretical basis for the alignment of objectives, activities, and assessments is described in the work of Wiggins and McTighe, who championed the “backward design” approach to course and curriculum design.⁵⁵ This approach suggests courses and curricula should be constructed by first clearly identifying the precise learning objectives. The next step is to identify appropriate assessments that would ensure students have achieved the learning objectives. Only then are learning activities developed that help students pass assessments. This approach has been generally accepted in higher education as the most robust and sound method to use when designing a course or curriculum. In addition, pharmacy educators have advocated for reviewing the alignment of course objectives with assessments and learning activities in order to improve assessment content validity and consistency, which improves the overall outcomes of the course.^56,57

Rather than assignments that simply expose learners to new information, Michaelsen argues for assignments that encourage depth of thinking and require active involvement in higher-level cognitive skills.⁵⁸ He states that using higher-level thinking and problem solving is the key to promoting both greater depth of understanding and retention. In addition, he cites a number of studies that demonstrate improved long-term educational impact of group work, when these types of assignments are used.⁵⁸ However, these forms of thinking may be difficult to precisely articulate because they can be quite wide-ranging and nuanced. For the purposes of this paper, it is appropriate to suggest that “deep thinking” can include both critical thinking and higher-order or complex thinking.

Bloom’s Taxonomy outlines a hierarchical structure of thinking where, as students continue to expand their knowledge in a subject, they are able to apply, analyze, and evaluate information related to that subject.⁵⁹ Many have suggested that the ability to critically consider all the available information (ie, analyze, evaluate) and apply it to a problem is among the most valuable skills a student can acquire.⁶⁰ This is certainly true in pharmacy education, in which critical thinking is among the most valuable demonstrable skills for recent graduates.⁶¹ With critical thinking as foundational, Peeters and colleagues describe higher-order, complex thinking as involving problem-solving, clinical reasoning, and moral reasoning.⁶² In pharmacy, this thinking is reflected in the ability to develop and apply sound ethical and clinical judgments.⁶³

The verification process identified several connections between the indicators defined by the expert panel and the literature on TBL practice, providing evidence of consistency between the findings and the literature. A key component of successful TBL is that students come to class having prepared to apply what they have learned (indicator 1).²⁹ When compared to traditional lecture, TBL increases the number of students prepared for class.² Proper student preparation has been shown to improve student participation in class and significantly improve test scores, regardless of student ability.^8,64 Literature indicates that well-prepared application activities promote more intense discussion among team members (consistent with indicator 13), thereby enhancing team cohesiveness.^27,28

The panel also agreed that quality applications should be of a difficulty level that is challenging (indicator 4), but that can be completed in a feasible timeframe (indicator 2). Although there is no specific length of time an application activity may take, Michaelsen and colleagues indicate the goal should be to promote a high level of engagement among team members, and create a high enough level of energy so that students are willing to take time to have meaningful team discussions and defend their answers as discussed.⁶⁵

The panelists agreed that the activity should follow the published rules on TBL “4S” activities (ie, significant problem, specific choice, same problem, and simultaneous report) (indicator 3) for the creation and implementation of application activities.^28,29,65 Team-based learning advocates assert that these attributes will result in student accountability and will generate discussion within and among teams.⁶⁶ Haidet and colleagues recognize the “4S” characteristics as a core element of TBL and state that these characteristics “foster individual and team motivation, a common frame of reference, critical thinking and conceptual depth, and energy during whole-class discussions.”⁶⁷ In an article proposing TBL best practices in pharmacy education, Farland and colleagues suggest that application activities be practice-based (indicators 6 and 7) and focused on overarching concepts that are significant and potentially integrative between or within courses (indicator 10).⁶⁸

Although few comments were provided for indicator 5, the panel agreed that quality application activities “incorporate or provide effective feedback to groups.” The educational research literature has documented that feedback is essential to content learning and retention.⁶⁹ There are many points in the TBL process where feedback is inherent or can be incorporated. For instance, the readiness assurance process provides feedback to individuals and teams on their level of preparation. Students may receive feedback from team members rating their professionalism⁷⁰ or from instructors attempting to support team leadership.^71,72 However, the feedback generated as part of the application process is also important to learning. Students receive feedback from peers during and after application activities when teams simultaneously report an answer (ie, task feedback) and discuss the analysis that led to their decision (ie, process feedback). Medina and colleagues examined the use of a problem-solving rubric to provide verbal and/or written feedback on a group’s ability to prioritize, organize, and defend the best and alternative options in TBL.²⁶ In discussing TBL facilitation, Gullo and colleagues suggest that facilitators provide adequate time for closure, commenting on difficult concepts and emphasizing take-home points, as well as highlighting student comments that corrected an inaccuracy or excelled in some manner.⁷³ More innovation in providing immediate or post-class feedback on applications is needed, in addition to investigation of effects.

Some panelists disagreed with the “specific choice” aspect of the “4S” rules (indicator 3). Comments supported the use of open-ended questions to: promote critical thinking, show the richness of clinical decision making and move students to higher levels of Bloom’s taxonomy. The TBL literature strongly advocates for application activities that force a specific choice to create “constructive controversy” between teams.⁷⁴ However, specific choice questions do not necessarily need to be written in a multiple-choice format, as several alternative methods have been described (eg, ranking, sorting, sequencing, matching).⁷⁵ According to Roberson and Franchini, limiting the options from which teams can choose gives the instructor the prerogative to guide the class discussion so that “feedback on the task can be directed at specific, anticipated discoveries and realizations” that are most beneficial to the learning process.⁷⁵ Panelists commented in round 2 and during the open comment period that adherence to the “4S” rules should not discourage innovation or impede creative approaches to application activities that remain consistent with the principles of TBL¹ described by Michaelsen and colleagues.

The panel’s consensus on indicator statements 6 and 7 is also consistent with and expands upon the “4S” rule of providing students with a “significant problem.” In pharmacy, a significant problem includes both an authentic pharmacy challenge (ie, an important, practical, real-life scenario that a pharmacist is likely to face, such as a question on rounds) and relevant pharmacist tasks (ie, one or more things that pharmacists do, such as interpreting a drug study’s relevance for a particular patient). Designing application activities with relatable contextual details (indicator 6) that highlight and exercise pertinent pharmacist duties/tasks/responsibilities (indicator 7) that are also perceived as germane by students is challenging. During the design process for application activities, instructors may find it helpful to consult additional faculty members and/or pharmacists to identify and hone rich and engaging challenges and meaningful, relevant tasks.

The indicator, “The activity has multiple right answers,” achieved 76% agreement but narrowly missed consensus. Comments around this potential indicator of quality involved correct answers vs best answers and the plausibility of distractors. Panelists suggested that an activity with multiple viable answers is more consistent with real-life problem-solving and is more likely to encourage rich discussion. Such an activity may also help students to realize that there might not always be one “right” answer and that there may be multiple ways to approach a problem. However, activities designed with a single best answer are supported by literature on specific-choice in TBL because they compel the group to defend their position against other groups that have made comparable decisions.¹ Other comments focused on the point that there is “learning value” in pursuing a best answer. Continued dialogue in this area is needed to help isolate any potential indicators and to understand the insights that may be gleaned from divergent opinions. Both approaches may have merit, assuming that both could be implemented in a way that addressed a particular learning need/goal and led to quality discussion of complexities or nuances.

In round 1, some panelists discussed the integration of specific faith-based or social justice objectives into TBL applications. However, in round 2, only 53% agreed that it is important to “incorporate elements from school-specific emphases (eg, faith, underserved).” Comments suggested that coverage of mission or specialized outcomes is desirable but may not be required in all application activities. Certainly, mandating that every application activity incorporate mission elements would be limiting. However, TBL applications might be an opportunity to help a school’s mission “come alive” for students, demonstrating the school’s commitment to its mission and providing opportunities to discuss the real-world complexities of living out those mission elements.

The results of this study might be further elaborated upon by investigation with student stakeholders. Students may have additional insights into the indicators of quality application activities from their perspective. In addition, because of the limited number of rounds, it is unclear whether items without consensus simply required more discussion and refinement or whether those indicators were, in fact, not strong indicators of quality. Given the asynchronous nature of the investigation, clarification could not be made with the panelists on the intent of the indicators, when questions arose. In particular, comments from later rounds suggested that some panelists were interpreting “indicator” as required (ie, must always be present). However, the intent was simply to emphasize that the item’s presence was likely to support (indicate) success. Clarifying this issue may have generated additional indicators for consideration in the design of application activities, allowing all panelists to brainstorm possibilities outside of the constraints of “those things required.” Future research can use these indicators to examine the quality of application activities, including the effects of training, checklists, and peer review on the quality of the application activities developed.

CONCLUSION

Fourteen quality indicator statements for application activities were identified by consensus among TBL experts. The indicators aligned well with TBL principles and higher education concepts. Pharmacy TBL experts identified quality indicators related to application activities, structure, content, design and intent, and learning actions and results. The indicators can assist pharmacy faculty members in creating, implementing, and improving application activities to provide high quality TBL exercises that promote deep thinking and generate engaged classroom discussion. The indicators could also be used to inform faculty development and quality improvement efforts.