Abstract
Objective. To evaluate the success of a teaching certificate program by qualitatively evaluating the content and extent of participants’ reflections.
Methods. Two investigators independently identified themes within midpoint and final reflection essays across six program years. Each essay was evaluated to determine the extent of reflection in prompted teaching-related topic areas (strengths, weaknesses, assessment, feedback).
Results. Twenty-eight themes were identified within 132 essays. Common themes encompassed content delivery, student assessment, personal successes, and challenges encountered. Deep reflection was exhibited, with 48% of essays achieving the highest level of critical reflection. Extent of reflection trended higher from midpoint to final essays, with significant increases in the strengths and feedback areas.
Conclusion. The teaching certificate program fostered critical reflection and self-reported positive behavior change in teaching, thus providing a high-quality professional development opportunity. Such programs should strongly consider emphasizing critical reflection through required reflective exercises at multiple points within program curricula.
INTRODUCTION
Reflection is an intentional, dynamic process that allows improvement in one’s actions, abilities, and knowledge by learning from past experiences.1-4 While this process can be useful in almost all aspects of life, reflection in the workplace can be particularly beneficial. Successful professionals must be able to reflect on their experiences in order to find solutions to complex problems encountered on a daily basis.1-2 Such reflection is not only necessary for pharmacists and other health care professionals to improve their practice, but also to further hone their expertise as educators.5 Reflection should be used by pharmacists committed to professional growth as lifelong learners.6
The practice of reflection during residency programs offers a valuable opportunity to observe and guide residents in this process at the beginning of their careers. Teaching certificate programs within pharmacy residency programs were founded on the idea that having specialized pharmacy knowledge does not necessarily equate with being an excellent teacher.7 Teaching certificate programs give participants general pedagogical knowledge to combine with their existing content knowledge. Participants complete didactic and experiential teaching activities to develop such teaching skills. A critical component of the teaching certificate program at the University of Arkansas for Medical Sciences (UAMS) College of Pharmacy is the extensive reflection required throughout the year-long program. Because reflection is such an integral part of professional development, evaluation of the content and extent of participants’ reflections is imperative.
Research on the topic of reflection is extensive, but investigation of reflection on teaching by pharmacy faculty members or faculty members in training is not described in the literature. Additionally, methods to assess teaching certificate programs focus on surveys but lack the details offered by more in-depth analysis. Qualitative analysis of reflective essays could provide better understanding of program benefits and participant growth. Our mixed-methods evaluation is the first to examine thematic composition of reflections as well as the extent of reflection evidenced in the written essays of potential future pharmacy faculty members as they participate in teaching certificate programs.
METHODS
This study was designed as a mixed methods thematic analysis of teaching certificate program participants’ reflective essays. A qualitative approach was chosen because this method allows deep analysis of the text not obtainable from survey-based research. A modified constant comparison method of analysis was employed, and categories and themes were constructed from open and axially coded data.8 The coding scheme arose from the data as researchers explored them. Resulting themes were evaluated in comparison to the typology derived from teacher reflection theoretical framework. Qualitative methods were quantified to provide comparison of themes and level of reflection. The data evaluation provides a descriptive evaluation of the effectiveness of this teaching certificate program activity and its impact on teaching development.
The school’s teaching certificate program was originally developed in 2005 to enhance the teaching skills of pharmacy residents but quickly expanded to include preceptors because of an increased demand for preceptor development in teaching. The program, described in a previous manuscript,9 facilitated development as an educator through the following experiences: formulating personal goals for development in teaching, tailoring teaching approaches to learning setting and audience, practicing effective assessment and feedback skills, receiving ongoing feedback from program faculty members, reflecting upon individual teaching experiences, developing a personal teaching philosophy, and documenting experiences through the development of a comprehensive teaching portfolio.
Over the course of the program (July to May), participants attended formal teaching seminars, self-selected teaching activities, and met with a faculty teaching mentor, who monitored their progression in the program. Participants also were required to write two reflective essays describing their teaching development. The midpoint reflection was submitted in December, and the final reflection was submitted in May of the program year. In these global reflections, participants were asked to discuss their teaching development and specifically include commentary on each of four topic areas: teaching-related strengths, teaching-related weaknesses, ability to effectively assess learners, and ability to provide effective learner feedback. Pharmacy residents and preceptors from across the state participated in all aspects of the program. All retrievable essays from participants who submitted written essays in December and May were analyzed. Participants were excluded if they did not complete the teaching certificate program by submitting December and May essays or if the data were not available.
De-identified electronic copies of reflection essays were used to extract themes and the extent of reflection using the four prompted topic areas of strengths, weaknesses, assessment, and feedback. The 4-category coding scheme described by Kember et al10 was adapted to determine the extent of reflection for each of the four prompted topic areas as well as the highest level obtained overall (Table 1). A fifth level of “0” was added to be assigned when the participant did not discuss a topic area. Because the instructions for the reflective assignment served only as prompts for reflection, participants were not required to write about each area, and therefore could receive a score of “0.”
Five-Category Scheme for Assessing Extent of Reflection in Written Essays10
Two investigators in the team independently identified the themes discussed in the essays along with the extent of reflection of each essay in the four topic areas using NVivo, v10 (QSR International Pty Ltd., Doncaster, Victoria, Australia). The primary coder used a sample of 10 essays to create initial categories of themes and shared this initial set of themes with the second coder. Coders independently analyzed essays for themes discussed and the extent of reflection obtained in each topic area. The coders met weekly to discuss the extent of reflection assigned to each essay and resolved any differences.
By using a constant comparative approach, coders agreed by consensus on every essay to achieve 100% inter-rater reliability. Additional reflective themes also were added based on coder agreement when a new topic was discussed that did not fit into one of the predetermined themes. A third investigator periodically reviewed the coding process and identified themes for verification to enhance agreement between coders. An electronic coding platform made it possible to efficiently code, analyze, and organize this large amount of data. Each essay, depending on length, took about 10-20 minutes to analyze.
Further patterns emerged from the data during qualitative analysis and were explored with additional detail using triangulation methods. Coder agreement on coding of themes was assessed using Cohen’s kappa. Data were analyzed to identify differences in the extent of reflection in the midpoint and final essays using Wilcoxon signed-rank test. These data were further analyzed to identify whether extent of reflection varied according to participant gender and experience (preceptor vs resident) using the Wilcoxon rank sum test with Stata/SE, v12.0 (StatCorp LP, College Station, TX). The thematic analysis was granted exempt status by the UAMS Institutional Review Board.
RESULTS
All available pairs of essays were evaluated from participants completing the teaching certificate program between 2006 and 2012. One hundred thirty-two essays were analyzed from 66 participants. Of the 66 participants, 53 (80%) were female, and 10 (15%) were preceptors.
The coders identified more than 11 000 references to 28 themes discussed within the 132 essays. Themes covered a broad range of professional development topic areas, including delivering educational content, interacting with students, evaluating success, and encountering challenges. A full list of identified themes is available from the authors. Agreement between the coders on themes discussed in each essay was high (kappa=0.74). Data saturation of the identified themes occurred early in the coding process, indicating that the list of themes was representative of the context of the essays.
Themes were tracked by the number of essays in which the theme was discussed (“mention”) as well as the total number of references to the theme in all essays (“weight”), as themes could be discussed multiple times within a single essay. Table 2 provides the 10 most common discussed themes and weight placed on theme according to total number of references to that theme. The same 10 themes ranked highly according to both mentions and weight, indicating the emphasis placed on these ideas by the participants was consistent between and within essays. (A detailed description of themes and results of analyses are available from the corresponding author.)
Ten Most Common Themes by Mention and Weight
High levels of reflection were exhibited in the participant essays, with 48% of essays achieving the level of critical reflection (level 4). Quotations from teaching certificate program participant essays specific to the area of assessment were extracted to illustrate the extent of reflection. An example of nonreflection (level 1) was: “Assessment questions are aimed at evaluating comprehension of stated objectives.” A statement that was consistent with understanding (level 2) was: “There are various characteristics that students display that allow assessment of their abilities.” Reflection was evident in the level 3 statement: “It is easier to see two students evaluate similar patients than it is to see students present two different cases and discern which is more proficient.” Critical reflection (level 4) was evidenced by the statement: “But then I realized that I didn’t know if I was being an effective teacher without an assessment of some type. So I have started incorporating pre- and post-lecture questions to see what the learner gained from the presentation.”
Figure 1 shows the levels of reflection obtained overall and in the four prompted topic areas of strengths, weaknesses, assessment, and feedback for midpoint and final essays. Mean levels of reflection achieved in combined midpoint and final essays in each topic area were 2.4, 2.9, 2.7, and 2.9, respectively, indicating the greatest extents of reflection in the weakness and feedback topic areas. Each topic area showed increases in the mean level of reflection between midpoint and final essays [strengths, 2.2 (1.1) vs 2.5 (1.1); weaknesses, 2.8 (1.0) vs 2.9 (1.0); assessment, 2.6 (1.0) vs 2.8 (0.9); and feedback, 2.4 (1.0) vs 3.0 (1.0)].
Level of Reflection10 Achieved in Midpoint and Final Reflection Essays. Number of essays according to the highest level of reflection (Level 0=Absent; Level 1=Non-reflection; Level 2=Understanding; Level 3=Reflection; Level 4=Critical Reflection) attained in the four prompted topic areas for both midpoint and final essays (n=132).
The extent of reflection between midpoint and final essays significantly increased in the prompted topic areas of strengths (p=0.03) and feedback (p=0.0002). In the strengths topic area, 29 (43.9%) participants did not show change in the extent of reflection from midpoint to final reflections, but 25 (37.8%) showed deeper reflection, compared with only 12 (18.2%), who had decreased reflection scores. In the feedback topic area, 28 (42.4%) participants did not show change in the extent of reflection from midpoint to final reflections, but 30 (45.5%) showed deeper reflection, compared with only 8 (12.1%) who had decreased reflection scores. The highest level of reflection achieved seemed to be higher in the final essays [mean value 3.53 (0.53)] when compared with the midpoint essays [mean value 3.36 (0.57)], but this change did not reach significance (p=0.055) despite increases in each individual topic area. There were no differences observed when comparing change in the extent of reflection at midpoint and final according to gender or preceptor/resident status.
Further patterns emerged from the data during qualitative analysis and were explored with additional detail using triangulation methods. Program participants often discussed others’ evaluation of their teaching activities, but focused primarily on receiving evaluations from students and faculty members (65% and 49% of essays, respectively). Just 24% of essays mentioned gaining evaluation by peers (despite a teaching certificate program requirement for obtaining a peer teaching evaluation), and only 4% commented on gaining feedback from patients on their teaching. In contrast to their discussion on receiving evaluations, only five (7.6%) participants discussed the process of evaluating others’ performance in teaching. Four of these regarded providing peer evaluations, and one participant commented on giving feedback to faculty members.
A frequent conclusion of the participants was that they did not have the opportunity to complete a specific type of teaching development activity. This theme was reported in 31 (23.5%) essays. As expected, just over 70% of these occurrences were found in midpoint essays, when participants had completed only the first half of the teaching certificate program. This theme often disappeared from the final essays. However, the perceived lack of opportunities to participate in formally grading students and writing test questions (n=4), serving as an experiential education preceptor for students (n=3), giving a formal presentation (n=1), and providing student feedback (n=1) remained in the nine final reflective essays.
DISCUSSION
Reflection is vital in the life of an educator.11 Even more, reflection is the key to learning, which occurs when we create meaning from a past event and use this to shape future experiences.4,12 While other disciplines such as education have been using reflective practices for some time, the health sciences professions have more recently adopted this concept in the training of future health care professionals. Medicine, nursing, and pharmacy are among the disciplines that are adapting these types of reflective processes in curricula to aid in learning as well as improve patient care.13-21
The extent of medical residents' reflection has been explored 13-15 and showed that physicians' decision-making skills were improved in complex clinical cases if they were able to critically reflect on those experiences.15 An evaluation of nursing students concluded that interview sessions (individual, paired, and group) on reflective practice were viewed as beneficial for the participating students and encouraged them to practice reflective thinking on their own.17 Like many other professions over the past few decades, accreditation standards for pharmacy education require student reflection and subsequent assessment of these skills.18-20
Additionally, the American Society of Health-System Pharmacists Required Competency Areas, Goals, and Objectives, for Postgraduate Year One Pharmacy Residencies address the need for residents to reflect on their personal performance and professional development.21 Published works on reflection in pharmacy center on pharmacy students, but the literature suggests reflection is necessary in the day-to-day practice of a pharmacist as well.22,23 However, little is known about the reflective abilities of pharmacy educators and how these skills affect teaching-related self-development.
Assignments in this teaching certificate program served as tools to facilitate and encourage reflective thinking in order to promote growth in teaching abilities, and participants achieved high levels of reflection when completing these assignments. Almost 50% of essays evidenced critical reflection by describing how change occurred in their practice of teaching (behavior change). This high level is rarely achieved by so many.10 This could be because participants in this program were highly achievement-oriented as supported by their admission to and completion of pharmacy school as well as pursuit of residency training.
Additionally, preceptors in the program were seeking teaching development and voluntarily participated. The teaching certificate program also emphasized reflection throughout the program. In addition to the reflections reviewed in this evaluation, participants were asked to reflect immediately after each teaching activity; this practice of reflective thinking and writing may have facilitated deeper reflection in the more summative midpoint and final essays. Another factor that could have contributed to the high levels of reflection was the mentorship of program faculty members, who reviewed reflection drafts and further prompted participants to consider their experiences, personal characteristics, and teaching development goals.
When analyzing themes discussed by participants, it was evident that they considered the reflection prompts when writing their essays because strengths, weaknesses, assessment, and feedback all appeared often. A closer look at the references to these categories revealed that participants wrote most frequently about the areas of assessment and feedback. These areas stood out because they were the areas in which participants lacked confidence early in the teaching certificate program, but they were also the areas in which participants recognized the most growth at the conclusion of the program. Collectively, the extent of reflection significantly increased from the midpoint essays to the final essays in the feedback category.
There was no intervention on reflective writing between the midpoint and final essays, so the change evidenced by our evaluation seems to have occurred naturally. Participants gained more practice with these skills over the course of the program year, and they subsequently reflected more and explained their growth in this area. This finding of deeper reflection in the area of feedback validated the design of the UAMS teaching certificate program and its focus on self-reflection as a method of teaching skills development.
The area discussed by all participants in the final essays was their confidence and comfort in teaching. This linked nicely with the significant increase in the extent of reflection seen in the strengths category. Participants seemed to be more deeply aware of their strengths as the program progressed, and they discussed areas of perceived confidence in these essays. This is consistent with our previous findings that demonstrated increased self-perceived teaching abilities during the program.9
Because the teaching certificate program provides the opportunity for gaining practical experience in teaching, confidence in these abilities is expected to grow. Gaining confidence can allow a deeper awareness of personal strengths. Growth was evidenced as experiences led to increased confidence, and prompted reflection on these experiences led to gains in self-perceived teaching abilities and strengths. Increased confidence through experiences was the aim of the teaching certificate program and explains why such programs are beneficial to participants, future employers, and academic institutions.
Participants most often solicited feedback from students and faculty members. Feedback from peers and patients were reported much less frequently. Because it is a teaching certificate program requirement for participants to receive feedback from faculty members, peers, and students, it is not surprising that these perspectives were discussed in the reflections. However, patient teaching also is encouraged by the program, and the perspective and feedback of patients should be valued. Although the number of participants soliciting feedback from patients was not collected in our evaluation, few discussed these perspectives in their reflections.
Twenty participants (30%) listed patient counseling as a teaching activity, but only three participants discussed receiving teaching evaluations from patients. This could be because this information was not solicited from participants and, therefore, they do not reflect on it specifically or because the participants perceived patient evaluations to be lower quality or less importance than those from faculty members or peers. Additionally, participants might have viewed the patient as a different type of learner than students and not seen the need for reflection on their evaluations. Curricular emphasis of patients as learners increased in later program years to encourage variation of feedback from learners. To foster the view of patients as learners, programs should consider requiring patient teaching activities along with assessment of patient learning and completion of teaching evaluations by patients.
It is also noteworthy that participants wrote more frequently about receiving feedback from students and faculty members and less frequently about feedback received from peers. This corresponds to the small number of participants who reported giving feedback to faculty members (n=1) and peers (n=4). Giving and receiving feedback from peers may be an area of discomfort for participants despite the teaching certificate program requirement to give and receive this type of feedback. Additionally, only one participant reflected on the program requirement to provide feedback to at least two faculty members after observing their teaching. Overall, reflection on giving feedback to peers and faculty members was lacking, and participants did not reflect on receiving feedback from all sources.
The extent to which feedback was sought is not known, so additional research is needed to clarify this finding. Many factors could contribute to this pattern, including the perceived need to obtain feedback from certain groups, the feasibility and convenience of sampling, the perceived relative importance of feedback from more experienced or more educated groups, or the comfort level of the participants to give feedback to and receive it from certain groups. Teaching certificate programs should consider increased emphasis on 360-degree evaluations of teaching and both giving and receiving feedback to help participants understand the necessity and value of feedback from others. If participants gain comfort in this activity, they could provide higher quality feedback to others and potentially could benefit more from the feedback they receive.
Participants were not prompted to write about opportunities they did not get to experience; therefore, it can be assumed that participants discussing this theme must have been expecting to participate in such activities. By identifying these patterns, program directors can get an idea of skills that some participants might want to gain from the program. As expected, more than 70% of the themes coded for no opportunity were expressed in the midpoint essays. Of the remaining nine items discussed in the final essays, grading students and writing test questions accounted for the most themes discussed. Our program has taken these comments into consideration, incorporating additional opportunities for examination item review sessions to give participants insight on how to develop and evaluate examination items.
Qualitative evaluation of participant reflections can provide feedback on teaching certificate program effectiveness. These reflections provide a detailed view of how participants develop teaching skills throughout the program and impart realizations of the effectiveness of the program. By using reflective essays as quality indicators, program directors can shape program content to better develop teachers. At the same time, the reflection itself can aid participants in the development of a well-informed, highly individualized written statement of teaching philosophy, another potential quality measure of the teaching certificate program. This research did not evaluate participants’ teaching philosophy statements. However, qualitative evaluations of these documents also could provide a proxy of the teaching certificate program’s effectiveness in increasing participants’ awareness of their own teaching style and ideals.
Although our qualitative evaluation was rigorous and followed the guidelines presented by Anderson,24 care must be taken when generalizing the results to other programs. Our data represent the products of a single teaching certificate program and are specific to the participants and experiences of this program. The characteristics of the residents and preceptors completing the program are varied, but they may reflect these groups at other institutions.
The use of quantitative and qualitative measures to evaluate this teaching certificate program is a strength of the study, and additional analyses are needed to determine if participants’ reflective abilities remain constant over time and predict future teaching performance. Finally, because of the extensive nature of our evaluation and the volume of the qualitative data, the potential exists for coding errors and inconsistencies between coders. Several mechanisms were in place to limit these inaccuracies and preserve the integrity of the data.
CONCLUSION
Participation in the teaching certificate program appeared to increase confidence and enhance awareness of strengths through participant reflection. Pharmacy residents and preceptors frequently achieved the highest level of reflection (critical reflection) in global self-assessments of teaching experiences. Such deep reflection is indicative of professional development because teaching certificate program participants evidenced change not only in teaching attitudes, but also teaching behaviors, as discussed in written essays. Just as reflective exercises are emphasized for pharmacy students and residents, findings from this analysis suggest that teaching certificate programs should strongly consider emphasizing purposeful critical reflection through required reflective exercises at multiple points within the program curricula. Qualitative evaluation of participant reflections can provide quality indicators to assist program directors in shaping program content.
- Received December 15, 2014.
- Accepted March 5, 2015.
- © 2016 American Association of Colleges of Pharmacy