Pharmacy Students’ Utilization of an Online Tool for Immediate Formative Feedback on Reflective Writing Tasks

INTRODUCTION

Pharmacy educators acknowledge that reflective-learning processes are not inherent or intuitive skills. Instead, they are skills that are only acquired through education and practice.^1-4 Developing reflective skills assists pharmacy students to perform better academically^5,6 and make better clinical decisions,^5,7-9 Additionally it may enhance self-directed learning.¹⁰ These skills are also essential for pharmacists to be able to self-critique particularly in professional practice.^11,12 Furthermore, the importance of reflective skills is not restricted to the pharmacy profession. They are equally important in other health settings, including nursing, medicine, physical therapy, and other allied health specialties.^9,13-21

Reflective writing is not an easy skill to develop and differs greatly from other forms of writing.¹ Studies have shown that students perceive reflection and reflective writing as a challenging and confronting task, particularly as it involves an introspective aspect and has personal elements.^11,22 Despite this, mastering the art of reflection is considered essential to the professional development of the pharmacy practitioner.^23,24 Through reflective writing processes, many health professions schools are assisting students with learning how to critique their own practice skills in order to improve their future practice.^16,19,25,26 However, the assessment of student performance on reflective tasks requires careful consideration and a clear understanding of the purpose and aim of these exercises.^11,27 If one of the primary aims of reflection is for students to be able to self-critique their professional practice skills, then an online tool that promotes this process, provides timely feedback, and can be used at any time may be beneficial.

Students’ reflective writing can be stimulated by formative feedback provided by their educators.²⁸ The sooner students receive feedback after submitting an assessment, the greater the effect the feedback has on their future learning.^29,30 However, with increasing numbers of students enrolled in higher education and increasing time constraints for academics as they balance teaching and research responsibilities, providing individual formative feedback to each student regularly or in a timely manner is often not possible.³¹ This poses problems for students who require more assistance with a task, particularly one as challenging and confronting as reflective writing.

There is a growing interest in integrating reflective-skill development within health education contexts. Furthermore, self-directed learning is an important skill for graduate students as well as health professionals. This, coupled with faculty members’ time constraints to provide students with consistent formative feedback was a catalyst for the development and launch of a novel online tool in the pharmacy curriculum.³²

Academic Writing Analytics (AWA) is a web application designed to provide immediate formative feedback to students on first drafts of their reflective-writing assignments. The app was developed by the Connected Intelligence Centre (CIC), University of Technology Sydney, in collaboration with Xerox Research Centre Europe (XRCE) and academics from diverse disciplines at the University of Technology Sydney (pharmacy, business, ecology, law, and engineering).

Academic Writing Analytics utilizes a range of natural language processing (NLP) techniques to analyze student writing. Natural language processing involves the use of computer algorithms to identify and extract specific features from human language text. These include syntactic features that describe the structure of the text and lexical features based on the words that are used, as well as features intended to capture semantic information. These features are used to identify high-level concepts, which in turn can be used as a basis for providing feedback to the student.

A prototype of AWA was evaluated for its precision and accuracy. The results of that evaluation in which computer output was compared with human annotations was encouraging.³¹ In particular, a second iteration showed promise comparing the parser output (analysis of syntactic features, computer language) with the human annotators of reflective statements among a group of students. Three hundred twelve reflective writing pieces (containing 2366 sentences) from Georgetown University (GU) were analyzed and coded. Coding elements used Carol Rodgers’ reflective model drawn from the work of Dewey and colleagues³³ to distinguish between shallow and deep reflection. Coding consisted of a review of assessment rubrics and independent coding by academics and linguistic graduate students, with a follow-up discussion and shared agreement. Using well-established metrics in classification methodology for precision, recall, accuracy analysis of the comparison between parser output and human annotations showed that accuracy did not decrease significantly with the second iteration compared to the first iteration. This showed promise with regards to reliability of the tool as the first iteration only analyzed 30 reflective pieces containing 382 sentences while the second iteration analyzed approximately 10 times as many reflective pieces as the first.³¹

This study aimed to identify pharmacy students’ perceptions of the benefits and utility of using a novel, online tool to provide formative feedback to them prior to submitting their reflective writing tasks for summative assessment.

METHODS

The AWA tool provides students constructive, formative feedback on drafts of reflective writing.³² This feedback is provided on screen in the form of tags (color-coded symbols) and bold type to indicate reflective writing elements. The AWA is not used for summative purposes and hence does not provide a score of student performance.³¹ As no grade is given, the user’s focus is directed toward learning with regards to the writing task and incorporating reflective elements into their writing. As AWA’s output was drawn from the literature and grading rubrics grounded in the reflection process, AWA’s output uses terminology aligned with the summative reflective rubric. Students can access the tool at any time, which may be particularly useful for when they are on experiential placement and require timely feedback, or in rural or remote areas where faculty members may not be available to provide feedback. Using AWA requires students to enter their reflective text into a text box in the web interface or upload a file. The student submits the text and within seconds a reflection feedback report is provided. The AWA web application sends the text to a text analytics pipeline (TAP) to be analyzed using the aforementioned NLP, resulting in analytics based on the input text. The resulting analytics are rendered by the web application in the form of a feedback report for the students to review. An example of the feedback AWA provided on an actual pharmacy student’s reflective statement is provided in Appendix 1. Primarily, feedback is provided through annotations in the text in the form of visual tags. This is accompanied by some textual feedback underpinned by the literature, ie, theories about the reflective process and drawn from grading rubrics^31,32 prompting the writer to self-critique their reflective writing. Three primary tags (color-coded symbols) were developed based on the theoretical conceptions of reflection, reflexivity, and reflective writing: a blue square tag (context) indicates reflection on initial thoughts and feelings about a significant event, situation, or experience; a pink circle tag (challenge) represents challenges within the reflective process, and a green triangle tag (change) represents a higher-order thinking process that indicates students have made a shift in perspective that will lead to change in behavior, attitude, or approach to future learning. As the Journal only publishes figures in black and white, the original tags are described in the example provided only in terms of shape (Appendix 1). A color version of Appendix 1 can be found at http://bit.ly/Lucas_AJPE_Apndx1, https://www.youtube.com/watch?time_continue=162&v=N8O6aIpB2f8, and https://www.youtube.com/watch?v=vl9fpKtbLfY&feature=youtu.be.

In addition to the tags, short groups of words or expressions are emphasized with plain lines indicating a belief, learning, or knowledge; dotted lines indicating self-critique; and dashed lines indicating strong feelings. Bolded words indicate deeper reflection, with one or more of the tagged sentences showing evidence of personal application. The colored symbols allow the student to instantly recognize the key elements of the reflective process, and a “guidance tab” (not shown in Appendix 1), which they can refer to at any stage, automatically protrudes onto the interface, prompting and providing students with the meanings for all tags and indicators. The AWA tool can be used multiple times as students revise their work prior to submission for final assessment.³²

The study was approved by the Research Ethics Committee of the University of Technology Sydney. Clinical Practice for Pharmacy is a required six-credit unit of study offered to first-year Master of Pharmacy (MPharm) students at the Graduate School of Health-Pharmacy, University of Technology Sydney. This unit of study focuses on developing reflective practice skills in the clinical environment (experiential placements), and students are required to maintain an ePortfolio and write about their weekly clinical experiences using reflective writing processes. The reflective statements are documented in the e-portfolio and graded midsemester and again at the end of the semester. The reflective statements are graded using a reflective rubric adapted from the previous work of Tsingos and colleagues (Table 1),¹¹ which was drawn primarily from Boud’s Stages of reflection³⁴ and Mezirow’s categories of reflection,³⁵ including elements that use a rubric by Wetmore and colleagues.³⁶

Students (20 years and older; N=67) who were enrolled in the unit of study were invited to participate in the study. A participant information statement (PIS) was provided for students to sign prior to the study indicating that if they chose to participate, it would not affect their grades. To maintain confidentiality, the PIS also indicated the students were to provide a pseudonym on the pretest and posttest, which would allow researchers to match data without identifying any students or their responses.

Prior to administering the survey, the course coordinator delivered a required, one-hour workshop to demonstrate use of AWA. Students were then provided with a link to an online, anonymous survey (pretest) where they were asked to provide a pseudonym that would be retained for use on the posttest. The online survey consisted of six quantitative items rated on a 6-point Likert-type scale ranging from 1=strongly disagree to 6=strongly agree. Students were asked to indicate to what extent they were in agreement with the statements related to the AWA tool (Table 2). After completing the pretest, students were instructed to use AWA to write and revise their weekly reflections for four weeks, and then complete the posttest.

Data from the completed questionnaires were collated and analyzed by a research fellow working in an area external to the pharmacy discipline. Data collected from question 3 was flipped during analysis to represent the positive orientation of this negative question, ie, AWA “is unlikely to be a useful tool for pharmacy education.”

RESULTS

Of the 67 final-year MPharm students, 60 volunteered to participate in the study, for a response rate of 90%. Because not all of the students created a pseudonym, which was required to match the pre- and posttest data, only 39 pre- and posttest survey instruments could be matched and analyzed. The average student agreement to questions 1 through 6 and standard deviations for each question on the pretest and for the posttest are presented in Table 2. There was a significant increase from pretest to posttest in students’ agreement with the following AWA benefits: assists with self-directed learning (p=.053), is relevant to pharmacy professional practice (p=.024), and enhances student confidence in their ability with writing their submissions (p=.05). Students’ responses to question 3 were also statistically significant (p=.016) (Table 2) indicating AWA as a useful educational tool.

DISCUSSION

This study aimed to identify pharmacy students’ perceptions of the benefits and utility of an online, formative feedback tool. The study findings support previous research and suggest that the AWA tool has the potential to assist students with developing their reflective writing skills by providing formative feedback to them in a timely fashion.³² The strengths of this study include that students volunteered for the study and used the tool in their own time. Also, because the students were asked to provide a pseudonym to de-identify themselves, the data collected were likely honest responses. In addition to the fair response rate, a noticeable increase in students’ agreement with survey items regarding AWA’s benefits after using the tool for four weeks suggest they were engaged in the task. The pharmacy students in this study perceived that the greatest benefits of the AWA tool related to its usefulness for pharmacy education (question 3); its potential to enhance students’ self-directed learning skills (question 4); its relevance to the pharmacy profession (question 5); and its capacity to increase students’ confidence in writing reflective statements (question 6). Although most pharmacy students agreed that the AWA tool assisted their development of reflective writing skills (question 1) and ability to critique their own work (question 2), these benefits were considered secondary compared to the benefits related to questions 3, 4, 5, and 6. Perhaps this was because students perceived AWA more as a useful, educational tool for their self-directed learning and feedback rather than one that developed their skills in critiquing their reflective writing.

The study findings may not be generalizable to all pharmacy students, thus presenting a potential limitation. Further studies investigating AWA’s use with students’ reflective-writing tasks at other pharmacy schools would be beneficial. A limitation of this study was that only students’ perception of the tool’s benefits and utility were reported while students’ performance was not measured. There are limitations to using artificial intelligence programs such as AWA; hence, students must develop a critical mindset when using such a tool. For example, students should be allowed to question AWA’s critiques of their writing, which is a positive effect of using such a tool. The AWA web application does not provide students with a performance score/grade; therefore, students and educators may find it difficult to track improvements in performance. Despite this limitation, the majority of students engaged with the task voluntarily and perceived value in the feedback that AWA provided. Enhanced performance may be measured in future studies via the collection of analytical data related to changes in students’ reflection writing (ie, comparing the first task submitted with the final task submitted) after self-critiquing with the assistance of AWA. Analytical data collection could include changes (increases) in the number of tags/symbols (indicating the various elements of the reflection process) and the “boldness” of the writing (indicating deeper reflection, personally applied). Future research could include investigating whether a relationship exists between students who score higher grades as assessed using the reflective writing rubric and their use of AWA. Qualitative analysis of students’ initial reflections and comparison with their final reflections prior to summative assessment could also be beneficial in tracking their performance.

CONCLUSION

The findings of this study suggest that the novel online reflective learning tool, AWA, has the potential to assist pharmacy students to become more confident with their reflective writing tasks. As reflective writing skills have gained traction within educational contexts, future studies including the use of the AWA tool in other health education or broader educational contexts may be beneficial.