Abstract
Objective. To explore pharmacy students’ perceptions of a novel web application tool (AcaWriter) implemented in a Master of Pharmacy curriculum to support reflective thinking in scientific research.
Methods. A qualitative research design involving a 50-minute focus group (n=12) was used. The focus group session was audio-taped, transcribed verbatim, and analyzed thematically using the Braun and Clarke framework.
Results. Analysis generated four themes related to AcaWriter’s utility in enhancing students’ research thinking and capacity. The themes identified included: ease of use to prompt reflection, tangible tool with non-judgmental capacity; benefits for enhancing self and peer reflection on research techniques and group dynamics; benefits of the reflective writing process to enhance research capacity compared with engaging in reflective dialogue; and benefits beyond the writing process: cultivating self-improvement and self-confidence.
Conclusion. The findings of this study show that a novel web application implemented within a pharmacy curriculum can assist students’ self and peer reflection on a research task. Further research is needed to explore the impact of using this tool and its relationship with academic performance and outcomes.
INTRODUCTION
Developing research skills in the pharmaceutical sciences is not an intuitive process; however, reflection has been used as a pedagogical strategy to assist students with developing critical thinking processes, such as those required for conducting research.1 Students’ research processes are developed and refined over time with better understanding of the topic area, by formulating the research question, by developing and mastering technical competencies, and by interpreting and analyzing data. This complex process may take time, guidance, training, and self-reflection. Reflecting on approaches, assumptions, and processes is often the first step to the deeper learning process.2,3 However, reflection is not necessarily an inherent skill.4 Previous research has shown that reflection can be taught through prompts, guides, and instruction, and can be facilitated using a variety of different reflection tools, including those that involve reflective writing processes.2 The choice of tool used is often dependent on its feasibility, utility, and ease of use, with importance placed on the self-directed learning experience.
With recent advances in cloud computing power and natural language processing, an explosion of automated text analysis in many sectors is occurring. Through the software tools we use in our daily lives, we are becoming accustomed to the idea that computers can “understand” (although in a very different way than humans) the topics in a document (hence the power of online search) and linguistic expressions (eg, highlighting in an email when it appears someone wants to schedule a meeting, and automated language translation). Natural Language Processing is establishing itself in specific educational contexts, one of which is instant, detailed, providing formative feedback to students about their writing. Clearly, no human can provide such a service instantly for hundreds of students at any hour of the day. This makes natural language processing, as a particular form of artificial intelligence (AI), in theory, an attractive addition to the educational ecosystem.
AcaWriter is a web application developed by the Connected Intelligence Centre at the University of Technology Sydney, Australia, that is used to assist learners with reflective writing (AcaWriter orientation website, University of Technology Sydney: https://uts.edu.au/acawriter).5 AcaWriter utilizes natural language processing techniques to detect elements of human reflective language text at the sentence level.6 AcaWriter is grounded in pedagogical reflective writing theory, which defines several common reflective elements, such as the writer’s experience of context, challenge and change.6 These reflective elements are extracted by using concept mapping rules which identify the constituent concepts in a sentence and the syntactic dependency between them. The development and evaluation of these concept mapping rules for detecting reflective elements have been described in a previous study.7 While rule-based approaches to detecting reflective writing can be designed and validated without large datasets, they also have limitations; thus, more recent work has evaluated machine learning approaches.8
Instead of merely showing students examples of other people’s writing to illustrate good and poor samples, AcaWriter provides students with immediate formative feedback on their own reflective writing. The feedback comes in the form of a report in which the student’s writing is annotated using icons, bold, font color, and underlining, along with written feedback (Appendix 1). The icons identify specific reflective elements in the text and are embedded and indicated within the text prior to the sentence. For example, a blue square indicates context (initial thoughts, feelings, and reactions about a significant experience); a pink circle indicates challenges (new surprising or unfamiliar ideas, problems, or learning experiences); a green triangle indicates change (a shift in perspective relating to new knowledge gained that leads to a change). Feedback from AcaWriter also comes in the form of written feedback that pertains to the whole text and prompts the learner to address areas that seem to have been overlooked. For example, the feedback may state, “It appears that you have not yet commented on what you would do differently should the same event occur in the future,” or “Perhaps think about changes in perspectives, strategies, behavior or approach” (Appendix 1 provides a visual of an exemplar reflection and related symbols and prompts).
Because reflective writing is complex, safeguards are built into the software to account for errors. AcaWriter is only used to provide on-demand formative feedback. The feedback is phrased in cautious language, and the tool prompts an alert to encourage the student to disagree, eg, if they feel that their writing does in fact convey an important move that AcaWriter has missed: “Computers don’t read writing like humans. So, if you’re sure your writing’s good, it’s fine to disagree with AcaWriter’s feedback, just like you’d ignore a poor grammar suggestion.” The intent here is for students to develop a critical attitude toward artificial intelligence.
A precursor tool to AcaWriter was shown to be successful in assisting pharmacy students in reflecting on their placement experiences in order to enhance their future professional practice.9 AcaWriter has also been used across a number of diverse disciplines (law, business, engineering) for the purpose of assisting students to develop reflective capacity.10 However, AcaWriter had never previously been used to explore its value to assist students’ research thinking capabilities in any educational discipline (although some work has supported utility in writing archetypally structured research abstracts in any field).11
A key insight gained from the prior work with AcaWriter in both pharmacy and other fields is that it is most effective when coherently integrated into a curriculum a rather than merely offered as an optional tool (ie, analogous to a grammar checker).9,10 The focus of this study was to explore pharmacy students’ perceptions of AcaWriter as they encountered it as part of the formal Master of Pharmacy curriculum, to enhance their reflective thinking processes related to pharmaceutical research project outcomes.
METHODS
Drug Disposition for Pharmacy is a required six-credit unit of study offered to first-year Master of Pharmacy (MPharm) students at the University of Technology Sydney, Australia, during their final semester. This course covers the principal topics in pharmacokinetics, pharmacogenetics, and pharmacodynamics required for applied therapeutics and practice. Core topics include: monoexponential kinetics, multiexponential kinetics, intravenous infusions, drug disposition, drug metabolism, drug elimination, and pharmacodynamics. The course also covers advanced topics, such as factors affecting drug efficiency, including drug interactions and sources of variability (eg, the role of pharmacogenetics) in therapeutic outcomes. Applied topics include therapeutic drug monitoring, drug individualization, and adverse drug reactions.
For the purpose of their research project associated with the unit of study, groups of students were expected to design and conduct a practice-relevant research project, draft a manuscript, and provide an oral presentation on their findings. Students were assigned to a group of seven or eight by the course coordinator (Table 1). The project was a semester-long activity (over 14 weeks). Students were requested to work within their groups and submit a draft proposal of their research to the course coordinator by mid-semester (week 5 of the course), for a small percentage (10% weighting for summative assessment) (Steps 5 and 6, Table 1). Students then had the opportunity to attend an interactive workshop, facilitated by two faculty members, one with an expertise in the pharmaceutical sciences and the other with an expertise in reflective practice, to discuss summative results and related research project issues, and to gain insights into and understanding of how to improve their research project for their final summative assessment. During this interactive workshop/laboratory session (Step 7, Table 1), students were asked to use AcaWriter, first as a self-directed learning activity and then as a peer reflection activity (first sharing their thoughts and AcaWriter parser output(s) with their research group and then with the entire class). The process undertaken for the self- and peer reflection activities using AcaWriter in a workshop/laboratory setting has been documented elsewhere.12 Prompt questions are included in AcaWriter that address the research project and guide students’ self-directed reflective learning activity and thinking processes related to their research project. The included questions related to methodology and calculations, teamwork and group dynamics for the group research project, students’ perceptions of their project management skills, and student application of scientific knowledge and ability to link pharmaceutical research to clinical practice (Step 7, Table 1). The parser output of a students’ reflection related to pharmacy practice is shown in Appendix 1.
Tiered Approach to a Research Project in Which Master of Pharmacy Students Used the AcaWriter Software to Improve Their Reflective Practice in Scientific Research
A qualitative research design (thematic analysis of a focus group) was chosen given the nature of exploring student perceptions of using a novel online tool for reflecting on their research processes.13 Approval was sought from and granted by the UTS Research Ethics Committee. Because conducting the pharmaceutical research group activity was a compulsory component of the course, all students were required to be involved and use AcaWriter as a tool to assist their reflective learning. However, participation in the focus group discussion conducted after completion of the project was voluntary. Recruitment for the focus group participants was conducted via the school’s learning management system by the lead researcher, who was not an instructor in the course. The recruitment notice outlined that the first 12 students to indicate their interest to participate in the focus group would be accepted. The recruitment notice was not sent out to students until the following semester (after students had received their grades). Prior to the focus group discussion, students who indicated their interest in participating were provided with a participant information sheet and written consent form. The form emphasized that if they contributed to the focus group session, it would not affect their grades for the course. The students signed written consent forms indicating their willingness to participate and returned them to the lead researcher. The lead researcher guided discussion using the focus group guided questions (Table 2). The discussion was audio-taped and transcribed verbatim by an external transcription company. Because understanding students’ perceptions was a desired outcome of this research, thematic analysis was considered the most appropriate method for data analysis,14 and was undertaken according to Braun and Clarke’s six-phase process.15 Themes were initially coded by the lead researcher. A second researcher who had also reviewed the raw data independently was consulted and final themes generated once consensus was reached.
Prompted Guided Questions Used in a Focus Group to Obtain Pharmacy Students’ Feedback Regarding the AcaWriter Software
RESULTS
Of the 50 Master of Pharmacy students (17 male; 33 female) enrolled in the course, 12 (24% of the cohort; 11 female; 1 male) participated in the voluntary 50-minute focus group discussion. Thematic analysis of the transcribed discussion generated four key themes: Ease of use to prompt reflection, tangible tool with non-judgmental capacity; Benefits for enhancing self and peer reflection on research techniques and group dynamics; Benefits of the reflective writing process to enhance research capacity compared with engaging in reflective dialogue; and Benefits beyond the writing process: Cultivating self-improvement and self-confidence. Quotes from nine of the 12 focus group participants (1 male; 8 female) are presented in Table 3. Examples of when AcaWriter performed well and poorly are presented in Appendix 1.
Emergent Themes and Student Quotes From a Focus Group on Pharmacy Students’ Opinions of the AcaWriter Software
AcaWriter is far from perfect and when it performs Appendix 1 provides some examples of when it performs well and poorly. As we discuss next, this is inherent to a form of writing as complex as personal, professional reflection, but did not prevent the tool from being positively received.
DISCUSSION
While AcaWriter has been used for self-directed reflective learning across several educational domains,6,10 to our knowledge, this is the first study to explore student perceptions of this novel open-source web application to assist the development of practice-based research capacity. Furthermore, previous research related to the reflective genre in AcaWriter and its utility only explored its use with self-reflection and not with the capacity to be used to enhance peer reflection.6,9,10 One of the findings of this study that is supported in previous studies is that engaging in reflection is a powerful tool that helps enhance relationships.16 Students perceived the peer reflection component during the interactive AcaWriter workshop/laboratory as a process to support and build trust within a team environment.
There is growing recognition that analytics/AI-powered educational tools need to be developed using robust techniques that give different stakeholders (including educators and students) a genuine voice throughout the design process.17 Without this, tools are not utilized to their full capacity and are often ignored. We detail elsewhere the use of co-design techniques early in the design process in which the lead educator was able to shape the tool’s design.18 We note that reflective writing is an extremely complex form of communication, with oneself and any others for whom this is intended. It is impossible for AI to “understand” concepts in the way that humans do, and even within the limitations of what AcaWriter seeks to do (ie, provoke student reflection by mirroring back to them which sentences appear to be making salient moves) it is imperfect. As shown in the screenshot in Appendix 1, a warning is given at the top of the feedback to remind students that the agency rests with them, not with the computer: “Computers don’t read writing like humans. It’s fine to disagree with AcaWriter’s feedback, just like you’d ignore a poor grammar suggestion.” Elsewhere, we exemplify and discuss the impossibility of developing automated reflective writing classification and feedback that is exactly the same as that which human teachers who perform these tasks could provide; rather, humans and machines are best understood as bringing complementary lenses.7
However, despite the imperfections of natural language processing, the results of our study confirm that students still perceived the tool to be not only user-friendly but valuable as an on-demand source of immediate formative feedback that prompted productive reflection about their research projects. Research skills included critical thinking processes, problem solving, and effective communication and collaboration, especially if conducting research using a team approach. Previous research has shown that engaging pharmacy students in reflective practice can strengthen relationships and rapport with faculty, colleagues, and peers; improve their clinical decisions; facilitate their critical thinking and problem-solving ability; and enhance their academic performance.16,19–22 Despite some students indicating that they initially regarded the task of engaging in reflective writing as “futile,” time consuming, and a “box ticking exercise,” which has also been acknowledged in previous research, following the interactive workshop/laboratory, participants appeared to have changed their views and realized that engaging with the reflective process and using AcaWriter had benefits, and that engaging with the self- and peer reflection activities assisted them with their learning and critical thinking.23 They perceived that the time allocated within the curriculum to reflect on various aspects of the project enabled them to move forward and reevaluate their project methodology, calculations, and team skills and resources in order to produce a better research output. These results would seem to support the argument that in the context of building higher order student capabilities, imperfect analytics and AI still have important contributions to make in educational technology, when embedded into robust learning designs, and if students are encouraged to mindfully question, rather than mindlessly accept, the automated feedback.24
Reflective writing is a pedagogical strategy to prompt critical thinking.25 Our study findings indicate student support for this process. While students perceived the debriefing session as useful, the addition of the writing process further cemented their learning. Previous research supports this, and has shown that using a combination of a writing approach and verbal discussions complement the learning process and improve communication skills. Through this combined communication strategy, students acquire skills to reflect, which often leads them to transform their ideas into words.26
Another finding of our study indicated that students felt that the web application had further benefits beyond enhancing reflective and research capacity. Students perceived that the online platform enabled them to build their own confidence in the writing process (given the immediate formative feedback provided), facilitated a greater understanding of oneself through identifying their own assumptions, beliefs, and approaches, and in some cases, some participants indicated that engaging reflective practice using this tool enhanced their mental health capacity. Based on the students’ feedback, this may be attributed to the fact that shy students, who would not normally engage in group discussions or hold back on their opinions for a group project because of self-doubt or fear of being judged, found solace in the self-reflective writing component which still allowed them to receive instant feedback. Students’ reflective writing can be refined several times until they are happy with their final feedback (ie, writing/feedback cycles that are only practical with automation), and this may have improved their confidence level to engage with the group. Thus, engaging with AcaWriter before a group discussion and peer reflection may enable a shy student to receive feedback (automated but grounded in theory) rather than through human interaction with a course coordinator or team member. The self-reflection activity (assisted with AcaWriter) prior to the peer reflection activity enables students to revise their thinking processes.
Limitations to this study include the fact that only one cohort of pharmacy students from one university were included in the sample and focus group discussions. The study findings may not be generalizable to pharmacy students at other pharmacy schools or in other programs of study. A further limitation was that only one focus group session was conducted. Data derived from two or more sessions may have resulted in the generation of different key themes. An additional limitation to the study relates to possible selection bias given the larger number of female students than male students who volunteered for the study. Furthermore, the quotes included in the study were derived from nine of the 12 student participants. This was because the quotes from the remaining three participants did not offer any additional information related to the themes generated from the discussion. Further research should include pharmacy students from other schools and students from other disciplines that include scientific research and teamwork as part of the curriculum. Methodologically, an opportunity for future work is to use system logs to build a more comprehensive picture of how students use AcaWriter. For instance, we have demonstrated the computational analysis of the edits students make to their drafts to visualize the changes.27
Finally, the design of this study does not permit us to make the strong causal claim that the integrated use of AcaWriter leads to higher grades. Such claims pose methodological challenges to evaluating educational innovations in authentic settings. We can envisage a crossover intervention-control group study, with repeated measures to establish statistical relationships with final grades, but this would require substantial changes to the current curriculum. Alternately, a between-subjects design could compare an earlier cohort’s grades (without AcaWriter) with the cohort described in this study, but the samples would need to be comparable and larger. Moreover, because the technology’s availability fundamentally shapes the student reflection activities, the task comparison would not be exact. These are the complexities of evaluating educational technology interventions outside of artificially controlled laboratory studies. An example of an analytics-intensive approach is in related work (with another discipline and writing genre), evidencing that the quality of writing improved from draft to draft, in terms of the presence/absence of salient rhetorical moves, for students who engaged deeply with AcaWriter’s feedback.28
CONCLUSION
The findings of this study show that when a novel web application was integrated into the pharmacy curriculum, it demonstrated potential to assist pharmacy students’ self and peer reflection on a research task. Students reported that the reflection activities, augmented by AcaWriter’s automated feedback enabled them to better critique their own research skills and capacity, teamwork, and collaboration approach, and refine their reflective and critical-thinking processes to improve the research outputs. This study highlights the limitations of the current work, and the prospects for future investigations. Future and expanded investigations into the use of artificial intelligence-based feedback for pharmacy students would be beneficial in enhancing generalizability within pharmacy education.
ACKNOWLEDGMENTS
The authors would like to acknowledge the student volunteers from the University of Technology Sydney, Australia, Masters of Pharmacy degree, who participated in this study.
Appendix 1. AcaWriter’s User Interface and Sample Feedback
These screenshots illustrate AcaWriter’s differential feedback on weak and strong samples of students’ reflective writing.
For details, see the AcaWriter orientation website: University of Technology Sydney: https://uts.edu.au/acawriter
Full size colour versions are available online at: https://cic.uts.edu.au/ajpe2021-lucas-etal/
- Received July 29, 2020.
- Accepted March 23, 2021.
- © 2021 American Association of Colleges of Pharmacy
REFERENCES
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.