Abstract
Objective. To implement and assess the effectiveness of the Strathclyde Computerized Randomized Interactive Prescription Tutor (SCRIPT) in teaching a competency-based undergraduate pharmacy course.
Design. Data on students' access to SCRIPT, collected by quantitative electronic data capture, were analyzed to determine student usage patterns and correlations between usage and grades in class assessments. Data on students' perceptions were collected by electronic questionnaire and semi-structured interviews. Teaching staff members also were interviewed.
Assessment. Two hundred forty-three students accessed SCRIPT a median of 23 times each. Students accessed SCRIPT predominantly at times outside normal teaching hours and tended to access the tool more often in the 48 hours preceding class assessments. Feedback from students indicated overall satisfaction with the tool to compliment the timetabled teaching sessions but highlighted that more specific feedback on the examples was required. All staff comments were positive.
Conclusions. Students and teaching staff members valued SCRIPT as a tool to compliment teaching of the competency-based pharmacy practice classes in the MPharm degree. Future developments of SCRIPT will include expanded feedback for students, the capability to link the release of each SCRIPT exercise with the date the content is taught in class, and additional tools to facilitate “just in time” teaching.
INTRODUCTION
Web-based teaching has developed as either a replacement for or an adjunct to classroom-based teaching. The benefits of Web-based teaching include flexibility that allows the student to choose the time and place where they study, ease of access for the student and academic staff regardless of distance, and documentation of use. There also are disadvantages to Web-based learning including the costs of equipment and Internet access, technical problems with software and hardware, and risk of students developing an isolated learning environment. Also, in some situations, technology has been/is used for technology's sake rather than to achieve sound educational goals.1 The Web has been used to teach a variety of subjects at all levels of education. Although there are numerous examples of Web-based education in healthcare,2–6 no publications describing a specific tool relating to identification of clinical and legal errors in prescriptions were identified in the literature. Computer-based learning does appear to have a role in teaching clinical decision making to students taking healthcare-related courses.5–7
In the School of Pharmacy at the University of Strathclyde, Glasgow, UK, we developed a Web-based teaching tool, SCRIPT, to supplement the laboratory-based teaching in a professional competency class in the undergraduate pharmacy degree. The professional competency class requires students to identify clinical and legal errors on a prescription, which is then dispensed and checked in a role-play scenario. In the past, students had commented that the class caused concerns and undue worry as the students felt underprepared because of the limited availability of study and revision materials. SCRIPT, which allows students to independently assess their ability to identify clinical and legal errors within a scenario, was designed as a study aid to supplement laboratory sessions and provide students with optional revision examples. The aim of this study was to evaluate student use and perception of SCRIPT and to assess how it could be developed further to enhance student learning.
DESIGN
SCRIPT was developed within our Web-based virtual learning environment, SPIDER (Strathclyde Personal Interactive Developmental Educational Resource). All programming was carried out in house, creating a modular extension to SPIDER, using the same open source technology used in the virtual learning environment (Linux; Apache, version 2.2.9., Apache Software Foundation, Forest Hill, MD; mySQL, version 5.0.5, MySQL/Sun, Uppsala, Sweden; PHP 5.2.6, The PHP Group). PHP was used for coding; a MySQL database was used to store the data; Apache was used as the Web server; and all programs were run on a Linux operating system. SPIDER allowed students remote access to SCRIPT 24 hours a day.
SCRIPT presented students with a series of random prescriptions, in groups of 3, allowing them to identify errors in prescribing, dosing, labeling, and register entry, and interactions. SCRIPT has a bank of 400 scenarios covering all the main prescription categories, labels, and registers, and the ability to cover prescription endorsing; a requirement to allow payment from the government. The names and addresses of the patients and prescribers were randomly generated, which gave the impression of an increased number of scenarios. Additional patient information that may or may not have been relevant to the scenario was provided to the student. After completion of a scenario, the program gave immediate feedback to the student, with marks based on the grading system used in the class. The feedback was standardized according to the errors in the prescription (Figure 1). At the end of each test (defined as a set of 3 prescriptions), students could review their performance by accessing an error summary (Figure 2). Also, the class coordinator and other academic staff members could access class data, which showed the most common errors made by a cohort of students accessing SCRIPT. (The demonstration site for SCRIPT, with access to a limited number of prescriptions and scenarios, is available at http://spider.science.strath.ac.uk/script/)
SCRIPT screenshot showing the feedback students receive after completing each prescription.
SCRIPT screenshot showing a student's personalized summary highlighting the areas where they make regular mistakes.
In academic session 2007–2008, there was a population of 243 students in 2 cohorts. The course was taught to a home cohort (126 students attending class at the Glasgow campus for the entire degree program) over the 2 traditional academic sessions from October through May, and to a collaborative cohort (117 international students attending class at the Glasgow campus for one calendar year) in a compressed summer semester from June through August. Students in the home cohort completed the competency course during the 3rd year (October 2007 to June 2008) of a 4-year integrated master of pharmacy (MPharm) degree program. Students in the collaborative cohort were enrolled in the same degree program but completed years 1 and 2 and part of year 3 at the International Medical University, Kuala Lumpur, Malaysia, before completing the remainder of the degree at the University of Strathclyde. The competency class was taught over 2 semesters for the home students but compressed into 1 summer semester for the collaborative students. The material studied was common, and as the degree is awarded by the University of Strathclyde, the same standards were expected of both cohorts. Thus, the intent of the study was not to compare the 2 cohorts. All data in the results section considered the population except where noted.
Data were analyzed to identify the total number of times that students attempted a set of 3 scenarios, the pattern of use based on a 24-hour day, the patterns of use in the 7 days before and after each class assessment, and any correlation between access and assessment marks. There were 4 possible assessments in this class. Class assessment 1, after 10 laboratory sessions, was formative to introduce the structure of the assessment; class assessment 2, at the end of 20 laboratory sessions, accounted for 10% of the final grade for the course. Students who attained in excess of 70% in the exemption assessment (assessment 3), approximately 2 weeks before the degree assessment (assessment 4), did not need to take the degree assessment. In the exemption and degree assessment all material covered in the class could be examined.
Students were also asked in questionnaires and semistructured interviews for comments about SCRIPT and how the program compared to the instructional portion of the class. Ethical approval for the semistructured interviews was obtained from the departmental ethics committee. The questionnaires were made available to all students on the virtual learning environment, SPIDER, after the final class assessment. As this was an in-course evaluation, the departmental ethics committee stated that no ethical approval was required. All students were made aware via an e-mail that completion of the questionnaire was voluntary, responders would remain anonymous, and responses would be used for ongoing evaluation of SCRIPT.
EVALUATION AND ASSESSMENT
In the 2007–2008 academic year, there were 6734 attempts, logging on to attempt a set of 3 prescriptions, made using the SCRIPT tool. Of these attempts, 2841 were made within the first 2 semesters and 3893 occurred during the summer semester.
Table 1 gives details of the overall access to SCRIPT throughout the academic year. Both cohorts made considerable use of the tool. A Mann-Whitney test (2 tailed) indicated that the collaborative cohort used the tool significantly more than the home cohort (p < 0.001). The main use of SCRIPT occurred between 8AM and 11PM with peaks of activity at 11AM and between 8PM and 10PM. Over 50% of the attempts were made outwith the traditional teaching hours of 9AM to 5PM. There was negligible use of SCRIPT between 3AM and 6AM.
Details of Student Attempts at SCIRIPT (2007–2008)
SCRIPT was available to the students from 1 week before class assessment 1 until the end of the academic year. Analysis of the total number of attempts highlights that the students rarely accessed SCRIPT at any time other than just before and/or after a class assessment. The majority of attempts occurred in the 24 hours immediately preceding an assessment, with the exception of a peak in usage that occurred approximately 5 days before class assessment 1, resulting from the collaborative cohort being introduced to SCRIPT.
Figure 3 shows that there was a significantly greater pass rate in class assessment 2 for users of SCRIPT compared to non-users (p < 0.05). The differences in pass rates among users and nonusers of SCRIPT for class assessment 1 the exemption assessment, and the degree assessment were not significant (chi-square).
The percentage pass rate for users and non-users of SCRIPT.
The credit weighted average mark (the mark for each class within year 3 year weighted by the effort required for the class) of each student was used as an index of their academic ability, and Figure 4 shows that there is no significant correlation between the credit-weighted average and the number of SCRIPT attempts made by the student. In this context, the credit-weighted average was based on the students' total performance in the third year, excluding the competency class which was reported as either pass or fail.
Number of SCRIPT attempts compared to the students' credit weighted average for the third year of the MPharm.
In addition to the usage data collected from SCRIPT, a questionnaire regarding the utility of the tool was administered to students. Ninety-three (38.3%) students completed the questionnaire (Table 2). Positive comments received included “a useful aid to studying,” “allowed me to look at examples that weren't in the class notes or labs,” and “needs additional feedback.” Negative comments included “I found the drop down menu confusing,” and “feedback could be improved”. All comments from staff members were positive.
Student Responses to the Evaluation Questionnaire (N=93)
The majority (84%) of students found that SCRIPT was user friendly, and had good concordance with the prescriptions in the competency-based class (55% thought the prescriptions were similar to those in the class and 42% thought the level of difficulty was comparable to the class). Nearly a quarter of respondents expressed a desire for the program to be expanded to include additional prescription types. With regard to the feedback offered after each test, 58% of students agreed that it was useful and 16% disagreed; some students expressed a need for feedback to be expanded. Sixty-eight percent of students found SCRIPT helpful at identifying problem areas in their knowledge and understanding of pharmacy practice.
The majority of the benefits identified were related to the speed with which feedback was received, the use of real prescription backgrounds to replicate those seen in community practice, and the usefulness of the tool to aid revision by identifying areas of weakness. The following is an example of a student comment “I thought [SCRIPT] was v[ery] useful. It is an interesting alternative to looking at textbooks and I liked … [that] it gives you immediate feedback on how you performed. I like how it details the errors and shows where and how many marks were lost to show how serious the mistake made was”.
The semistructured interviews identified a number of potential additions to the feedback: identifying resources, such as the students' textbooks, where additional information can be found; highlighting “counseling points” that students should consider when talking to the patient; and “personalizing” feedback for each prescription by highlighting key points to consider.
DISCUSSION
Both cohorts accessed SCRIPT numerous times over the academic year. The 2 cohorts show different patterns of SCRIPT use, with students in the collaborative cohort accessing SCRIPT more often than those in the home cohort. The home cohort used SCRIPT for the first time while the staff members involved in teaching the class were becoming familiar with the technology. During the teaching period for the collaborative cohort, the staff members had become more comfortable with the technology and appeared to encourage the students to make use of the tool. Lack of appropriate buy-in from academic staff members is a well-known barrier to the introduction of new technology into education8; thus, increased staff familiarity may have influenced the collaborative cohort's more frequent use of the tool. The home cohort also had longer periods between both the teaching sessions and assessments and were able to discuss course concepts with their peer group, tutors, and possibly with pharmacists during their part-time employment in pharmacies. The absence of these additional learning opportunities for students in the collaborative cohort has been recognized and may account for that cohort's greater usage of SCRIPT.
As expected, the majority of access to SCRIPT occurred during normal waking hours. This was more evident in the home student cohort, as there were no attempts made between 3:00 am and 7:00 am in this cohort. The collaborative students did access SCRIPT overnight. This may have been a reflection of the compressed nature of the collaborative course, where the students had an intense and full timetable, resulting in the need to study overnight, or may have been because many of the collaborative students were contacting relatives in Malaysia via the Internet during this time so it was a convenient time to access SCRIPT as well. Within the home cohort, there was increased use of SCRIPT between 10:00 am and noon, which can be explained by free periods in their timetable. For the collaborative cohort, the peak in use between 8:00 pm and midnight appears to relate to their preferred study time, as they have a full timetable of classes between 9:00AM and 5:00PM, with few free periods. Many of the attempts were made at times when academic staff members would not have been available, thus highlighting the importance of SCRIPT as a revision aid and Web-based tool to encourage self-paced learning.9 Many of the staff members believe this is a key benefit of the program as it reduces the demand placed on staff members and allows the students more independent learning.
During the academic year, there was a correlation between the number of attempts and the date of the class assessments, with an increase in the number of attempts in the 2 days before the class assessments. Students appeared to be using SCRIPT as a tool to aid revision immediately before the assessments; however, only a few students appeared to be using it for immediate reinforcement of learning after the scheduled teaching events. This pattern of use has also been observed among business and Internet technology students who used online resources in preparation for examinations.10 The collaborative cohort was first introduced to SCRIPT 6 days before the first assessment and this may explain the peak in use at this time. Although the majority of students accessed SCRIPT before the assessments, a few students appeared to use SCRIPT in a reflective manner after the assessments, checking the answers they had given to determine how well they had performed on the test. There was no correlation between SCRIPT usage and the academic performance of students.
The students thought that SCRIPT was user friendly, a useful supplement to the competency class, and should be expanded to include other types of prescriptions. Students agreed that the tool was helpful in identifying problem areas and useful for revision purposes, but only a little more than half agreed that the feedback the program provided after their attempt was helpful. The feedback provided by SCRIPT was intentionally kept simple and limited to whether the student's had identified the errors correctly or not. However, the lack of in-depth feedback resulted in numerous e-mail queries from students. As a result, comments addressing legal aspects and more specific clinical feedback tailored to individual patient scenarios will be added to SCRIPT. Although this will enhance the students' learning experience, it will require ongoing updates to the system as changes in legislation occur.
As many of the students indicated that they could not see the link between SCRIPT and the material taught in class, the tutorial examples will be expanded to include all potential types of prescriptions and then group and link the examples to specific laboratory sessions. This can be achieved easily as the virtual learning environment allows tutorials to be released on specific dates and “switched on and off” as appropriate. We also intend to add additional features for class tutors. This will be able to limit the dates between which they can see errors to allow this to be linked to the individual groups of scenarios and thus provide “just in time” teaching to address problems as they arise during the use of SCRIPT.
Another planned refinement for SCRIPT will allow students to search for the correct error by typing a partial name or description of the error and searching a retrieved shorter list rather than searching through the entire alphabetical list of 40 possible errors. Once these modifications and upgrades have been completed, the commercial potential for this tool will be investigated with the University of Strathclyde's Research and Innovation unit.
CONCLUSION
The analysis of SCRIPT use over the academic year of 2007–2008 has demonstrated that the program is highly valued by the students, particularly as a review tool before class assessments. Students appreciated the benefits of such a tool, which was accessible any time, but they did highlight areas in need of further development, such as increasing the detail of the feedback provided.
- Received March 23, 2009.
- Accepted August 8, 2009.
- © 2010 American Journal of Pharmaceutical Education
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.