A Distance Education Course in Statistics

INTRODUCTION

There has been an explosion in the number of new pharmacy colleges and schools in the United States in the past decade. However, the high cost of starting a college or school of pharmacy and the difficulties in finding qualified faculty members have spurred exploration of other options for training pharmacists. Existing colleges and schools have increased the size of their on-campus classes and expanded their distance education programs.

In September 2002, the University of Florida College of Pharmacy in Gainesville opened distance campuses in Jacksonville, Orlando, and St. Petersburg, FL. The college started the distance program to meet the demand for more pharmacists, address the lack of space on the founding campus, and fulfill the college's strategic plan for improving cultural diversity and access to a pharmacy education for persons with geographic limitations. In compliance with requirements of the Accreditation Council for Pharmacy Education (ACPE), articles describing the effectiveness of the program as a whole^1,2 and educational technology assessments³ have been published. This study examines the innovations and assessment of a course that was restructured from a face-to-face delivery format to a primarily Internet-based delivery format presented to students on both the founding and distance campuses.

At the time the university launched its distance programs, the evidence regarding online distance education was sparse. Since then, studies have been published and a meta analysis conducted by the Department of Education.⁴ Given ACPE's concerns about student learning in online/hybrid versus traditional face-to-face classroom courses, we conducted an outcome evaluation of a course that was designed to prepare students to critically evaluate the statistical analyses of articles found in the primary literature. The primary objective of the study was to determine whether there was a significant difference between the average scores for the baseline and final examinations. The secondary objectives were to examine the internal validity of the study and factors associated with successful completion of the course.

DESIGN

The Introduction to Quantitative Methods in Pharmacy course was designed primarily to be a Web-based course. The course's lectures (Appendix 1) were produced by the course faculty member using Camtasia (TechSmith, Okemos, MI), converted into digital format (mp4 file), placed within the college's course management system, and video streamed to students. This gave students the ability to view the lectures at any time, start and stop the lectures, and repeat the lecture material whenever they wanted. The blueprint for each lecture was to: (1) provide 1 or more scenarios that describe the types of empiric questions appropriate for the statistical test that is the lecture topic, (2) provide the formulas for the statistical test, (3) describe conceptually the components of the formula for the test (eg, difference between the group means divided by the groups' pooled variability for the t test), (4) calculate an example, (5) use a 6-step algorithm to demonstrate testing the research question, (6) interpret correctly the result of the statistical test, and (7) explain the appropriate use and application of the statistical test in the recent literature.

In addition to the lectures, the course had elements that required students to interact with the course content beyond the lectures. The first opportunity was with the course's workbooks and simulations (http://www.media.cop.ufl.edu/ids/workbooks/wb3/index.htm), which illustrated the principles and specific topics for the week. The workbook items were formatted as multiple-choice questions. If students answered correctly, they were provided with immediate feedback to further clarify issues associated with the question and then allowed to move to the next question. If students answered the question incorrectly, they also were provided with immediate feedback on why their choice was incorrect, but they were unable to move to the next question until they answered the current question correctly. Students had to continue to choose an answer option and view the feedback provided until they answered the question correctly.

Next, as a weekly assignment, students were required to formulate a multiple-choice question that was related to the current week's content. Each student's question and answer were placed in a database embedded in the college's course management system. The correct answer was verified by 1 of the course teaching assistants and combined with previous years' questions. Using the software embedded in the course management system, nearly an infinite number of 50-item, multiple-choice practice examinations consisting of students' submitted questions could be generated.

Since previous examinations were not returned to students, 2 activities were conducted to prepare students for each examination. First, 3 or 4 days before the examination, live and Internet-based reviews were conducted. The review was formatted to imitate the popular television game show, “Who Wants to be a Millionaire?” Our game show was entitled, “Who Wants a Pizza Party?” (http://media.cop.ufl.edu/az/ried/gs.mp4). Students on all 4 campuses competed against one another and the course instructor sponsored a pizza party for students on the winning campus after the conclusion of the course. Because the college encourages faculty members to visit each distance campus once per semester, the faculty member often arranged to conduct the game show examination review live on that campus during his visit. Students at the other 3 campuses participated in the game show review online using VoIP software (Elluminate, Pleasanton, CA). In the end, each campus participated live in the game show examination review once and participated online 4 times. During the game show, 15 questions were asked that illustrated the principles and specific topics to be included on the examination. The instructors immediately provided students with feedback regarding why the answer to each question was either correct or incorrect. Online attendance for the game show was a course requirement and students were penalized if they did not attend. Students were called on during the session, and if they did not respond, points were deducted from their total score. The game show was designed to guide students in studying for the examination.

If students did not understand the material after the game show, they were encouraged to study that material before the Internet-based examination review, which took place the night before the examination. The Internet-based examination review also was conducted using Elluminate and included other features to create a virtual classroom (http://media.cop.ufl.edu/az/ried/er/review.swf). The examination review was voluntary and the number of students attending generally ranged from 10–75.

ASSESSMENT

Assessment Tools and Administration

A prospective, non-equivalent group, quasi-experimental study was conducted during the 2008 fall semester.⁵ Four comparison groups of approximately equal size were formed to reduce concerns associated with threats to internal validity common to most quasi-experimental studies (Table 1). Second-year (P2) students who were taking the statistics class during that semester, were randomly assigned to group 1, group 2, or group 3 of the study. The students in group 1 were asked to complete a survey instrument that contained demographic questions and questions about their prior experience with statistics courses in their secondary school and prepharmacy curricula. In addition, students' first and last names and university identification numbers were gathered so that data could be linked at the end of the semester. The students in group 2 were asked to complete a survey instrument that contained the same demographic and identification questions asked of group 1, plus 10-items regarding principles of statistics.⁶ The students in group 3 were asked to complete a survey instrument that included the same demographic questions and 10 statistics-related items, but also contained 10 items taken from the previous year's final examination for the statistics course (survey items are available from the author). These 3 comparison groups were formed to determine the learning that occurred during the semester and to control for recall bias. A fourth comparison group was constituted by selecting 40% of students from the first-year (P1) class. Group 4 was effectively a nonintervention control group and theoretically their knowledge level would be equivalent to that of the P2 students at baseline.

• Download figure
• Open in new tab
• Download powerpoint

Table 1.

Study Design

One week prior to beginning the fall 2008 semester, P2 students registered for the course were sent an e-mail that urged them to complete an online survey to determine the need for a statics prerequisite and provided them with a link to 1 of 3 survey instruments, according to the group to which they had been randomly assigned. The items in the baseline survey instruments were taken down from the Web site as soon as the semester started and none of the items were used in the 4 midterm examinations for the course. At the end of the statistics course, all students in the class (groups 1, 2, and 3) were given the same final examination consisting of the same 20 items included in the precourse survey administered to group 3. To further avoid recall bias, 2 versions of the examination that differed only in the order of the 20 items were distributed randomly to students.

At the end of the semester, the college's Office of Student Affairs abstracted additional data on participants' demographics, prepharmacy academic performance, and cumulative first-year grade point average (GPA) in core courses. These data, along with students' responses to the online survey, were linked with the students' final examination scores. Finally, P1 students (group 4) were surveyed at the beginning of spring semester 2009 using the same Internet-based survey instrument containing the same 20 items included on the final course examination in the statistics course.

Outcomes

Three hundred one students enrolled in the statistics course in fall 2008. Among those enrolled, 102 students (group 1) completed the demographic questions only, 94 students (group 2) completed the 10-item survey instrument and demographics questions, and 105 students (group 3) completed the 20-item survey instrument and graphics questions. Approximately 58% (n = 175) of the students were Caucasian and the majority of the rest (n=126) were either Hispanic or Asian. Sixty-five percent of the students were female and 44% had earned a bachelor's degree or higher prior to starting the pharmacy program. Students' distribution among the college's 4 campuses was similar to that in previous years, with 44.5% in Gainesville, 15.3% in Jacksonville, 20.3% in Orlando, and 19.9% in St. Petersburg. Students' average age was 26 ± 6 years. Students attending the Gainesville campus were younger on average than students at the distance campuses (t = −3.99, p < 0.001).

The measures of academic performance included the PCAT composite score (84 ± 9.9), science/math GPA (3.5 ± 0.3), and first academic year GPA (3.3 ± 0.5). Approximately 82% (n = 245) of students in the class had taken a statistics course. Thirteen percent (n = 39) of the students had taken a statistics course during high school. The remainder had taken a statistics course in community college (20.6%), another 4-year college (45.6%), or during graduate school (2.0%). The time since the statistic course had been taken ranged from less than 1 year (9.8%) to 5 or more years (12.9%); however, the majority of the students had taken a statistics course from 2–5 years previously (47.4%) and approximately 12% had taken more than 1 statistics course. The proportion of students who previously had taken a statistics course was similar across the 4 campuses (p = 0.38).

A similar proportion of students in each of the groups had taken a statistics course prior to the current course (p = 0.39). The 10 original items on the survey instruments completed by groups 2 and 3 were compared and the mean scores were similar (group 2 = 3.1 ± 1.7; group 3 = 3.2 ± 1.7; p = 0.79).

Students who previously had taken a statistics course generally scored higher on the baseline camparisons with one exception for both the 2008 and 2009 classes (Table 2). At the end of the semester, the average score on the 20-item final examination was similar among those who had taken statistics beforehand and those who had not.

• Download figure
• Open in new tab
• Download powerpoint

Table 2.

Scores of Pharmacy Students Prior to and After Completing Required Online Statistics Course

The baseline scores of students who took the course in 2008 (groups 1, 2, and 3) were similar to those of P1 students (group 4) who would be taking the course in 2009 (Table 3). A similar pattern was seen for the 2 sets of 10-items from the original AJPE study⁶ and the 10-items created for the IQMP course; the average scores were not significantly higher.

• Download figure
• Open in new tab
• Download powerpoint

Table 3.

Comparison of IQMP Classes on Average Baseline Scores

The mean final examination scores on all 20 items were significantly higher than students' scores on the same questions from the precourse survey (Table 4). Group 2 students' average scores were 5.5 points higher than their scores on the 10-item pretest, and group 3 students' scores were more than 11 points higher than their scores on the 20-item pretest.

• Download figure
• Open in new tab
• Download powerpoint

Table 4.

Comparison of IQMP Average Baseline Scores on the 10 and 20 Items

The average final examination scores of students in groups 1, 2, and 3 were similar (Table 5). Students averaged 80% to 82% on the 20 items in the examination, indicating that prior exposure to the questions at baseline did not impact the final scores. Group 1 students (demographic information only) who were not exposed to any of the items at baseline scored as well on the final examination as students in both group 2 (10-item precourse survey) and group 3 (20-item precourse survey).

• Download figure
• Open in new tab
• Download powerpoint

Table 5.

Analysis of Variance Comparing Baseline Groups on Final Examination Scores

DISCUSSION

Students' gain in knowledge between the start and finish of the course was significant. The average baseline score of students in groups 2 and 3 was 20% to 25%, and at the end of the course, their average score for the 3 intervention groups was 81%.

The secondary objective was to examine the comparison groups to assess the internal validity of the study. Given the non-equivalent control group design of this study, the purpose of this comparison was to examine whether the groups were potentially equivalent. In each case, the groups were similar in their baseline and end-of-course scores. Notably, the average baseline score of the students scheduled to take the course the following year was the same as it was for the baseline scores of the students currently taking the course. This finding indicates that P2 students taking the course were similar in terms of baseline knowledge of statistics to P1 students scheduled to take the course the following year.

The comparisons among the 3 groups taking the course tested the students' comparability at the conclusion of the course. The comparisons examined the issue of recall bias from baseline to conclusion and follow up. In each case, the average scores at the end of the course were similar. This means that students who did not see the 20 test items at baseline and answered only the demographic questions scored the same as students who previously had seen and answered the 10-item and 20-item that were included on the final examination. Similarly, the additional 10 new items did not provide the students taking the 20-item survey with an advantage on the original 10 items administered at baseline. The interpretation is that recall bias did not influence the final outcomes. This is not surprising since almost 5 months elapsed between administration of the baseline survey instruments and the end of the course. The differences among the baseline assessment alone contributed less than 2% to the final score (average 81%). Our interpretation of this finding was that only a small portion of the final score was likely due to recall and more likely due to sample variability. Consequently, the difference of 11 points (55%) in scores between baseline and final examination was likely due to knowledge learned in the course. The findings from this study were used to make the decision that a statistic course was unnecessary as a prepharmacy requirement.

In course evaluations at the end of the semester, many students commented that the workbooks were the most useful component of the course. Students' evaluations of the textbook and lecture format were mixed. Many students would have preferred video capture of the live lectures rather than PowerPoint slides and audio because, for one reason, the AVI files produced using Camtasia did not allow them to speed up the lectures when reviewing them. This problem now has been addressed by switching to mp3 audio files and mp4 video files, which can be replayed at faster speed. Student also stated in their evaluations that they liked the game show examination review held once at each campus; thus, we added additional live game show article and examination reviews in subsequent course offerings.

There are several practical barriers that other colleges or schools might encounter should they want to implement this course and achieve these student learning gains. This course could be delivered entirely live, entirely via the Internet, or using a combination of delivery methods (hybrid). The practical barriers to implementing this course at another college or school depend on how the instructor decided to deliver the course. With respect to delivering it entirely via Internet, the traditional factors will impact its delivery (eg, server capacity, connection speed, examination security, synchronous versus asynchronous delivery, course development time, and expense). Other than the workbooks and simulations, live and hybrid delivery methods would involve live lectures, critical article evaluations, game show reviews, and examinations. The delivery of live and recorded lectures and use of various teaching strategies would require additional faculty members at each distance location. After the course lectures and assignments are developed, the costs of delivering the online curriculum would be relatively low compared to the cost of delivering it live or using a hybrid format, which would require more faculty members.

Although course faculty members developed and recorded the lectures, all of the other activities in the course were completed online and at a distance by graduate students, including examination and game show item writing, double-checking student submitted items, creating items for the workbooks, responding to questions on the discussion board, holding office hours, moderating/delivering the game show examination and article reviews, conducting reviews the evening before the examinations (with a faculty member present as backup), reviewing examination items before administration, and proctoring the examinations. Consequently, we believe this course could be used or replicated at other colleges or schools of pharmacy nationally and internationally.

The findings of this study have to be interpreted in the context of its limitations. The biggest limitation is its quasi-experimental design. Obviously, it would have been impossible to randomize students in a PharmD program to taking a statistics course or not. Consequently, the next best option was to demonstrate equivalence/representativeness by developing comparison groups. Twenty assessment questions cannot represent the entirety of the material taught in the course. However, it was necessary to keep students' motivation as high as possible for the low-stakes precourse assessment, especially when the postcourse assessment (final examination) had higher stakes. Consequently, including a sufficient number of questions to assess learning and at the same time reduce respondent fatigue was necessary. The goal was to balance the ideal with what was practical, as is usually the case with utilization-focused evaluation.⁷ Next, this study addressed only the period between precourse and postcourse knowledge levels. Assessing knowledge retention and learning decay afterward was beyond the scope of this study. Measuring retention of knowledge learned from this single course would have required taking multiple measurements over time and ensuring that the students had no additional learning opportunities or reinforcement of learning (eg, in advanced pharmacy practice experiences), which would have been impossible and counterproductive within a professional education program.

In summary, online, distance education is a relatively new phenomenon in pharmacy education, with most programs initiated in the past decade. With the increase in the number of programs, significant questions have arisen about the effectiveness of courses taught online. Although it was not a randomized trial, the students taking this online clinical statistics course appeared to learn the course material as they scored significantly better on the postcourse assessment than on the precourse assessment.

SUMMARY

Students can be taught the basic principles of statistics and how to use statistics to read the pharmacy and medical literature entirely online. Students showed significant knowledge and competency gains using the online approach. This study has significant implications for how classes traditionally taught in the classroom might be taught online using innovative instructional technologies.