Predictive Relationships Between Students’ Evaluation Ratings and Course Satisfaction

INTRODUCTION

Pharmacy education has undergone a major transformation since the 1980s with the adoption of the doctor of pharmacy (PharmD) degree.¹ The Accreditation Council for Pharmacy Education (ACPE) is responsible for accrediting professional programs in pharmacy. The ACPE requires college and school of pharmacy administrators to evaluate the curriculum systematically and sequentially.² Institutional administrators must evaluate the structure, content, and organization of each course offered and must use and analyze course evaluation data to aid in continuous improvement of the curriculum. By fostering data-driven continuous quality improvements of the curriculum, institutional administrators are able to meet ACPE standards.

To maintain academically strong and effective programs responsible for training student pharmacists to be responsive to the changing health care needs of society, institutional leaders must meet the greater demand for accountability.² The accreditation standards that regional accreditors, the ACPE, and the US Department of Education set forth necessitate transparency and accountability for all stakeholders. Guidance is provided by ACPE through its accreditation standards and descriptive guidelines.

Student course evaluations are often the only data source available regarding the course’s effectiveness.^3-8 Faculty members and administrators often ignore or misunderstand course evaluation results.^5,9-12 Administrators use course evaluations in almost all higher education institutions, but often use these assessment tools incorrectly.¹³ Despite faculty protests, higher education administrators use student evaluations in making administrative decisions, including promotion, tenure, termination, and course offerings.^13-17 Although higher education administrators have been successful in developing processes to measure course effectiveness through course evaluations, their understanding of how to use these tools and who should use the data to make judgments and decisions is poor.^5,9,12,18,19

Some higher education administrators lack strong understanding about using course evaluation data to make judgments and decisions. Most previous course evaluation research focuses on traditional undergraduate degree programs and settings. Pharmacy schools have a different structure, which introduces unique influences on evaluations. Pharmacy schools have a mixture of didactic, laboratory, and experiential courses focused on practice and clinical education.

Medical and allied health educational administrators often look to general higher education literature for guidance when developing course evaluations.²⁰ As course evaluations in medical education serve to assess teaching effectiveness, medical education administrators must determine whether to use a global assessment of teaching assessment or a multidimensional scale.²⁰ The medical and allied health education literature has addressed both multidimensional and global assessment approaches.²⁰ The decision between global or multidimensional assessment is also important in pharmacy education.

No published research exists about the predictive relationship between the identified course evaluation rating variables and the overall course evaluation rating. Although many studies have focused on the relationship between course evaluations and student grades, none identified which components of a course predict student pharmacists’ overall rating for the course on the course evaluation.^21-28

Because higher education administrators and instructors are likely to continue using the overall course evaluation score as a single rating for comparing courses, knowing the extent to which the course evaluation variables in the current study are predictive of the overall course ratings could be useful. By identifying what aspects are most predictive, administrators and faculty members will have a better understanding of student course ratings and use the ratings to improve courses, the faculty promotion process, the tenure process, and course offerings.

No published studies address which extraneous variables moderate student course evaluation ratings in a professional school of pharmacy. One previous study in particular demonstrated that evidence indicating a single rating of teaching effectiveness is reliable when identifying instruction that needs improvement.²⁹ The focus of most of the previous research was on traditional undergraduate degree programs and university settings.^30,31 Medical schools, like pharmacy schools, have a different structure from traditional degree programs and therefore introduce unique influences on evaluations.^20,31,32

Beckman and colleagues noted medical and allied health education course evaluations should consist of two areas: interpersonal teaching and clinical teaching.³² In contrast, several scholars determined a multidimensional course evaluation approach is most appropriate when evaluating medical and allied health education.^33,34,35 Within medical and allied health education, multiple scholars have described characteristics of instructional effectiveness and how best to measure it.²⁰ Forms such as the Course Experience Questionnaire are used in medical and allied health settings to produce valid and reliable results.²⁰ Some evaluation instruments in the field focus on the effectiveness of teaching technical skills.²⁰ In regard to medical and allied health education, Litzelman and colleagues designed an instrument to evaluate clinical education focused on seven categories of teaching effectiveness: establishing a positive learning environment, controlling the teaching session, communicating goals to the learner, promoting understanding, evaluating achievement of goals, providing feedback to the learner and promoting self-directed learning, and promoting retention.³³

The most commonly used measuring technique for pharmacy college course effectiveness is student pharmacists’ ratings of courses.^36,37Although pharmacy education administrators focus on course evaluation items collectively, they should not ignore variables that moderate course evaluation items.²⁰ If a factor such as type of course significantly contributes to more favorable course evaluation ratings, pharmacy administrators should consider compensating for lower evaluation scores resulting from a less popular type of course when using course evaluation ratings to evaluate faculty members.³⁷

The goal of the current study was twofold. The first goal was to assess evidence of reliability for precision of measurement and validity for intended use of the course evaluation survey instrument using archival data. The second goal was to quantify the results of the relationships among predictor, criterion, and moderating variables by means of statistical analyses using archival data. The method was quantitative; the research design was descriptive correlational.

DISCUSSION

In higher education, evaluation has always been critical to the effective delivery of education and a crucial part of personnel and programmatic decision-making.³⁹ Given the importance of evaluation, administrators monitor course evaluation results to assess course and instructor performance.³⁹ In theory, instructors and administrators use evaluation results as practical tools for improving the program. However, if the design of the course evaluation instrument has flaws and the course evaluation items and other variables contribute to the relationship between the variables measured, then the usefulness of the tool is problematic. The usefulness of course evaluation items depends on the content and coverage of the items in conjunction with the variables contributing to the relationship between the variables measured.^39,40 Higher education administrators need to consider other things that may explain or influence the relationship between the course evaluation variables, such as GPA, course type, and course level.³⁹

Although higher education administrators focus on overall course evaluations, administrators and faculty members can identify means to improve course performance in individual course evaluation items. The results support prior research indicating individual course evaluation item ratings are predictive of how well a course performs.^40-44

Although higher education administrators focus on course evaluation items collectively, they should not ignore the possibility that any one item, ie, GPA, course type, or course level, may moderate course evaluation items. The current results both supported and differed from prior research indicating GPA, course level, and course type moderate the relationship between ratings of various course evaluation components and the overall course evaluation rating. If something such as type of course contributes to more favorable student course evaluation ratings, administrators must compensate for course type differences when using course evaluation ratings to evaluate faculty members.³⁷

Driscoll and Cadden indicated that course evaluation ratings were significantly different across departments.³¹ Type of course and level of course can affect students’ rating of courses.^32,34 In the current study, pharmacy practice type courses somewhat negatively moderated the relationship between the eight individual course evaluation items and the overall course evaluation item. Higher ratings were associated with pharmacy practice courses, thus higher education administrators and faculty members must take into account how course type may influence students’ ratings. However, Subramanya found that faculty members did not believe there was differentiation between course factors such as course level, course type, and class size that needed to be considered when interpreting course evaluation data.¹³ Faculty members and administrators must assert appropriate weights to account for the inherent course differences when interpreting course evaluation data.¹³

Critical limitations of the current study include the inability to draw causal inferences, the lack of control over predictor variables, and the inability to examine the contribution of other variables, such as faculty and student characteristics. Another limitation was the focus on one pharmacy school. Additionally, higher education administrators continue to use student evaluations of teaching as an essential tool, despite the controversy over their use. In higher education institutions, standardized course evaluation forms are often available for use within all courses, and therefore administrators will continue to use them.³⁹ Because programmatic accreditation requirements include assessment, faculty members in health professions education have embraced assessment as a useful tool to provide feedback about instructional practices.³⁹ Further research into the factors that influence how student pharmacists rate courses is needed.

The results both supported and differed from prior research indicating GPA, course level, and course type moderate the relationship between ratings of various course evaluation components and the overall course evaluation rating.

In the ordinal logistic regression model, each of the moderating variable outputs had one attribute with parameters set at zero. Thus, another limitation was the inconclusive results about the specific contributions of GPA, course level, and course type variables in explaining the significant relationships found between the eight individual course evaluation items and the overall course evaluation item. Replicating the study with a larger data set to ensure each attribute has sufficient numbers would help to provide a clearer picture of the moderating relationships.