Abstract
Objective. To evaluate whether a relationship exists between the degree of student state-anxiety and pass rates on moderate-to-high stakes objective structured clinical examinations (OSCE).
Methods. Third-professional year (P3) students were assessed using the Speilberger State-Trait Anxiety Index (STAI) three weeks prior to their first moderate-to-high stakes OSCEs. Students’ OSCE station pass rates, individual station analytical scores, and the overall pass rate for the class were compared with student responses on STAI surveys to measure their association.
Results. Seventy-three students (100%) provided consent to participate in the research; 64 (87%) sufficiently completed the STAI survey. Degree of student state-anxiety or train-anxiety was not associated with any of the outcomes assessed in this study. Overall pass rate, individual station pass rates, and station analytical checklist scores were not inversely correlated with state- or trait-anxiety scores.
Conclusions. Efforts to assist students in OSCE performance should focus on means other than reducing associated anxiety. Future research in this area should focus on what interventions beyond instruction could be put in place to help students be more successful during OSCEs.
INTRODUCTION
For schools or colleges of pharmacy in the United States, student ascension from classroom to experiential education is an important mile marker on the road to graduation and licensure. Under the current accreditation model, students must complete an equivalent of four full-time academic years of instruction, with the last year accounting for at least 1,440 hours of clinical activity, also called advanced pharmacy practice experiences (APPEs). The transition from a predominantly classroom-centric experience to a practice-centric experience requires that students demonstrate minimal competency in the four major domains described by the Center for Advancement of Pharmacy Education (CAPE) in 2013; these domains are Foundational Knowledge, Essentials for Practice & Care, Approach to Practice & Care, and Personal and Professional Development. Summative assessment of these areas prior to this transition is routine through the use of traditional in-class activities, quizzes, and examinations. The assessment of students’ execution of CAPE domains prior to this transition has been gaining attention, with some colleges and schools of pharmacy developing progression examinations that assess one or several of them.1 Additionally, the Accreditation Council for Pharmacy Education (ACPE) calls for schools and colleges of pharmacy to demonstrate documentation and achievement of student competence in the 11 pre-APPE domains (Table 1) listed as part of Appendix A of the Guidance for Standards 2016 prior to entering their experiential year. One method of assessing this is with the use of objective structured clinical examinations (OSCE).2,3
ACPE Pre-APPE Domains2
Objective structured clinical examinations provide the opportunity for curriculum assessment and feedback to learners. In their least-threatening form, OSCEs are used as a formative means of providing feedback to students on their application of knowledge, skills, and behaviors. In their most threatening form, OSCEs are used as a high-stakes method of evaluating student performance and affect students’ progression through a curriculum. In the latter, a student’s failure to successfully perform adequately on an OSCE may result in remediation or inability to advance in the curriculum.
External variables that may influence student performance should be considered when using OSCEs. One of these is test (or performance) anxiety. Anxiety may be further subdivided into two categories, namely state anxiety and trait anxiety. Speilberger describes trait anxiety as an individual’s propensity for anxiety.4 Trait anxiety is a relatively stable factor that reflects individual differences in perceiving threats, stress, and dangers. State anxiety is variable and is in response to a current situation or scenario. In short, trait anxiety describes a person longitudinally, while state anxiety describes the person situationally.
Test anxiety is seen at all levels of education including elementary, secondary, undergraduate, and graduate education.5,6 In undergraduate- and graduate-level education, 15% to 17% of students report experiencing test anxiety. When evaluating OSCEs as part of a professional curriculum, rates and degree of student state anxiety varies based on the type of examination administered, with performance-based examinations appearing to produce significantly more anxiety than written examinations.7 The presence of test anxiety has an influence on student performance in some settings.7-11 Results are mixed and inconsistent; some research reports an inverse relationship between anxiety and examination performance while others fail to demonstrate a relationship. In most research, a relationship exists with traditional examinations (eg, written, multiple choice) and does not exist with performance-based assessments (ie, OSCEs). Even deeper, some research reports that the consequences of poor performance on the examination have an influence on anxiety and, in turn, overall performance.12
With the growing number of programs using or planning to use OSCEs as a form of performance-based assessment, anxiety should be considered as a possible influence on student outcomes.13 The purpose of this research is to evaluate the influence of chronic (trait) and acute (state) anxiety on performance-based assessment outcomes.
METHODS
Wilkes University’s Nesbitt School of Pharmacy is a four-year professional program. Students are enrolled in two, 16-week academic semesters per year for their first three professional pharmacy years (P1 through P3) for their didactic and introductory pharmacy practice experiences. After the P3 year, students complete at least 36 weeks of experiential education over the following 12 months.
In the summer of 2014, the Department of Pharmacy Practice participated in a workshop for developing and implementing OSCEs in the School of Pharmacy. Over the course of the academic year, stations were written, reviewed, and scored in working groups during two half-day retreats. Eight stations were developed based on the instructional topics from a skills-based course (Pharmacy Care Laboratory) and the Accreditation Council for Pharmacy Education (ACPE) pre-APPE domains and were implemented as a moderate-to-high-stakes progression examination for the P3 year (class of 2016). The numbers of stations required to pass was determined by the examination administrator and not shared with evaluators or students. For the first iteration of the examination, the number of stations required to pass was intentionally set lower than it would be for future years to allow for exclusion of stations that could not be validated with post-examination quality metrics. Students who were unable to successfully pass the OSCE would receive a grade of “incomplete” for the Pharmacy Care Laboratory and would not be eligible to progress to APPEs. An individualized remediation plan would be put in place for students who could not successfully pass the OSCE and complete the course. The examination was administered during finals week in the spring semester, approximately six weeks before the start of a new APPE-year. All students completed the OSCE on the same day and schedules were managed so that student-testing groups were not able to share examination information details prior to all students completing the examination.
We conducted an analytic observational study of P3 students enrolled in a skills-based, patient-care course. A validated survey to measure anxiety was administered to students, and the results were then evaluated against OSCE performance data to identify any relationships. The following protocol was reviewed and approved by the University's Institutional Review Board.
Students in the School of Pharmacy had not been tested formally using OSCEs prior to this study. During the week of finals in December 2014, the semester immediately preceding this study, the P3 class participated in a two-station OSCE experience. It was a zero-stakes examination (not graded) and more for evaluating the examination process than assessing student performance. However, students were provided informal feedback on their performance after each station. In the three weeks leading up to the spring OSCE, students took part in a three-part laboratory series in their Pharmacy Care Laboratory course to introduce them to OSCEs by teaching them how to write, review, and set standards for a station. On the day of the first laboratory session, prior to starting instruction, students were informed by the researcher about the study and asked to provide informed consent and to complete the 40-item, Spielberger State-Trait Anxiety Index (STAI).11 The STAI contains 40-items (20 items associated with state-anxiety and 20 associated with trait-anxiety), each answered using a score of one through four, with one indicating the least anxiety about an item and four indicating the most anxiety about an item. Scores on the anxiety index range from 20 to 80 and remain independent. Higher test scores indicate a higher level of anxiety. Though not typically used to establish the presence of or monitor for clinical anxiety, some researchers suggest a score of 39 to 40 as the cut point for clinically significant state-anxiety.14 Examples of state-anxiety (ie, “right now”) items include, “I feel tense,” and, “I feel secure.” Examples of trait-anxiety items include, “I feel nervous and restless,” and “I make decisions easily.”
After completing the final laboratory session in the series, the STAI was again administered to students and their responses were recorded. The differences between scores on the pre-and post-laboratory series surveys will be presented in a future paper. The results of the post-laboratory series survey will be used in this research report.
One week later, students in the P3 class took part in the required moderate-to-high-stakes, eight-station OSCE. The OSCE was conducted after students had completed all final examinations. After students completed the OSCE, their scores and pass rates were recorded and matched to their pre-and post-laboratory STAI results.
The STAI and OSCE results were evaluated in several different ways. First, the student’s overall OSCE score was evaluated for an inverse correlation to his or her post state-anxiety and trait-anxiety scores. Overall scores were calculated as the percent of stations in which the student achieved a passing score. A relationship would result in a negative correlation, as higher anxiety scores should yield lower overall OSCE pass rates, should the hypothesis hold true. A Pearson’s correlation coefficient was used to evaluate the relationship.
Second, students were divided into quartiles based on post-examination state-anxiety and trait-anxiety scores (Q1: lowest anxiety score, Q4: highest anxiety score). Each quartile was evaluated compared to the entire cohort (ie, all respondents) to determine if there was a difference in overall scores between the groups. Then, the quartiles were compared to each other individually, focusing on students in the lowest quartile of anxiety scores versus those in the highest quartile of anxiety scores. A student’s t test and one-way analysis of variance (ANOVA) were used to test for the presence of significant differences.
Finally, students’ OSCE scores at individual stations were evaluated to identify any influence of post state-anxiety and trait-anxiety on their performance. For this analysis, the students’ analytical checklist raw scores (number of checklist items addressed in an individual OSCE station) as well as the station pass rate (percent of students successfully passing an individual station) were used to evaluate for an influence. For the relationship between anxiety scores and analytical checklist scores, a Pearson’s correlation coefficient was used; an inverse number supporting the hypothesis. For the comparison of anxiety scores and achieving a passing score for the station, a point biserial was used; again, an inverse number supporting the hypothesis. All statistical tests were completed using SPSS 21 (IBM Corp. Released 2015). IBM SPSS Statistics for Mac, Version 23.0. Armonk, NY: IBM Corp). Results were assessed using a two-tailed alpha, with significance established at less than 0.05.
RESULTS
Seventy-three students from the P3 class (100%) completed the informed consent for participation and were enrolled in the study. Of those, 64 (87%) completed the Spielberger STAI twice, once before and once after completing the laboratory series. Nine participants were not included in the analysis because they either did not complete the STAI or left more than two items unanswered. Fifty-nine of these students identified their gender on the survey instrument. For overall evaluation, data were included for all students who completed the survey instrument. For gender-based evaluation, only students who identified as a male (n=25) or female (n=34) were included.
State-anxiety scores for the student population were significantly higher than the published means for STAI. The expected range for a college student for state-anxiety is 35.7 to 36.5 for men and 35.2 to 38.8 for women. In our cohort, the mean state-anxiety scores were 47.4 men and 48.5 for women, respectively (one-sample t test p values: men=0.0003 and women=0.0028).
The expected range for college students for trait-anxiety is 34.9 to 38.3 for men and 34.8 to 40.4 for women. In our cohort, the mean trait anxiety scores were 38.2 (one-sample t test p=0.95) and 43.6 (one-sample t test p=0.18) for men and women, respectively.
Neither student state-anxiety nor trait-anxiety was significantly correlated with their overall OSCE performance. Of the eight stations in the OSCE, students passed a mean of 4.5±1.3 stations (men, 4.6±1.2, women, 4.4±1.2; p=0.42). The Pearson’s correlation coefficient (r) to state-anxiety scores was 0.11 (p=0.41) and to trait anxiety was 0.028 (p=0.82). A post hoc analysis of the same data set based on gender did not show a significant correlation for state-anxiety or trait-anxiety in either men or women (Table 2).
State-Anxiety and Trait-Anxiety to Overall OSCE Score Correlation
Students were divided into quartile groups based on their state-anxiety and trait-anxiety scores. Between these groups, there was not a demonstrable difference between anxiety scores and overall OSCE performance (p=0.38 for state anxiety; p=0.55 for trait-anxiety). Additionally, there was not a significant difference in performance between individual groups when comparing them head-to-head (Table 3). Finally, individual station performance was not influenced by student state-anxiety or trait-anxiety (Table 4).
Mean Number of OSCE Stations for Which Pharmacy Students Received a Passing Grade Based on Individual Anxiety Score Quartiles (n=64)a
Correlation Between Pharmacy Students’ OSCE-Station Pass Rates and Anxiety Scores (n=64)
DISCUSSION
We conducted this study to determine whether there was an association between student state-and/or trait-anxiety and OSCE performance. We chose to use the Speilberger State-Trait-Anxiety Index (STAI) in this study because it is a validated tool, only one page in length, easy to administer and score, and allowed us to evaluate both chronic (trait) and acute (state) anxiety in the student population.4
At the conclusion of this study, we were unable to identify an association between the state- or trait-anxiety that students reported experiencing prior to and while taking the OSCE and their performance on the OSCE. Even when comparing the station pass rate of students who experienced the least state- and trait-anxiety (lowest quartile) with that of those who experienced the most state- or trait-anxiety (highest quartile), the difference was not significant. These findings are consistent with the findings from other investigations in health professional education programs that sought to evaluate the association between performance-based assessment success and anxiety.7-11
Prior to making general conclusions regarding test anxiety and student performance on a global scale, it is important to point out that this lack of association has been demonstrated only in performance based assessments such as OSCEs or other simulation examinations. It has not been demonstrated in written or knowledge-based examinations used for academic progression or board licensure. In instances such as these (eg, board examinations), there is an inverse correlation between examination performance and anxiety.12,15
There was not a significant difference between the trait-anxiety scores for our student cohort and the norms published in the STAI manual. However, state-anxiety was significantly higher. One of the factors that may have contributed to this difference is the generation characteristic for the original cohort studied by Speilberger and this current college cohort. The Wilkes University School of Pharmacy uses a guaranteed enrollment model where students may interview for placement in the school as high school seniors. If accepted into the school as a “guaranteed seat” and if the student maintains sufficient academic performance over two pre-pharmacy years, he or she is automatically enrolled in the school. Because of this, a majority of our incoming classes do not have previous degrees and are only two to three years post-high school graduation. The data from the STAI scoring manual was generated in 1983, at the cusp of the birth of the current Millennial Generation (or Generation Y). Thus, the results published in the manual more than likely applied to Generation X. In the most recent report from the American Psychological Association, the levels of stress and anxiety/nervousness are higher among the Millennial Generation, though not significantly greater than those of the previous generation.16 However, what is different is the incidence of these symptoms, as members of the Millennial Generation are more likely to experience these symptoms than their Generation X and Baby Boomer counterparts.17 The available research also suggests an increased incidence of these symptoms among medical students, leading this author to question if the same can be said for pharmacy students as suggested by the higher than expected state-anxiety scores in this research report.18 Given the scarcity of reported data on anxiety in pharmacy education using the STAI, this data may provide a baseline to which others can compare data from their schools.
In addition to overall OSCE results, our research also looked at individual OSCE station performance for students and the possibility of a correlation between state-anxiety and individual station performance. The stations were sequenced so that each student was faced with the same clinical scenario in the same order. When determining station order, there was some concern that students would be less successful with the initial stations in the sequence due to immediate test anxiety. Like in the overall results, student state-anxiety did not correlate with performance on individual stations, regardless of their placement in the sequence. This finding is consistent with research on pretest and in-test anxiety influence on student performance, where there was no direct correlation.19
From our experience, schools and colleges of pharmacy planning to implement OSCEs as part of their formative or summative assessment plan may experience an increase in student anxiety toward examinations at any level. Our results add to the body of evidence that exists on the topic, though do not offer assessors any clearer understanding on how to manage this increased student anxiety when preparing students for a high-stakes simulation testing. At the Nesbitt School of Pharmacy, our first thought was that increasing student exposure to moderate-to-high stakes performance-based assessments throughout the didactic curriculum might alleviate student anxiety over the new testing format, ie, OSCEs. However, this was just an assumption and no formal data are available to support this hypothesis. The cohort of students reported in this research (class of 2016) underwent OSCEs for the first time as part of their final examinations prior to beginning their APPEs. This will also be the case for the group of students currently in their third professional year (class of 2016). Starting with the class of 2017, students will have at least four experiences with OSCEs linked to course progression prior to the pre-APPE OSCEs. A follow-up inventory will be warranted for the experienced group (class of 2017) to evaluate their state- and trait-anxiety leading up to the final OSCEs.
In the available qualitative research on OSCEs and anxiety, results demonstrate that this testing style causes more anxiety than traditional multiple-choice or knowledge-based assessments.5 In addition, there is a good bit of research addressing how much student anxiety influences performance on multiple-choice or professional written examinations (eg, National Medical Examiners Board).6,7 However, there is limited information available regarding the association between student anxiety and success with performance-based assessments.11 That which is available fails to demonstrate an appreciable influence. The research in this area should be ongoing and include pharmacy education.
Future research in the area of anxiety and student performance on examinations should consider the strengths and limitations of this research in their protocol design. First, the inclusion of the STAI as the tool to measure anxiety was good in that it looked at both the propensity for anxiety as well as the response to situational stimuli. Using homegrown questionnaires about student perceptions regarding their feelings prior to and during the OSCEs may seem appropriate, but a validated survey instrument that has performed consistently well as reported in the literature should be used to produce reliable results and a way to measure the studied population versus the norm. Second, the timing of survey administration was good because it captured student anxiety only days prior to the OSCE. However, not all of the surveys were administered on the same day in relation to the OSCEs. The survey was administered in three different class sections on three different days leading up to the OSCEs. These temporal differences may be a limitation to the reliability of the state-anxiety score and future research should aim to administer the STAI to all participants on the same day. Third, the inclusion of a laboratory series on OSCE development may have had an effect (increasing or lessening) on students’ anxiety and test performance. It would be interesting to see a parallel-group study design used to evaluate whether a preparatory session focusing on OSCE design would have an effect on levels of student anxiety and success (ie, pass rates). Finally, this research was designed only to evaluate a relationship between state-anxiety and OSCE scores. It did not determine if any intervention to reduce anxiety would be of benefit to student performance. Future research in this area should focus on what interventions beyond instruction could be put in place to help students be more successful during OSCEs.
CONCLUSION
Students in schools or colleges of pharmacy may experience anxiety regarding OSCEs and other performance-based assessment methods. Based on the results of this study and other research outside of pharmacy education, neither state-anxiety nor trait-anxiety was significantly correlated with students’ ability to successfully pass these examinations. If schools are concerned with implementing measures to improve student performance on OSCEs, efforts to reduce the burden of testing anxiety is likely not the best intervention and efforts should focus on other areas.
- Received September 12, 2015.
- Accepted February 2, 2016.
- © 2017 American Association of Colleges of Pharmacy
In this issue
Jump to section
Similar AJPE Articles
Cited By...
- No citing articles found.