A Cost-Benefit Analysis of Teaching and Learning Technology in a Faculty of Pharmaceutical Sciences

INTRODUCTION

There is increasing recognition and expectation in postsecondary contexts that teaching and learning technology (TLT) should be used to improve the quality of teaching, student learning, and the student experience.¹ Consistent with scholarly approaches to teaching,² this quality improvement agenda is evident among pharmacy educators in the delivery of pharmacy curricula as well as associated accreditation guidelines and standards. The Canadian Council for Accreditation of Pharmacy Programs (CCAPP), for example, requires the use and integration of teaching technologies that support the needs and learning preferences of diverse learners.³ In the United States, the Accreditation Council for Pharmacy Education requires that colleges and schools be equipped with technologies that support student training, reflect contemporary practice, and commit faculty members to effective teaching and enhanced student learning.⁴ In the past decade, the increasing focus on TLT reflects an awareness of the new generation of learners entering undergraduate programs⁵ who have grown up with technology and expect it to be used for greater collaborative and experiential learning.⁶ Increasingly, technologies incorporated into educational settings are broadly classified as Web 2.0 tools (eg, blogs, podcasts, wikis, social networks, and learning management systems), digital simulations, and virtual patients.^7,8 While using TLT is increasingly important, successful implementation often involves changes in teaching practice, which requires the conditions necessary to support changes, including faculty champions, administrative leadership, faculty development opportunities, and adequate resources to guide the adoption of technology.⁹

Provision of TLT and related infrastructure represents a key investment of resources, which are often limited in educational settings. In an age of increasing accountability and shrinking budgets, postsecondary decision makers (eg, deans, associate deans, and department heads) are expected to demonstrate efficient use of finite resources, whereby objectives and outputs are maximized subject to financial constraints.¹⁰ Investing resources in TLT precludes alternative uses of that funding, where the best alternative use of resources forgone is known as the opportunity cost. Rigorous economic evaluation offers a systematic, structured approach to decision-making through critical analysis of the costs and consequences of resource allocation, weighing possible improvements in outcomes against opportunity costs. Economic evaluation is used to support resource allocation decision-making in publically funded areas where market forces cannot be relied upon to maximize the benefit to a community. Examples include the transportation, environment, health care, and education markets.^11,12 For resourcing decisions to be effective, however, objectives and output metrics need to be explicit. Unfortunately, when economic analysis has been applied in dynamic, higher-education contexts this is rarely the case.^6,13 Also, in the rare occasion that objectives and outputs are clarified, approaches are typically constrained to cost-considerations. These objectives promote return on investment (ROI)-type evaluations that seek to justify the upfront and ongoing costs by offsetting them against capital cost savings or increases in revenues (eg, tuition) and profits.^6,13 In higher education settings, purely cost-focussed ROI evaluations are relatively easy to measure, but ignore a multitude of other potential benefits of investment, for example, the impact on student satisfaction, teaching quality, and achievement of educational outcomes and/or competencies.^14,15

Moving away from purely cost-focused ROI-type evaluations of TLT investments shifts attention to identifying and valuing improvements in the learning environment (eg, enhanced teaching practices, student learning and educational outcome achievements) as opposed to costs saved and/or revenues/profits generated. A key distinction here is the difference between price, cost, and value: price is the monetary amount paid for the technology (as set by the producer or manufacturer); cost is the total amount incurred to have the technology installed and available for teaching, and; value is the utility or perceived worth of the technology by users. The value of investment in TLT might include, for example, improvements in the quality of teaching, student learning, and achievement of educational outcomes and/or competencies associated with TLT use. In a practical sense, economic evaluations that focus on the value of investment can be done either through cost-effectiveness analyses, which value specific educational outcomes (eg, time required to acquire a specific skill or competency)^13,16 or through a broader cost-benefit analysis (CBA) which places no restriction on the outcome(s) to be valued.^17,18 Cost-benefit analysis is arguably one of the most theoretically sound and comprehensive methods of economic evaluation.¹² Cost-benefit analysis seeks to compare the cost of an investment with a monetary estimate of the social value placed upon it by stakeholders. Monetary estimates of the value of outcomes are elicited by creating a hypothetical market under survey conditions by asking stakeholders to estimate their maximum willingness to pay (WTP) for the outcomes. A CBA on an investment in technology for teaching and learning, for example, would ask faculty users to estimate their maximum WTP for the technology based on its perceived utility and worth to their curriculum and pedagogical practices and to their students’ learning. Valuing outcomes in monetary terms allows benefts to be compared with costs on a common denominator. Individual WTP estimates are summed and averaged, and compared against actual costs of the investment to allow an assessment of whether the cost of having the technology in place is offset by the value placed on it by users. If the average maximum WTP for TLT (which reflects the social value placed on it by users) exceeds the actual cost, then the technology is deemed to have a positive net benefit and can be considered a worthwhile investment, provided there is sufficient budget available.^19,20 Cost-benefit analysis allows informed decision-making about whether investments should be continued and/or pursued, and for alternative investments to be ranked according to their net benefit and pursued in order, as budgets permit. While this approach has long been used successfully in economic studies within environment,²¹ transport,²² and health²³ contexts, to our knowledge, no studies to date have attempted to quantify the value of investment in TLT in higher education contexts using CBA analysis.

To address this gap in the literature, the aim of this study was to conduct a CBA of investment into TLT by a Faculty of Pharmaceutical Sciences in a large research-intensive university, and demonstrate how this type of economic evaluation can be operationalized in this context. To meet this aim the study objectives were: to identify and describe the national, institutional, and college objectives and/or goals of investment in TLT; to identify the opinions of faculty members about the value of TLT in their pedagogical practices, the metrics and other qualities they used to inform their opinions, and how their opinions aligned with national, institutional, and college goals for TLT use; and to conduct a CBA to determine if technology investments were worthwhile. The context of TLT is used as an exemplar of an investment where allocating resources to one specific use precludes the opportunity to allocate resources to other potential uses (eg, recruitment of additional faculty and staff members).

METHODS

To address the study aim and objectives, both qualitative and quantitative methods were applied. The study was conducted in the Faculty of Pharmaceutical Sciences at the University of British Columbia (UBC) and received ethics approval from UBC’s Behavioral Research Ethics Board. The college currently employs approximately 70 faculty members and offers an undergraduate “one plus four” BSc(Pharm) program, a new “two plus four,” an entry-to-practice Doctor of Pharmacy (PharmD) program (launched September 2015), and postgraduate PharmD, Master of Science, and PhD programs. The college’s new state-of-the-art building, which opened in 2012, had significant investments in technology infrastructure and specific learning technologies, which aligned with its long history of technology innovation and use to support and enhance student learning.^24,25 An in-house Office of Educational Technology and Learning Designs provides day-to-day support and faculty development opportunities to facilitate learning technology use by both faculty members and students.²⁶

To give historical context to current perspectives on TLT, documents published by the college, university, and national governing bodies from the last decade (2005-2016) were analyzed for any stated aims for new educational technologies. Researchers used a modified version of Lincoln and Guba’s constant comparative method²⁷ to establish the goals and objectives for TLT use in the college (the 2005-2009 Strategic Plan (2005)²⁸; The UBC Review of the Faculty of Pharmaceutical Sciences (2007)²⁹; the 2012-2017 Strategic Plan (2012)³⁰; Faculty of Pharmaceutical Sciences BSc(Pharm) Self-Assessment Report for Accreditation (2012),³¹ at the University (Place and Promise: The UBC Plan (2012),³² and nationally (CCAPP Accreditation Standards for the First Professional Degree in Pharmacy Programs (2014)³; Association of Faculties of Pharmacy of Canada (AFPC) Strategic Plan, 2014-2016 (2014).³³ Documents were analyzed in a multi-step process. To find relevant sections in the documents related to TLT goals and objectives, each was first searched electronically using specific keywords (eg, technology, virtual, simulations, Web 2.0, Internet, teaching, learning management systems). Each section that contained a keyword was read carefully and, if applicable, extracted into a spreadsheet for further analysis. A narrative synthesis of the extracted sections from each document was created to form one vignette per document that encapsulated the document’s main themes, context, and goals (both implicit and explicit) for TLT. Documents were analyzed by at least two different researchers to ensure inter-rater reliability. The seven vignettes were then analyzed to characterize the overall goals for TLT in the college, both currently and historically.

All full- and part-time faculty members teaching in the BSc(Pharm) and/or entry-to-practice PharmD degree programs at the time of the study made up the potential sample of interviewees (approximately 70 individuals). Purposive sampling was used. The sampling criteria included administrative status, academic ranks, discipline affiliations, and teaching responsibilities and context. To seek diversity of opinions, the faculty members’ years of service, practice experience, and level of involvement in pharmacy affairs locally, provincially, and nationally were also included in the sampling criteria.³⁴ Recruitment of study participants from the volunteer pool took place in three waves using individualized e-mail invitations. For those that responded, additional information about the study and the interview protocol was provided and any questions were answered. Those faculty members who agreed to participate were asked to provide written consent and were included in the study unless they chose to withdraw.

Interviews took approximately 45 minutes and consisted of three sections: structured questions to determine the demographics of the study sample, open-ended questions enquiring about the participant’s goals and perspectives on TLT use and their methods of evaluating its impact on the quality of their teaching, and a WTP section.^20,27,34-36 Interviews were audio-taped, transcribed, coded, and analyzed for themes using the constant comparative method.^27,35

The WTP activity involved a random card sort method to elicit the maximum price the interviewee felt that the teaching and learning technology available to them was worth.³⁷ It allowed participants to consider all potential benefits within their evaluation of TLT and provided a framework for them to reveal their preferences. For example, one question stated: Imagine you are sent back to 2010-11 as an advisor to the planning committee that makes decisions regarding Faculty investment in the types of technology and supporting infrastrutcture that could be available (e.g., doc cam, patient simulators, AV equipment in each room, blogging and other applications, and learning management systems), as well as staff to support ongoing maintenance of this technology. Please consider your current knowledge about how these technologies have enabled you to teach and deliver your content.

Participants were asked to reflect on their current knowledge and opinions of TLT, and then they were shown several cards with varying monetary values. They were then asked to choose, from a college-level perspective, how much the college should be willing to pay for TLT, appreciating other potential uses of that money, such as recruitment of additional faculty and staff members, paying external clinical experts/practice educators for teaching contributions, and updating existing research space and equipment. Cards were shown in random order to minimize anchoring bias.^38,39 The values presented on the cards were chosen based on a pilot study conducted on six faculty members, all of whom would be included in the full study. The pilot study used random cards with increments of Can$1M from a minimum of Can$0 to a maximum of Can$10M; the mean WTP from the pilot survey was Can$2.8M (SD=Can$1.9M), with a maximum WTP of Can$6M. Based on data from the pilot study, the cards used in the full study were centered on Can$2.8M, and presented values ranging from Can$0M to Can$5.8M in increments of Can$0.4M. Participants were not told the total cost of TLT (Can$4.25M) nor the cost of the new building (Can$133.3M) to minimize contextual bias. For each card, the participant decided whether they would be willing or unwilling to recommend that the college spend that amount on TLT. They were asked to consider the timeframe for investment in TLT from September 2012 to September 2016 to reflect the amount of time the new pharmaceutical sciences building had been open. Once all cards were sorted, the participants were asked to choose the maximum amount that the college should invest in TLT over the specified timeframe, thereby providing the maximum WTP value.

The costs associated with TLT investment in the new building were calculated from detailed summaries of expenditures compiled and maintained by the college’s Office of Operations & Facilities Management and Finance Department. Categories of expenditures summarized on the orginal financial documents included hardware, software, and staffing requirements. For this study, costs were reorganized into four aggregate categories: research and development, initial investment costs, operations and maintenance, and salvage and disposal. The timeframe for the calculations of cost was September 2012 to September 2016.

Summaries of the WTP and net benefit estimates (WTP minus estimated cost) for each participant were summarized using means and bootstrapped confidence intervals for mean WTP and net benefit calculations. As the mean WTP was a sample mean, there was likely to be error in this estimate. We used bootstrapping to try to quantify the potential size of this error. Bootstrapping is a way of estimating robust standard errors needed to generate confidence intervals. Bootstrapping works by treating the sample as if it were the population of interest and then drawing repeated random samples with replacement to estimate the mean values. Bootstrapping allowed us to estimate a 95% confidence interval around the sample mean. Effectively, this 95% confidence interval represents the range that, if we were to have drawn 100 samples from our target population, 95% of our estimates of the mean would have fallen within this range. A demand curve was generated using WTP estimates to calculate the proportion of participants whose estimates of WTP for TLT exceeded increasing specified values of WTP. In any CBA, assumptions are made about how different estimates of cost and WTP are included. These assumptions can affect the conclusions that might be drawn from the analysis. In sensitivity analysis, these assumptions are varied to assess the impact any assumptions had on the recommendations from the study. The results are then presented accordingly to allow a decision maker to decide which assumptions seem most reasonable or relevant to their context. All analyses were performed using STATA (StataCorp LLC, College Station, TX).⁴⁰

RESULTS

The document analysis showed that the goals set by the college, UBC, and governing bodies of Canadian pharmacy education regarding TLT use were in agreement but general in nature. For example, the UBC Strategic Plan and CCAPP standards call for increased use of TLT; however, they offer minimal explanation of or guidance about what this means and/or how this expectation should be implemented or assessed.^3,28 The same perspectives emerge in documents authored by the college (eg, the October 2007 Faculty Review and November 2012 Self-Assessment Report).^29,31 Perhaps not surprisingly, more recent college-authored documents (eg, the 2012 Self-Assessment report and 2012-2017 Strategic Plan) strongly espouse that TLT will be crucial for increasing the prominence of the college on the world stage through technology-enhanced pedagogical innovation.^30,31 In these documents, educational innovation appears to be a key motivator for increasing the use of TLT.^28,30,31 Conversely, not all documents refer specifically to TLT as a method of innovating. For example, AFPC’s 2014-2016 Strategic Plan simply states that they seek to “inspire Faculties […] to develop innovative pharmacy education strategies and scholarship,”³³ implying that TLT use would inspire such educational innovation. Across these documents, however, there is consensus that TLT should be implemented in order to improve the quality of the entry-to-practice programs, whether by diversifying methods of teaching or by supporting the achievement of professional competencies.^28,30,31 Furthermore, support for TLT is steadily increasing. While the 2005-2009 Faculty Strategic Plan contains essentially no vision for the use of TLT, the 2012-2017 Faculty Strategic Plan is more explicit, attempting to outline why and how TLT should be used.^28,30 Overall, the goals for TLT within these documents are diffuse and gave the authors no specifics on how TLT should be integrated or the metrics by which implementation and impacts on teaching quality should be assessed.

A sample of 21 faculty members, highly representative of the faculty roster with regard to academic rank, role, disciplinary expertise, and teaching experience participated in the semi-structured interviews (Table 1). All participants had teaching roles and reported using TLT in their practice. Over 70% of participants claimed “intermediate to competent” proficiency with TLT, and the most common TLT used were presentation software and course management systems, prerecorded media, and audience response systems. Interestingly, much of the original TLT infrastructure in the new building (eg, Smart Boards [Smart Technologies Corporation, Calgary, Alberta, Canada], lecterns with a PC built in, and document cameras) were either underutilized or not mentioned.

Faculty members’ goals for TLT use in their curriculum and pedagogical practices were varied and diffuse although all felt growing pressure and/or encouragement to use more TLT. Most did not know if the college, university, or national governing bodies had specific goals for TLT use, so they relied instead on personal metrics and/or qualities to assess the value of TLT approaches in their teaching practices (Table 2). Student feedback (formal and informal), perceived improvements in student engagement, and enhanced student performance on assessments were most often cited as indicators of the value of TLT.

While all interviewees reported that TLT added value to their teaching and had a positive impact on student learning, a divergence of opinions emerged on many issues regarding TLT use. Ten participants, for example, discussed how time-consuming TLT use was (ie, learning how to use TLT and incorporate it effectively in the classroom, as well as creating appropriate support materials). In contrast, an equal number of participants discussed the time-saving aspects of TLT use, for example, using online examination software, which allowed for rapid grading of assessments, and in-class audience response systems (eg, PollEverywhere [PollEverywhere Inc., San Francisco, CA]) that supported quick, succinct, and anonymous ongoing formative evaluation of student understanding.

Participants agreed that TLT should be used for specific purposes, and should not be used “for technology’s sake.” Many felt that the use of TLT should be secondary, even inconsequential, to content that was adequately delivered through more traditional means by content experts. Also, many believed that the “fun” aspect of technology may in fact distract from student learning. One participant stated, “I would only incorporate more technology to solve a problem that I identify in my current approach to teaching.” Contrasting views suggested that the changing postsecondary climate and technology-native student demographic were reason enough to increase the use of TLT. These TLT proponents felt that content expertise was not adequate enough to maintain the college’s reputation, and that innovation in content delivery and teaching practice, facilitated in part by TLT, was a necessary part of teaching practices in pharmacy classrooms today.

Finally, the availability and quality of support for TLT use in the college was an key area of discussion. Those describing current inhouse TLT support as adequate felt they would benefit from more frequent training to enable better learning outcomes, such as knowledge retention by students. Of those that felt current TLT support was inadequate, six felt that the degree and quality of TLT use would be greatly improved by having TLT experts attend lectures or tutorials that recommend ways to incorporate TLT. There were also differences in opinion regarding who should be responsible for incorporating TLT into programs and teaching practices. Some believed faculty members should be responsible for innovation and incorporation of TLT, whereas others believed that TLT experts should instruct faculty members on how and where to incorporate TLT. All felt that in-house support should align with the college’s implicit aims to increase use of TLT.

Overall, opinions on TLT were predictably heterogenous given the diversity of the faculty members who participated. In addition, the general lack of specific goals and objectives for TLT, either by the college and/or faculty members, made it clear that both lacked well-developed organizing frameworks for the inclusion of TLT into their curriculum and pedagogical practices.

Beyond the qualitative assessment of the 21 faculty members that participated in interviews, 20 participants provided usable WTP values. One participant did not provide an upper limit on their WTP for TLT; therefore, their WTP was excluded from the primary analysis. For the primary analysis, the mean WTP value was estimated at Can$4,380,000 (95% CI Can$3,050,227 to Can$5,709,773), while the college’s mean cost estimate for TLT was Can$4,245,544 (Table 3). The mean WTP value was positively skewed, or inflated, by a small number of very high valuations, resulting in a mean value considerably higher than the median (median WTP, Can$3.65M). For the primary analysis, the WTP exceeded the cost of investment. This difference was not significant. The calculation of net benefits of TLT was positive Can$134,456 (95% CI: -Can$1,270,693 to Can$1,539,605), showing that the investment in TLT was in the realm of being cost beneficial (ie, indicative of WTP exceeding the cost of investment). A demand curve was derived from individual net benefit statistics (Figure 1). If the cost of investment was Can$0 (at the origin of the x-axis), as expected, 100% of individuals would be willing to pay that amount for the TLT in the college. As values increase across the x axis, the proportion of respondents that would be WTP decreases; between Can$3.25M and Can$4M, the proportion that would be WTP falls below 50%. The results indicate that 40% of participants would recommend the college should be willing to pay up to the value of Can$4.25M, the amount the college had actually invested in these technologies, as denoted by the vertical line in Figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Demand Curve Based on the Individual Levels of Willingness to Pay (WTP) of the 20 Respondents in our Primary Analysis. Vertical Dashed Line Indicates Actual Cost to the College

Including the participant who provided no upper-limit on their estimate of WTP by setting their value to Can$10M (the highest value on our cards) increased the mean WTP to Can$4,647,619 and the mean net benefit to Can$402,075 (95% CI: -Can$943,284 to Can$1,747,434). This still indicated that the amount the college had invested into TLT was not significantly different from the amount the participants were willing to pay (p=.56). Analysis of the demand curve suggested that by including this participant, the proportion of participants that recommended the college should be willing to pay up to the amount that the college had invested in technology for teaching and learning increased to 43%.

Results were most sensitive to the proportion of support staff time allocated to the cost of TLT; software and personnel costs were calculated with the assumption that they were being accrued when the building was first opened (ie, their maximum potential cost) (Table 4). By reducing the allocation of staff time from 100% in the base case to 50% or 25%, the net benefit increased from Can$134,456 (base case) to Can$1,220,286 (at 50%) and Can$1,763,200 (at 25%), respectively. The only scenario that produced a positive net benefit that was significant was when staff allocation was reduced to 25%.

DISCUSSION

This study sought to understand whether investment in TLT within a Faculty of Pharmaceutical Sciences in a large, research-intensive university was an efficient use of money. It is the first study to use CBA to value TLT investments in pharmacy education, and is unique in broader, higher-education contexts. While discussions of cost and value are often viewed as antithetical to the purpose of health professions education,¹⁵ our results suggest that the principles and practices used in welfare economics can be successfully applied to refocus efforts toward generating context-specific evidence for making informed monetary decisions about curriculum. Consistent with this, a recent study of the cost-effectiveness of active learning in a pharmacy school generated costs per additional unit of outcome (eg, the cost per student to increase the mean score of a cohort of students passing assessment), but concluded that whether this investment was truly cost effective could not be answered without an assessment of WTP.⁴¹ Our study provides an example of how WTP can be applied within a CBA framework.

The results from our primary analysis showed a small, positive net benefit (Can$134,456). However, because only eight of the 20 participants would be willing to be pay in excess of the estimated cost of TLT, we conclude that the cost and value of investment in TLT were approximately equal and, more specifically, that the college’s monetary investment in TLT was approximately equal to the social value placed on TLT by faculty users. There was considerable variation in the net benefit estimates as all participants provided positive TLT valuations; mean WTP estimates varied widely and appeared skewed by a small number of very large TLT values. The variability in WTP valuations led to confidence intervals for the base case that ranged from a negative net benefit of -Can$1.25M (where cost greatly exceeds value of investment) to a positive net benefit of Can$1.55M (indicating that value greatly exceeds cost of investment) (Table 4). The demand curve also emphasizes the inherent variability in the results, highlighting that, despite the close proximity of cost and WTP estimates, only 40% of participants would recommend the level of investment spent by the college on TLT (Figure 1). The college investments in TLT would have to be between Can$3.75M and Can$4M (or Can$250K-Can$500K lower than the original Can$4.25M cost) before 50% of faculty members would agree with existing TLT expenditures. Sensitivity analysis showed that only one scenario led to a significant net benefit of investment in TLT. Although these findings could be viewed as encouraging, in the absence of any clear college, institutional, or national goals for TLT use, drawing robust conclusions would be difficult.

There are a number of possible explanations for our finding that, in the absence of a clear institutional goal, the faculty members’ perceived value of the investment in TLT by the college appears to reflect good value for the money spent. One is that the faculty roster comprises excellent instructors who provide high-quality teaching and embrace contemporary uses of TLT in ways that are expected and intended.^1,2 However, results from the faculty interviews suggest there is a divide between TLT users and non-users; ie, many faculty members are still catching up to the tech-saavy student body.^5,6,9 A more germane concept for this study is the notion of efficiency of investment, a construct used extensively in economic analyses.^10,36 The full value of the investments in TLT may not be recognized yet. For example, the WTP estimates from our sample reflect the value of active use of TLT, not the value of full potential use. Considerable investment was made in lecture hall technology, such as document cameras, lecturn cameras, and smartboards. However, no users mentioned using these technologies in their teaching practice. This apparent underutilization implies that there is room to improve on the effectiveness of TLT use and efficiency. In other words, without investment in these technologies, the perceived value of TLT might have been unchanged. In the interviews, faculty members also suggested more frequent training as a way of improving the effectiveness of their TLT use. Thus, using support staff time to deliver faculty development opportunities on the effective use of TLT could have increased faculty members’ perceived value of TLT for the same level of investment.

Our study is also useful in highlighting the variety of metrics that faculty members consider when assessing value. While previous evaluations have focused on either cost or student performance on assessments or grades, we have been able to show that feedback from students, whether formal or informal, and student engagement in class are at least as important if not more important to our faculty members. A narrow focus on measurable outcomes, such as cost or mean grade point average,^14,18,41 can lead to an organizational focus on content and the mastery of content, rather than the move toward implementation and execution-based learning.⁶ Defining the value of investment using relevant outcomes, which may be improving teacher-learner communication, enhancing large lecture courses, and enabling more flexible learning can help direct investment towards achieving important institutional goals.¹³ More generally, the method of CBA used in our study could be applied to any investment decision in other post-secondary contexts.

The study had a number of limitations. First, it was set within a single college within a research-intensive university, which may limit the generizability of the findings to other settings. The college is, however, representative of 10 such Faculties in Canada, and one third of all faculty members across instructional and professorial ranks and across clinical practice and basic pharmaceutical sciences disciplines were sampled. Second, the study only considered the value of TLT from the perspective of faculty members; a comprehensive CBA should also consider value from the perspective of students. An area of priority for future research would be to repeat the study using PharmD students to understand the value of investment in TLT from the perspective of the “consumers” (ie, the students) whose learning and experience we are seeking to optimize. The WTP approach rests on hypothetical survey methods that can be prone to methodological bias. However, these methods have been validated for nearly six decades, and the findings regularly inform public policy action.²⁰ Furthemore, the use of these methods are increasingly common. Several assumptions were made in this study in estimating the cost and value of investment in TLT that could have affected the estimates of net benefit. The primary analysis used the most conservative of these assumptions, and we tested the impact of these assumptions one at a time (known as one-way sensitivity analyses). A more detailed method of sensitivity analysis, eg, probabilistic sensitivity analyses, which allows multiple inputs to vary simultaneously, could have been applied. Finally, it is also plausible that the WTP responses were biased upwards, particularly for longer-standing faculty members, as overall expenditure of the college on the new building may have been known to them. However, there was no knowledge of how this money was apportioned to specific investment on TLT within the building.

CONCLUSION

To date, economic evaluations in postsecondary education contexts have been narrowly focused on ROI-type studies.^6,13 In an age of increasing accountability and finite resources, financial decisions are made whereby any amount expended in one area represents an opportunity cost in another. While discussion of cost, value, and commerce are often viewed as antithetical to the purpose of health professions education,¹⁵ more focus should be placed on value for money with the explicit intent of generating context-specific evidence for making informed resource allocation decisions about curriculum. There has been an increasing volume of work into the scholarhip of teaching, learning, and educational leadership in recent years; however, the evaluation of the efficiency of investments has not been seen in education, despite proliferating in other sectors including health, transportation, and the environment. This study attempted to fill this gap by demonstrating that the methods used in economic studies can be applied successfully to educational settings. It showed that the college’s monetary investment in TLT was approximately equal to the social value placed on TLT by faculty users and provided an example of methods to bring greater understanding to curriculum and pedagogical practices and decision-making.

Converging educational and health economic research practices have helped identify areas for continuous quality improvement in pharmacy programs. Providing greater clarity about the goals and objectives of investments such as TLT, consideration of factors like what metrics faculty members use to assess success of an innovation, and supporting change with strategic decisions about the need for faculty development would help maximize the value of investments to support the optimal delivery of curriculum.