SAVE | EMAIL | PRINT | MOST POPULAR |  RSS | | REPRINT
|
Maybe ROI Really Is a Waste of Time
June 09, 2009
By Saul Carliner
Although members of the Return on Investment (ROI) Industrial Complex stress the urgency of "proving" the value of training, the empirical evidence suggests otherwise. In fact, most training managers are not even asked to provide return on investment. Yes—you read this correctly. In a study by my research team (published in Carol Hansen and Yih-Teen Lee's 2009 book, "The Cultural Context of Human Resource Development," 58 percent of training managers responded they were not expected by their sponsors (upper management) to report on the effectiveness of their work, and 69 percent said they were not expected to report productivity.
A 2005 ASTD study of C-level executives, including chief learning officers in large corporations, suggests why: The C-level executives "placed greater emphasis on perceptions as measures" and expressed concern about "the challenge of measurement" for financial and other performance measures.
Part of the challenge is methodological. Even those who have proposed methodologies for measuring ROI of individual courses acknowledge serious flaws with them. One of the most significant flaws is accounting. Because much training is intended to reduce errors, such as scrap, the associated expenses are never incurred. But accounting systems—the ones ultimately used to track ROI—can only handle expenses that are tracked on the books. Any resulting savings from reduced scrap is, at best, a speculation and causes credibility issues with financial experts. Furthermore, many evaluation experts acknowledge that, even when performance improves, other factors—such as new machinery that produces fewer flawed products—could have contributed to the resulting savings.
Other challenges are practical. Except for the largest training departments, few have staff available to conduct the evaluation required, much less people skilled in analyzing and reporting the resulting data. Many training managers work in small departments (10 people or fewer) that do not have staff to conduct a full, four-level course-by-course evaluation. Nor do they have the skills on staff to evaluate and report that data. Many also report to senior managers who do not know how to assess training.
Even when senior management knows how to assess training, ignorance often is preferred. Is that because, as researchers Elwood Holton and Sharon Naquin discovered, "there are many organizations that have been incredibly naïve in spending large amounts of money, or are [they] using different decision-making processes"?
Do You Really Want to Know?
Consider the evaluation of learning. Although many texts stress the importance of assessing learning as part of a complete evaluation of a learning program, most organizations really don't want to formally collect and track that information. In fact, only 38 percent of organizations do so. That's because if, at the end of a learning program, the organization conducts a test and then links personnel decisions to learners' performance on that test without validating it (as few organizations do), the organization risks litigation from an employee who misses out on a promotion or similar opportunity. The employee can sue the organization, charging that the test was biased. It stands to reason that if organizations do not want to know whether learning occurred, they also are comfortable not knowing specifically what the return on investment is.
But sometimes, senior management is not interested in the evaluation because training isn't a significant part of their portfolio. Responsible for a successful, fee-based training program that generated more than $1 million annually, a training manager interviewed for my study observed that the vice president to whom she reported was pleased about the success of the program. But that meant little to him because "he had a $50 million quota," and her program barely represented 2 percent of it. So whatever happened with her revenue quota (positively or negatively) had little impact on his.
Perhaps therein lies the real issue: Despite the combined $50 billion to $60 billion per year spent on workplace training each year by organizations in the U.S., that's only a small part of these organizations' expenditures. The exact percentage varies by organization, but, according to ASTD, organizations spend between 1 and 3 percent of payroll expenses on training. In contrast, organizations spend as much as 20 percent of expenses on marketing. If an organization needs to cut expenses, cutting out all training would have less impact than cutting marketing by 10 percent.
That doesn't mean organizations lack interest in seeing a return on their investment in training. But the effort of showing how each course returned ROI provides little ROI—and that's why senior organizational managers don't actively seek it as much as some of the trade press might have us believe.
Assessment Perception
As suggested by the C-level study mentioned earlier, perceptions play a significant role in assessing the training function. Indeed, the survey by my research team found that word of mouth was the most common measure of effectiveness that training managers felt was important to their sponsor, ranking higher than ROI and course evaluations.
That perception is not always positive. In her 2006 study of perceptions of HRD, Randhir Auluck found that "the overall ranking of the status of the HRD/training profession, when compared with that of other organizational functions and occupations, is worryingly low." Perhaps the real issue is that human resources is dismissed by others as "touchy-feely" as InBev CEO Carlos Brito did in comments to a group of MBA students at Stanford.
Besides, if trainers need to demonstrate that their training investment provides a return, empirical evidence suggests that figures that are more credible and easier to get can do the job. For example, Laurie Bassi and her research team found a direct correlation between overall investments in training and human resources, and the longer-term financial performance of organizations. These figures both can be found on an organization's financial records; they do not need to be derived through an additional ROI study. Other studies have drawn conclusions similar to Bassi's.
So why the focus on evaluating the ROI of individual courses if the return on training might be more easily and credibly assessed as a whole (that is, on the basis of the entire investment in training rather than the investment in an individual course)? That's partly because our industry tends to think in terms of individual courses. For example, look at most publications on designing and evaluating training. Almost all focus on designing and evaluating courses. Curricula (collections of courses) and departments (collections of curricula) receive little, if any, attention. Furthermore, most of the competitions for outstanding instructional product focus only on individual courses, not broader curricula.
But it's also partly explained by who's writing and speaking about ROI. My research found that articles on the topic in training trade publications are overwhelmingly written by consultants. Because consultants work outside of the organization, and often on a single course for their clients at a time, their focus is going to be on an individual course or project. And because they are removed from the daily pressures of the small, internal training departments where the majority of training professionals work, these consultants have devised ever-sophisticated ROI methods that are increasingly out of touch with the needs and resources of resource-stretched training departments.
In short, perhaps trainers should stop trying to collect ROI on individual courses because information about the ROI of individual courses requires resources that are rarely available to collect; does not provide a credible measure; and, most significantly, senior managers are not asking for it.
Furthermore, because perceptions of training seem to have more credibility—and are more challenged among senior managers—and because other financial data is both more credible and provides more established evidence of the actual return on training, perhaps trainers instead should focus efforts on tracking and managing perceptions and following the broader measures that provide insights into the effectiveness of an organization's total investment in training rather than investments in individual courses.
Saul Carliner is an associate professor with the Graduate Program in Educational Technology at Concordia University, a training consultant, and author of six books on training and e-learning, including the recent “E-Learning Handbook: Past Promises, Present Challenges,” co-edited with Patti Shank (Pfeiffer).
|
SAVE | EMAIL | PRINT | MOST POPULAR | RSS |
|
|
| Back to Training Index |
|
|
