SAVE | EMAIL | PRINT | MOST POPULAR |  RSS | | REPRINT
|
50 Years for Four Levels
October 22, 2009
By Lorri Freifeld
Half a century after Don Kirkpatrick created his Four Levels of Evaluation, they are still a guiding force in training. And training is a pretty big force within the Kirkpatrick family: Don's son, Jim, worked as a career consultant and training director at a bank before being bitten by the Four Level bug and now co-authors books and co-hosts conference sessions on the subject with his dad. Don's daughter, Sue, has been in sales training for Northwest Airlines for years. Jim's wife, Wendy, created the Kirkpatrick Goal Achievement Model, and brother Ted is an entrepreneur with a burgeoning interest in training. With a son and nephew now in college and showing signs of catching the training bug, Jim says, "it's kind of like succession planning."
"I was always afraid my father would ask me what each level was," Jim admits. "I didn't even know the order of the levels. I had been in careers where Four Levels weren't really relevant. Then one day a friend of mine was hosting an event with Dad. I went there to see Dad—and to get a free lunch. I started writing things on a napkin and realized this stuff really was relevant and useful in my work as a training director at a bank. Eventually, I decided I wanted to help Dad spread the word about the Four Levels."
Don adds, "I was so patient with Jim. 'Get on the ball, Jim!' I would think. 'Expand those levels. I expect you to be a good chip off the old block.'"
Jim, indeed, has picked up the torch and continued Don's Four Levels work; in fact, his next book, "Kirkpatrick Then and Now: A Strong Foundation for the Future," is due on shelves this fall. And you can download the free Kirkpatrick 50th anniversary white paper on a fresh look at the Kirkpatrick model. Just click here.
In celebration of the golden anniversary, Training magazine checked in with the father of the Four Levels and his son to talk about the model and how training professionals should be using it today—and tomorrow.
Training: Why do you suppose the Four Levels are still alive and strong after 50 years?
Don Kirkpatrick: Because, according to a quote from a Ford Motor Company trainer who attended one of my sessions, "The Four Levels are practical and easy to understand and have replaced the ambiguous word, 'evaluation,' which had many different meanings."
Training: You have been training and speaking about the Four Levels for decades. How is it that you still enjoy your speaking engagements and workshops?
DK: I love to teach, and the Lord has given me a special gift in doing it. I like to see people learn, and I like to see them enjoy the workshops I teach.
An illustration is my conversation with a trainer at Dana Corporation, who hired me to teach a one-day workshop on managing change. Before the program, he called and asked me if I were going to use the "SMART" approach. I was somewhat familiar with it but not sure what each letter stood for, so I asked him to remind me. He did.
At that moment, I thought I would embarrass him a little as he did me, so I said, "No, Bill, I am going to use the 'PIE' approach." I had made it up at that moment.
There was silence for a few seconds on the other end of the line. Finally he said, "I beg your pardon. What approach did you say you were going to use?"
So, again I said, "The PIE approach." I waited a beat or two, then said, "I trust you know what the PIE approach is."
And he said, "No, I never heard of it."
So I explained that it stood for "Practical, Interesting, and Enjoyable." And that has been my approach to teaching with a slight modification. My son, Jim, suggested I change the "I" to Interactive because Interesting is similar to Enjoyable. So I did, and both of us use this approach.
Training: What are your pet peeves about training today?
DK: Jack Phillips has been a good friend and a promoter of the Four Levels by calling his pet project, ROI, "Level 5." But I have always listed ROI under Level 4. And I don't like the fact that he has changed the wording of Level 3 from "Behavior" to "Application" and Level 4 from "Results" to "Impact." This has caused confusion.
Jim Kirkpatrick: My pet peeve is the use of the word "learning" in the training journals. Everything is about "Learning 2020." That's why training is on trial today—because too many training professionals believe their job is done when people learn. We need to extend our role and not pat ourselves on the back when learning has occurred. CEOs are looking for performance results, not just learning.
Training: What are the biggest challenges organizations face today in successfully using the Kirkpatrick Four Levels? What are some of the solutions?
DK: First, everyone must learn the Four Levels and the guidelines. Second, training professionals must get managers on board. People won't apply the Four Levels on the job unless managers are on board. Plus, you need managers to evaluate. Third, identify the people who will judge the effectiveness of the training program—I call them the "jury" —find out what their expectations are, and get them on board. If you want people to accept what you are doing, you must give them a sense of ownership, and you must communicate results.
JK: The first barrier that comes to my mind is the long history of thinking the Four Levels start with Level 1. In tandem with that is the notion that organizational change and business value begin with training. I believe the trainer-centric approach to our industry has led us to believe we are the end, not a means to the end.
Because learning professionals typically start with Kirkpatrick's Level 1 (Reaction), they soon can get to Level 2 (Learning) but seldom beyond. The main reason for this is we have tended to believe our responsibilities end when participants leave the classroom, are satisfied, and have learned. In our industry, notice how much emphasis there is on Levels 1 and 2—learning objectives, skills gap, learning and development departments, training departments, generational learning styles, etc.
I find the group that offers the biggest objection to extending our roles beyond the classroom and the computer are the learning professionals themselves. Evidently there is a lot at stake in maintaining our role as learning leaders as opposed to strategic business partners.
A related challenge is that business leaders also see our role as designers, developers, and deliverers of training programs. "Can you put together a program for us on change management?" "Sure!" And when we attempt to cross over to the business to show them the merits and value of a true business partnership, we often are met with, "Go back to where you belong."
The solution? Find ways to make a strong business case to fellow training professionals, business executives, managers, supervisors, support staff (IT and HR), clients, and prospects for what is known as the Kirkpatrick Business Partnership Model. In a nutshell, it points out that training in and of itself does little to create a positive Return on Expectations (ROE), and that a variety of additional interventions and actions are required to produce significant ROE for our business partners and clients.
I can think of at least four ways to make a strong case:
1. Through a customized impact study on one of your flagship programs.
2. Presenting the research data that show training effectiveness comes from a variety of factors we can positively influence.
3. Benchmarking other organizations that successfully have implemented the full Kirkpatrick Business Partnership Model (starting with Level 4 and working backward).
4. Lively discussions centered on the testimonies of satisfied and engaged employees.
One solution to the challenge is the creation of an advisory or learning council. This is a committee of 10 to 15 people—both in training and various business functions (i.e., IT, HR, marketing, etc.) —who look at how to execute company strategy through training. Clarion Healthcare and Georgia Pacific have good advisory board committees that have helped create an understanding in the company of what training is all about.
Generating that feeling of ownership Don mentioned is the biggest challenge. It's not the science of the Four Levels; it's the art of getting people—managers, executives, and even staff—to buy into it.
Training: So how do you get that buy-in?
DK: When I was running a training program at the Management Institute in Madison, WI, I asked participants what would happen when they went back to their jobs after the program. Most of them had no idea. They didn't know if their boss would talk to them about the experience and ask about new ideas the program sparked, or just tell them to get back to work. So I wrote a booklet about preparing managers for their employees to participate in this program, including "here's why I'm approving you for this program, here's what I'm going to ask you when you get back, here's what I expect you to apply when you return, etc." This was to get managers on board before the program and keep them involved afterward.
JK: I would show management the strong evidence that training, in and of itself, will not give them the ROE they are expecting. I would show them that while it is clear that training—at Levels 1 and 2—is the foundation for ultimate impact, those of us in the training department (or consulting world) will not stop there, but make a great effort to equip all other business partners to be able to perform their roles as strategic learning partners. I would never purposely diminish the role of training, but would point out that going forward, the old definition needs to be put aside for the new, expanded definition and role of effectively working into the business at Levels 3 and 4.
I also would have a case study or two in my back pocket to show how other companies have done it successfully.
In the end, the evidence doesn't always have to be numbers. Sometimes the strongest evidence is the testimony from people who have been through it and can show how it works.
DK: That said, some think all you have to do is get testimonials. But they're just an addition to the metrics. They add to the impact of the program.
JK: It's just like a real trial when the lawyer brings in the "weeping widow" to bring richness to the evidence.
Training: Once an organization has mastered all Four Levels, how does it achieve consistent success year after year, training program after training program?
JK: I think the keys are to continue to sit at the business table during discussions of strategy development and execution, and to continue to drive and fully evaluate the most strategic programs. Keep the pressure on the drivers and critical behaviors of all programs. Keep communicating. Keep your team visible—an active part of what happens at Levels 3 and 4. And improve your dashboard and scorecards so you continually are enhancing the management and communication of your business partnership efforts and successes.
Our model was designed to be sustainable. Evaluation data and information are not gathered only to demonstrate our value at the end, but work in a feedback loop to ensure effectiveness and efficiencies are maintained throughout. It promotes continuous improvement by answering the all-important question along the way, "If not, why not?" If metrics or indicators anywhere along the way fall below standards, training leaders quickly will be able to find out what the cause(s) is, and intervene to get things back on track.
DK: I think the key is keeping in constant contact with the "jury," especially as the members of the jury continually change, and make sure you continually ask, "What can training help us accomplish?" It's a variation on my belief that every comment sheet should end with the question: "How can this program be improved?"
JK: Yes, communicating with the jury is crucial. I suggest doing so through dashboards to reassure them the results are coming and keep them apprised of how things are going, what help or coaching you need, and any other developments. That way, there are no surprises.
Training: Most organizations use the Kirkpatrick Four Levels to measure the impact of training after it is complete. But you advocate making them a part of every phase of a training initiative, from beginning to end. Can you give an example of how to do that?
JK: Sure. Let's say an executive in your organization asks for a new program on leadership. Instead of conducting a cursory needs assessment, and putting together and delivering a program, start with the concept we call "the end is the beginning." You must begin with a clear picture of what a successful leadership program ultimately will bring to the organization in terms of Level 4. This means you first negotiate your stakeholders' expectations so they are satisfying to them and achievable for you, then you convert those expectations to observable, measurable success outcomes by using various forms of the question, "What will success look like?" (Level 4 Results). Then you determine the critical new leadership behaviors training grads will have to perform on the job (Level 3, Behavior) to bring about those success outcomes.
Also crucial at this stage is to determine, implement, and monitor the required drivers of those behaviors—the processes and systems that will serve to get grads to perform those critical behaviors on the job. Next, it is important to identify and reinforce pre-training necessities for success, for example, a culture of support and accountability, coaching guides, and job aids. Then, and only then, you develop and deliver the training that will produce competent employees (Level 2, Learning, and Level 1, Reaction) who will be able to perform the targeted, critical behaviors. In other words, start with Level 4 and work backward.
It is also important to monitor not only Levels 1 and 2, but the required drivers and critical behaviors at Level 3, and the outcomes at Level 4. This will allow for a good system to make sure learning to behavior transfer occurs, and also provide you with a compelling chain of evidence at the end to demonstrate the impact of the initiative.
Training: So organizations that have mastered Levels 1, 2, and 3 but are having trouble getting to Level 4 are having trouble because they didn't start with Level 4?
JK: Correct. I would tell them to pick one of their successful Level 1 to Level 3 programs, and recreate what the expectations were, what success outcomes were, and how those hopefully were impacted by the good work done at Level 3. This would involve them identifying which business and human resource "needles" ultimately moved. If it were hard to do that, I would say start fresh with a new program, work it Level 4 to Level 1, demonstrate it at those levels, and use that to stick their flag in the ground.
DK: We replaced the concept of ROI, Return on Investment, with ROE, Return on Expectations. To do this and measure Level 4, Results, we suggest trainers ask members of the jury what they expect to be accomplished by conducting training programs. To be specific, probably the most common training program is on leadership. Many times it is suggested and/or urged by the jury. Therefore, they must have something in mind to be accomplished.
So ask them, "What do you expect us to accomplish by conducting a program on leadership?" Some of the answers might be ambiguous such as, "Better leaders." Then follow up by asking, "What would that look like? Can you be more specific?" They then might answer with objectives such as "more participation by employees, higher morale, and less turnover."
By asking, discussing, and even negotiating, trainers can get a good idea of what the program should accomplish and agree that the expectations are realistic. From this, trainers can develop a program to accomplish those expectations.
The byproduct of such discussions will give the jury a feeling of "ownership," which is important to the program's success.
Training: Does the trend of training moving from classroom instruction to more social networking and collaborative delivery methods have any impact on the Four Levels? Why or why not? If yes, what is the impact?
JK: Yes, and I think it is a terrific impact. For decades, we have been locked into the notion that instructor-based training was the holy grail, followed by e-learning. Until recently, both still represented the old guard of "we train you." Now with the onset of many types of social networking opportunities and an improvement in e-learning methods, there can be a healthy balance of responsibility—on us for training and on our participants for learning. I am excited about a future where there is more responsibility on the learner for learning, performing, and contributing. My colleague, Erica Fox, from Google describes their culture as being very much that way. And a culture that emphasizes personal responsibility seems to have a better chance at creating employee ownership, engagement, commitment, and increased productivity and retention.
Informal learning and social networking modes have, of course, become the rage. That is a good thing, particularly when that self-guided learning is well connected to the foundational knowledge and skills previously taught through more traditional means. They do a great job of extending learning, and the influence of the training industry, into the world of the worker.
These systems also promote the truth that learning is a process, not an event. They only will help by extending learning beyond the classroom and into the workplace. Of course, it will be our job to corral and connect all of these possibilities to maximize learning, performance, and impact.
Training: What possible changes in the training industry do you see coming in the next 5 to 10 years that might impact the Four Levels? How would you change the model in response?
DK: I see no possible changes in the training industry that will impact the basics of the Four Levels. There have been several additions in implementing them, such as testimonials and focus groups. And there will be more modifications. But I believe the Four Levels are here forever. The changes will be in implementing them.
Through LinkedIn, I asked the question: "After 50 years since the Four Levels were introduced, are they out of date?"
Some 30-plus trainers responded almost unanimously, "NO!" The problem is not with the Four Levels as such; it is with implementation, especially at Level 3, Behavior, and Level 4, Results.
JK: I honestly think the cat is out of the bag, and business leaders will demand that training needs to be more business oriented. There will be no more free passes. We talk about it a lot and may kid ourselves into thinking we are truly business partners, but I think over the next months and years we actually will become strong in both delivering and demonstrating value. I think we will make good on the demands from our "corporate juries." I also believe the Four Levels and similar models will lead the charge.
I think there will be more emphasis on the hope of the "quick fix" that some believe social networking and e-learning can bring. We must stand firm on the practice of incorporating these and other ways to augment training, but not to replace the tried-and-true.
I already see a stronger emphasis on Level 3. The only way to get from training to results is through the "missing link" —Level 3. Instead of both training and the business treating it like it is owned by the other, there will become stronger joint ownership and a great increase in ultimate business impact.
Training: Jim, you have a new book coming out— "Training on Trial." Why did you write it and what is it about?
JK: Don used the words, "…before the day of reckoning arrives" in an article he wrote in 1959—referring to the need for training managers to demonstrate the value of their programs to business leaders. We believe these words largely have gone unheeded over the last five decades. This book is about bad news—that we are being charged by business executives and clients with "costs exceeding value." It is also about good news—never before has the learning industry had such an opportunity to show the power of what can be done if the old definition of training is discarded for the new business partnership approach. The rest of the book is about how to execute it.
Training: Jim, how are you and your wife, Wendy, moving the Four Levels beyond training?
JK: We find ourselves talking more about working with business leaders and how we can help them leverage their training departments more effectively. We believe the best way to get everyone working together is to attack it from both ends. This interview is obviously for training professionals, but things might move along faster if we can help to educate business leaders—the people who make strategic decisions about how their businesses are managed, leveraged, and funded. We are working in a few areas to see if we can help to make that happen. Wendy has taken the Kirkpatrick principles and has created what she calls the Kirkpatrick Goal Achievement Model, which shows individuals how to set and attain professional, career, or personal goals by using the Four Levels—again, starting with the end as the beginning with Level 4.
Finally, we hope to work more in areas of faith-based organizations and other not-for-profits.
Training: Don, how do you want the training industry to view your work over the last 50 years?
DK: Just the way it has recognized me, simply as the originator of the "Four Levels." I give credit to the Lord for introducing a concept that has helped many people and stood the test of time. I also would like to be remembered for receiving the awards for Lifetime Achievement in Workplace Learning and Performance from ASTD in 2003 and Special Lifetime Achievement from the Asia HRD Congress in 2007. Finally, I would like to be remembered for my son, Jim, who followed in my footsteps and took the Four Levels to new heights.
For more information, visit www.smr-usa.com or contact Jim Kirkpatrick at jdkphd50@msn.com or 317.402.3882. To download the free Kirkpatrick 50th anniversary white paper on a fresh look at the Kirkpatrick model, click here.
|
SAVE | EMAIL | PRINT | MOST POPULAR | RSS |
|
|
| Back to Training Index |
|
|
