By Jeanie McKay
NOVA Quality Communications
As professional process "improvers" and data analysts, we spend our weeks reviewing manufacturing processes, determining plans for measurement, charting and making decisions based upon what our data shows us. But the funny thing is, when it comes to applying the same systematic methodology to our own training, we fall flat.
You know the routine. The exhausting training day winds to a close, and trainees gather their materials before bolting for the door. We humbly pass out our evaluations and remind everyone that we have kids in college. The trainees rapidly circle numbers or smiley faces and hit the road. We swipe erasers over the board, slide the chairs under the tables, wipe up the left-over donut crumbs and dart our curious eyes over to the stack of evaluations to see how well we did.
As our eyes quickly scan the columns, we let out a sign of relief as we see many circled "10s." And when we encounter an occasional low-end mark on someone’s rating sheet, we quickly do a mental roll call to figure out who the class sorehead was this time around.
Funny thing... the evaluations somehow tell us how "we" did...and not "them." After all, our management grades us on these subjective indicators!
What exactly are we evaluating?
When it comes to evaluating the true impact of the training upon employee learning and performance change, we often fail to analyze the right stuff or even collect the right feedback at the right time. At best, evaluation has been a perfunctory task with little planning and even less usefulness. Yet evaluating the true impact of this costly venture is paramount to quantifying the return-on-investment.
Just as it takes planning and skill to analyze our manufacturing processes effectively, knowing what to measure and when to measure in terms of training is equally complex. What are we actuality trying to discover through evaluation? Are we attempting to determine:
Training Evaluation is Multi-Faceted
It is impossible to evaluate or isolate all above results with one simple end-of-course survey. In order to logically analyze training, we have to look at four distinct levels of evaluation.
Level 1: Reaction Evaluation—How does the learner feel about the training?
Evaluation at this level generally measures what trainees liked about a particular training program at the close of the course or workshop. Perceptive instructors usually are able to somewhat monitor the general reaction during the class. The important thing to remember is that this in no way measures what learning took place.
Level 2: Learning Evaluation—What knowledge can be recalled? What skills can be demonstrated?
Learning is defined as the "principles, facts and techniques that were understood and absorbed by participants." Obviously this is much more difficult to measure than reactions. In order to adequately measure what was learned in training, there must be an entry pre-test.
Level 3: Transfer-of-learning Evaluation—What new skills or information is the learner using on the job after training?
Because it is difficult to develop a transferred knowledge measurement tool, much training is delivered without any plan whatsoever for measuring this training component. This check must occur in the workplace at a pre-determined period following training.
Level 4: Overall Impact Evaluation—What results were achieved? How did the training benefit the organization?
Management is most concerned with training evaluation on a broad-scale organizational level. This evaluation must somehow determine the results and whether it was worth the investment. This evaluation is often the determining factor for continuing or terminating training efforts.
What would help us to design effective evaluations at each level?
Level 1: Reaction Evaluation—How does the learner feel about the training?
Evaluation at this level generally measures what trainees liked about a particular training program at the close of the course or workshop. Perceptive instructors usually are able to somewhat monitor the general reaction during the class. The important thing to remember is that this in no way measures what learning took place.
Level 2: Learning Evaluation—What knowledge can be recalled? What skills can be demonstrated?
Learning is defined as the "principles, facts and techniques that were understood and absorbed by participants." Obviously this is much more difficult to measure than reactions. In order to adequately measure what was learned in training, there must be an entry pre-test.
Level 3: Transfer-of-learning Evaluation—What new skills or information is the learner using on the job after training?
Because it is difficult to develop a transferred knowledge measurement tool, much training is delivered without any plan whatsoever for measuring this training component. This check must occur in the workplace at a pre-determined period following training.
Level 4: Overall Impact Evaluation—What results were achieved? How did the training benefit the organization?
Management is most concerned with training evaluation on a broad-scale organizational level. This evaluation must somehow determine the results and whether it was worth the investment. This evaluation is often the determining factor for continuing or terminating training efforts.
What would help us to design effective evaluations at each level?
Level One
Evaluating reaction is a good measure of "customer satisfaction." But we have to be aware of two factors that may bias these kinds of evaluations:
The expectation that training must be entertaining
Because reaction sheets measure how the participants felt about the
training, the trainer may tend to emphasize participant enjoyment rather than substantive
information.
Because trainer ratings are a major influencing factor in the rewards that a trainer receives from his or her management, these circumstances can often lead to a conflict of interest.
The perception that learning is passive
There is a commonly held belief that it is the trainer’s responsibility to ensure that participants learn. Measuring how well this responsibility has been met with a reaction sheet is problematic. Typical reaction-evaluations ask questions about the trainer’s performance and questions about the course design—not questions about the participant’s efforts to learn. Keep in mind that it is ultimately the responsibility of the trainer to provide information and the opportunities to learn. It is the responsibility of the participant, not the trainer, to process that information.
Level TwoTo evaluate learning, each participant should be measured by quantitative means. A pre-test and post-test should be administered so that any learning that takes place gets attributed to the training program. Without a baseline for comparison of the as-is, you will never be able to reveal exactly how much knowledge has been obtained.
Although we first think of paper-and-pencil tests for measurement, simulations and demonstrations can also be used to measure knowledge before and after. If set-up correctly, these activities can closely approximate the participants’ work environment and can help them better relate to the learning.
Below are four ideas to help you design a learning measurement tool:
Remember that there are many complex factors that influence transferability back to the job. These factors may include the manager, the culture, the environment, peer pressure, and the infrastructure’s readiness to accommodate change. Many studies have shown that incentives to performance may be wiped out when the environment does not reward or reinforce what was learned during training.
For skills to be transferred successfully from the classroom back to the job, certain features must be designed into the training. (Stay tuned! This topic will be covered in next month’s feature article.)
Below are some suggestions for evaluating training programs in terms of
behavioral change:
Level Four
Attempting to evaluate organizational impact is not for the faint-hearted! There are many complicating factors that make it extremely difficult to isolate and evaluate certain kinds of programs in terms of long-term results.
When we attempt to assess organizational value, we must be careful to compare apples with apples. In order to do this, we must have an up-front agreement with management about what the results should and will be at the point of evaluation. Management must understand what training can reasonably do to transfer knowledge and teach skills and some possibly change some attitudes along the way. Management must also be coached to understand what performance gap closures lie beyond the scope of training.
Effective training should provide value back to the organization; however, quantifying those gains with direct costs is difficult. But without clarification of goals and a comprehensive needs analysis prior to training, this will be even harder. A thorough needs analysis should answer the question, "What good WILL training do?" A thorough training evaluation will answer the question, "What good DID training do?"
To conduct a Level Four evaluation you need to:
Before attempting to determine the worth of training, it may be wise to determine and clarify what benefits the organization perceives the changed on-the-job will produce. Some examples are shown in the table below:
| Organizational Benefits | Indicators |
| Increased outputs |
|
| Time savings |
|
| Improved quality |
|
As you can see, there is no easy cookbook approach to evaluating training. To properly evaluate our efforts, we are required to think through the purpose of the training, the purpose of the evaluation, the audience for the results of the evaluation, and the points and spans at which measurement will be taken. Evaluation methods must be carefully planned and measurement tools must be designed. Consider using a training evaluation tool such as the example below as you begin to better analyze your the effectiveness of your training:
Stage |
Tasks |
Time-frame |
Focus |
|
|
Plan |
|
|
Implement |
|
|
Evaluate the Evaluation |
|
Is this really worth the effort?
"Why are we being asked to measure? Who uses this data anyway? I barely have time to get my work done as it is!"
Sound familiar? We improvement junkies rely on measurements to mitigate risks, determine andmaintain improvement. We frequently hear the whines from those from whom we require process/product measurements for other efforts. Yet, the training process demands no less effort. Without this information and feedback, training effectiveness remains another company overhead expense that is difficult to justify or improve.
ŠJeanie McKay,
2001
NOVA Quality Communications
Proceed to Our Articles Library
NOVA
Quality Communications
60280 Sunset View Drive
Bend, OR 97702
541-382-8785