Evaluation

Relative Merits of The Various Evaluation Levels 
Narayan van de Graaff, Managing Director, Advanced HR Solutions

Level 1: Reaction
This is by far the most frequent type of training evaluation. Participant feedback can be useful, particularly if specific training is going to be frequently repeated. If participants are dissatisfied with the training, they will be less likely to apply key learnings in the workplace. The converse also applies. Some words of warning:

• Asking just open-ended questions can make analysis that much harder, and simply having a generic evaluation questionnaire (one size fits all) which asks people to tick the boxes, will usually not yield valuable information. Try to find a balance between the two.

• All too often participants are given an evaluation questionnaire minutes before the finish time, when they're all keen to get out of the workshop. Allow enough time for thoughtful feedback.

• As an alternative to evaluation questionnaires, consider some focus groups conducted by a third party without a vested interest in the outcomes. Contrast this with one management consultant I knew, who asked participants to be kind with him in their feedback, because of possible implications for him!

• Try not to skew evaluation questionnaires with leading questions, or questions that are either all positively or negatively worded.

• Recognise that participants have a tendency to over-rate - facilitators shouldn't let their egos get too carried away by positive feedback!

• Positive reactions to training are certainly no guarantee that there will be any sustained transfer of the learnings to the workplace.

Level 2:Learning
One approach in Level 2 is to evaluate the knowledge and skills of participants in specific areas at the outset prior to the training, and then evaluate the same skills and knowledge after the training - if the workshop effectively covered those areas, one would expect a demonstrable improvement in knowledge.

This can be useful. However, bear in mind that increased knowledge just after the workshop is often not sustained, nor does it necessarily translate into enhanced workplace behaviours or improved business results. It can also be a challenge to do this properly. An evaluation questionnaire that asks participants to subjectively rate their own knowledge and skills before a workshop, and again after the workshop, may have very low validity.

Level 3:Behaviour
This evaluation level looks at the extent to which participants behave differently in the workplace as a consequence of the training. The extent to which participants will behave differently after training is influenced by various factors, such as:

	The opportunities provided to practice their new-found knowledge and skills
	Their positive or negative feelings about the training and what it achieved (or didn't!)
	The support and constructive feedback provided by their manager and other key people both before and after the training
	The extent to which the training gave them the opportunity to reinforce and practice the key knowledge and skills
	The degree to which there are punishments or rewards related to (non) practice of the desired behaviours after the training.

It is important to defer assessment of the post-training behaviour until well after the training is completed (at least three months) - this will help gauge the extent to which there is sustained improvement.

Level 4: Results/Impact
Evaluation Levels 4 and 5 are clearly the ones of most interest to senior management. Participants may have liked the training, they may have shown improved knowledge and skills since the workshop, and are even behaving differently, but how has this impacted on business performance? What are the KPIs showing? And how can we be sure that any improvement in KPIs are the result of the training?" Relevant questions indeed!

I carried out a series of performance management workshops for all managers in a large council some time ago. As part of my initial discussions, I asked the General Manager what he saw as a successful training outcome. Without hesitation, he said, "That the managers carry out the performance appraisals in a timely manner." Timely completion of performance appraisals isn't the only relevant KPI by any means. However, the General Manager was clearly a key stakeholder in the workshop outcomes, and we therefore tracked this KPI (as well as others) both before and some months after the training.

What are some of the key indicators linked to Level 4 evaluation? A few are listed below:

	Customer satisfaction		Income
	Customer complaints		Costs
	Customer retention		Overtime
	Staff satisfaction		Productivity
	Number of staff grievances		Budget variances
	Work stoppages		Projects within budget/on time
	Workplace accidents		Staff turnover
	Performance appraisal ratings		Sick leave/Staff absenteeism

The challenge in evaluating the impact of training on results is to ensure that changes are due to the training rather than other factors. One way to do this is to have an experimental group (which does the training) and a control group (which doesn't do the training), and then compare the two groups some months after the training in terms of relevant key indicators.

Level 5: ROI
We have already briefly discussed ROI. Much relevant material is available on the web. According to Phillips, we need to heed the following points when seeking to measure ROI:

1. Keep the process simple

2. Use sampling for ROI calculations

3. Always account for the influence of other factors

4. Involve management in the process

5. Educate the management team

6. Communicate results carefully

7. Plan for ROI calculations

8. Give credit to participants and managers.

We have already highlighted the fact that only 10 - 20% of training is effective (in other words, that it leads to sustained workplace improvement). Elsewhere, we have also indicated how you can help ensure the participants in your organisation's workshops to become part of the 20%. The further good news is that effective training can often yield very high ROIs. Phillips and others have provided some well-documented workplace examples that demonstrated high ROI, sometimes even exceeding 1000%!

<<PREV PAGE The Real Effectiveness of Training

We can help you to more effectively evaluate
the effectiveness of your training and HR interventions.
Call 0438 792 300 or alternatively email us
at training@advanced hr.com.au.