What Level of Evaluation?
Narayan van de Graaff, Managing Director, Advanced HR Solutions
Over thirty years ago, Donald Kirkpatrick pioneered modern thinking about training evaluation, by advocating a model that comprises four levels of evaluation. It is still used by many, and the evaluation levels he stated are as follows:
Level 1: Reaction. Feedback from participants on the perceived effectiveness of the workshop (content, facilitator, etc.) - typically through questionnaire, but also through focus groups, one-on-one interviews with outside persons, and informal feedback throughout the training.
Level 2:Learning. Evaluation of participant learning as a result of the training - e.g. by checking learning throughout the training; comparing control with experimental groups, and comparing pre- and post-training measures.
Level 3:Behaviour. Evaluation of the positive behaviour changes that occurred as a result of the training - typically through interviews with participants and their managers (or others), questionnaires, observation, and control versus experimental groups.
Level 4: Results/Impact. Evaluation of the impact of the training program on key organisational indicators, such as reduced costs, increased revenue, customer satisfaction and work output, - through comparing KPIs (key performance indicators), interviews with key people, questionnaires, etc.
Kirkpatrick did indicate the difficulties associated with level 4, and (perhaps because of this) focused mainly on the first three levels.
ROI: The Critical 5th Level.
Dr Jack Phillips (1997) uses the above four levels and adds ROI as the fifth, and 'ultimate' level of evaluation. He also highlights the fact that measuring ROI has in recent years become a critical issue at many HRD (Human Resource Development) and related conferences.
Relative Merits of The Different Levels
Phillips (and various others) identifies ROI as the ultimate level of evaluation, but also recognises that the other levels have their place. He does state that there is an inverse relationship between frequency of use/ease of assessment and the value of the information.
In other words, Reaction (Level 1) is easy to assess and very frequently used, but is seen as less valuable than the next levels, and in particular, ROI (Level 5), which is much more difficult to assess and much less frequently used.
Introduction to ROI
Many studies have shown that training returns are greatest when:
-
Senior management is committed
-
There is supervisory support
-
It is aligned with HR practices (e.g. recruitment, appraisal, pay)
-
It is closely linked to corporate strategies and work practices
-
The training is very specific, and linked to a well-identified need.
ROI is simply defined as the ratio between program net benefits and related costs. An easy example is as follows:
Total costs of program: $75,500
Total benefits (1st yr): $210,000
Net benefits = $210,000 - $75,500
= $134,500
ROI = $134,500/$75,500 = 178%
Phillips encourages us to only proceed with training initiatives if ROI is at least 25%. The above example is dramatically more than this, of course.
He also states that an effective ROI process must:
-
be simple
-
be flexible
-
be credible
-
be economical
-
be theoretically sound
-
include program costs
-
have a good track record
-
account for other factors
-
apply with all types of data.
-
be relevant to various programs
|
<<PREV PAGE The Real Effectiveness of Training
Merits of The Various Evaluation Levels NEXT PAGE>>
We can help you to more effectively evaluate
the effectiveness of your training and HR interventions.
Call 0438 792 300 or alternatively email us
at training@advanced hr.com.au.
|
