Kirkpatrick Model

What the Kirkpatrick Model is

The Kirkpatrick Model (نموذج كيركباتريك) is a framework for evaluating a training programme on four graded levels. Each level asks a question further from the programme than the one before it: did the participant react favourably, did the participant learn, did the participant change behaviour at work, and did a measurable work result change. The framework is named after its author, Kirkpatrick. It is a methodological tool for an establishment and not a provision of the Saudi Labor Law (نظام العمل), so the establishment chooses which levels it measures and how.

The order matters. An establishment that stops at the first level learns whether participants liked the programme and does not learn whether it helped. The financial return of a programme is calculated separately, as training ROI explains. That calculation follows level 4 and does not replace it.

The four levels

• Level 1: reaction. The question is how the participant saw the programme, and whether the participant found it relevant to the job. The evidence is a questionnaire completed straight after the programme. It records satisfaction and does not show benefit: a participant may be satisfied with a programme that changed nothing in the work.
• Level 2: learning. The question is what knowledge or skill the participant gained. The evidence is a test before and after the programme, or a practical exercise that is assessed. This level needs a baseline, because a score after the programme does not show what the programme added.
• Level 3: behaviour. The question is whether the participant applies in daily work what was learned. The evidence is a manager’s observation, or a follow-up checklist, some time after the participant returns to work, because behaviour does not show on the day of the programme.
• Level 4: results. The question is whether a named work indicator changed because of the programme. The evidence comes from an indicator that was measured before the programme, such as the error rate, the time taken to complete a task, or the share of transactions that have to be repeated.

The interval between the levels is part of the design. Levels 1 and 2 are measured at the end of the programme or shortly after it. Levels 3 and 4 are measured after a period long enough for an effect to appear. The establishment fixes that period before the programme starts.

A worked example: a programme for 20 customer service employees

A programme on handling calls was attended by 20 employees. Before the programme began, the establishment had measured the rate at which customers called again about the same problem. All the figures below are assumed, to show the arithmetic.

• Level 1. The average rating given by participants was 4.4 out of 5.
• Level 2. The average test score was 55 before the programme and 78 after it. The increase is 23 points (78 minus 55), which is 41.8% of the first score (23 ÷ 55).
• Level 3. After 90 days, managers observed that 14 of the 20 participants applied the five steps of the call, which is 70% (14 ÷ 20). In a comparison group of 20 employees who had not attended, six applied them, which is 30% (6 ÷ 20). The difference is 40 points (70 minus 30).
• Level 4. The rate of repeat calls in the group that attended fell from 18% to 14%, a fall of four points. In the comparison group it fell from 18% to 17%, a fall of one point. The net effect attributed to the programme is three points (four minus one).

All four levels point to a positive effect. At level 4, however, the effect is three points and not four, because the group that did not attend improved by one point without the programme. An establishment that had stopped at level 1 would have reported satisfaction of 4.4 out of 5 and would not have learned that the effect was three points.

Consider a second programme. Participants rated it 4.8 out of 5 at level 1. At level 2 the average test score was 60 before the programme and 63 after it, an increase of only three points (63 minus 60). The participants liked the programme, but they did not learn much from it.

Reading the gaps between levels

When two levels diverge, the gap helps to locate the problem. The three readings below are ways of interpreting the levels, and they are not rules of the model. They do not replace looking at the facts of each programme:

• Level 2 is high and level 3 is low. The participants learned and did not apply what they learned. The question moves from the quality of the programme to the work environment: whether the manager allows the participants to apply the learning, and whether the tools are available.
• Level 3 is high and level 4 is flat. Behaviour changed and the indicator did not move. The establishment reviews the choice of indicator, or asks whether the programme targets the behaviour that moves that indicator.
• Level 1 is low and level 2 is high. Learning took place although participants did not like the experience. This is not a failure.

What may be said after each level

Each level supports one sentence that does not go beyond its evidence:

• After level 1: participants found the programme relevant.
• After level 2: participants gained what they were tested on.
• After level 3: participants apply what they learned in their work.
• After level 4: the indicator moved by a stated amount, and the movement is attributed to the programme in a stated way.

An establishment that measured levels 1 and 4 and did not measure the levels between them has assumed that participants learned and that they applied what they learned. No measurement supports those two assumptions, so the report states them as assumptions. In the example above, the three points at level 4 are supported by levels 2 and 3 and by a comparison group that controls for the season and for conditions shared by both groups. The comparison group does not rule out that the difference arose by chance, and no statistical test is run on the assumed figures.

Who can collect the evidence

The model does not name who collects the evidence at each level. The following parties can do it:

• Level 1. The trainer or the delivery body can collect the evidence at the end of the programme.
• Level 2. The trainer can collect it from the two tests, or human resources can collect it if the test is the same across programmes.
• Level 3. The direct manager can collect it, because the manager sees the participant’s daily work.
• Level 4. The owner of the indicator in the department concerned can collect it, and human resources can help to assemble the data. HR analytics describes the use of employee data to answer management questions with numbers.

Each level needs a record that remains after the programme. A learning management system records the completion of learning paths. That is closer to participation data than to levels 3 and 4: level 3 needs the manager’s observation, and level 4 needs the work indicator from outside the system.

What the Saudi texts record of these levels

Article 44 of the Labor Law provides that a training programme includes the skill in which the worker trains, the rules and conditions of the training, its duration and number of hours, the theoretical and practical programmes, and the method of testing and the certificates granted.

Three provisions of the Implementing Regulations of the Labor Law bear on the same subject:

• Article 15 of the Implementing Regulations of the Labor Law. In implementation of Article 42 of the Labor Law, it requires every employer to evaluate the trainee at the end of the training or during its periods and to include the evaluation in the worker’s file at the establishment.
• Article 16 of the Implementing Regulations of the Labor Law. In implementation of Article 44 of the Labor Law, it requires every employer whose establishment is bound to provide training to document the training data of each worker. The data include the name of the programme, its type, its duration, the provider, and the mechanism and method of delivering the training. The documentation is due within 30 days of the date of the training, on the electronic platform that the Ministry specifies.
• Article 5 of the Implementing Regulations of the Labor Law. It lists a record of the training of Saudis among the required records. The record holds the name of the programme, its type, its duration, the year of training, the number and names of the workers trained, and their share of the workforce.

What Article 16 of the Implementing Regulations of the Labor Law says about the mechanism and method of delivery is discussed in blended learning. The two instruments use some of the same Article numbers, so each number needs its instrument: Article 15 of the Labor Law concerns the establishment data that are reported to the labour office and has no link to training, while Article 15 of the Implementing Regulations of the Labor Law is the provision that requires the trainee to be evaluated.

These provisions cover the data of the programme and its participants, and the evaluation of the trainee. The closest counterpart of trainee evaluation in the model is level 2, and that match is between two frameworks and is not a text in either. What the worker does after returning to work, and what the work produces, are levels 3 and 4, and they remain a decision for the establishment.

What spoils an evaluation

• Stopping at level 1. A questionnaire is quick and does not prove learning. Putting participant satisfaction into the benefit calculation produces a return based on an impression and not on a measured change, a point developed in training ROI.
• A test after the programme with no test before it. A score of 78 says nothing unless the score before the programme is available.
• Measuring behaviour before there has been a chance to apply it. An employee whose job has not allowed use of the skill shows no new behaviour, even if the employee learned the skill.
• Attributing the whole result to the programme. An indicator moves for other reasons. The establishment needs a comparison group, as in the example, or attendance in phases, which the example does not use.
• Setting the measure after the programme. The level 4 indicator is chosen before the programme. The question that comes before training is the subject of training needs analysis, and the stage at which evaluation sits in the design of a programme is the subject of the ADDIE model.

Does the Labor Law oblige an establishment to measure the effect of training?

The Labor Law does not oblige an establishment to use the four levels. Article 15 of the Implementing Regulations of the Labor Law requires the evaluation of the trainee and its inclusion in the trainee’s file. Article 16 of the Implementing Regulations of the Labor Law requires an establishment bound to provide training to document its training data. Article 5 of the Implementing Regulations of the Labor Law requires the record of the training of Saudis. The closest of these duties to level 2 is the evaluation of the trainee. An establishment that wants to measure behaviour and results does so by decision of the establishment.

In the provisions we reviewed, we found no mention of a worker’s behaviour after training or of work results. We also found no definition of the phrase «منشأته ملزمة بالتدريب» (an establishment bound to provide training), which opens Article 16 of the Implementing Regulations of the Labor Law, so we do not state which establishments these duties apply to.

This is an explanation of the concept and of the statutory provisions cited, not legal advice.