Paired Comparison Method

What the paired comparison method is

The paired comparison method (طريقة المقارنة الزوجية) is a way of appraising employees in which each employee is compared with every colleague in a separate and independent comparison of two, and the whole group is then ranked by the number of times each person was preferred.

The unit of judgement in the paired comparison method is a pair, not an individual. The appraiser is not asked “how many marks out of ten does this employee deserve?” but “which of these two?”. That is the source of the method’s strength, but it is also the source of its limit.

How the paired comparison method is carried out

1. Set one declared criterion on which the comparisons are made, such as the quality of output or punctuality, and write it down before any pair is judged. A criterion that has not been written down can shift from one pair to the next, and the appraiser may not notice having shifted it.
2. Form every possible pair within a single group.
3. Judge each pair separately, without regard to the other pairs.
4. Count, for each employee, the number of times they were preferred. It sets their position in the ranking.

Two further rules can be written down at the same stage, alongside the criterion: how a tie will be handled, and, where more than one appraiser takes part, how their results will be combined. The worked example below shows that neither situation is remote, even in a group of five.

How the number of comparisons grows in the paired comparison method

The number of pairs in a group of n employees is n × (n minus 1) ÷ 2. That arithmetic can change the mind of anyone who assumes the method suits a group of any size:

• 5 employees: 10 comparisons.
• 10 employees: 45 comparisons.
• 20 employees: 190 comparisons.
• 50 employees: 1,225 comparisons.

Where there is more than one criterion, these figures are multiplied by the number of criteria, because each criterion needs a separate matrix. An appraiser facing hundreds of comparisons in succession can begin to fall back on a general impression, and at that point the method has lost the very thing it was chosen for.

So the count comes first: anyone considering the paired comparison method can calculate the number of comparisons for their group and their criteria before the first pair is formed. In the sources we reviewed, we found no published limit on group size for the paired comparison method on which to base a recommendation. The formula shows how quickly the burden of comparisons grows as the group grows, but it does not mark a size beyond which the method should not be used.

A paired comparison matrix worked for five employees

Take a group of five people who perform the same tasks: Ahmed, Badr, Khalid, Saad and Fahd. There is a single declared criterion: the quality of output. By the formula above, the number of pairs is 5 × 4 ÷ 2 = 10 comparisons. Here they are with their results:

1. Ahmed and Badr: Ahmed preferred.
2. Ahmed and Khalid: Khalid preferred.
3. Ahmed and Saad: Ahmed preferred.
4. Ahmed and Fahd: Ahmed preferred.
5. Badr and Khalid: Badr preferred.
6. Badr and Saad: Badr preferred.
7. Badr and Fahd: Badr preferred.
8. Khalid and Saad: Khalid preferred.
9. Khalid and Fahd: Khalid preferred.
10. Saad and Fahd: Saad preferred.

The number of times each person was preferred is then added up: Ahmed 3 (over Badr, Saad and Fahd), Badr 3 (over Khalid, Saad and Fahd), Khalid 3 (over Ahmed, Saad and Fahd), Saad 1 (over Fahd) and Fahd 0. The points total 3 + 3 + 3 + 1 + 0 = 10, which equals the number of comparisons, because each comparison awards one point and no more. That total is the first thing to check: if it differs from the number of pairs, a pair has either not been judged or been judged twice.

What a paired comparison matrix shows, and what it does not

The ranking that was wanted has not been produced. Three people stand on 3 points and the matrix cannot separate them, and the reason is neither closeness in performance nor an error in the arithmetic. Look at the three judgements between these three names, in comparisons 1, 5 and 2: Ahmed was preferred to Badr, Badr was preferred to Khalid, and Khalid was preferred to Ahmed. The loop is closed.

Breaking the tie with a rule that puts first whoever beat their direct opponent leads back into the same loop, because each of the three beat one of the other two and lost to the remaining one. Breaking it with a fresh comparison among the three only repeats judgements already made, and if the second result differs from the first it does not overturn it, because the method has nothing with which to prefer one result over the other.

The loop is not a fault in the arithmetic. It follows directly from the structure of the paired comparison method: each pair is judged separately, without regard to the other pairs, which is what the method is praised for, and that same feature allows judgements to accumulate that are inconsistent with one another. The appraiser does not sense this while working, having never seen the three judgements together at a single moment.

What, then, can be done? The honest answer is to record the result as it stands. The group produced three people in one position, one after them and one at the end, and the tie is documented together with its loop rather than hidden behind a preference that has no basis. Anyone who needs to separate the three needs something outside this matrix: a second criterion with a separate matrix, or a measurement against a declared level outside the group, such as the performance standards written for a job.

In the sources we reviewed, we found no rule for breaking a tie, or for combining the matrices of several appraisers, that we could present as the preferred one. The rules described here are set out with their arithmetic effect, not as recommendations.

What the paired comparison method is not

Three concepts can be confused with the paired comparison method, and in each case the difference lies in the question being asked, not in the name:

• Forced ranking against set proportions. It is a constraint on the shape of the output: how many employees each band can hold is fixed in advance, and the ranked list is then cut at those proportions. The paired comparison method is one of the ways in which that list may be built, so the relationship between the two is one of layers, not of alternatives. Put more precisely, the paired comparison method answers “how do we arrive at a ranked list?”, while the proportional constraint answers “what do we do with the list once we have it?”. Dropping the paired comparison method does not remove the constraint, and adopting it does not impose one.
• Rating scales, on which each employee is given a score on a declared scale, as on a graphic rating scale. They measure the employee against a level outside the group, so a whole team can be rated high or rated low. The paired comparison method cannot do that in any circumstances, because its only point of reference is the rest of the group.
• Straight ranking, in which a manager writes the names in order from highest to lowest in a single sitting. The output resembles that of the paired comparison method, but the route is different: straight ranking asks for one overall judgement, while the paired comparison method breaks that judgement into documented judgements of two, each of which can be reviewed separately. The benefit of the method lies in that reviewability, not in its accuracy.

In the sources we reviewed, we found no measurement on which to base a claim about the effect of the paired comparison method on appraisal accuracy compared with other appraisal methods.

What the paired comparison method does not produce

• The size of the gap. A ranking says that the first person comes before the second; it does not say how far ahead.
• The level of performance. A group that is weak throughout still produces someone in first place, and a group that is strong throughout still produces someone in last place. The ranking is internal and is not measured against any standard outside it.
• The reason for the judgement. The comparison produces a result but no feedback: the employee learns their position but not what would change it.

The paired comparison method is a ranking tool, and a ranking is not a judgement of competence. For that reason its result is not handed over as a final appraisal. It is one input, examined alongside others, and a decision built on it alone is built on a ranking without a measure.

When the paired comparison method fits

The paired comparison method fits a small group with similar tasks, a single declared criterion, and a need for a ranking with a limited purpose: choosing a candidate for an assignment, or allocating a limited number of opportunities. It does not fit where the jobs within the group differ, because the question “which of these two?” then becomes a question about two jobs rather than about two employees.

There is a further case in which it does not fit even when the tasks are similar: when the group includes someone the appraiser has not worked with for long enough. A comparison of two leaves no room for the answer “I don’t know”, and an appraiser obliged to judge a pair they know nothing about can end up judging it on something else, such as seniority, how near someone sits, or an impression passed on by others. That point then enters the total with nothing to distinguish it from a point based on observation.

So the question to ask before the pairs are handed out is whether this appraiser has seen the work of everyone in the group. If the answer is no, any pair whose two members the appraiser does not know is passed to an appraiser who knows them both, or left out, and leaving it out is more honest than a point with no basis.

The paired comparison method with more than one appraiser

The matrix above is a single matrix for a single appraiser. If two appraisers rate the same group, there are two matrices and twenty judgements rather than ten, and a declared rule for combining the two matrices is needed before the appraisers begin, not after they disagree. One simple rule is to add up each employee’s points from both matrices. All the points then total twenty, and the most any member of a group of five can reach is 8 points.

That addition hides something worth seeing. A pair on which the two appraisers disagreed gives each of its members one point, exactly as a pair on which they agreed the two were level would, if the rule allowed a tie. The result is the same, but the two cases are not: the first is a disagreement in judgement, and the second is an agreement that the two are close. Reading the combined matrix correctly means counting the disputed pairs separately, because their number shows whether the criterion has been understood in one way or in two.

Where the paired comparison method sits among an organisation’s documents

Performance reports are a chapter of the model work regulation (النموذج الموحّد), the model in accordance with which organisations prepare their work regulation (لائحة تنظيم العمل). So the appraisal method an organisation chooses is examined alongside what it has adopted in its work regulation, not as an isolated administrative arrangement.

The distinction that carries consequences is the one between the tool and its result. The matrix is an internal worksheet; what needs a written basis in the adopted document is any effect on the worker’s rights that is built on it. Anyone who intends an employee’s position in an internal ranking to become the reason for an effect on that employee should first ask where that effect is written down, before asking how accurate the matrix is.

In the sources we reviewed, we found no provision dealing with the paired comparison method specifically. So we do not write that the paired comparison method is required, and we do not write that it is prohibited.

This is an explanation of the concept, not legal advice.

Related terms

• Graphic Rating Scale
• Performance Calibration
• Factor Comparison Method
• Central Tendency Error
• Peer Appraisal