Rater Bias

What rater bias is

Rater bias, also called appraisal bias, is a systematic tendency in performance appraisal that makes a rating reflect something in the rater, or something in the way the appraisal is carried out, instead of reflecting the employee’s performance alone.

The source of the distortion can therefore sit in either of two places. It can sit in the person giving the rating: in what they remember, in what they would rather avoid, in whom they find easier to favour. It can also sit in the method: in a form or a procedure that pushes the rating in a particular direction whoever fills it in. In both cases the result is the same. The rating stops being a record of the employee’s performance by itself, because something that did not come from the employee has been added to it.

Patterns of rater bias

Rater bias takes several recognisable forms. Each one moves the rating away from the performance by a different route:

• The recency effect in appraisal. It gives the last months of the period more weight than the year as a whole, because those months are the nearest to memory. The rating then describes the end of the year while standing for all of it.
• Central tendency error. It gives everyone a middle rating in order to avoid the difficult conversation that a rating at either end of the scale would bring. The team ends up gathered in the middle of the scale, whatever the differences between its members.
• Leniency or severity. One rater gives high ratings across the board and another gives low ratings across the board. Once the two sets are placed side by side, a comparison between the two raters’ teams no longer means anything, because a rating from the first rater and the same rating from the second do not describe the same level of performance.
• Similarity. It favours whoever resembles the rater, in career path or in working style. The employee who has followed the route the rater followed, or who works the way the rater works, gains from that resemblance, and the gain does not come from what the employee delivered.
• The halo effect. It lets a single quality, positive or negative, colour the whole appraisal. One quality ends up setting the tone of the rating on every item, so the form shows a single impression carried across all the items instead of separate judgements about each one.

Rater bias, implicit bias and the halo effect

Implicit bias is a further source of rater bias, and it overlaps with some of the patterns above. Similarity is one of them: favouring someone who resembles the rater can come from a tendency of which the rater is not aware.

Implicit bias differs from the halo effect in where it originates. The halo effect is an exaggerated generalisation from an observation that is real: the rater saw something the employee actually did or actually is, and then let it spread too far across the rest of the form. Implicit bias is a judgement that comes before any observation, and it comes from the group to which the person belongs.

Why a warning does not remove rater bias

Telling raters that rater bias exists does not remove it, because it operates without the awareness of the person who holds it. A rater who has been warned still has the same memory, the same wish to avoid a hard conversation and the same inclinations. What has changed is what they know, not what shapes their rating.

What reduces rater bias is a change in the instrument and the procedure, not in intentions. The changes are these:

• Criteria written before the appraisal. They fix what is being rated before any rating is given, so the standard cannot be adjusted afterwards to fit an impression the rater already holds.
• Behavioural examples for each rating. They describe what performance at each level looks like, so that a rating points to something observable instead of to the rater’s private sense of what the level means.
• A record of observations kept throughout the year. It is written down as the year goes on, not recalled at its end, so the appraisal draws on what was noted at the time instead of on what memory happens to supply when the form is opened.

What reveals rater bias

Rater bias shows itself in the distribution, not in the individual case. A single rating, taken alone, cannot show whether it was biased, because there is nothing beside it against which to measure it. A set of ratings can.

Comparing the average rating given by each rater exposes the lenient rater and the severe one: a rater whose average sits above the others, or below them, stands out once the averages are set next to each other. Comparing the distribution of ratings across teams exposes central tendency: a team whose ratings are all gathered in the middle of the scale looks different from a team whose ratings spread across it.

This is the work of a performance calibration session. The raters meet over shared examples before the ratings are confirmed, and the meeting adjusts the ruler, not the ratings.

When a difference between raters is not rater bias

Not every difference between two raters is rater bias. One rater’s team may really be stronger, or its tasks may really be harder. A higher average from one rater can describe a team that performed better, and a lower average can describe a team that faced more difficult work.

The error opposite to rater bias is to assume it. Forcing the distributions into line until they match produces a fairness that exists only on the surface, and it penalises the good team: its members lose ratings they earned so that their distribution looks like everyone else’s.

So the examination of a difference begins with a question, not with a correction. A gap between two raters is a reason to ask what lies behind it, and the answer may be rater bias or may be a real difference in the teams or in the work.

Rater bias in self assessment

Rater bias is not confined to the manager. Self assessment, in which employees rate their own performance, is also subject to what distorts a rating, although there the source is the incentive rather than an unconscious tendency. An employee rating their own work has a stake in the result, and it is that stake, rather than a tendency below awareness, that can move the rating.

The incentive is one of the reasons for preferring to use self assessment as an input to the conversation between employee and manager, and not as a figure that enters the calculation of the final rating.

This is an explanation of the concept, not legal advice.

Related terms

• Performance Calibration
• Contrast Error in Appraisal
• First Impression Error
• Peer Appraisal
• Subordinate Appraisal