Leniency and Strictness Error

What Leniency and Strictness Error is

Leniency and Strictness Error (خطأ التساهل والتشدد) is a rater placing the ratings of everyone they assess at the top of the scale, which is leniency, or all of them at the bottom, which is strictness. The level of the ratings then says more about the rater’s own standard than about the team’s performance. This is our wording of the concept, not a quotation from any one source.

Its mechanism is a shift in the ratings, not a compression of them: the whole team moves to one end of the scale, and the order among its members holds. The top employee under a strict rater stays the top of that team, yet their rating reads as weak once it is set next to a rating from another team. The definition describes a pattern in a distribution. It is not a verdict on any particular rater or on any single rating.

Which sources did we check?

We checked the bibliographic data of these articles in Crossref, which confirms that they exist and gives their details, not their content. We left out the identifiers that contain a hyphen.

• Saal, Downey and Lahey. The article “Rating the Ratings: Assessing the Psychometric Quality of Rating Data” in Psychological Bulletin, volume 88, issue 2, 1980, pages 413 to 428.
• Landy and Farr. The article “Performance Rating” in Psychological Bulletin, volume 87, issue 1, 1980, pages 72 to 107.
• Bernardin and Buckley. The article “Strategies in Rater Training” in Academy of Management Review, volume 6, issue 2, 1981, pages 205 to 212, identifier 10.5465/amr.1981.4287782.
• Woehr and Huffcutt. The article “Rater Training for Performance Appraisal: A Quantitative Review” in Journal of Occupational and Organizational Psychology, volume 67, issue 3, 1994, pages 189 to 205.
• Kane, Bernardin, Villanova and Peyrefitte. The article “Stability of Rater Leniency: Three Studies” in Academy of Management Journal, volume 38, issue 4, 1995, pages 1036 to 1051, identifier 10.2307/256619.
• Jawahar and Williams. The article subtitled “The Performance Appraisal Purpose Effect” in Personnel Psychology, volume 50, issue 4, 1997, pages 905 to 925.
• Bernardin, Cooke and Villanova. The article “Conscientiousness and Agreeableness as Predictors of Rating Leniency” in Journal of Applied Psychology, volume 85, issue 2, 2000, pages 232 to 236.

We read the abstracts of Saal and colleagues, of Bernardin and Buckley, of Woehr and Huffcutt, of Jawahar and Williams, and of Bernardin and colleagues as OpenAlex displays them, and we read the text of none of these articles. Of the article by Kane and colleagues, only the opening of its abstract was available to us. OpenAlex holds no abstract for the article by Landy and Farr, so we go no further than its title.

How does leniency or strictness show in a distribution of ratings?

Not in a single rating. An outstanding employee deserves the top rating, and giving it to them is not leniency. The entry on Rater Bias lists leniency or severity among the patterns of bias, describing one rater who gives high ratings across the board and another who gives low ratings across the board, so that a comparison between their two teams no longer means anything. The same entry states that comparing the average rating given by each rater exposes the lenient rater and the severe one.

These are the signs by which we read the pattern, and they are ours, not taken from a source:

• An average far from those of colleagues: A rater working on the same form, across a comparable set of roles, whose average sits clearly higher or lower than the others.
• One end missing for one rater: Nobody at the bottom of the scale under the lenient rater, and nobody at the top under the strict one, while differences between team members remain.
• A gap that repeats across cycles: The same rater at the same end year after year, while the people being rated change.

We do not offer the third sign as a rule. The opening of the abstract of Kane and colleagues (1995) reports that research has counted leniency among the most troublesome rating errors, and states that little is known about how stable it is as a tendency in a rater. We did not see the rest of the abstract or the results of the three studies.

How does Leniency and Strictness Error differ from neighbouring appraisal errors?

Compressing the range and shifting it are separated in the account given under Central Tendency Error: compression pulls the ratings towards the middle, so the difference between people is lost, while a shift moves all the ratings upward or all of them downward, and the spread between them may stay wide. The same account says that a shift is corrected against an external reference the rater can be compared to, and that compression is corrected by widening what the rater is able to support with evidence.

Unlike the rating in Contrast Error in Appraisal, which by its entry’s account changes when the order of the files changes and nothing else does, the lean of a lenient or strict rater holds whatever order the files come in. This distinction is our inference from the two definitions.

Leniency and Strictness Error also parts from First Impression Error in where the rating comes from. That entry describes an early judgement about a person which settles, after which the evidence that follows is interpreted through it, so its locus is one particular employee. The locus of leniency and strictness is the rater, and their effect reaches everyone that rater assesses.

Leniency and Strictness Error is not to be confused with what is known as the Halo and Horn Effect, whose entry defines the halo side as a single positive impression of a person carrying over into the assessment of all their other qualities. The halo travels from quality to quality within one employee’s form, and leniency travels from employee to employee within one rater’s team. This pairing is our wording.

What do study abstracts report about the causes of leniency?

The abstract of Jawahar and Williams (1997) reports that Taylor and Wherry hypothesised in 1951 that ratings collected for administrative purposes, such as a pay rise or a promotion, are more lenient than ratings collected for research, for feedback or for employee development. The abstract states that a meta analysis of 22 studies with a combined sample of 57,775 supported the hypothesis, with administrative ratings on average one third of a standard deviation higher than research or development ratings. It adds that the difference was larger in field studies, with practising managers, and with real subordinates.

The abstract of Bernardin, Cooke and Villanova (2000) reports that the study covered 111 students rating one another, that scores on agreeableness were associated with higher ratings given, with a coefficient of 0.33, and that scores on conscientiousness were associated with lower ratings, with a negative coefficient of 0.37. The sample is students in peer assessment, so we do not carry the result over to managers rating employees.

The abstract of Saal and colleagues (1980) describes research on rating errors as having suffered from confusion in its definitions, its measurement and its methods, and documents this in more than 20 articles published between 1975 and 1977. In our reading, that means “leniency” in one study is not necessarily what another study measures under the same name.

A worked hypothetical example: two raters, two teams

Suppose an organisation in which two raters each assess a team of five employees on a scale of 1 to 5, with the same form and comparable roles:

• The first team: Ratings of 5, 5, 4, 4 and 3, which sum to 21, so the mean is 21 ÷ 5 = 4.2.
• The second team: Ratings of 3, 3, 2, 2 and 1, which sum to 11, so the mean is 11 ÷ 5 = 2.2.

The difference between the two means, 4.2 minus 2.2, is two full points. The range is the same in both teams: from 5 to 3 in the first and from 3 to 1 in the second, two steps in each. Neither distribution is compressed; both are shifted. The mean of all ten together is the sum of 21 and 11 divided by 10, that is 32 ÷ 10 = 3.2.

If the organisation required a rating of 4 or higher for a promotion shortlist, 4 of the 10 employees would be shortlisted, all of them from the first team. The rating of 3 is the lowest in the first team and the highest in the second. Measured by distance from each rater’s mean, a 3 in the first team sits 4.2 minus 3, or 1.2 points, under its mean, and a 3 in the second team sits 3 minus 2.2, or 0.8 points, over its mean.

That calculation assumes the two teams are equal in actual performance, and nothing in the table establishes that. The figures are hypothetical, set out to show how the table is read, and they represent no organisation and no study. A gap between two means opens a question about the raters, and on its own it is not enough to change any rating.

What narrows the effect of Leniency and Strictness Error?

Differences between raters’ averages are dealt with in a session known as Performance Calibration, in which the raters meet before the ratings are confirmed and agree through shared examples what each rating means. Its entry states that the session makes the lenient rater and the strict one visible, because comparing their averages with those of others reveals them, and warns that levelling distributions by force until they match produces an apparent fairness that penalises the team that genuinely performed better.

Among the tools that hold the ruler itself steady is the scale defined in the entry on Behaviourally Anchored Rating Scale (BARS) as one on which each point is described by a specific behaviour that can be observed at work, rather than by a general word such as “good” or “poor”. In our estimate, a behavioural description narrows the space that a rater’s own standard fills, and we found no source that measures this.

Setting a shared reference differs from the rule of fixed proportions described under Forced Ranking, which according to its entry obliges an organisation to distribute its employees across performance categories in proportions fixed in advance, whatever the results turn out to be. Imposed proportions erase the shift from the table, and with it they erase the real difference between a strong team and a weaker one, in our estimate.

On training, the abstract of Bernardin and Buckley (1981) reports that there is no evidence that training raters to change their rating distributions improves accuracy or validity, and it proposes three alternatives: keeping diaries to strengthen observation, a shared frame of reference among raters for what effective performance means, and training that builds a rater’s confidence in situations calling for a negative appraisal. The abstract of Woehr and Huffcutt (1994) describes a quantitative review of four training strategies and four measures, leniency among them. That abstract does not give its results, so we attribute no effect to it.

What does the evidence allow us to say?

We do not state that any particular average is the correct one for a team, nor a threshold past which high ratings become leniency or under which low ratings become strictness. All we have are abstracts of some of the articles, the opening of an abstract for another, and the title of a third. The figures from Jawahar and Williams and from Bernardin and colleagues are reported from their two abstracts as given, with their samples and settings.

In our sources we found no Saudi study we could rely on about leniency or strictness in performance appraisal. Crossref lists an article by Alharbi titled “Criteria for Performance Appraisal in Saudi Arabia, and Employees Interpretation of These Criteria” in International Journal of Business and Management, volume 13, issue 9, 2018, starting on page 106, and its abstract describes a qualitative study of ten employees in public bodies in the city of Tabuk. The abstract does not deal with leniency or strictness, so we build no description of organisations in the Kingdom on it.

The error remains a question about the ruler before it is a question about the rating: do two raters mean the same level of performance by the same rating? The answer does not come from one rater’s table. It comes from setting the raters’ tables side by side, then asking what lies behind each difference before correcting it.

This is an explanation of the concept, not legal advice.