Content Validity

What content validity is

Content validity is the degree to which the content of a test represents the content of the job: its questions and tasks form a sample of the domain performed in the role, rather than a set chosen on no particular basis.

It is the first of the three routes to evidence set out under test validity. What sets it apart from the other routes is its method. It rests on a judgement about the content itself; it does not wait for an outcome to arrive, and it does not compute a relationship between two scores.

Content validity starts by defining the job domain

Representation cannot be assessed until it is known what is being represented. The domain, for content validity, is what the job actually involves: its tasks, its knowledge and its decisions, each carrying the weight it carries in the work.

That domain comes from job analysis. A job analysis describes the job as it is, not as it ought to be, and that is exactly what a test needs, because a test built on an imagined job represents something that does not exist.

Weight is part of representation, not an addition to it. A task that takes a tenth of the working time and carries half the marks on the test gives an inverted picture of the work.

The table of specifications behind content validity

So that talk of representation does not stay at the level of impression, a table is drawn up before a single question is written. Its rows are the task groups as they emerged from the analysis. Its columns give, for each group, its weight in the job, the number of items that correspond to it, and the marks those items carry.

The value of the table is that it turns a question answered by opinion into one answered by comparison. Someone shown a written test and asked whether it represents the work answers with what they suppose. Someone given a table with the weight of every group beside its share of the marks answers with what they can see.

Content validity in a worked example: a test that looks sound but is inverted

Suppose the analysis of an accounts payable accountant role produced five task groups, weighted by the time spent on each: matching invoices to purchase orders 40%, preparing supplier payments 25%, reconciling supplier statements 20%, answering supplier queries 10%, and preparing ageing reports 5%.

A test of 20 equally marked items was then written, and its items fell as follows: four on matching, two on payments, two on reconciliation, six on queries and six on reports.

The arithmetic shows the problem. Matching has 4 items out of 20, a 20% share of the test, against a 40% weight in the work: a shortfall of 20 percentage points. Reports have 6 items out of 20, a 30% share, against a 5% weight: an excess of 25 percentage points. The two tasks that occupy 65% of the role’s time, matching and payments, are covered by six items out of 20 between them.

The result is that the task performed least in the role received a share of the test as large as any other. A candidate weak at matching and strong at reports finishes ahead of a candidate with the opposite profile, and the test did not contain a single wrong question.

Correcting the table while keeping 20 items gives eight for matching, five for payments, four for reconciliation, two for queries and one for reports. That split matches the weights exactly: 8, 5, 4, 2 and 1 out of 20 are 40%, 25%, 20%, 10% and 5%. It will not work out so neatly in every table. Had the analysis produced a weight of 33% for one group, no whole number of items out of 20 would have matched it, and at that point what is called for is a reasoned rounding with its reason written down, not arithmetical precision.

The weights and counts in this example are assumed in order to show the calculation. They do not describe an actual job, and they are not a recommended distribution for any test.

Weighting by time and weighting by consequence in content validity

The example carries a qualification that should not be dropped. Time is the easiest measure of weight, but it is not the weight in every case.

A task performed once a month that takes an hour may carry consequences, if done wrongly, that a full month of daily tasks does not: approving a large payment, closing a period, or sending a statement to an outside body. A task that takes a third of the working time and follows the same steps every day may leave little room for error once it has been learned.

The domain is therefore examined from two angles together: how much time is spent on each part, and what happens if that part is done wrongly. Representation breaks down when a rare task with heavy consequences is left out because its share of time is small, and the test is still described as representing the work by its weights.

This is settled in the table, not in the questions. Beside the time weight sits a column for the consequence of error, and any decision to raise a group’s share above its share of time is justified by what that second column records. What matters is that the adjustment is written and reasoned, rather than made in the course of writing items and leaving no trace.

Who judges content validity

The judgement behind content validity is a reasoned human judgement, and it is not extracted from data. The people qualified to make it know the job from practice: its current holder, the line manager, and whoever receives the role’s output in another department.

Each item is put to the same question: is this something performed in this job, and with this degree of importance? Once every item has been reviewed, the opposite question follows: what is performed in the job and does not appear in the test?

Judges work separately before they meet. If they meet first, the first to speak can draw the rest towards a view, and what the quieter judges would have said is lost. If each judges alone and the judgements are then compiled, the points of disagreement come to the surface, and those points are the useful output of this step.

A disagreement is addressed, not put to a vote. An item that the jobholder sees as central and the manager sees as marginal may not reflect a difference of assessment at all. It can be a sign that the role is performed differently from how it is understood from above. The question to ask then is not which of the two is right but what is actually happening in the role.

The conclusions are kept in writing: who judged, when, and on which version of the test. Someone who changes an item a year later cannot tell whether the judgement still stands without a record to compare against.

In the sources we reviewed, we found no minimum number of judges for a content validity judgement, no rate of agreement between judges at which an item counts as representative, and no numerical index of how far a test represents the job. We do not reproduce figures on these points that circulate outside those sources.

Excess and missing content in content validity

Two errors mirror each other:

• Excess content. An item that measures something the role does not require. It excludes candidates on a point they do not need, and it raises the cost of the test for no return.
• Missing content. A core part of the work that the test does not touch. It is harder to see, because the test looks sound in everything it contains, and the fault lies in what it leaves out.

Reviewing the items one by one exposes the first error and not the second. Missing content only shows when the test, taken as a whole, is set against the output of the job analysis.

Excess content can enter a test because it is easy to mark, because it survived from an earlier test written for another job, or because its author happens to be good at it. Missing content can arise when an aspect is dropped because it is hard to measure, such as working with incomplete data or handling conflicting requests, both of which can arise in the role.

Why content validity can be settled before anyone is hired

Content validity does not have to wait for results. A judgement on content is possible on the day the questions are written, before a single candidate sits the test.

That is what makes it the practical route for a small organisation, and for any job filled by a handful of people a year. It requires no sample sizes, no waiting period and no later data. What it does require is an honest analysis of the job and a judgement from people who know it. The same reasoning lets a work sample test make its case without cohorts or a waiting period.

It also has an effect on cost that is worth stating. An error of representation is found before anyone has been excluded by it. Whoever reviews the table and finds a task group with no items corrects a sheet of paper. Whoever waits for the results makes the same correction after decisions have already been made on papers that did not represent the role.

When content validity has to be judged again

Representation is a property of a relationship between two things that both change, so it lapses when either of them changes. The judgement is revisited when any of four events occurs:

• The tasks of the role, or their weights, change.
• A new work tool removes a step or adds one.
• The test items are amended, even if only by deletion.
• Time has passed since the last analysis of the job.

Of the four, the new tool is the one that can pass unnoticed. A task that used to be done by hand and is now completed in a single step in a system no longer carries weight in the work, yet the test still gives it items because that is what it weighed when the test was written.

Where content validity stops

The strength of this route weakens the further the thing measured lies from what can be observed. A test that presents one of the job’s tasks allows a productive discussion of its content. A test that claims to measure a trait in the person that cannot be seen in any task gains nothing from an argument about content, which is the position integrity testing is in.

Representing the content is one thing, and a stable score is another, which is the subject of test reliability. A test that represents the work well but is marked by two people who disagree is not a basis for a decision.

What content validity is not

Something that is not representation can be put forward in its place, and one form this takes is the claim that a test “looks related to the job”. That is an impression held by whoever reads the test, and it may coincide with representation or depart from it. A test made up entirely of questions on the terminology of a trade can look wholly connected to the trade and still contain not one task that is performed in it.

The difficulty of the questions is not part of content validity either. An item taken from the core of the role stays representative even when it is easy, and a hard item from outside the role does not become representative through being hard. Confusing the two can produce tests whose difficulty is raised whenever applicant numbers grow, using tasks that are not performed in the job.

This is an explanation of the concept, not legal advice.