Skill assessments rest on two properties: every instrument has to earn both

A skill assessment is a measuring instrument, and every instrument prints on its side a range, a precision, and conditions under which a reading is meaningless.

Assessments rarely say those things about themselves, but this page does.

Every assessment produces a number, a grade or a verdict, and the number is a fact about the assessment as much as about the person. Reading it well means knowing what the instrument could see, what it could only infer, and what was outside its field of view altogether.

What to take away

What the words mean

Assessment, test and evaluation are used as synonyms and are not. A test is one instrument: a set of tasks with a scoring rule. An assessment is the whole process of gathering evidence about a skill, which may use several tests and other sources. An evaluation is a judgment made on the basis of the assessment. Test, assess, evaluate: instrument, evidence, verdict.

Formative assessment is done to guide learning: it tells the learner what to work on. Summative assessment is done to certify: it tells someone else what the learner can do. The same task can serve either purpose, but the stakes differ, and every limitation of an instrument matters more in the summative case.

Criterion-referenced scoring compares the person against a fixed standard: did they do the task to the required level. Norm-referenced scoring compares them against other people: where did they rank. The second tells you nothing about whether anyone can actually do the thing, only who did it relatively better. A cohort in which everyone is weak still has a top decile.

Two properties every instrument has

Reliability is consistency. If the same person took the assessment twice,

or two assessors scored the same performance, would the result be the same? An unreliable instrument produces noise, and a single reading from it is partly luck.

Validity is whether the instrument measures what it claims to. A perfectly reliable instrument can be measuring the wrong thing: a written test of negotiation reliably measures the ability to write about negotiation, which is correlated with negotiating and is not negotiating.

The two are independent in one direction: an instrument cannot be more valid than reliable, because noise limits how well it tracks anything.

It can be entirely reliable and not valid at all. When told an assessment is "accurate", ask which of the two is meant; the answer is usually the first, and the second is the one that matters.

What each type can see

The eight instruments below are the ones most working assessments are built

Read the third column as the honest boundary of the instrument. Nothing in that column can be fixed by improving the instrument, because it is a consequence of what the instrument is. It can only be addressed by adding a different instrument whose blind spot is elsewhere, which is what a real assessment, in the sense defined above, does.

What no assessment can measure

Some things are outside the reach of any instrument, and a claim to measure them should be read as a claim about something else.

How a result can be wrong

Two kinds of error, and every decision rule has to choose between them.

A false positive passes someone who lacks the skill. A false negative fails someone who has it. Stricter assessment reduces the first and increases the second. No setting removes both.

Which error is worse depends on the assessment's purpose. A formative check should tolerate false positives, since the learner will soon find out. A safety-critical certification should not.

Three further distortions are worth naming because they are so common.

Practice effects. People who have taken similar tests before do better on them regardless of skill. The instrument is partly measuring familiarity with instruments.

Teaching to the test. When an assessment has consequences, preparation targets it. Scores rise; the skill may not. This is the same mechanism as a proxy metric becoming a target, which business acumen treats as a general hazard, and the same warning applies: the closer the assessment is to the real task, the less room there is for the two to come apart.

Assessor variation. Any instrument that involves human judgment inherits the judge's habits. Two assessors watching the same performance can differ, which is a reliability problem, and both can share the same bias, which is a validity problem that agreement between them hides.

Reading a result well

When a result lands, six questions, in order.

The last question is the test of a useful assessment. A result that says "below standard" and nothing else has done half its job. One that says which component failed, on which kind of task, has given the person the first line of their next learning plan, which is the purpose the whole exercise is best suited to.

The US federal personnel agency publishes a plain guide to assessment and selection methods, a sober description of trade-offs between instrument types.

It is written for hiring, but the reasoning about what each method can support carries over. Its separate notes on other assessment methods are a fair sample of how a careful institution describes its own instruments' limits.

Common questions

Can I assess my own skills?

You can assess your knowledge, by testing recall against a source. You can assess a skill only partially, because the errors you cannot see are the ones you most need to know about. For anything beyond knowledge, add one external source: a known-answer task, a recording compared with a model, or a person.

Is a certificate proof of a skill?

It is proof that a summative assessment was passed on a date. How much that says about the skill depends on how close the assessment was to the real task, which the certificate rarely states. Treat it as evidence, weighted by what you can find out about the instrument.

Which type is best?

To find what to work on, use a work sample with immediate feedback. To predict real performance, use direct observation over time, which is also the most expensive.

To certify, use whichever type is closest to the real task and can be scored consistently. No method is best in general, only best for a purpose. The vocabulary for keeping purposes apart is the same one decision skills uses for a decision's objective.

How should assessment results be discussed with the person?

As feedback, not as a verdict: which task, what the gap was, what the next step is. The difference between those two conversations is set out under communication skills, and it decides whether the assessment changes anything.