Industry
Part of Skill assessments rest on two properties: every instrument has to earn both
Skill assessments framework: four purposes, what each demands, and who pays for error
A skill assessments framework by purpose: selection, development, diagnosis and certification, what each demands, who pays for error, and questions before choosing.
Before asking whether an assessment is any good, ask what decision it is feeding. The same instrument can be adequate for one decision and indefensible for another, and most arguments about assessment are really arguments about a purpose nobody stated.
This page sets out the four purposes, what each one demands, and the questions that come before choosing anything.
What to take away
- Selection, development, diagnosis and certification are four different purposes with four different tolerances for error. Naming the purpose settles most of the choice.
- The question is never whether an instrument is accurate. It is which kind of error you can live with, and who pays for it.
- An assessment used for a purpose it was not built for is not a compromise. It is a different instrument, with unknown properties.
The four purposes
The decision it feeds
- Selection
- Who to choose
- Development
- What to work on next
- Diagnosis
- What is going wrong now
- Certification
- Whether a standard is met
The error that hurts most
- Selection
- Rejecting someone capable
- Development
- Pointing at the wrong thing
- Diagnosis
- Missing the real cause
- Certification
- Passing someone who is not ready
Who pays for it
- Selection
- The candidate, invisibly
- Development
- The learner, in wasted effort
- Diagnosis
- The work
- Certification
- Whoever relies on it
The right-hand column is the one people skip, and it is the one with the ethical
Four purposes and their costs
Purpose
- Selection
- Rejecting capable
- Development
- Wrong target
- Diagnosis
- Missing cause
- Certification
- Passing unready
Error
- Selection
- Candidate invisibly
- Development
- Learner wasted effort
- Diagnosis
- The work
- Certification
- Whoever relies
Who pays
- Selection
- Development
- Diagnosis
- Certification
What each purpose demands
Selection. Demands the strongest evidence of any of the four, because the
The decision is consequential. The subject cannot appeal it, and the errors are invisible. Public guidance exists because instruments do harm.
In the United States, the Equal Employment Opportunity Commission's guidance on employment tests sets out what an employer is expected to be able to show. Requirements differ by jurisdiction. This is an area for professional advice rather than judgment.
Development. Demands much less, because the cost of being wrong is a redirected effort rather than a lost opportunity. A rough instrument that starts a good conversation is defensible here and not in selection.
Diagnosis. Demands specificity rather than accuracy. A diagnostic that distinguishes four causes badly is more useful than one that measures overall ability precisely, because the decision it feeds is about which cause to address.
Certification. Demands a defined standard and a defensible cut point, and that is a harder problem than it looks. The standard is a judgment about what is good enough, and it should be documented and revisited rather than treated as a property of the test.
The questions before choosing anything
- What decision will this feed, and who makes it?
- What would we do differently depending on the result? If nothing, stop here.
- Which error can we tolerate, and who bears it?
- What is the instrument evidence for, according to whoever built it?
- What will we do about the people the result is wrong about?
Question two eliminates a large share of proposed assessment. Question five is
Questions before choosing anything
- What decision, and who makes it?
- What would we do differently?
- Which error can we tolerate?
- What is the instrument evidence for?
- What about people it is wrong about?
The two properties, and where they bind
Any instrument has to be consistent and has to measure the thing claimed. The
Where reliability and validity bind
Purpose
- Selection
- Matters most
- Development
- Matters less
- Diagnosis
- Matters
- Certification
- Matters most
Consistency
- Selection
- Matters absolutely
- Development
- Matters
- Diagnosis
- Matters
- Certification
- Matters absolutely
Measures claim
- Selection
- Development
- Diagnosis
- Certification
assessment guide sets those out in full; what matters here is that the two bind differently by purpose.
Consistency, reliability in the technical sense, matters most where a single result stands alone. In development it matters less, because the result is one input into a continuing conversation and errors get corrected by the next attempt.
Whether the instrument measures what it claims matters everywhere and matters absolutely in selection and certification, where the result is used as though it were a fact about a person.
Building the program rather than choosing the tool
- Write the outcome the assessment serves, as a performance rather than as a state. The twelve rewritten learning outcomes show what that sentence looks like.
- Decide what will happen for each band of result, in advance, in writing.
- Decide who sees the result, and for how long it is used.
- Decide what a person can do if they think the result is wrong.
- Set a date to look at whether the assessment predicted anything.
The last one is nearly always omitted, and it is the only step that produces evidence about the instrument in your own setting. Without it a program can run for years on the strength of a supplier's claims.
What no framework fixes
None of this tells you whether a person will do well in a role. That depends on conditions the assessment cannot see: the team, the support, what else is happening in their life, and what the job turns out to actually be.
The honest position is that assessments narrow uncertainty and do not remove it, and that a program which acts as though they do will make confident decisions that are wrong at a rate nobody measures. The nine decision errors cover the general form of judging by a result rather than by the reasoning behind it.
Common questions
Can I use one instrument for two purposes?
Only if the evidence supports both, which is rare. More often the instrument was built for development and gets used for selection because it is already paid for, and the properties do not transfer with it.
What about assessments people take about themselves?
Useful for starting a conversation and weak as evidence about anything, because the result reflects self-perception. That is a legitimate thing to measure, and it is not skill.
How do I know what an instrument is evidence for?
Ask the supplier for the technical documentation and read what it claims. Public material on assessment methods, such as the Office of Personnel Management's overview of other assessment methods, describes what the categories are and what each is used for.
Where does this leave a very small business?
With the development purpose, mostly, and with structured conversations rather than instruments. The learning plans guide covers the planning side, which is where the return is for a small team.
