5 things worth knowing about decision skills metrics

You cannot measure whether a decision was right. You can measure several things around it, and the value of the exercise depends entirely on not confusing those things with the thing itself.

What to take away

Process and outcome, kept apart

The two diagonal boxes are the ones people can talk about. The two off-diagonal

This is why the recording matters more than the reviewing. A review with no contemporaneous record can only assess the outcome, and assessing the outcome tells you almost nothing about how to decide better next time.

What can honestly be counted

Time to decide. From the decision being stated to the decision being made.

Worth watching because it exposes decisions that were never actually made, and because it can be compared against how reversible the choice was.

Share reopened. How often a settled decision is reopened without new information. A high share points at a process problem, usually a missing decider or a missing revisit trigger.

Share revisited on trigger. The opposite, and a good sign. It means triggers were set and honored. Setting the trigger at the time of deciding is covered in the roles and process guide.

Reasoning that held. For each decision, whether the thing you said would change your mind either happened or did not, and whether you noticed. This is the closest thing to a real measure of decision quality, and it requires nothing but the record.

None of these is a score, and none should have a target attached. Putting a target on time to decide produces fast decisions; putting one on share reopened produces decisions nobody admits reopening.

Calibration, in plain terms

Calibration is the match between how confident you were and how often you turned

out to be right. It is worth building because it is one of the few things about your own judgment you can actually check.

The method is unglamorous. When you decide, write down what you expect and how sure you are, in words rather than a number if numbers feel false. Later, read the ones you were most sure about. The pattern that emerges is usually specific: people are not uniformly overconfident, they are overconfident about a particular kind of question.

The general phenomenon of extreme results moving back toward the ordinary, regression toward the mean, is worth knowing here, because it explains why a run of correct calls is weaker evidence than it feels.

What not to measure

Each of these is a proxy that moves as soon as people know it is watched, which is the practical content of Goodhart's law and the reason the metric words guide treats the promotion of a proxy to a target as a category error rather than a mistake of degree.

A review that is worth holding

Step four being fourth is the whole design. Every step after the outcome is known is contaminated by hindsight bias, which makes the result feel like it was obvious at the time and makes the people who did not predict it look careless.

Common questions

Is this worth doing for a business with three people?

The recording is. The reviewing can be a ten-minute conversation once a quarter. What does not work is the reviewing without the recording, which is the version most small teams try first.

How many decisions should be recorded?

The ones that are hard to reverse, plus the ones that surprise you. A record of every choice is a filing exercise; a record of the twelve that mattered this year is a resource.

What if the record shows I was wrong repeatedly?

Then it has paid for itself. The value is entirely in the specific pattern, and patterns are only visible in writing. The nine decision errors list the shapes that show up most often.

Can any of this tell me whether I am a good decision maker?

No, and treating it as an assessment of a person is where the practice usually dies. It measures a process you can change, which is more useful and less comfortable. The decision record fields are where it starts.