An LLM's Confidence Score Is a Meter With No Cal Sticker

On a test floor, every meter has a sticker. It says who calibrated it, against what, and when it’s due again. A meter without one isn’t an instrument. It’s a box that shows numbers.

Picture one on the bench with a sticker dated February 30. Nobody ever explains it. Everyone uses the meter anyway.

When you ask an LLM how sure it is and it says “92%,” you’re reading a meter with no sticker.

Where the number comes from

The clearest write-up I’ve seen on this is Asking LLM for confidence , posted October 20. The core point is simple. An LLM produces text one token at a time. When you ask “How confident are you?”, the answer “0.85” is not read from a gauge inside the model. In the post’s words, it is “a plausible continuation of the conversation. It is not a measurement of anything.”

The model was trained against outcomes, but the outcome was the next token of text. It learned what confidence scores look like in text. It never got scored on whether its stated confidence matched how often it was right.

The post lists three things you can check yourself:

Ask again and the number changes. The same question can get 0.85, then 0.7. Nothing about the problem changed.
Change the wording and the number changes. Ask “Are you sure?” and the score usually moves.
The words before the number decide the number. If the model already wrote “This is clearly the right answer,” a high score is the natural next thing to write.

On the floor we’d call that a reading that depends on how you hold the probe. You don’t ship on it.

What a cal sticker actually certifies

A sticker means two things. The instrument was checked against a known reference. And it was checked recently, under the same kind of conditions it’s used in.

The rumple post says the same about any confidence score worth trusting. It needs two properties: it reflects how much information the prediction is based on, and it “has been checked against what actually happened, on the same kind of question.” The LLM’s number has neither.

Build the instrument, then calibrate it

The fix in the post is the one every test engineer already knows. Don’t ask the device under test to grade itself. Build a measurement from things you can observe, then keep checking it against outcomes.

The post walks it through with a support agent:

Pick observable inputs. Did search find an article on this topic? How many? Has the agent answered this type of question before? Ask twice: do the two replies agree?
Start with the base rate. If a person checked 200 past replies and 150 were right, every reply starts at 75%.
Record every reply and what happened. The inputs, the confidence the rule gave, and whether it was right.
Update the rule from the record. If replies with a matching article and two agreeing answers were right 95% of the time, the rule says 95% for the next one like that.
Keep checking. Every week or so, for each confidence level, how often was the reply right? If the 90% replies were right only 70% of the time, the rule is overconfident.

The LLM still does the work. It just doesn’t sign the cal sticker.

What I saw when I measured one

I did a version of this in Is Jev actually calibrated? , with a scorer built to output a confidence, not a chat model asked for one. Even there, the shape mattered more than the headline. On 240 labelled agent tool calls, a confidence of exactly 1.000 was right 133 times out of 134. Scores between 0.90 and 0.99 averaged 0.970 and were right only 87.7% of the time. The overall ECE was 0.057, which averages both of those into one small number.

That’s the reason to recheck on a schedule. A single number can look fine while one band of the meter is off. You only see it by binning real outcomes on your own kind of task.

The rule for agents

If an agent routes, escalates, or auto-approves on a confidence number, ask where that number came from.

If the model wrote it, treat it as text.
If a rule built from evidence and past outcomes produced it, ask when it was last checked, and on what.
If nobody records whether answers turned out right, there’s no sticker. The post puts it plainly: the rule is only as good as the record.

Same idea as your tests are lying to your agent and who signs the robot test report . And the same idea as the golden unit : the thing being measured doesn’t get to certify the measurement.

Check the sticker. Then check the date on it.

Sources

Asking LLM for confidence , Some Random Thoughts (rumple.github.io), October 20, 2026.
Is Jev actually calibrated? The reliability curve on 240 cases , my own measurement, with the data in themsquared/jev-calibration .

Frequently asked questions

Can you trust an LLM's self-reported confidence score?

Not on its own. The model writes the number token by token like any other text, so it is a plausible continuation, not a measurement. Ask again and it can change, for example 0.85 one time and 0.7 the next. Rewording the question, like asking ‘Are you sure?’, usually moves it too. You only know if the numbers mean anything by grading many of them against an answer key.

How do you get a confidence score for an AI agent without asking the LLM?

Build a rule from things you can observe about each answer, such as whether search found a matching source and whether two runs of the same question agree. Start from the historical rate of correct answers, record every answer and whether it turned out right, update the rule from that record, and recheck calibration on a schedule.

How often should you check an agent's calibration?

On a schedule, on the same kind of task. The rumple.github.io post suggests making the table every week or so: for each confidence level, how often was the answer correct. If the 90% answers were right only 70% of the time, the rule is overconfident and the record shows how to fix it.