Confidence and calibration

A model can make a prediction.

But how much should we trust it?

Suppose two systems both predict:

Outcome A

One says:

55% confidence

The other says:

99% confidence.

Those are very different claims.

A useful predictive system should therefore tell us not only:

What do I think will happen?

but also:

How strongly do I believe it?

That leads us to two important ideas:

confidence

and:

calibration.

Confidence describes how strongly a model believes its own prediction. Calibration asks whether that confidence is actually justified.

Confidence is not accuracy

Suppose a model predicts:

Cat — 95% confidence.

That is not the same as saying:

This prediction is definitely correct.

It means the model is assigning a very high probability to the cat class.

The prediction may still be wrong.

Likewise:

60% confidence

does not mean the model is bad.

It may simply recognise that the example is ambiguous.

A good model should be uncertain when the evidence is uncertain.

Confidence should reflect evidence

Imagine two photographs.

Image A

A clear, close-up photograph of a cat.

The model predicts:

Cat: 99%

Image B

A dark, blurry image showing only part of an animal.

The model predicts:

Cat: 54%

This makes sense.

The second image contains less information.

The model's confidence reflects that.

Ideally:

clear evidence → concentrated probability

ambiguous evidence → spread-out probability

Confidence is a probability distribution

For classification, confidence often comes from the model's probability distribution over possible labels.

For example:

Prediction A

Cat: 0.97

Dog: 0.02

Fox: 0.01

The model is highly confident.

Prediction B

Cat: 0.40

Dog: 0.35

Fox: 0.25

The model is uncertain.

The final label might be:

Cat

in both cases.

But the underlying beliefs are very different.

A label can hide uncertainty

Suppose an application displays only:

APPROVED

or:

REJECTED.

Underneath, the model may have predicted:

Person B

P(default) = 0.49

If the threshold is:

0.50

both may be approved.

But the confidence in those decisions is very different.

Reducing probabilistic predictions to discrete decisions can hide important uncertainty.

Calibration asks a different question

Suppose a model makes:

1,000 predictions

with:

80% confidence.

If the model is well calibrated, we would expect roughly:

800 of those predictions

to be correct.

Not necessarily exactly 800.

Random variation still exists.

But over many comparable cases, confidence should correspond reasonably well to observed frequency.

This is calibration.

What does well calibrated mean?

A well-calibrated model behaves roughly like this:

Predictions at 50% confidence → correct about 50% of the time

Predictions at 70% confidence → correct about 70% of the time

Predictions at 90% confidence → correct about 90% of the time

The probability becomes meaningful.

When the model says:

90%

we know what that number means empirically.

Overconfidence

Suppose a model makes 100 predictions at:

90% confidence.

But only:

60 are correct.

The model is overconfident.

It behaves as though its knowledge is much stronger than it really is.

This can be dangerous.

Downstream users may treat uncertain predictions as near-certainties.

Underconfidence

Now suppose a model makes 100 predictions at:

60% confidence

and:

95 are correct.

The model is underconfident.

That may sound harmless.

But underconfidence can also create costs.

A system may:

request unnecessary human review,
collect unnecessary measurements,
maintain too much reserve,
avoid useful actions.

Calibration matters in both directions.

Accuracy and calibration are different

Suppose two classifiers each achieve:

90% accuracy.

Model A

When correct, it usually reports around:

90% confidence.

When uncertain, its confidence falls appropriately.

Model B

Reports:

99.9% confidence on almost everything.

They have the same accuracy.

But Model A is much better calibrated.

This matters because Model A gives us useful information about when it is likely to be wrong.

A perfectly accurate model would be easy to calibrate

If a model were literally correct every time, confidence could be:

100%.

But real models are imperfect.

So useful confidence estimates should distinguish:

easy cases

from:

hard cases.

The system should know something about where its own limits lie.

Calibration turns confidence into something operational

Suppose an autonomous system says:

99.99% probability road is clear.

Should we trust that?

Only if the system's probability estimates are calibrated.

If events assigned:

99.99%

occur correctly only:

95% of the time,

then that probability is deeply misleading for safety-critical use.

Calibration turns probability from a decorative number into an empirical claim.

Predicted confidence	Actual success rate
50%	52%
60%	59%
70%	71%
80%	79%
90%	89%

Predicted confidence	Actual success rate
50%	40%
60%	45%
70%	50%
80%	55%
90%	60%

Reliability diagrams

A common visual tool is a reliability diagram.

We plot:

predicted probability

against:

observed frequency.

A perfectly calibrated model would lie close to a diagonal line:

predicted 70% → observed about 70%

Deviations reveal:

overconfidence,
underconfidence.

This allows calibration to be measured rather than guessed.

Calibration needs many examples

We cannot judge calibration from one prediction.

Suppose a model says:

90% probability of rain.

It does not rain.

That does not prove the model was overconfident.

A 90% event should fail around:

10% of the time.

Calibration requires repeated comparable forecasts.

Rare predictions are harder to calibrate

Suppose a model predicts:

0.01% probability of catastrophic failure.

How do we know whether that number is accurate?

We may need:

enormous datasets,
simulations,
physical models,
stress tests.

Rare-event calibration is difficult precisely because observations are rare.

This is a major challenge in safety-critical systems.

Calibration can vary across groups

Suppose a healthcare model outputs:

80% probability of disease.

For Group A, predictions at 80% are correct around:

80% of the time.

For Group B, they are correct only:

55% of the time.

The model may be calibrated overall while poorly calibrated for one group.

So calibration should sometimes be examined across:

populations,
locations,
operating conditions.

Global calibration can hide local miscalibration

Imagine:

Group A → overconfident

Group B → underconfident

When combined, the errors may partially cancel.

Overall calibration appears excellent.

But the system behaves differently for different groups.

Again:

Average performance can hide local failure.

Calibration can vary through time

A model may be well calibrated when deployed.

Then the world changes.

Perhaps:

user behaviour changes,
climate conditions shift,
new technology appears.

Now:

90% predictions

are correct only:

70% of the time.

The model has become miscalibrated because the deployment distribution changed.

Calibration therefore needs ongoing monitoring.

Calibration can vary across space

Suppose a weather model is calibrated well in regions with dense sensor coverage.

In remote regions:

fewer observations,
more uncertain terrain effects.

The same nominal probability may be less reliable.

A useful system may need location-specific calibration.

Calibration and distribution shift

Suppose a credit model was trained during:

stable economic conditions.

Then a major recession begins.

Historical relationships between:

income,
employment,
default

change.

The model may still output precise probabilities.

But those probabilities may no longer correspond to observed frequencies.

Distribution shift can break calibration.

Confidence should fall outside familiar territory

Suppose a model was trained on temperatures between:

0°C and 30°C.

Now it receives:

50°C.

A well-designed system should ideally recognise:

This is far outside my training experience.

Its epistemic uncertainty should increase.

A system that remains extremely confident simply because its mathematical function still produces an output is dangerous.

Out-of-distribution inputs

An input is often described as out of distribution when it differs substantially from the kinds of examples seen during training.

Examples might include:

new environments,
new populations,
unusual sensor conditions,
entirely new categories.

Models can behave unpredictably in these regions.

Confidence should therefore be interpreted with caution.

High confidence can be wrong

Neural networks can sometimes produce:

very high confidence

for examples they classify incorrectly.

This reveals an important point:

The raw probability output of a model is not automatically a trustworthy measure of epistemic certainty.

A softmax probability, for example, can be highly concentrated even when the model is outside familiar territory.

Calibration must be evaluated.

Confidence can be calibrated after training

Sometimes a model predicts the right classes but its probabilities are systematically too confident.

We may apply calibration methods after training.

The aim is not necessarily to change which class wins.

Instead, we adjust the probability values so that:

stated confidence better matches observed accuracy.

Examples include methods such as:

temperature scaling,
Platt scaling,
isotonic regression.

The details can wait.

The principle is:

Probability outputs themselves can be calibrated.

Calibration should use separate data

Suppose we train a model on one dataset.

If we also calibrate it using exactly the same data, we risk overfitting the calibration.

A better workflow uses separate data for:

training,
calibration or validation,
final testing.

This connects to the later machine-learning module.

Confidence intervals and confidence scores are different

The word confidence is used in several ways.

A classifier might say:

90% confidence this is a cat.

Statistics might report:

95% confidence interval for a population mean.

These are not the same concept.

A confidence interval concerns uncertainty in an estimated quantity.

A model confidence score may represent a predicted probability.

Terminology needs to be interpreted carefully.

Prediction intervals express uncertainty differently

Suppose a demand model says:

Expected demand = 40 GW

with:

90% prediction interval = 36–45 GW.

The interval expresses uncertainty about the future realised value.

Calibration asks:

Do approximately 90% of comparable future outcomes actually fall inside our 90% prediction intervals?

Again, the uncertainty statement can be tested.

Interval calibration

Suppose a model generates:

1,000 90% prediction intervals.

If:

895 actual outcomes

fall inside, the model is reasonably calibrated.

If only:

600

fall inside, the intervals are too narrow.

If all:

1,000

fall inside enormous ranges, the intervals may be too wide to be useful.

Calibration and sharpness

A good probabilistic forecast needs both:

calibration

and:

sharpness.

Calibration asks:

Are the probabilities honest?

Sharpness asks:

Are the predictions as specific as they can reasonably be?

Consider:

Model A

Temperature tomorrow: between -100°C and +100°C with 99.9% confidence.

Probably calibrated.

Not useful.

Model B

Temperature tomorrow: 19–21°C with 90% probability.

Much sharper.

If calibrated, it is far more informative.

Uncertainty should be narrow only when justified

The goal is not to maximise confidence.

It is not impressive for every prediction to be:

99.9% confident.

The goal is:

Be as confident as the evidence justifies — no more and no less.

A model that says:

I am uncertain

when uncertainty is genuine is behaving well.

Calibration and decision thresholds

Suppose a hospital uses a model predicting disease probability.

Treatment is recommended if:

P(Disease) > 20%.

If the model is badly calibrated, that threshold loses meaning.

A predicted:

20%

might actually correspond to:

or:

50%.

Decision rules relying on probability require calibrated probabilities.

Calibration can matter more than classification accuracy

Suppose an insurance system uses predicted probabilities to set prices.

It needs to distinguish between:

1% risk

5% risk

20% risk.

Simply knowing:

high risk / low risk

is not enough.

If probabilities are miscalibrated, prices may be systematically wrong even if ranking accuracy is good.

Ranking and calibration are different

Suppose Model A correctly ranks customers:

Customer X riskier than Customer Y.

But it predicts:

80% versus 70%

when true risks are:

8% versus 7%.

The ranking is excellent.

The calibration is terrible.

Different applications need different model properties.

Recommendation systems may care more about ranking

A recommendation system might mainly care:

Which item is most likely to interest the user?

Absolute probability may matter less.

An insurance system might care greatly about:

What is the actual probability of loss?

The importance of calibration depends on the service.

Calibration and expected value

Suppose a system calculates:

expected loss = probability × consequence.

If the probability is badly calibrated, the expected value is badly calibrated too.

A prediction of:

1% risk

versus:

10% risk

can radically change the economic decision.

Expected-value optimisation relies on meaningful probabilities.

Calibration and resource planning

Suppose a hospital predicts:

10% probability that demand exceeds capacity.

It decides whether to open additional beds.

If the true probability is:

40%,

the service will be underprepared.

If it is actually:

0.1%,

the service may waste resources.

Calibrated uncertainty helps allocate resources efficiently.

Calibration and reliability

Suppose an electricity system estimates:

1% probability of shortage.

Reliability planning depends on that probability having real meaning.

If the system systematically underestimates tail probabilities, apparently efficient planning can produce frequent failures.

Calibration is therefore not just a statistical nicety.

It can be an engineering requirement.

Calibration and autonomous systems

Suppose an autonomous vehicle estimates:

P(Pedestrian ahead) = 5%.

The action may depend strongly on whether 5% really means:

approximately 5%.

If the system is overconfident and true probability is closer to:

40%,

the consequence can be severe.

Safety-critical systems require careful treatment of probabilistic confidence.

Confidence should influence behaviour

Suppose a robot predicts:

object = cardboard box

with:

99% confidence.

It may proceed normally.

Now:

object = cardboard box

with:

52% confidence.

A good system might:

slow down,
look again,
use another sensor.

The label is the same.

The uncertainty changes the action.

Abstention

Sometimes the correct decision is:

Do not make the automated decision yet.

A model can abstain when confidence is too low.

For example:

high confidence → automatic decision

medium confidence → additional measurement

low confidence → human review

This can make services much safer.

Selective prediction

A model does not necessarily need to answer every case.

Suppose it is:

99% accurate on the 80% of cases where it is confident

but much less accurate on the remaining 20%.

The service might automate only the confident cases.

The uncertain cases go elsewhere.

This is sometimes called selective prediction.

Human review is a resource too

But sending uncertain cases to humans creates another problem.

Human attention is finite.

Suppose:

10 million predictions per day

and:

20% require review.

That is:

2 million reviews.

The service may not have enough people.

So confidence thresholds become part of resource allocation.

Confidence thresholds create trade-offs

Lower the automation threshold:

more cases automated

but:

more errors.

Raise it:

fewer errors

but:

more cases require human review.

This is another example of prediction becoming service design.

The model output is only one piece.

We still need to allocate scarce human or computational resources.

Confidence can determine priority

Suppose a medical AI triages cases.

It may distinguish:

high probability of urgent condition,
low probability,
highly uncertain.

The uncertain group might deserve additional testing rather than simply being placed in the middle.

Uncertainty itself can be a reason for priority.

Uncertainty is not the same as low risk

Suppose:

Patient A

P(disease) = 5%

with strong evidence.

Patient B

P(disease) = somewhere between 1% and 60%

because data is poor.

Patient B may have a lower point estimate than some threshold.

But uncertainty is much greater.

A good decision system should distinguish:

low estimated risk

from:

poorly known risk.

Confidence and epistemic uncertainty

This is where the distinction from the previous lesson matters.

A model can be uncertain because:

the outcome is inherently variable

or because:

the model does not know enough.

The second form — epistemic uncertainty — may be especially relevant to deciding whether to:

collect more data,
request human review,
avoid acting.

Confidence should fall when evidence is sparse

Suppose a model predicts customer demand for:

a familiar city with ten years of data.

Then it predicts demand for:

a newly opened location with one week of data.

Even if the point predictions are similar, confidence should probably differ.

The evidence base differs.

Confidence should reflect data quality too

Suppose the inputs contain:

missing sensor values,
uncertain measurements,
stale information.

The model should ideally carry that uncertainty into its output.

Recall:

UNCERTAIN INPUT

↓

UNCERTAIN STATE

↓

UNCERTAIN PREDICTION

The model should not convert poor evidence into false precision.

Garbage in, confidence out?

A dangerous system can behave like:

poor data

↓

complex model

↓

99.8% confident prediction

The complexity of the model does not magically improve the evidence.

Confidence should ultimately be grounded in:

data quality,
model validity,
historical calibration.

Confidence and hallucinations

Large language models present a particularly interesting challenge.

They can produce:

fluent,
grammatical,
authoritative-sounding text

even when the factual basis is weak.

Humans often use tone as a cue for confidence.

But generated linguistic confidence is not necessarily calibrated factual confidence.

This is one reason hallucinations can be persuasive.

Fluency is not probability of truth

Suppose an LLM responds:

The treaty was signed in 1847.

The sentence may be produced fluently.

That does not imply:

P(statement is true) = 99%.

The model may simply have generated a plausible sequence of tokens.

We should distinguish:

probability of a token sequence under the language model

from:

probability that the factual claim corresponds to reality.

Those are not automatically the same thing.

This distinction is fundamental

A language model is trained primarily to model patterns in language.

If:

plausible language

and:

true statement

were always identical, there would be no problem.

But human text contains:

fiction,
mistakes,
speculation,
misinformation,
contradictions.

So linguistic likelihood is not a direct truth score.

This helps explain why confident-sounding hallucinations are possible.

Calibration for generative AI is difficult

Classification has a relatively clear outcome:

correct / incorrect.

Generative answers can contain:

several claims,
partial correctness,
uncertainty,
interpretation.

What does:

80% confidence in this paragraph

even mean?

Calibrating generative AI is therefore more complicated than calibrating simple classifiers.

Claim-level confidence

One possible approach is to evaluate individual claims.

For example:

Claim A → strong evidence

Claim B → moderate evidence

Claim C → uncertain

This may be more useful than assigning one confidence score to an entire response.

It also connects to provenance and retrieval.

External evidence can improve confidence

Suppose an AI does not rely only on internal model knowledge.

It retrieves:

current documents,
databases,
authoritative sources.

Now the system can ground its answer in external evidence.

Conceptually:

MODEL PRIOR KNOWLEDGE

RETRIEVED EVIDENCE

↓

ANSWER

The quality of the evidence can affect how confident the system should be.

But retrieval does not guarantee truth

A retrieved source may itself be:

wrong,
outdated,
misleading.

Several sources may copy the same error.

So confidence still requires evaluating:

provenance,
independence,
reliability.

Evidence quality matters.

Calibration can deteriorate after deployment

Suppose a model is carefully calibrated before launch.

Then:

users adapt,
inputs change,
policies change.

The model's probability outputs drift away from reality.

This is why calibration is not a one-time certification exercise.

It is something that may need to be monitored continuously.

Calibration monitoring

A deployed system can compare:

predicted probabilities

with:

realised outcomes.

For example:

Predicted failure risk: 10%

Over time:

Do approximately 10% of such cases actually fail?

If not, recalibration or retraining may be needed.

The future becomes evidence about whether the model's earlier beliefs were justified.

This closes the loop

At time t:

MODEL PREDICTS PROBABILITY

Then:

REALITY HAPPENS

At time t+1:

OBSERVED OUTCOME

becomes:

EVIDENCE ABOUT MODEL CALIBRATION.

So:

PREDICTION

↓

OUTCOME

↓

EVALUATE CONFIDENCE

↓

UPDATE MODEL

Prediction systems should learn not only:

what happens

but also:

whether their own confidence was appropriate.

Calibration and trust

People sometimes say:

We need users to trust AI.

A better objective is:

Users should be able to trust the AI to the degree that its evidence justifies.

Blind trust is undesirable.

Blind distrust is also undesirable.

Calibration provides one way of aligning:

system confidence

with:

actual reliability.

Trust should be calibrated too

Suppose an AI is extremely reliable at:

translation,
summarisation.

But less reliable at:

obscure factual recall,
novel legal interpretation.

Users should not apply one global trust level to every task.

Trust should vary with:

domain,
evidence,
uncertainty,
observed performance.

Confidence interfaces matter

How should a system communicate uncertainty?

Possibilities include:

probabilities,
confidence bands,
labels such as low / medium / high confidence,
alternative hypotheses,
explicit warnings.

The interface matters because people interpret uncertainty differently.

A technically calibrated model can still be misunderstood if uncertainty is communicated badly.

False precision should be avoided

Suppose an AI says:

Probability = 73.428%.

That looks extremely precise.

But perhaps the underlying evidence supports only something like:

roughly 60–80%.

Decimal places can create an illusion of certainty.

Good communication should match the precision of the evidence.

Confidence categories also lose information

On the other hand:

Low

Medium

High

may be easier to understand.

But these categories compress the probability distribution.

The choice depends on:

user,
task,
consequence.

Again, representation is a design decision.

Confidence and service levels

Suppose an AI service offers:

Standard service

Automation allowed above:

90% confidence.

High-assurance service

Automation allowed only above:

99.9% confidence.

The second may require:

more computation,
more sensors,
more human review.

Confidence requirements can therefore become part of service design.

Reliability has a price

Higher assurance often requires more resources.

For example:

redundant sensors,
backup models,
additional testing,
human review.

So the desired confidence level has economic consequences.

There is no free certainty.

Different tasks justify different confidence requirements

For:

suggesting a restaurant

moderate confidence may be fine.

For:

stopping a train

much higher assurance may be required.

For:

autonomous surgery

the standard may be higher still.

The appropriate confidence depends on:

consequence of error.

Calibration and fairness interact

Suppose the same confidence threshold is applied to everyone.

That sounds equal.

But if model calibration differs across groups, the threshold may produce unequal error rates.

For one population:

80% prediction really means 80%.

For another:

80% prediction really means 60%.

Equal thresholds can then create unequal outcomes.

Fairness may require calibration analysis by group

This does not mean every group must always have identical probability distributions.

It means we should understand whether:

the same numerical confidence has the same meaning.

If not, users may be receiving systematically different quality of service.

Confidence can influence access to resources

Suppose scarce treatment is allocated partly according to predicted probability of benefit.

If one group's probabilities are systematically underconfident, they may receive less treatment.

Calibration errors can therefore become allocation errors.

This is where probabilistic modelling becomes a societal issue.

Confidence should not become a moral score

A predicted probability describes uncertainty about a defined outcome.

For example:

P(default).

It does not describe:

someone's worth,
deservingness,
moral character.

The distinction matters because numerical risk scores can easily acquire social meaning beyond what they actually measure.

Calibration is always relative to a target

Suppose a model is calibrated for predicting:

loan default within 12 months.

That does not imply it is calibrated for:

financial hardship

or:

long-term creditworthiness.

Calibration only applies to the outcome actually defined.

Again:

What is the label?

The target can change

Suppose the definition of default changes.

Now the historical calibration may no longer apply.

A change in:

policy,
label definition,
measurement process

can break calibration even if the underlying model has not changed.

Confidence and human judgement

Humans are often poorly calibrated too.

Experts can be:

overconfident,
underconfident.

One advantage of explicit probabilistic forecasting is that calibration can be measured.

For example:

When this expert says 70%, how often is the event actually occurring?

This can improve decision-making.

Prediction tournaments

Some forecasting systems ask people to assign probabilities to future events.

Over time, forecasters can be scored on:

accuracy,
calibration.

This encourages people to distinguish:

I think this will happen

from:

How strongly do I think this will happen?

That is a useful discipline.

Scoring confidence

If we want models or humans to provide good probabilities, we need evaluation methods that reward honest uncertainty.

Suppose an event occurs.

A prediction of:

60%

should be scored differently from:

99%.

Being confidently wrong should usually be penalised more than being cautiously wrong.

This leads to proper scoring rules.

We will return to them in Module 5.

Overconfidence should be costly

Suppose:

Prediction A: 51% probability of event

and the event does not occur.

The forecaster was barely leaning toward the wrong answer.

Now:

Prediction B: 99.99% probability

and the event does not occur.

That is a much more serious failure of probabilistic judgement.

A good scoring rule captures this difference.

Calibration and learning

A model can use realised outcomes to improve its own confidence estimates.

Suppose it notices:

My 90% predictions are only correct 75% of the time.

It can recalibrate.

So confidence itself can be learned.

The system is learning not only:

how to predict

but:

how much to trust its predictions.

Confidence as part of state

We can now extend our state representation.

An intelligent system may maintain:

estimated state

plus:

uncertainty about state

plus:

confidence in its model.

For example:

position estimate = x

state uncertainty = small

model confidence = high

This richer representation can guide better action.

The decision may be confident even when the prediction is not

Suppose an AI estimates:

P(rain) = 45%.

That prediction is uncertain.

But the decision may be:

Carry an umbrella.

If the cost of carrying it is tiny, the decision can be obvious despite uncertain prediction.

This is an important distinction:

prediction confidence

and:

decision confidence

are not the same.

A highly confident prediction can still produce a difficult decision

Suppose an AI says:

99% probability Patient A benefits from treatment.

But:

treatment is extremely scarce,
several other patients have similarly high probabilities.

The prediction is confident.

The allocation decision is still difficult.

Again:

prediction ≠ decision.

Calibration belongs between prediction and action

A useful pipeline is:

DATA

↓

MODEL

↓

PROBABILITY

↓

CALIBRATION CHECK

↓

DECISION RULE

↓

ACTION

We should not treat raw model scores as automatically meaningful probabilities.

The course framework

We can now enrich the full loop:

PAST DATA

↓

MODEL

↓

PROBABILITY DISTRIBUTION

↓

CONFIDENCE

↓

IS THAT CONFIDENCE CALIBRATED?

↓

DECISION

↓

ACTION

↓

REAL OUTCOME

↓

COMPARE PREDICTED PROBABILITY WITH REALITY

↓

RECALIBRATE / LEARN

The system receives feedback not only on:

whether it was right

but:

whether it was appropriately confident.

Ask calibration questions

Whenever a system reports a probability or confidence score, ask:

What does the confidence number mean?
Is it a genuine probability or just a model score?
How was it calibrated?
Are 80% predictions actually correct around 80% of the time?
Does calibration vary across groups?
Does it vary across locations?
Does it vary through time?
What happens outside the training distribution?
Does missing or noisy data reduce confidence?
Is the system overconfident?
Is it underconfident?
Can it abstain?
What happens when confidence is low?
Does the decision threshold depend on the consequence of error?
Is uncertainty communicated clearly to users?

These questions are crucial whenever probabilities influence real decisions.

The central idea

Confidence is useful only if it means something.

A model saying:

90%

should not merely mean:

My internal calculation produced the number 0.9.

It should mean something closer to:

Across comparable situations where I make predictions like this, outcomes behave roughly as this probability suggests.

That is calibration.

A good model should not only know what it predicts. It should know how much trust its own prediction deserves.

Calibration connects:

belief

to:

observed reality.

The loop becomes:

PREDICT

↓

STATE CONFIDENCE

↓

OBSERVE OUTCOME

↓

WAS THAT CONFIDENCE JUSTIFIED?

↓

UPDATE

This is especially important for modern AI, because systems can produce outputs that sound equally confident even when the underlying evidence differs dramatically.

In the next lesson, we will examine a distinction that is closely related but conceptually different:

risk versus uncertainty — the difference between situations where we can meaningfully estimate the probabilities of outcomes and situations where even the probability model itself may be uncertain.