5.10 Scoring Rules

Suppose two weather forecasters make predictions about whether it will rain tomorrow.

Forecaster A

90% chance of rain.

Forecaster B

60% chance of rain.

Tomorrow arrives.

It rains.

Both forecasters predicted that rain was more likely than not.

So were they equally good?

Probably not.

Forecaster A assigned much more probability to the outcome that actually occurred.

But now imagine it does not rain.

Forecaster A should presumably be penalised more heavily, because they expressed much greater confidence in the wrong outcome.

This creates an interesting problem.

Once we allow models to make probabilistic forecasts, simply asking whether they were right or wrong is no longer enough.

We need a way of evaluating the entire probability forecast.

That is what scoring rules do.

From accuracy to scoring

For a simple classification system, we might calculate:

[ \text{Accuracy}

\frac{\text{correct predictions}} {\text{total predictions}} ]

If a model predicts:

Rain

and it rains, the prediction is correct.

If it does not rain, the prediction is incorrect.

But suppose the model actually predicted:

[ P(\text{rain})=0.51 ]

Another model predicted:

[ P(\text{rain})=0.99 ]

If it rains, ordinary classification accuracy treats both identically.

Both predicted:

[ \text{rain} ]

But their probabilistic predictions were very different.

Likewise, if it does not rain, both classifications are wrong.

Yet the model that assigned 99% probability to rain was much more confidently wrong.

A scoring rule allows us to preserve this information.

What is a scoring rule?

A scoring rule is a mathematical function that evaluates a probabilistic forecast after the actual outcome becomes known.

Conceptually:

[ S(\text{forecast},\text{outcome}) ]

The forecast might be a probability:

[ p=P(Y=1) ]

and the eventual outcome might be:

[ y\in{0,1} ]

The scoring rule compares the two.

Depending on the convention being used, a better forecast might receive:

a higher score, or
a lower loss.

The important idea is the same:

Probabilistic forecasts should be rewarded for assigning high probability to outcomes that occur and penalised for assigning high probability to outcomes that do not.

The Brier score

One of the simplest scoring rules for binary probabilistic forecasts is the Brier score.

For a predicted probability (p) and an observed outcome (y):

[ BS=(p-y)^2 ]

where:

[ y= \begin{cases} 1 & \text{if the event occurs}\ 0 & \text{if the event does not occur} \end{cases} ]

A lower Brier score is better.

Let's see how it works.

Example: it rains

Suppose:

[ p=0.9 ]

and it rains:

[ y=1 ]

Then:

[ BS=(0.9-1)^2 ]

[ BS=0.01 ]

That is a very good score.

The forecast assigned high probability to the event that actually occurred.

Now consider the forecaster who predicted:

[ p=0.6 ]

The Brier score becomes:

[ BS=(0.6-1)^2 ]

[ BS=0.16 ]

Still reasonable, but worse.

The 90% forecast receives more credit because it assigned greater probability to the observed outcome.

Example: it does not rain

Now suppose the event does not occur:

[ y=0 ]

The 90% forecast receives:

[ BS=(0.9-0)^2 ]

[ BS=0.81 ]

while the 60% forecast receives:

[ BS=(0.6-0)^2 ]

[ BS=0.36 ]

The highly confident forecast is penalised more severely.

That makes intuitive sense.

Saying:

I'm 90% certain it will rain

should carry more responsibility than saying:

Rain seems slightly more likely than not.

Compare several predictions

Suppose the event occurs:

[ y=1 ]

Different probability forecasts would receive:

Predicted probability	Brier score
0.99	0.0001
0.90	0.0100
0.75	0.0625
0.50	0.2500
0.25	0.5625
0.10	0.8100
0.01	0.9801

The pattern is clear.

The more probability the model assigned to the outcome that actually occurred, the better its score.

But notice something important.

The scoring rule does not merely ask:

Was the forecast correct?

It asks:

How much probability did the forecast assign to reality?

Why evaluate probabilities?

Suppose two medical models predict whether patients will develop a complication.

Both achieve:

[ 80% ]

classification accuracy.

At first glance, they appear equally good.

But perhaps Model A produces sensible probabilities:

Patient 1: 52%
Patient 2: 68%
Patient 3: 91%

while Model B routinely produces:

Patient 1: 99%
Patient 2: 99%
Patient 3: 99%

Whenever Model B is wrong, it is extremely confidently wrong.

If those probabilities are subsequently used to:

prioritise patients,
allocate hospital capacity,
recommend treatment,
communicate risk,

the distinction matters enormously.

Probabilistic predictions are information.

Scoring rules help us evaluate the quality of that information.

Log loss

Another widely used scoring rule is logarithmic loss, often called log loss or cross-entropy loss.

For a binary outcome:

[ L

-\left[ y\log(p) + (1-y)\log(1-p) \right] ]

Again:

(p) is the predicted probability of the event;
(y) is the actual outcome.

If the event occurs:

[ y=1 ]

then:

[ L=-\log(p) ]

If the event does not occur:

[ y=0 ]

then:

[ L=-\log(1-p) ]

Lower values are better.

Log loss strongly punishes confident mistakes

Suppose the event occurs.

A model predicts:

[ p=0.9 ]

Then:

[ L=-\log(0.9) ]

which is small.

Now suppose the model predicted:

[ p=0.01 ]

but the event occurs anyway.

Then:

[ L=-\log(0.01) ]

which is much larger.

And as:

[ p\rightarrow0 ]

for an event that actually occurs:

[ -\log(p)\rightarrow\infty ]

So log loss has a particularly important property:

Being confidently wrong can be extremely expensive.

This discourages models from assigning probabilities close to zero to outcomes that remain genuinely possible.

Why punish confidence?

At first this might seem harsh.

If the model gets something wrong, why should it matter whether it predicted:

[ 49% ]

or:

[ 1% ]

Because those statements contain very different information.

A prediction of:

[ P(Y)=0.49 ]

says:

This outcome is slightly less likely than not, but I am very uncertain.

A prediction of:

[ P(Y)=0.01 ]

says:

I believe this outcome is extraordinarily unlikely.

If the supposedly extraordinary event keeps happening, something is badly wrong with the model's representation of uncertainty.

Confidence should therefore have consequences.

Proper scoring rules

This leads to one of the most elegant ideas in probabilistic forecasting.

A proper scoring rule is designed so that, on average, a forecaster does best by reporting their true probability belief.

Suppose you genuinely believe:

[ P(\text{rain})=0.7 ]

A properly designed scoring rule should make:

[ p=0.7 ]

your best strategy.

You should not benefit, on average, from pretending:

[ p=0.9 ]

or:

[ p=0.4 ]

if 0.7 really represents your belief.

A strictly proper scoring rule makes truthful reporting uniquely optimal.

This is a remarkably powerful property.

The evaluation mechanism creates an incentive for honesty.

Scoring rules as incentive design

This connects forecasting to something much broader.

Suppose forecasters are rewarded according to how well they perform.

If the reward mechanism is poorly designed, they may have incentives to manipulate their predictions.

Perhaps cautious predictions are rewarded.

Perhaps extreme predictions receive attention.

Perhaps forecasters can maximise their apparent accuracy by avoiding difficult cases.

A proper scoring rule tries to align the incentive with what we actually want:

Report the probability you genuinely believe best represents the uncertainty.

This is no longer merely a statistical problem.

It is a small example of mechanism design.

We are designing rules so that desirable behaviour emerges from the incentives faced by participants.

That idea will become increasingly important later in this course.

A forecast can be useful even when the unlikely thing happens

Suppose a model predicts:

[ P(\text{failure})=0.05 ]

The machine subsequently fails.

It would be tempting to say:

The forecast was wrong.

But a 5% event should happen approximately:

[ 1\text{ time in }20 ]

under comparable conditions.

The important question is not whether one particular low-probability event occurred.

It is whether the model's probabilities are systematically useful.

Across many predictions:

do 5% events occur about 5% of the time?
do 50% events occur about 50% of the time?
do 90% events occur about 90% of the time?

And does the model assign greater probability to outcomes that subsequently occur?

This is why scoring probabilistic forecasts requires many observations.

Scoring rules and calibration

Scoring rules connect closely to the previous lesson on calibration.

Calibration asks:

When a model predicts 70%, does the event actually happen about 70% of the time?

Scoring rules ask something slightly different:

How good was the entire set of probability forecasts?

A model can therefore be evaluated along several dimensions.

We might care about:

calibration,
discrimination,
sharpness,
overall probabilistic accuracy.

A good forecasting system should not merely produce probabilities.

Those probabilities should contain useful information.

Sharpness

Consider two weather models.

Model A

Almost every day:

[ P(\text{rain})=0.5 ]

Model B

Depending on conditions:

[ P(\text{rain})=0.05 ]

or:

[ P(\text{rain})=0.95 ]

If Model B's probabilities are well calibrated, it is much more informative.

Model A is effectively saying:

I don't know.

every day.

There is nothing inherently wrong with admitting uncertainty when uncertainty genuinely exists.

But if useful information exists that allows us to distinguish likely rain from unlikely rain, we would like the model to capture it.

This idea is sometimes described as sharpness.

A useful forecast should ideally be:

well calibrated and as informative as possible.

We do not want confidence for its own sake.

We want justified confidence.

Different scoring rules value mistakes differently

There is no single scoring rule appropriate for every problem.

The choice of scoring rule determines how different errors are treated.

For example, log loss strongly penalises predictions that assign extremely low probability to events that subsequently occur.

The Brier score treats probability errors quadratically.

Other scoring rules can evaluate:

continuous distributions,
quantile forecasts,
prediction intervals,
multiple possible categories.

The choice therefore depends partly on what kind of forecast we are evaluating.

But there is a deeper issue.

Whenever we choose a metric, we are making a decision about:

What counts as a good prediction?

That decision affects which models appear best.

Continuous forecasts

So far we have considered binary events such as:

[ \text{rain} ]

versus:

[ \text{no rain} ]

But many forecasts concern continuous quantities:

temperature,
electricity demand,
wind generation,
rainfall,
travel time,
asset prices.

A probabilistic model might predict an entire distribution:

[ F(y) ]

rather than one probability.

We therefore need scoring rules capable of comparing a predicted distribution with the value that eventually occurs.

One example is the Continuous Ranked Probability Score, or CRPS.

Conceptually, CRPS evaluates how closely the predicted probability distribution aligns with the observed outcome.

You do not need to memorise the formula here.

The important point is that we can score:

[ \text{entire probability distributions} ]

rather than merely point predictions.

Scoring prediction intervals

Suppose Model A predicts tomorrow's electricity demand as:

90% interval: 39–45 GW

Model B predicts:

90% interval: 0–100 GW

Tomorrow's demand is:

[ 42\text{ GW} ]

Both intervals contain the observed outcome.

Does that mean both forecasts were equally good?

Clearly not.

Model B has protected itself by giving an almost uselessly wide interval.

A good scoring system for prediction intervals should therefore balance two things:

coverage — did the interval contain the outcome as often as intended?
sharpness — was the interval reasonably narrow?

Otherwise a forecaster could always say:

Somewhere between negative infinity and positive infinity.

Technically, the outcome would always fall inside the interval.

Practically, the forecast would tell us nothing.

The metric changes the model

This connects back to something we encountered when discussing loss functions.

Before training a model, somebody has to decide:

What counts as being wrong?

Suppose Model A is optimised for:

[ \text{classification accuracy} ]

Model B is optimised for:

[ \text{log loss} ]

Model C is optimised for:

[ \text{Brier score} ]

They may learn different behaviours.

The evaluation function is therefore not merely something we apply after modelling.

It can influence what the model learns in the first place.

This gives us a recurring chain:

[ \text{objective} \rightarrow \text{incentive} \rightarrow \text{behaviour} ]

That applies to machines.

It applies to organisations.

And it applies to people.

Goodhart's Law

There is a famous observation often summarised as:

When a measure becomes a target, it can cease to be a good measure.

Imagine a school evaluated entirely on examination pass rates.

It may begin teaching narrowly to the test.

A hospital evaluated entirely on waiting times may reorganise behaviour around the waiting-time metric.

A social-media algorithm optimised entirely for engagement may discover that outrage is extremely engaging.

An AI benchmark becomes prestigious, and developers begin optimising specifically for performance on that benchmark.

The metric begins shaping the system.

This is why evaluation deserves its own module.

Choosing how to measure success is not a boring administrative detail.

It can fundamentally change what the system does.

Scoring forecasts in high-stakes systems

Imagine an electricity system forecasting the probability of shortage.

Model A predicts:

[ P(\text{shortage})=0.001 ]

Model B predicts:

[ P(\text{shortage})=0.01 ]

Suppose a shortage occurs.

A scoring rule may tell us Model B made the better probabilistic forecast.

But that still does not tell us whether the electricity system should have prepared additional reserves.

That depends on:

the cost of reserves,
the cost of shortages,
reliability requirements,
available resources,
acceptable risk.

This distinction matters.

A scoring rule evaluates the forecast.

A decision rule determines the action.

These are not the same thing.

Prediction quality and decision quality

Consider the full chain:

[ \text{world} \rightarrow \text{data} \rightarrow \text{model} \rightarrow \text{forecast} \rightarrow \text{decision} \rightarrow \text{action} ]

A scoring rule evaluates one particular part:

[ \text{forecast} \leftrightarrow \text{observed outcome} ]

But an excellent forecast can still produce a terrible outcome if the subsequent decision mechanism is badly designed.

Likewise, a decision system may sometimes perform reasonably despite imperfect forecasts.

We therefore need to distinguish:

[ \text{forecast quality} ]

from:

[ \text{decision quality} ]

This distinction becomes extremely important as AI systems move from merely predicting the world to acting within it.

Can we score an AI's confidence?

This becomes particularly interesting for modern AI systems.

Suppose an AI answers:

The capital of Australia is Sydney.

and says:

Confidence: 99%.

The answer is wrong.

That should be treated differently from:

I think the answer may be Sydney, but I'm uncertain.

A genuinely useful AI system would ideally be able to associate its answers with meaningful uncertainty.

Across many questions, statements associated with:

[ 90% ]

confidence should be correct roughly:

[ 90% ]

of the time.

We could then evaluate those confidence estimates using scoring rules.

But this is difficult.

The probability a language model assigns to generating a sequence of words is not necessarily the same thing as:

[ P(\text{statement is true}) ]

That distinction will become important later when we examine large language models, hallucination and reasoning.

The deeper lesson

Scoring rules may initially look like a technical detail.

They are much more interesting than that.

They force us to confront a fundamental question:

What exactly do we mean by a good prediction?

Is it enough to choose the most likely outcome?

Should uncertainty matter?

Should confident mistakes be punished more severely?

Should a forecaster be rewarded for honestly expressing uncertainty?

How do we prevent systems from gaming the metric?

These are questions about mathematics.

But they are also questions about incentives and system design.

A scoring rule converts an abstract objective:

Make good forecasts.

into a mathematical signal:

[ \text{forecast} + \text{outcome} \rightarrow \text{score} ]

And once a score determines rewards, training, selection or behaviour, the scoring rule becomes part of the system itself.

That gives us another principle for the course:

You do not merely get the behaviour you ask for. You tend to get the behaviour your measurement and incentive system rewards.

So whenever somebody tells you:

This AI model is better.

there is an extremely important question to ask:

Better according to what score?