Mean absolute error

In the previous lesson, we looked at accuracy.

Accuracy makes intuitive sense when a model predicts categories:

• spam or not spam;
• disease or no disease;
• fraud or legitimate transaction;
• cat or dog.

But many predictions aren't categories.

Instead, we might want to predict a number.

For example:

• tomorrow's temperature;
• the price of a house;
• electricity demand at 6pm;
• how long a journey will take;
• how much energy a wind farm will generate;
• how many customers will visit a shop;
• how much rainfall will occur tomorrow.

Suppose the actual temperature tomorrow is:

18°C

and our model predicts:

20°C

The model wasn't simply "right" or "wrong".

It was wrong by:

2°C

For numerical predictions, therefore, we need a way of measuring how far predictions are from reality.

One of the simplest ways of doing this is Mean Absolute Error.

---

Prediction error

Suppose we want to predict tomorrow's temperature.

Our model predicts:

$$\hat{y} = 20$$

and the actual temperature turns out to be:

$$y = 18$$

The prediction error is:

$$e = y - \hat{y}$$

so:

$$e = 18 - 20 = -2$$

The negative sign tells us something useful.

The model overpredicted the temperature by 2°C.

If instead the model predicted 16°C:

$$e = 18 - 16 = 2$$

The positive error tells us the model underpredicted by 2°C.

So the sign of the error tells us the direction of the mistake.

But sometimes we simply want to know:

How wrong was the prediction?

For that, we can ignore the direction and measure the size of the error.

---

Absolute error

The absolute error is:

$$|y - \hat{y}|$$

The vertical bars mean:

take the absolute value.

In other words, ignore whether the error is positive or negative.

If:

$$y = 18$$

and:

$$\hat{y} = 20$$

then:

$$|18 - 20| = 2$$

If instead:

$$\hat{y} = 16$$

then:

$$|18 - 16| = 2$$

In both cases, the prediction is 2°C away from reality.

That is the absolute error.

---

Why use the absolute value?

Imagine a model makes four predictions with the following errors:

Prediction	Error
1	+4
2	-4
3	+3
4	-3

If we simply calculate the average error:

$$\frac{4 - 4 + 3 - 3}{4} = 0$$

we might conclude:

Average error = 0

That sounds like perfect performance.

But the model hasn't made a single perfect prediction.

The positive and negative errors have simply cancelled each other out.

This is why measuring prediction error requires some care.

Taking the absolute value prevents this cancellation:

Prediction	Error	Absolute error
1	+4	4
2	-4	4
3	+3	3
4	-3	3

Now the errors can be meaningfully averaged.

---

Mean Absolute Error

For many predictions, we calculate the absolute error for each prediction and then take the average.

This gives us the Mean Absolute Error, usually abbreviated to MAE.

$$\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i-\hat{y}_i|$$

where:

• $n$ is the number of predictions;
• $y_i$ is the actual value;
• $\hat{y}_i$ is the predicted value.

Despite the notation, the idea is extremely simple:

Calculate how far every prediction was from reality, ignore the direction of the mistake, and take the average.

---

A simple example

Suppose we are predicting the temperature over five days.

Day	Actual	Predicted	Absolute error
Monday	18°C	20°C	2°C
Tuesday	20°C	19°C	1°C
Wednesday	17°C	20°C	3°C
Thursday	16°C	16°C	0°C
Friday	19°C	17°C	2°C

The total absolute error is:

$$2 + 1 + 3 + 0 + 2 = 8$$

There are five predictions.

Therefore:

$$\text{MAE} = \frac{8}{5} = 1.6$$

The model's Mean Absolute Error is:

1.6°C

This has a wonderfully intuitive interpretation:

On average, the model's predictions were 1.6°C away from the actual temperature.

---

MAE stays in the original units

One particularly useful property of Mean Absolute Error is that it has the same units as the thing being predicted.

If we predict temperature:

MAE = 1.6°C

If we predict house prices:

MAE = £24,000

If we predict journey times:

MAE = 6.4 minutes

If we predict electricity demand:

MAE = 120 MW

This makes MAE relatively easy to explain to people who aren't machine-learning specialists.

Imagine telling an electricity system operator:

"The forecasting model has an MAE of 180 MW."

That has a physical interpretation.

The model's forecasts differ from actual demand by around 180 MW on average.

Compare that with being told:

"The model has an error score of 0.037."

Without additional context, that number means very little.

Interpretability matters.

---

Smaller is better

Unlike accuracy, where larger values generally indicate better performance, Mean Absolute Error works in the opposite direction.

An MAE of:

$$0$$

would mean every prediction was exactly correct.

As errors increase, MAE increases.

So:

Lower MAE generally means more accurate numerical predictions.

But there is an important caveat.

Whether an MAE is actually good depends entirely on what is being predicted.

An MAE of £10,000 might be excellent when predicting £5 million houses.

It would be terrible when predicting £20,000 cars.

An error of 100 MW might be tiny when forecasting the electricity demand of an entire country.

It could be enormous when forecasting demand for a small town.

The number needs context.

---

Compare against a baseline

The lesson from accuracy applies here too.

A model should not be judged in isolation.

Suppose an electricity-demand forecasting model has:

MAE = 250 MW

Is that good?

We don't know.

Perhaps the existing forecasting method has:

MAE = 400 MW

In that case, the new model represents a substantial improvement.

But perhaps a simple model that predicts:

Demand tomorrow will be the same as demand today

achieves:

MAE = 180 MW

Then our sophisticated machine-learning model isn't particularly useful.

Again, the right question isn't simply:

What is the model's error?

It is:

How does the model perform compared with a sensible alternative?

---

MAE treats errors linearly

There is an important property hidden inside Mean Absolute Error.

Consider two errors:

10

and:

20

The second contributes exactly twice as much to MAE as the first.

Similarly:

• an error of 5 contributes 5;
• an error of 10 contributes 10;
• an error of 50 contributes 50;
• an error of 100 contributes 100.

The penalty grows linearly with the size of the mistake.

We can think of the relationship as:

$$\text{Penalty} = |\text{Error}|$$

Doubling the error doubles its contribution to MAE.

This sounds obvious.

But it represents a design choice.

---

Are errors really linear in consequence?

Suppose an electricity-demand forecast is wrong by:

100 MW

Perhaps that causes a small operational inconvenience.

What if it is wrong by:

1,000 MW?

Is that simply ten times worse?

Perhaps.

But perhaps the larger error creates disproportionately serious consequences.

The system might need expensive emergency reserves.

Network constraints might become binding.

Reliability could be threatened.

The economic cost might increase much faster than the prediction error itself.

The same issue appears elsewhere.

A journey-time prediction that is wrong by 5 minutes might be mildly annoying.

One that is wrong by 50 minutes could cause someone to miss a flight.

A medical dosage estimate being slightly wrong might have little consequence.

A very large error could be dangerous.

This raises an important question:

Should an error twice as large always be considered exactly twice as bad?

Mean Absolute Error effectively says:

yes.

At least mathematically.

---

Large errors and small errors

Consider two models.

Model A

Its absolute errors are:

$$4,\ 4,\ 4,\ 4,\ 4$$

Therefore:

$$\text{MAE} = 4$$

Model B

Its absolute errors are:

$$0,\ 0,\ 0,\ 0,\ 20$$

Its MAE is also:

$$\text{MAE} = 4$$

According to Mean Absolute Error, these models have identical average performance.

But they behave very differently.

Model A is consistently a little wrong.

Model B is usually perfect but occasionally very wrong.

Which model would you prefer?

That depends on the application.

For some systems, occasional large errors may be acceptable.

For others, they may be catastrophic.

MAE alone doesn't tell us which behaviour we should prefer.

---

Average performance can hide extremes

This brings us back to a recurring theme.

Averages compress information.

Suppose a model has:

MAE = 2.3

That tells us something useful about its typical error.

But it doesn't tell us:

• the largest error;
• how frequently large errors occur;
• whether errors are concentrated at particular times;
• whether the model systematically overpredicts;
• whether it systematically underpredicts;
• whether certain populations experience larger errors;
• whether performance deteriorates under unusual conditions.

Two models can therefore have the same MAE while behaving very differently.

This is why serious model evaluation usually involves examining multiple metrics and the distribution of errors, rather than reporting a single number.

---

Direction can still matter

Remember that MAE deliberately removes the sign of the error.

That is useful when we care about the magnitude of prediction mistakes.

But sometimes direction matters too.

Suppose an electricity-demand forecasting model consistently underestimates demand.

Its errors might look like:

$$+100,\ +120,\ +90,\ +110,\ +130$$

The MAE tells us the average size of those mistakes.

But it doesn't immediately tell us that the model is systematically biased in one direction.

Similarly, consistently overestimating:

• rainfall;
• medical risk;
• delivery times;
• electricity generation;
• financial losses

may have different consequences from consistently underestimating them.

MAE answers:

How far away were we?

It does not answer:

Which direction were we wrong?

Both questions can matter.

---

Different errors may have different costs

There is an even deeper limitation.

MAE assumes that an error of a particular size has the same importance regardless of when or where it occurs.

Suppose an electricity-demand forecast is wrong by 500 MW.

At 3am on a mild summer night, that error might be relatively easy to manage.

At 6pm during a freezing winter evening when the system is already close to its limits, exactly the same forecasting error could be much more consequential.

Mathematically:

$$|\text{Error}| = 500\text{ MW}$$

in both cases.

But the system consequence is different.

This is a fundamental distinction:

Prediction error and consequence are not the same thing.

A metric measures some mathematical property of prediction performance.

Whether that property corresponds to what we actually care about depends on the system in which the prediction is used.

---

Why MAE is still extremely useful

None of these limitations make Mean Absolute Error a bad metric.

Quite the opposite.

MAE is widely useful precisely because it asks a very simple and interpretable question:

How far away from reality are our predictions, on average?

It is:

• easy to calculate;
• easy to understand;
• expressed in the original units;
• resistant to positive and negative errors cancelling;
• straightforward to compare between models predicting the same quantity.

For many problems, that is exactly the information we want.

The important thing is to understand what the metric measures and what it doesn't.

---

From measurement to values

There is a deeper theme emerging across this course.

When we train a machine-learning system, we choose a loss function.

When we evaluate it, we choose an evaluation metric.

Neither choice is entirely neutral.

Choosing MAE means saying:

I care about the average absolute distance between prediction and reality.

That might be perfectly appropriate.

But another application might care much more about rare, very large errors.

Another might care more about underprediction than overprediction.

Another might care primarily about whether errors cross some critical threshold.

The mathematics doesn't independently tell us which of these matters.

We have to decide what kind of mistake we care about.

---

The bigger idea

Mean Absolute Error gives us a simple way of evaluating numerical predictions:

$$\text{Prediction} \rightarrow \text{Error} \rightarrow \text{Absolute error} \rightarrow \text{Average error}$$

It answers:

How far from reality are our predictions, on average?

That is enormously useful.

But consider our two models again.

One makes lots of small mistakes.

The other is usually perfect but occasionally makes an enormous mistake.

Mean Absolute Error may judge them equally.

Should it?

That depends on whether we believe large mistakes should be disproportionately costly.

And that leads directly to our next evaluation metric:

Mean Squared Error

Instead of asking every unit of error to count equally, what happens if we make large errors hurt much more than small ones?