Confusion matrices

In the previous lessons, we introduced four possible outcomes for a binary classifier:

• true positives;
• true negatives;
• false positives;
• false negatives.

We also saw that:

• precision depends on false positives;
• recall depends on false negatives;
• different kinds of mistakes can have very different consequences.

A confusion matrix brings all four outcomes together in one place.

Despite the slightly unfortunate name, it is one of the clearest tools for understanding how a classification model is actually behaving.

Instead of reducing performance to a single number, a confusion matrix shows us the structure of the model's decisions.

---

The basic idea

Suppose a model predicts one of two classes:

• positive;
• negative.

Reality can also be:

• positive;
• negative.

We can arrange these possibilities in a table.

	Predicted Positive	Predicted Negative
Actual Positive	True Positive	False Negative
Actual Negative	False Positive	True Negative

This is a confusion matrix.

It compares:

what actually happened

with:

what the model predicted

The four cells tell us exactly how the classifier succeeded and failed.

---

Reading the matrix

Let's label the four outcomes.

	Predicted Positive	Predicted Negative
Actual Positive	TP	FN
Actual Negative	FP	TN

where:

• TP = true positive;
• TN = true negative;
• FP = false positive;
• FN = false negative.

The diagonal cells contain the correct predictions:

• true positives;
• true negatives.

The off-diagonal cells contain the mistakes:

• false positives;
• false negatives.

So at a glance, the matrix separates:

correct predictions

from:

incorrect predictions

while also showing which kind of mistake occurred.

---

A medical example

Suppose 1,000 people are screened for a disease.

In reality:

• 100 have the disease;
• 900 do not.

The model produces the following results:

• 80 true positives;
• 20 false negatives;
• 60 false positives;
• 840 true negatives.

The confusion matrix is:

	Predicted Disease	Predicted No Disease
Actual Disease	80	20
Actual No Disease	60	840

Now we can immediately see much more than a single accuracy figure would tell us.

The model:

• detected 80 sick patients;
• missed 20 sick patients;
• incorrectly flagged 60 healthy patients;
• correctly reassured 840 healthy patients.

That is a much richer description of behaviour.

---

Accuracy from the confusion matrix

Accuracy counts all correct predictions.

So:

$$\text{Accuracy} = \frac{TP+TN} {TP+TN+FP+FN}$$

Using our example:

$$\text{Accuracy} = \frac{80+840} {80+840+60+20}$$ $$= \frac{920}{1000}$$ $$= 92\%$$

So the model is:

92% accurate

That sounds good.

But the confusion matrix tells us something accuracy alone does not:

The model missed 20% of the people who actually had the disease.

That may or may not be acceptable.

---

Recall from the confusion matrix

Recall asks:

Of all the real positive cases, how many did we detect?

So:

$$\text{Recall} = \frac{TP} {TP+FN}$$

Using our example:

$$\text{Recall} = \frac{80} {80+20}$$ $$= 80\%$$

This means:

The model detected 80% of the patients who actually had the disease.

Equivalently:

It missed 20%.

---

Precision from the confusion matrix

Precision asks:

When the model predicted positive, how often was it correct?

So:

$$\text{Precision} = \frac{TP} {TP+FP}$$

Using our example:

$$\text{Precision} = \frac{80} {80+60}$$ $$= \frac{80}{140}$$ $$\approx 57.1\%$$

So when the system says:

DISEASE

it is correct only about:

57% of the time

That may be acceptable for an early screening tool.

It might be unacceptable for an automated final diagnosis.

Again, context matters.

---

One matrix, many metrics

This is one reason confusion matrices are so useful.

From the same four numbers:

• TP;
• TN;
• FP;
• FN;

we can calculate several evaluation metrics.

These include:

• accuracy;
• precision;
• recall;
• specificity;
• false-positive rate;
• false-negative rate;
• F1 score.

The confusion matrix acts like the underlying accounting table.

The metrics are different ways of summarising it.

---

Specificity

Recall tells us how well the model identifies positives.

There is a corresponding metric for negatives.

It is called specificity.

Specificity asks:

Of all the actual negative cases, how many did the model correctly identify as negative?

Its formula is:

$$\text{Specificity} = \frac{TN} {TN+FP}$$

Using our example:

$$\text{Specificity} = \frac{840} {840+60}$$ $$= \frac{840}{900}$$ $$\approx 93.3\%$$

So the model correctly identifies about 93% of healthy patients as healthy.

---

False-positive rate

The false-positive rate asks:

Of all the actual negative cases, how many were incorrectly predicted as positive?

Its formula is:

$$\text{False Positive Rate} = \frac{FP} {FP+TN}$$

Using our example:

$$\text{FPR} = \frac{60}{60+840}$$ $$= \frac{60}{900}$$ $$\approx 6.7\%$$

Notice that:

$$\text{Specificity} + \text{False Positive Rate} = 1$$

So:

$$\text{FPR} = 1-\text{Specificity}$$

---

False-negative rate

Similarly, the false-negative rate asks:

Of all the actual positive cases, how many did the model miss?

Its formula is:

$$\text{False Negative Rate} = \frac{FN} {FN+TP}$$

Using our example:

$$\text{FNR} = \frac{20}{20+80}$$ $$= 20\%$$

And because recall measures the proportion of positives we correctly found:

$$\text{Recall} + \text{False Negative Rate} = 1$$

So:

$$\text{FNR} = 1-\text{Recall}$$

---

Why the matrix is so revealing

Suppose two models both achieve:

95% accuracy

At first glance, they appear equally good.

But their confusion matrices could look very different.

Model A

	Predicted Positive	Predicted Negative
Actual Positive	95	5
Actual Negative	45	855

Model B

	Predicted Positive	Predicted Negative
Actual Positive	55	45
Actual Negative	5	895

Both models make:

50 mistakes

out of:

1,000 predictions

So both have:

95% accuracy

But Model A makes mostly false-positive errors.

Model B makes mostly false-negative errors.

If the task is medical screening, those systems could have radically different consequences.

The same headline performance can hide very different behaviour.

---

Confusion matrices expose imbalance

Return to the rare-disease example.

Suppose 10,000 people are screened.

Only 100 have the disease.

Imagine a model that predicts:

NO DISEASE

for everyone.

Its confusion matrix is:

	Predicted Disease	Predicted No Disease
Actual Disease	0	100
Actual No Disease	0	9,900

Its accuracy is:

$$\frac{9900}{10000} = 99\%$$

That looks excellent.

But the confusion matrix makes the failure obvious.

The model detects:

zero cases of disease

It has:

$$TP = 0$$

and:

$$FN = 100$$

Its recall is:

$$0\%$$

The confusion matrix prevents the 99% accuracy figure from hiding the problem.

---

Counts versus rates

Confusion matrices usually show counts.

For example:

	Predicted Positive	Predicted Negative
Actual Positive	80	20
Actual Negative	60	840

But sometimes it is useful to convert these counts into rates or percentages.

For example:

	Predicted Positive	Predicted Negative
Actual Positive	80%	20%
Actual Negative	6.7%	93.3%

This makes it easier to compare performance when the number of examples differs.

Counts answer:

How many cases were affected?

Rates answer:

What proportion of cases were affected?

Both are useful.

---

Scale matters

Suppose a false-positive rate is:

$$0.1\%$$

That sounds tiny.

If the model processes:

1,000 cases

that means roughly:

1 false positive

But if the model processes:

1 billion cases

that means roughly:

1 million false positives

So a confusion matrix should be interpreted in the context of scale.

A small rate applied to a huge system can still create a huge operational burden.

---

The matrix depends on the threshold

Suppose a classifier outputs probabilities.

For example:

Case	Probability of Positive
A	0.95
B	0.81
C	0.72
D	0.63
E	0.49
F	0.31
G	0.18

If we use the threshold:

$$0.5$$

then A, B, C and D are classified as positive.

But if we use:

$$0.8$$

only A and B are classified as positive.

The model has not changed.

But the confusion matrix changes.

That means:

• TP changes;
• TN changes;
• FP changes;
• FN changes.

So every threshold creates a different confusion matrix.

This is a very important point.

A confusion matrix describes a classifier at a particular operating threshold.

It does not necessarily describe the full behaviour of the underlying prediction model.

---

Changing the threshold moves errors around

Suppose we lower the decision threshold.

More cases become positive.

Usually:

• true positives increase;
• false positives increase;
• false negatives decrease;
• true negatives decrease.

If we raise the threshold:

• true positives decrease;
• false positives decrease;
• false negatives increase;
• true negatives increase.

So moving the threshold shifts the balance between different errors.

We can think of it as moving predictions between cells in the confusion matrix.

---

There is rarely a perfect threshold

Imagine a fraud-detection system.

If we lower the threshold far enough, we might catch almost every fraudulent transaction.

Recall becomes very high.

But many legitimate transactions may also be flagged.

False positives rise.

If we raise the threshold far enough, we may almost eliminate false alarms.

But more fraudulent transactions escape.

False negatives rise.

There is usually no magical threshold where:

• false positives disappear;
• false negatives disappear;
• precision is perfect;
• recall is perfect.

Real classification problems often involve overlap and uncertainty.

The system designer must therefore choose an acceptable operating point.

---

The confusion matrix is a snapshot

This gives us a useful way to think about confusion matrices.

A confusion matrix is not necessarily the whole story.

It is a snapshot of classifier behaviour at a particular threshold.

Change the threshold and you get a different snapshot.

That raises a natural question:

Instead of looking at only one threshold, can we examine how the classifier behaves across all possible thresholds?

Yes.

That is exactly what an ROC curve helps us do.

But before we get there, there is another useful idea to understand.

---

Confusion matrices for more than two classes

So far, we have considered binary classification.

But models often predict more than two categories.

Imagine an image classifier predicting:

• cat;
• dog;
• horse;
• bird.

A confusion matrix can be extended.

For example:

Actual \ Predicted	Cat	Dog	Horse	Bird
Cat	90	7	1	2
Dog	6	88	3	3
Horse	1	4	92	3
Bird	3	2	4	91

The diagonal still represents correct predictions.

For example:

90 cats were correctly classified as cats.

The off-diagonal cells show the specific confusions.

For example:

7 cats were classified as dogs.

This can reveal patterns that overall accuracy cannot.

Perhaps the model frequently confuses:

• cats with dogs;
• one medical condition with another;
• similar manufacturing defects;
• visually similar road signs.

The confusion matrix tells us what the model is confusing with what.

That is where the name comes from.

---

Some mistakes are more understandable than others

Suppose an animal classifier occasionally confuses:

wolf → dog

That may be understandable.

But if it frequently predicts:

wolf → toaster

something more fundamental may be wrong.

A multiclass confusion matrix can reveal this structure.

It can help us investigate:

• poor training examples;
• ambiguous classes;
• insufficient features;
• data-quality problems;
• overlapping categories;
• model limitations.

Evaluation can therefore help us diagnose why a model performs poorly, not merely measure how poorly it performs.

---

Confusion matrices and fairness

Confusion matrices can also be calculated separately for different populations.

Suppose an overall model has:

90% accuracy

But we create separate confusion matrices for different groups.

We may discover that:

• one group experiences more false positives;
• another experiences more false negatives;
• recall differs significantly;
• specificity differs significantly.

The overall matrix may have hidden those differences.

This becomes especially important when predictions affect access to:

• credit;
• healthcare;
• employment;
• education;
• insurance;
• public services.

A single aggregate metric may not be enough.

Later in the course, when we study fairness, we will return to this idea in much more detail.

---

The confusion matrix as an accounting system

A useful way to think about a confusion matrix is as an accounting system for classification decisions.

Every prediction has to go somewhere.

It must be one of:

$$TP,\ TN,\ FP,\ FN$$

And therefore:

$$N = TP+TN+FP+FN$$

where:

$$N$$

is the total number of predictions.

From those four quantities, many classification metrics can be derived.

The confusion matrix is therefore more fundamental than many of the headline numbers built from it.

---

But it still doesn't tell us everything

Even a confusion matrix leaves important questions unanswered.

It does not directly tell us:

• how confident each prediction was;
• whether probabilities are well calibrated;
• how performance changes with the threshold;
• whether performance changes over time;
• whether performance changes across locations;
• whether mistakes have different real-world costs.

It describes what happened.

It does not fully describe:

why it happened

or:

what would happen if we changed the decision rule.

That is where other evaluation tools become useful.

---

The deeper lesson

Accuracy compresses classification performance into one number.

A confusion matrix expands that number back into its components.

Instead of asking:

How many predictions were correct?

we can ask:

• How many positives did we correctly detect?
• How many negatives did we correctly reject?
• How many false alarms did we create?
• How many real cases did we miss?

This turns evaluation from a scoreboard into a diagnosis.

And that matters because different systems care about different cells.

A cancer-screening system may care enormously about:

false negatives

A fraud system may care about both:

false positives and false negatives

A spam filter may strongly prioritise avoiding:

false positives

The matrix itself is neutral.

The importance we attach to each cell comes from the system around it.

---

The bigger idea

A confusion matrix gives us a structured picture of classification behaviour:

$$\text{Reality} \times \text{Prediction} \rightarrow TP,\ TN,\ FP,\ FN$$

From there we can calculate:

• accuracy;
• precision;
• recall;
• specificity;
• false-positive rate;
• false-negative rate.

But remember:

The confusion matrix changes when the classification threshold changes.

So a single confusion matrix tells us how the model behaves at one chosen operating point.

What if we want to understand its behaviour across many possible thresholds?

What if we want to see the trade-off between:

• detecting more real positives;
• and creating more false positives?

That leads us to the next tool:

ROC Curves

Rather than evaluate the classifier at one threshold, we can trace how its behaviour changes as the threshold moves.