Module 5 — When is a prediction good?
Lesson 4 of 14
Precision and Recall
So far, we have looked at ways of evaluating predictions of numbers.
Mean Absolute Error asks:
How far away from reality are our predictions, on average?
Mean Squared Error asks:
How large are our errors if we want to penalise big mistakes more heavily?
But many machine-learning systems do not predict numbers.
They predict categories.
For example:
- fraud or legitimate;
- disease or no disease;
- spam or not spam;
- pedestrian or no pedestrian;
- defective or not defective.
In these cases, the important question is not simply:
How many predictions were correct?
We also need to ask:
When the model says something is positive, how often is it right?
and:
Of all the positive cases that really exist, how many did the model actually find?
These two questions lead us to two of the most important classification metrics in machine learning:
precision and recall.
First, what does "positive" mean?
In classification, one class is often treated as the positive class.
The word "positive" does not necessarily mean good.
It simply means:
the thing we are trying to detect
For example:
| Problem | Positive class |
|---|---|
| Fraud detection | Fraud |
| Cancer screening | Cancer present |
| Spam detection | Spam |
| Fault detection | Fault present |
| Pedestrian detection | Pedestrian present |
The other class is called the negative class.
So in fraud detection:
- positive = fraud;
- negative = legitimate transaction.
In medical screening:
- positive = disease present;
- negative = disease absent.
This terminology lets us describe different kinds of prediction outcomes precisely.
Four possible outcomes
Suppose a model predicts whether someone has a disease.
There are four possibilities.
True positive
The model predicts:
Disease
and the person actually has the disease.
The prediction is correct.
This is called a:
True Positive
or:
TP
True negative
The model predicts:
No disease
and the person does not have the disease.
The prediction is correct.
This is called a:
True Negative
or:
TN
False positive
The model predicts:
Disease
but the person does not actually have the disease.
The prediction is wrong.
This is called a:
False Positive
or:
FP
False negative
The model predicts:
No disease
but the person actually does have the disease.
The prediction is wrong.
This is called a:
False Negative
or:
FN
Together, these four outcomes give us the basic language of classification evaluation.
| Reality | Prediction | Outcome |
|---|---|---|
| Positive | Positive | True Positive |
| Negative | Negative | True Negative |
| Negative | Positive | False Positive |
| Positive | Negative | False Negative |
These four quantities will appear repeatedly throughout the rest of this module.
Precision
Precision asks:
When the model predicts positive, how often is it correct?
Its formula is:
or:
The denominator contains everything the model predicted as positive.
Precision then asks:
What fraction of those positive predictions were actually positive?
A simple precision example
Suppose a fraud-detection model flags 100 transactions as fraudulent.
Of those:
- 80 really are fraud;
- 20 are legitimate transactions.
Then:
and:
So:
or:
80% precision
This means:
When the model says "fraud", it is right 80% of the time.
Another way to say it is:
20% of its fraud alerts are false alarms.
High precision means fewer false alarms
A model with high precision does not make many false-positive predictions.
Suppose an email spam filter has extremely high precision.
When it says:
SPAM
it is almost always correct.
That means legitimate emails are rarely sent to the spam folder.
This may be particularly important if false positives are costly.
Imagine a spam filter that incorrectly hides:
- an employment offer;
- an invoice;
- a hospital appointment;
- an important message from a customer.
In such a system, we may care strongly about precision.
We do not want the system confidently labelling legitimate things as positive.
Recall
Recall asks a different question:
Of all the positive cases that really exist, how many did the model find?
Its formula is:
or:
The denominator now contains all the cases that are actually positive.
Recall asks:
What fraction of the real positive cases did our model successfully detect?
A simple recall example
Suppose there are actually:
100 fraudulent transactions
in a dataset.
The model successfully detects:
80
but misses:
20
Then:
and:
So:
or:
80% recall
This means:
The model detected 80% of all the fraud that actually occurred.
It missed the remaining 20%.
High recall means fewer missed positives
A model with high recall does not make many false-negative predictions.
This matters when missing a real positive case is particularly costly.
Consider cancer screening.
Suppose 100 patients genuinely have a disease.
A system with:
99% recall
detects 99 of them and misses 1.
A system with:
60% recall
detects 60 and misses 40.
Even if both systems had reasonable overall accuracy, their practical usefulness could be dramatically different.
In some applications, the central objective is:
Do not miss the thing we are looking for.
That means prioritising recall.
Precision and recall ask different questions
This distinction is crucial.
Precision looks at the model's positive predictions.
Recall looks at the world's actual positive cases.
Precision asks:
When I say yes, how often am I right?
Recall asks:
Of all the real yeses, how many did I find?
They sound similar.
They are not.
A medical example
Suppose 1,000 patients are screened.
In reality:
- 100 have the disease;
- 900 do not.
The model predicts disease for 80 people.
Of those 80:
- 70 genuinely have the disease;
- 10 do not.
That means:
But there were 100 people with the disease in total.
So the model missed:
Its precision is:
or:
87.5%
Its recall is:
or:
70%
So we can say:
When the model predicts disease, it is correct 87.5% of the time.
But:
It only detects 70% of the people who actually have the disease.
Those are very different statements about model performance.
The precision–recall trade-off
There is often a trade-off between precision and recall.
Suppose a model assigns every transaction a probability of fraud.
For example:
| Transaction | Predicted fraud probability |
|---|---|
| A | 0.98 |
| B | 0.91 |
| C | 0.82 |
| D | 0.67 |
| E | 0.55 |
| F | 0.43 |
| G | 0.28 |
| H | 0.12 |
We need some threshold above which the system says:
Fraud
Suppose we choose:
Only transactions A and B are flagged.
The model is being very cautious.
Those alerts may be highly reliable.
So precision may be high.
But many fraudulent transactions with lower predicted probabilities could be missed.
So recall may be low.
Lowering the threshold
Now suppose we lower the threshold to:
The system now flags:
- A;
- B;
- C;
- D;
- E;
- F.
It will probably detect more genuine fraud.
So recall increases.
But it may also flag more legitimate transactions.
So precision may decrease.
This produces a common pattern:
Increasing recall often reduces precision.
and:
Increasing precision often reduces recall.
The exact relationship depends on the model and the data, but the trade-off appears frequently.
A fishing analogy
Imagine you are trying to catch a particular species of fish.
You cast a very small, highly selective net.
Almost everything you catch is the species you want.
That is like high precision.
But many of the target fish escape.
That means low recall.
Now use an enormous net.
You catch almost every target fish.
That is high recall.
But you also catch many fish you did not want.
That means lower precision.
The question becomes:
Would you rather catch almost everything and accept more false alarms, or be very selective and risk missing things?
The correct answer depends on the application.
When precision matters most
High precision is particularly valuable when false positives are costly.
For example:
Spam detection
If a system labels an email as spam, we want strong confidence that it really is spam.
Otherwise important messages disappear.
Automated accusations of fraud
If an account is frozen whenever the system predicts fraud, false positives can seriously inconvenience innocent customers.
Content moderation
Automatically removing legitimate content because a model incorrectly labels it harmful may create significant costs.
Predictive maintenance
If every predicted fault causes an expensive shutdown and inspection, false alarms can become extremely costly.
In these cases, we may want the model to be cautious before declaring something positive.
When recall matters most
High recall becomes particularly important when false negatives are costly.
For example:
Medical screening
Missing a disease may have serious health consequences.
Fire detection
A system that rarely raises false alarms but frequently misses actual fires would be dangerous.
Cybersecurity
Missing a genuine intrusion might be much more costly than investigating some harmless activity.
Safety systems
An autonomous vehicle perception system should be extremely reluctant to miss an actual pedestrian.
In such cases, the priority may be:
Find as many genuine positive cases as possible.
Even if that means accepting more false alarms.
Context determines which matters more
There is no universal rule saying:
precision is more important
or:
recall is more important.
Imagine a security system scanning bags at an airport.
A suspicious bag is the positive class.
If the system has low recall, dangerous items could be missed.
That is serious.
But if we tried to achieve perfect recall by flagging every single bag, precision would become terrible.
Every passenger would require manual inspection.
The airport would effectively stop functioning.
So the real question is not:
Should we maximise precision or recall?
It is:
What balance between them produces an acceptable system?
This is a design problem.
Can we get 100% recall?
Yes.
There is a very easy way.
Simply predict:
POSITIVE
for everything.
Suppose we are detecting fraud.
If every transaction is labelled:
FRAUD
then we are guaranteed to detect every fraudulent transaction.
There can be no false negatives.
Therefore:
Fantastic?
Not really.
Every legitimate transaction is also labelled fraudulent.
Precision could therefore be extremely poor.
This is another reminder that:
A metric can look excellent while the overall system is useless.
Can we get very high precision?
We could take the opposite approach.
Only predict positive when the model is almost completely certain.
Perhaps we flag only cases with:
Most of those predictions might indeed be correct.
Precision could be extremely high.
But we might detect only a tiny fraction of the real positive cases.
Recall would be poor.
Again, optimising one metric in isolation can produce undesirable behaviour.
Accuracy, precision and recall
We now have three different questions.
Accuracy
How many predictions were correct overall?
Precision
When the model predicts positive, how often is it right?
Recall
Of all the genuinely positive cases, how many did the model detect?
Each metric describes a different aspect of model behaviour.
Why accuracy can still look excellent
Return to our rare-disease example.
Suppose 10,000 people are screened.
Only 100 actually have the disease.
Imagine the model produces:
- 60 true positives;
- 40 false negatives;
- 20 false positives;
- 9,880 true negatives.
Its accuracy is:
That looks outstanding.
But its recall is:
The model misses:
40% of people who actually have the disease.
So the statement:
"The model is 99.4% accurate"
is technically true.
But it gives a deeply incomplete picture of performance.
Precision and recall expose what accuracy hides.
Metrics encode consequences
There is a deeper pattern here.
Precision and recall are not merely alternative mathematical formulas.
They correspond to different kinds of concern.
Precision becomes important when we care about:
false positives
Recall becomes important when we care about:
false negatives
And which kind of error matters more depends on the consequences.
A false positive in a spam filter may be annoying.
A false positive in a criminal-risk system could have serious consequences.
A false negative in an advertising system may mean losing a sale.
A false negative in a cancer-screening system could mean missing a life-threatening illness.
The numbers alone do not tell us how serious those consequences are.
Precision and recall are properties of a system
There is another subtle but important point.
Precision and recall do not belong only to the underlying machine-learning model.
They also depend on the decision threshold.
Imagine the model outputs:
The model itself has produced a probability.
Whether the system ultimately says:
DISEASE
depends on the threshold we choose.
If:
the prediction is positive.
If:
the prediction is negative.
So the same model can produce different:
- precision;
- recall;
- false-positive rates;
- false-negative rates
depending on how we choose to operate it.
That means evaluating machine learning involves more than evaluating a model.
We are evaluating a model plus a decision rule.
Prediction and decision are beginning to separate
This distinction will become increasingly important later in the course.
The model might tell us:
There is a 73% probability that this transaction is fraudulent.
That is a prediction.
But then somebody has to decide:
- approve the transaction;
- decline it;
- ask for additional authentication;
- temporarily hold it;
- send it for human review.
Those are decisions.
The prediction informs the decision.
It does not uniquely determine it.
So the complete system looks more like:
DATA → MODEL → PROBABILITY → THRESHOLD / RULE → DECISION → ACTION
Precision and recall depend partly on where we place that decision boundary.
What if we want both?
Naturally, we would like:
- very high precision;
- very high recall.
A perfect classifier would have:
and:
Every positive prediction would be correct.
And every real positive case would be found.
Sometimes models can achieve high values for both.
But difficult classification problems often require trade-offs.
This raises another question:
Can we combine precision and recall into a single measure?
There are metrics that attempt exactly that.
One common example is the F1 score.
But before combining anything, it is important to understand what is being combined.
Precision and recall are valuable precisely because they force us to look at the two different ways a classifier can fail.
The deeper lesson
We began this module by saying:
Accuracy isn't enough.
Precision and recall show us exactly why.
A classifier can fail in two fundamentally different ways:
False positive
The model says something is there when it isn't.
False negative
The model says something isn't there when it is.
Precision focuses our attention on the first problem.
Recall focuses our attention on the second.
Neither is universally more important.
The question depends on the consequences of each mistake.
And that means evaluating a model ultimately requires us to understand the system in which the model operates.
The bigger idea
Machine-learning evaluation is slowly moving us away from the simple question:
Was the prediction correct?
towards richer questions:
How was it wrong?
How often does each kind of mistake occur?
Which mistakes matter most?
What happens when those mistakes enter a real decision-making system?
That is the real purpose of evaluation.
A model is not useful merely because it produces impressive numbers.
It is useful when its behaviour is appropriate for the consequences of the decisions built around it.
And to understand that behaviour properly, we need to look more closely at the two kinds of classification error we have just introduced:
False Positives and False Negatives
Because calling something positive when it is not, and failing to identify something that is, can produce radically different consequences.