False positives and false negatives

In the previous lesson, we introduced precision and recall.

Both depend on understanding two different kinds of classification mistake:

• false positives;
• false negatives.

These sound like technical details.

They are not.

In many real systems, the difference between a false positive and a false negative is the difference between:

• inconvenience and danger;
• wasted money and missed opportunity;
• unnecessary intervention and failure to act;
• a harmless alarm and a catastrophic oversight.

So before we move further into classification metrics, we need to look at these two errors more carefully.

---

Start with the four possible outcomes

Suppose a model is trying to detect whether something is positive.

For example:

• fraud;
• disease;
• spam;
• a defective product;
• a pedestrian;
• a cyberattack.

There are four possible combinations of reality and prediction.

Reality	Prediction	Outcome
Positive	Positive	True Positive
Negative	Negative	True Negative
Negative	Positive	False Positive
Positive	Negative	False Negative

The two correct outcomes are:

• True Positive
• True Negative

The two incorrect outcomes are:

• False Positive
• False Negative

These two mistakes are fundamentally different.

---

False positives

A false positive happens when the model predicts that something is present when it is not.

The model says:

YES

Reality says:

NO

Examples include:

• predicting fraud when a transaction is legitimate;
• predicting cancer when the patient is healthy;
• predicting spam when an email is legitimate;
• predicting a fault when equipment is working normally;
• predicting a pedestrian when there is none.

A false positive is therefore sometimes called a:

false alarm

---

A simple example

Suppose an airport security scanner inspects 1,000 bags.

Ten genuinely contain prohibited items.

The system flags 40 bags as suspicious.

Of those 40:

• 10 really contain prohibited items;
• 30 do not.

Those 30 are:

false positives

The system raised 30 alarms that turned out not to correspond to a real threat.

---

False positives create unnecessary action

A false positive often causes a system to do something that was not actually needed.

That might mean:

• blocking a bank transaction;
• ordering another medical test;
• sending a technician to inspect equipment;
• stopping a production line;
• deleting an email;
• escalating a case for human review;
• triggering an alarm.

These actions may have costs.

Some are small.

Some are large.

The important point is:

A false positive consumes resources because the system reacts to something that was not really there.

---

False negatives

A false negative is the opposite.

The model predicts that something is absent when it is actually present.

The model says:

NO

Reality says:

YES

Examples include:

• missing a fraudulent transaction;
• missing a cancer diagnosis;
• allowing spam through;
• missing an equipment fault;
• failing to detect a pedestrian;
• failing to identify a cyberattack.

A false negative is therefore sometimes described as a:

missed detection

---

A simple example

Return to our airport scanner.

Suppose ten bags genuinely contain prohibited items.

The scanner detects eight of them.

But two pass through without being flagged.

Those two are:

false negatives

The system failed to detect something that was really there.

---

False negatives create missed action

A false negative often causes the system to fail to act when action was needed.

That might mean:

• allowing fraud to continue;
• failing to investigate a disease;
• missing a dangerous equipment failure;
• allowing a cyberattack through;
• failing to brake for an obstacle;
• failing to intervene in a dangerous situation.

This can be much more serious than a false positive.

But not always.

Again, context matters.

---

The same model can fail in two directions

Imagine a medical screening system.

A false positive means:

The system says a patient may have the disease when they do not.

Possible consequences include:

• anxiety;
• more testing;
• additional cost;
• unnecessary treatment.

A false negative means:

The system says a patient is healthy when they actually have the disease.

Possible consequences include:

• delayed diagnosis;
• delayed treatment;
• worsening illness.

Both are errors.

But they are not equivalent.

This is why simply counting:

correct versus incorrect

can be misleading.

---

The cost of each error depends on the system

Consider several examples.

Spam filtering

A false positive means:

A legitimate email is sent to spam.

A false negative means:

A spam email reaches your inbox.

For many users, the false positive may be worse.

Missing an important message can be more costly than seeing an annoying advertisement.

---

Fraud detection

A false positive means:

A legitimate transaction is blocked.

A false negative means:

Fraud is allowed through.

Both matter.

The bank may strongly care about false negatives because fraud causes financial loss.

Customers may strongly care about false positives because being unable to use their own money is frustrating and disruptive.

---

Medical screening

A false positive means:

A healthy patient is told they may have a disease.

A false negative means:

A sick patient is told they are healthy.

Here, a false negative may be particularly dangerous.

But reducing false negatives usually means accepting more false positives.

---

Fire alarms

A false positive means:

The alarm activates when there is no fire.

A false negative means:

There is a fire and the alarm does not activate.

The consequences are clearly asymmetric.

A system that occasionally produces nuisance alarms may still be preferable to one that misses real fires.

---

Autonomous vehicles

A false positive might mean:

The system believes an obstacle exists when it does not.

The vehicle may brake unnecessarily.

A false negative might mean:

The system fails to detect an actual pedestrian.

The consequences could be catastrophic.

Again:

The errors are mathematically similar but operationally very different.

---

Why there is usually a trade-off

Suppose a model produces a probability:

$$P(\text{fraud}) = 0.62$$

We need a threshold to decide whether to label the transaction as fraud.

If the threshold is:

$$0.5$$

the transaction is flagged.

If the threshold is:

$$0.8$$

it is not.

Now imagine lowering the threshold across the whole system.

More cases will be classified as positive.

That usually means:

• more true positives;
• fewer false negatives;
• but also more false positives.

Raise the threshold and the opposite usually happens:

• fewer false positives;
• but more false negatives.

So we often cannot reduce one type of error without increasing the other.

---

Decision thresholds encode preferences

This is an important idea.

The model may output probabilities.

But the threshold converts those probabilities into action.

For example:

$$P(\text{disease}) > 0.2 \Rightarrow \text{send for further testing}$$

Choosing 0.2 instead of 0.8 is not simply a mathematical detail.

It reflects a judgment.

It says:

Missing a possible case is costly enough that we are willing to investigate many uncertain cases.

A higher threshold would encode a different preference.

So the threshold partly determines the balance between:

• false positives;
• false negatives.

And that means threshold selection is a design decision.

---

False positives and false negatives are not properties of the model alone

This distinction matters.

Suppose a model outputs a continuous probability score.

The model itself may remain unchanged.

But if we change the decision threshold, we change:

• the number of true positives;
• the number of true negatives;
• the number of false positives;
• the number of false negatives;
• precision;
• recall;
• accuracy.

So classification performance depends on:

MODEL + THRESHOLD

not merely:

MODEL

This is another example of the broader principle:

Prediction and decision-making are not the same thing.

---

Some systems use more than two actions

The situation can become even more interesting.

Suppose a fraud model produces a probability.

Instead of using only:

• approve;
• decline;

the system could use several actions.

For example:

$$P(\text{fraud}) < 0.2$$

→ approve automatically

$$0.2 \leq P(\text{fraud}) < 0.7$$

→ request additional authentication

$$P(\text{fraud}) \geq 0.7$$

→ block and investigate

Now prediction is being translated into a richer decision policy.

This can help reduce the consequences of uncertainty.

Rather than forcing every uncertain prediction into a simple yes/no decision, the system can create intermediate actions.

---

Not every error has the same cost

We can represent this idea more formally.

Suppose a false positive has cost:

$$C_{FP}$$

and a false negative has cost:

$$C_{FN}$$

Then the expected classification cost might be represented as:

$$\text{Total cost} = C_{FP}\cdot FP + C_{FN}\cdot FN$$

If:

$$C_{FN} \gg C_{FP}$$

then false negatives are much more serious.

The system may therefore be designed to prioritise recall.

If:

$$C_{FP} \gg C_{FN}$$

then false positives are much more serious.

The system may prioritise precision.

This makes something explicit that accuracy hides:

Different mistakes can carry different costs.

---

But cost is not always money

The word "cost" should not be interpreted only financially.

A cost might be:

• money;
• time;
• risk;
• inconvenience;
• lost opportunity;
• environmental impact;
• harm;
• unfairness;
• loss of trust.

Suppose a model incorrectly flags someone as high risk.

The cost may include:

• a rejected application;
• increased scrutiny;
• reputational harm;
• reduced access to a service.

Those consequences may be difficult to express as a single number.

This is why real-world model evaluation is often much harder than simply calculating a metric.

---

Errors can affect people differently

There is another important complication.

False-positive and false-negative rates may differ across groups.

Imagine a facial-recognition system.

Suppose its overall accuracy is high.

But perhaps false-positive rates are much higher for one demographic group than another.

The overall metric may hide this.

Likewise, a medical model might have:

• excellent recall overall;
• but much worse recall for a particular population.

Then the model's errors are not distributed evenly.

This introduces questions of:

• fairness;
• representation;
• bias;
• system design.

We will return to these later in the course.

---

Rare events make everything harder

False positives become especially important when the thing we are looking for is rare.

Suppose we screen:

100,000 people

for a disease that occurs in only:

100 people

Imagine a system that correctly identifies:

90 of those 100 cases

but also incorrectly flags:

1,000 healthy people

Then:

$$TP = 90$$ $$FN = 10$$ $$FP = 1000$$

The recall is high:

$$\frac{90}{100} = 90\%$$

But the number of false positives is much larger than the number of true positives.

This is one reason rare-event detection is difficult.

Even a small false-positive rate can create many false alarms when the negative population is enormous.

---

A tiny false-positive rate can still mean many false alarms

Suppose only:

$$1\%$$

of negative cases are falsely classified as positive.

That sounds excellent.

But imagine there are:

1,000,000 negative cases

Then:

$$0.01 \times 1,000,000 = 10,000$$

false positives.

So percentages need context.

A low error rate applied at enormous scale can still create a very large number of mistakes.

This matters in systems such as:

• online platforms;
• cybersecurity;
• financial services;
• healthcare screening;
• infrastructure monitoring.

Scale changes the consequences.

---

Error rates and absolute numbers both matter

Suppose two systems both have a false-positive rate of:

0.1%

System A processes:

1,000 cases per year

System B processes:

1 billion cases per year

For System A:

$$0.001 \times 1000 = 1$$

false positive per year.

For System B:

$$0.001 \times 1,000,000,000 = 1,000,000$$

false positives per year.

The rate is identical.

The operational burden is not.

So evaluation should often consider both:

• proportions;
• absolute numbers.

---

Sometimes false positives create feedback

Suppose a predictive policing system flags certain locations as high risk.

More police are sent there.

More offences are therefore observed there.

That new data enters the training dataset.

The model may then become even more confident that those locations are high risk.

A false positive is no longer just a one-off error.

It can influence future data.

This creates a feedback loop.

The same can happen in:

• credit scoring;
• recommendation systems;
• fraud detection;
• hiring;
• healthcare;
• predictive maintenance.

The prediction changes the system.

The changed system generates new data.

The new data influences future predictions.

This is why mistakes in deployed systems can sometimes reinforce themselves.

---

False negatives can create feedback too

Suppose a system repeatedly fails to detect a particular kind of fault.

Those faults may never be labelled correctly.

The training dataset therefore continues to contain too few examples of them.

Future models remain poor at detecting the same faults.

Again:

prediction → decision → observation → new data

The error can become embedded in the system.

---

Which error is worse?

There is no universal answer.

Consider these questions:

• What happens after a false positive?
• What happens after a false negative?
• Who bears the cost?
• How large is the cost?
• Is the cost reversible?
• Is the harm temporary or permanent?
• Does the mistake affect future data?
• Are the errors concentrated on particular groups?
• Does scale amplify the mistake?

Only then can we decide what balance is acceptable.

---

The language of "positive" can be misleading

Remember that "positive" does not mean desirable.

In a disease model:

positive = disease

In fraud detection:

positive = fraud

In fault detection:

positive = fault

So:

true positive

does not mean:

good outcome

It simply means:

The model correctly detected the target class.

This distinction becomes important because the terminology can otherwise be confusing.

---

Why these errors matter for precision and recall

Now we can connect back to the previous lesson.

Precision is:

$$\text{Precision} = \frac{TP}{TP+FP}$$

So precision is damaged by:

false positives

Recall is:

$$\text{Recall} = \frac{TP}{TP+FN}$$

So recall is damaged by:

false negatives

That is why:

• high precision means relatively few false positives;
• high recall means relatively few false negatives.

The metrics are simply different ways of looking at the error structure.

---

A classifier is more than one number

A classification model produces a pattern of:

• true positives;
• true negatives;
• false positives;
• false negatives.

Accuracy compresses those outcomes into one number.

Precision and recall expose more detail.

But there is an even clearer way of displaying all four outcomes together.

We can arrange them in a simple table.

That table is called a:

Confusion Matrix

Despite the slightly unfortunate name, a confusion matrix is one of the clearest ways to see how a classifier is actually behaving.

It lets us stop asking merely:

How accurate is the model?

and instead see:

Exactly what kinds of mistakes is it making?