ROC Curves

In the previous lesson, we introduced the confusion matrix.

For a binary classifier, every prediction falls into one of four categories:

• true positive;
• true negative;
• false positive;
• false negative.

But we also discovered something important:

The confusion matrix depends on the classification threshold.

If we change the threshold, some predictions move from negative to positive or from positive to negative.

That changes:

• true positives;
• false positives;
• true negatives;
• false negatives;
• precision;
• recall;
• accuracy.

So evaluating a classifier at only one threshold gives us only one snapshot of its behaviour.

What if we want to understand how the classifier performs across all possible thresholds?

One of the most common tools for doing this is the:

Receiver Operating Characteristic curve

usually shortened to:

ROC curve.

---

Start with probability scores

Many classification models do not naturally produce a simple:

YES

or:

NO

Instead, they produce a score or probability.

For example, a fraud model might output:

Transaction	Predicted probability of fraud
A	0.97
B	0.89
C	0.76
D	0.61
E	0.48
F	0.32
G	0.17
H	0.04

We then choose a threshold.

For example:

$$P(\text{fraud}) \geq 0.5$$

means:

classify as fraud

while:

$$P(\text{fraud}) < 0.5$$

means:

classify as legitimate

At this threshold:

• A is positive;
• B is positive;
• C is positive;
• D is positive;
• E is negative;
• F is negative;
• G is negative;
• H is negative.

But 0.5 is only one possible threshold.

We could choose:

$$0.9$$

or:

$$0.7$$

or:

$$0.2$$

Each threshold produces a different set of classifications.

---

Imagine moving the threshold

Start with an extremely high threshold:

$$1.0$$

Almost nothing will be classified as positive.

That means:

• very few false positives;
• but also very few true positives.

Now gradually lower the threshold.

More cases become classified as positive.

We begin detecting more real positive cases.

So:

true positives increase

But we also begin incorrectly classifying more negative cases as positive.

So:

false positives increase

Continue lowering the threshold until:

$$0$$

Now virtually everything is classified as positive.

We detect every positive case.

But we also classify every negative case as positive.

An ROC curve traces this changing behaviour.

---

Two quantities

An ROC curve compares two rates:

1. True Positive Rate
2. False Positive Rate

Let's look at each.

---

True Positive Rate

The True Positive Rate asks:

Of all the actual positive cases, what proportion did the model correctly identify?

Its formula is:

$$\text{True Positive Rate} = \frac{TP}{TP+FN}$$

You may recognise this formula.

It is exactly the same as:

recall

So:

$$\text{TPR} = \text{Recall}$$

It is also sometimes called:

sensitivity

A high True Positive Rate means the classifier successfully detects most of the positive cases.

---

False Positive Rate

The False Positive Rate asks:

Of all the actual negative cases, what proportion did the model incorrectly classify as positive?

Its formula is:

$$\text{False Positive Rate} = \frac{FP}{FP+TN}$$

Remember that specificity is:

$$\text{Specificity} = \frac{TN}{TN+FP}$$

Therefore:

$$\text{FPR} = 1-\text{Specificity}$$

So the ROC curve compares:

how many real positives we detect

against:

how many real negatives we incorrectly flag

---

Plotting the ROC curve

An ROC curve places:

False Positive Rate

on the horizontal axis and:

True Positive Rate

on the vertical axis.

So:

$$x = \text{FPR}$$

and:

$$y = \text{TPR}$$

Both range from:

$$0$$

to:

$$1$$

or equivalently:

0% to 100%

Each possible classification threshold produces a point on this graph.

By moving the threshold from very high to very low, we trace out the ROC curve.

---

The bottom-left corner

Consider the point:

$$(0,0)$$

This means:

$$\text{FPR}=0$$

and:

$$\text{TPR}=0$$

The classifier has:

• detected none of the positive cases;
• created no false positives.

This could happen if the threshold is so high that the model predicts:

NEGATIVE

for everything.

It creates no false alarms.

But it detects nothing.

---

The top-right corner

Now consider:

$$(1,1)$$

This means:

$$\text{FPR}=1$$

and:

$$\text{TPR}=1$$

The classifier detects:

every positive case

but also incorrectly labels:

every negative case

as positive.

This could happen if the threshold is so low that the model predicts:

POSITIVE

for everything.

Recall is perfect.

But the classifier is not useful.

---

The ideal corner

The most desirable point is:

$$(0,1)$$

This means:

$$\text{FPR}=0$$

and:

$$\text{TPR}=1$$

The model detects:

100% of the positive cases

while incorrectly flagging:

0% of the negative cases

That would be a perfect classifier.

So when looking at an ROC curve, we generally want the curve to bend toward the:

top-left corner

---

A perfect classifier

Imagine a disease-detection model that perfectly separates healthy and sick patients.

Every sick patient receives a higher risk score than every healthy patient.

We could choose a threshold that gives:

$$TPR = 1$$

and:

$$FPR = 0$$

The ROC curve would rise directly toward the top-left corner.

This represents perfect discrimination.

The model can completely separate the two classes.

---

What does random guessing look like?

Now imagine a classifier with no useful information.

Its predictions are effectively random.

As we lower the threshold, it detects positives and incorrectly flags negatives at approximately the same rate.

Its ROC curve therefore follows roughly a diagonal line:

$$(0,0) \rightarrow (1,1)$$

Along this line:

$$TPR \approx FPR$$

The model is not meaningfully distinguishing positives from negatives.

It is behaving approximately like random guessing.

---

Better than random

A useful classifier should generally produce an ROC curve above the diagonal.

For example, perhaps at some threshold:

$$FPR = 0.1$$

while:

$$TPR = 0.8$$

This means:

The model detects 80% of the positive cases while incorrectly flagging 10% of the negative cases.

That may represent useful discrimination.

Whether it is good enough depends on the application.

---

A numerical example

Suppose we have:

100 positive cases

and:

900 negative cases.

At one threshold, the model produces:

$$TP = 80$$ $$FN = 20$$ $$FP = 90$$ $$TN = 810$$

The True Positive Rate is:

$$TPR = \frac{80}{80+20} = 0.8$$

The False Positive Rate is:

$$FPR = \frac{90}{90+810} = 0.1$$

So this threshold produces the ROC point:

$$(0.1,\ 0.8)$$

Now lower the threshold.

Perhaps we get:

$$TP = 95$$ $$FN = 5$$ $$FP = 270$$ $$TN = 630$$

Now:

$$TPR = \frac{95}{100} = 0.95$$

and:

$$FPR = \frac{270}{900} = 0.30$$

So our new point is:

$$(0.30,\ 0.95)$$

We have increased the True Positive Rate.

But we have also increased the False Positive Rate.

That is the trade-off the ROC curve makes visible.

---

Every point represents a different operating policy

This is an important way to interpret the curve.

Each point corresponds to a different threshold.

And each threshold represents a different balance between:

• detecting positives;
• creating false alarms.

So an ROC curve does not tell us:

Here is the correct threshold.

Instead, it shows:

Here are the trade-offs available to you.

Choosing among them requires information that the ROC curve itself does not contain.

---

The ROC curve does not know the consequences

Suppose two thresholds are available.

Threshold A

$$TPR = 0.70$$ $$FPR = 0.02$$

Threshold B

$$TPR = 0.95$$ $$FPR = 0.15$$

Which is better?

We cannot answer from those numbers alone.

If we are detecting a life-threatening disease, perhaps detecting an additional 25% of cases is worth the additional false positives.

If every false positive triggers an extremely expensive and invasive procedure, perhaps not.

The ROC curve describes the trade-off.

It does not tell us how to value it.

---

The curve describes ranking ability

There is another useful way to understand ROC curves.

Imagine a model gives every case a score.

A good classifier should generally assign:

higher scores to positive cases

and:

lower scores to negative cases.

The ROC curve tells us how well the model can separate or rank those two groups as we move the threshold.

A strong model tends to place positive cases toward the high-score end.

A weak model mixes positives and negatives together.

This idea becomes particularly important when we summarise the whole ROC curve with a single number.

That number is called:

Area Under the Curve

or:

AUC.

---

Area Under the ROC Curve

The Area Under the ROC Curve is usually written as:

ROC-AUC

or simply:

AUC

It measures the area underneath the ROC curve.

Its value usually ranges between:

$$0$$

and:

$$1$$

A perfect classifier has:

$$AUC = 1$$

A classifier behaving like random guessing has approximately:

$$AUC = 0.5$$

Generally:

The larger the AUC, the better the model is at separating positive and negative cases.

---

An intuitive interpretation of AUC

There is a particularly useful interpretation.

Imagine randomly choosing:

• one positive case;
• one negative case.

Ask the model to assign both a score.

The AUC can be interpreted as the probability that the model gives the positive case a higher score than the negative case.

So if:

$$AUC = 0.90$$

then, roughly speaking:

There is a 90% chance that a randomly selected positive case receives a higher score than a randomly selected negative case.

This makes clear that AUC measures ranking ability.

It does not directly tell us whether the model's probabilities are accurate.

---

AUC does not measure calibration

Suppose two models evaluate the same cases.

Model A predicts:

$$0.51,\ 0.52,\ 0.53,\ 0.54$$

for increasingly risky cases.

Model B predicts:

$$0.60,\ 0.70,\ 0.80,\ 0.90$$

If both models rank the cases in exactly the same order, they can have the same ROC-AUC.

But their probability estimates are very different.

Perhaps an event predicted with:

$$90\%$$

probability actually occurs only:

$$60\%$$

of the time.

That model may rank cases well while producing poorly calibrated probabilities.

This distinction will become important later in this module.

Discrimination and calibration are different properties.

ROC curves primarily measure discrimination.

Calibration asks whether predicted probabilities correspond to observed frequencies.

---

AUC does not choose the threshold

Another common misunderstanding is that a high AUC automatically tells us how the classifier should be deployed.

It does not.

Suppose:

$$AUC = 0.94$$

That tells us the model is generally good at ranking positive cases above negative ones.

But we still need to choose a threshold.

Should we classify positive when:

$$P > 0.3$$

or:

$$P > 0.5$$

or:

$$P > 0.9$$

AUC does not answer that question.

Threshold selection depends on:

• false-positive costs;
• false-negative costs;
• available resources;
• operational capacity;
• risk tolerance;
• the purpose of the system.

---

AUC can hide the region we actually care about

Suppose a cybersecurity system processes millions of events.

Perhaps the organisation can tolerate a False Positive Rate of at most:

$$0.1\%$$

Anything higher would produce too many alerts for analysts to investigate.

In that case, we care primarily about a tiny part of the ROC curve:

$$FPR \leq 0.001$$

A model could have excellent overall AUC but perform poorly in precisely this region.

So a single AUC number can hide operationally important detail.

Again:

A summary metric compresses information.

Always ask whether the compressed information contains what you actually care about.

---

Rare events create another complication

ROC curves can sometimes appear optimistic when positive events are extremely rare.

Suppose a fraud system processes:

1,000,000 transactions

and only:

1,000 are fraudulent.

There are therefore:

999,000 legitimate transactions.

Imagine the classifier has:

$$FPR = 1\%$$

That sounds relatively low.

But:

$$0.01 \times 999,000 = 9,990$$

legitimate transactions would be incorrectly flagged.

If the system detected all 1,000 fraudulent transactions, we would have:

• 1,000 true positives;
• 9,990 false positives.

So most fraud alerts would still be false alarms.

This is why the False Positive Rate alone does not always tell us the full operational story.

For highly imbalanced datasets, other tools — particularly precision-recall curves — can sometimes be more informative.

---

ROC curves compare models

One particularly useful application of ROC curves is comparing classifiers.

Suppose:

Model A

and:

Model B

both predict the same outcome.

We can plot both ROC curves.

If Model A's curve consistently lies above Model B's, then Model A generally achieves:

• higher True Positive Rates;
• for the same False Positive Rates.

That suggests Model A has better discriminatory ability.

But if the curves cross, the answer becomes more interesting.

Perhaps:

• Model A performs better when very low false-positive rates are required;
• Model B performs better when high recall is the priority.

Then asking:

Which model is better?

again becomes incomplete.

The correct question is:

Which model is better for the operating conditions we actually care about?

---

ROC curves separate model capability from one decision threshold

This is perhaps the most useful conceptual role of an ROC curve.

A confusion matrix tells us:

How does this classifier behave at this threshold?

An ROC curve tells us:

What trade-offs can this model achieve across many thresholds?

So we can think of the progression as:

Probability scores

↓

Choose one threshold

↓

Confusion matrix

or:

Probability scores

↓

Sweep across thresholds

↓

ROC curve

The ROC curve therefore gives us a broader view of the model's classification capability.

---

But deployment still requires a decision

Eventually, however, a real system has to act.

A bank cannot merely admire an ROC curve.

It eventually has to decide whether to:

• approve a transaction;
• request authentication;
• block it;
• investigate it.

A medical system eventually has to decide whether to:

• reassure a patient;
• request another test;
• refer them to a specialist;
• begin treatment.

So model evaluation and system design eventually meet.

The ROC curve helps us understand the available trade-offs.

Humans, organisations or other decision systems must determine which trade-off is acceptable.

---

Why is it called "Receiver Operating Characteristic"?

The name has an unusual history.

ROC analysis originated in signal detection, particularly radar systems.

Operators needed to distinguish:

real signals

from:

noise.

Changing the detection threshold created the same trade-off we see in machine learning:

• detect more real signals;
• but create more false alarms.

The technique was later adopted widely in:

• medicine;
• statistics;
• engineering;
• machine learning.

The terminology sounds unusual because it comes from this earlier signal-detection context.

But the underlying problem is exactly the same:

How well can we distinguish signal from noise as we change the threshold?

---

ROC curves and the bigger picture

We now have several layers of classification evaluation.

Accuracy

asks:

How often are we correct overall?

Precision

asks:

When we predict positive, how often are we right?

Recall

asks:

Of all the real positives, how many do we detect?

Confusion matrix

shows:

What kinds of correct and incorrect predictions are we making?

ROC curve

asks:

How does the trade-off between detecting positives and creating false positives change as we move the threshold?

ROC-AUC

asks:

How well does the model generally rank positive cases above negative cases?

Each reveals something different.

None tells us everything.

---

The deeper lesson

ROC curves introduce an important shift in how we think about machine-learning models.

The model does not necessarily produce a decision.

It may produce a:

score

or:

probability

The decision appears only when we introduce a rule such as:

$$P(\text{positive}) > \tau$$

where:

$$\tau$$

is the decision threshold.

So:

MODEL → SCORE → THRESHOLD → CLASSIFICATION

Changing the threshold changes the system's behaviour without changing the underlying model.

This means there may be no single answer to:

How good is this classifier?

Its usefulness depends partly on how we choose to operate it.

---

The bigger idea

An ROC curve lets us explore many possible operating points.

It exposes a trade-off:

detect more real positives

versus:

create more false positives

But it still leaves a deeper question unanswered.

Suppose a model says:

There is an 80% probability of this event occurring.

What does that 80% actually mean?

If we collect 100 cases for which the model predicts an 80% probability, should the event occur approximately 80 times?

A model might rank cases extremely well and therefore have an excellent ROC curve while still producing probabilities that are systematically too high or too low.

So we need to evaluate not only:

Can the model distinguish high-risk cases from low-risk cases?

but also:

Can we trust the probabilities themselves?

That brings us to the next lesson:

Calibration

When a model says 80%, should we actually believe 80%?