Enleashed

A machine-learning model learns from examples.

But there is a danger.

The model may become so good at the training data that it starts learning:

noise,
accidental patterns,
quirks,
exceptions

that do not generalise.

This is overfitting.

Overfitting happens when a model learns the training data too specifically and performs worse on new, unseen data.

At its simplest:

TRAINING PERFORMANCE ↑

while:

TEST PERFORMANCE ↓

The model appears to improve.

But only on the data it already knows.

A simple analogy

Imagine a student preparing for an exam.

They memorise the exact answers to:

100 practice questions.

On those questions:

100% accuracy.

Then the real exam contains:

new questions testing the same underlying concepts.

The student performs poorly.

They learned:

the examples

rather than:

the underlying ideas.

That is overfitting.

Machine learning can do the same thing

Suppose a model is trained on:

1,000 houses.

It becomes extremely good at predicting the sale prices of those exact houses.

But for new houses:

performance is poor.

The model has captured details specific to the training set instead of learning general relationships that hold more broadly.

Training error can be misleading

Suppose two models produce:

Model	Training error	Test error
A	1%	18%
B	5%	6%

Model A appears better if we look only at:

training performance.

But Model B is much better on unseen data.

The real objective is not:

Perform perfectly on the training dataset.

It is:

Learn something that generalises.

Overfitting is a generalisation failure

Recall the learning objective:

PAST EXAMPLES

↓

LEARN PATTERN

↓

NEW EXAMPLE

↓

USEFUL PREDICTION.

Overfitting breaks the middle step.

Instead:

PAST EXAMPLES

↓

LEARN SPECIFIC DETAILS OF THOSE EXAMPLES

↓

NEW EXAMPLE

↓

POOR PREDICTION.

The model has learned too much of the wrong kind of information.

Signal and noise

A useful way to think about data is:

OBSERVED DATA = SIGNAL + NOISE.

The signal contains relationships we hope will persist.

The noise contains:

random variation,
measurement error,
one-off events,
accidental correlations.

A useful model learns mostly:

signal.

An overfit model starts learning:

noise.

A simple regression example

Suppose we observe:

x	y
1	2.1
2	4.0
3	6.2
4	7.9
5	10.1

The underlying relationship appears roughly:

y ≈ 2x.

There is some measurement noise.

A straight line might capture the general pattern well.

Now fit a complicated curve

Suppose we use an extremely flexible polynomial.

The curve passes through:

every training point exactly.

Training error:

zero.

It looks perfect.

But between the training points, the curve swings wildly.

For a new value:

x = 3.5,

the prediction may be absurd.

The complicated model has fitted:

the noise around the relationship

instead of only:

the relationship.

Complexity makes overfitting easier

A simple model can represent only a limited range of functions.

A highly flexible model can represent many.

That flexibility can be useful.

But it also gives the model enough freedom to capture accidental details of the training data.

Very roughly:

MORE MODEL FLEXIBILITY

↓

MORE ABILITY TO FIT SIGNAL

but also:

MORE ABILITY TO FIT NOISE.

Too simple versus too complex

We can imagine three cases.

Too simple

The model cannot represent the real relationship.

This is:

underfitting.

Appropriate complexity

The model captures the important structure.

This produces:

good generalisation.

Too complex

The model captures both:

structure + noise.

This creates:

overfitting.

The goal is not simplicity for its own sake

A simple model is not automatically better.

Suppose the world really contains a complex relationship.

Then a simple model may perform badly.

The objective is:

Enough complexity to capture useful structure, but not so much that the model becomes dominated by training-specific detail.

Overfitting depends on both model and data

A model that is overly complex for:

100 observations

may be entirely appropriate for:

100 million observations.

So overfitting is not determined by parameter count alone.

It depends on the relationship between:

model capacity,
amount of data,
diversity of data,
regularisation,
optimisation.

More data can reduce overfitting

Suppose a model has many parameters.

With only:

100 examples,

there may be many ways to fit them perfectly.

With:

10 million diverse examples,

the model has much less freedom to exploit accidental quirks.

More data can force the model to learn more general patterns.

But more duplicated data may not help much

Suppose we add:

1 million nearly identical examples.

The dataset becomes larger.

Its information content may not increase very much.

What often matters is:

diverse relevant evidence,

not merely:

row count.

Overfitting can happen to features

Suppose we predict whether students pass an exam.

A model notices:

Every student with ID number ending in 7 passed in the training data.

That pattern may be accidental.

A flexible model might use it.

On the training data:

performance improves.

On future students:

the relationship disappears.

This is overfitting to a spurious feature.

Accidental correlations are everywhere

Large datasets contain many variables.

Given enough features, some will correlate with the target purely by chance.

For example:

random identifier,
data-recording artefact,
timestamp quirk.

A sufficiently flexible model may exploit these patterns.

The fact that something predicts the training labels does not prove that it will continue predicting them.

Small datasets are especially vulnerable

Suppose only:

20 observations exist.

One unusual example can strongly influence the learned model.

With limited data, it is difficult to distinguish:

stable relationship

from:

random accident.

So small datasets tend to create greater overfitting risk.

Noise creates opportunities to overfit

Suppose measurements contain substantial random error.

A flexible model may try to explain every fluctuation.

But the random error will not repeat in future data.

So fitting it harms generalisation.

This is why perfect fit is not always desirable.

Overfitting can occur in classification

Suppose we classify:

cats

and:

dogs.

Training data contains:

cats photographed indoors,
dogs photographed outdoors.

A model may learn:

indoors → cat

outdoors → dog.

Training accuracy may be excellent.

Now show:

dog indoors.

The model fails.

Shortcut learning and overfitting overlap

Shortcut learning means the model uses an unintended predictive signal.

If that shortcut works only in the training environment, it becomes a generalisation failure.

Examples include:

snow → wolf,
hospital identifier → disease,
camera type → class.

The model learns what reduces loss, not necessarily what humans intended.

Overfitting can happen in time

Suppose an electricity-demand model is trained on:

2015–2025.

It learns highly specific patterns involving:

holidays,
weather,
economic conditions.

Then behaviour changes.

Future relationships differ.

A model tuned too closely to historical details may generalise poorly.

Overfitting can happen in space

Suppose a housing model is trained in:

Dublin.

It learns extremely detailed spatial patterns.

Deploy it in:

Cork.

The relationships do not transfer.

The model may have over-specialised to the geography of the training region.

Overfitting to people

Suppose a health model contains repeated observations from the same patients.

The model may learn:

patient-specific patterns.

If those same patients appear in the test set, performance looks excellent.

But for completely new patients, performance drops.

This is why grouped train-test splits matter.

Overfitting to devices

An IoT model may learn quirks specific to:

particular sensors,
firmware,
calibration.

If the same devices appear in both training and testing, the test may not reveal the issue.

Deploy to new sensors and performance falls.

Overfitting can happen to a benchmark

Overfitting does not happen only inside one model-training run.

An entire research community can overfit to:

a benchmark dataset.

Researchers repeatedly:

test ideas,
inspect scores,
modify architectures.

Eventually models become extremely optimised for that benchmark.

But benchmark improvement may not translate equally well to the real world.

The test set can become part of training indirectly

Suppose we evaluate:

100 different models

on the same test set.

Then choose the best one.

The test performance has influenced model selection.

It is no longer completely independent.

Repeatedly doing this can create:

test-set overfitting.

This is why validation sets exist.

Train, validate, test

Recall:

TRAINING DATA

→ learn parameters

VALIDATION DATA

→ choose model and hyperparameters

TEST DATA

→ final evaluation.

Using the test set only at the end protects its role as a genuinely unseen benchmark.

Validation curves can reveal overfitting

Suppose training continues.

At first:

training loss ↓

and:

validation loss ↓.

Good.

Then:

training loss continues ↓

but:

validation loss ↑.

That is a warning sign.

The model is improving its fit to the training examples while becoming worse at generalisation.

Early stopping

One response is:

stop training when validation performance stops improving.

This is called:

early stopping.

Conceptually:

EPOCH 1

Training improves
Validation improves

↓

EPOCH 10

Training improves
Validation improves

↓

EPOCH 20

Training improves
Validation worsens

↓

STOP.

More training can make the model worse

This can feel counterintuitive.

We might expect:

More learning must be better.

But if learning increasingly focuses on idiosyncrasies in the training data, more optimisation can reduce generalisation.

The goal is not:

maximum training.

It is:

maximum useful learning.

Regularisation

Another major approach is regularisation.

Regularisation discourages the model from becoming unnecessarily complex or extreme.

Conceptually:

TOTAL TRAINING OBJECTIVE

PREDICTION LOSS

PENALTY FOR COMPLEXITY.

The model now balances:

fit the data

against:

remain controlled.

L2 regularisation

One common approach penalises large parameter values.

This is often called:

L2 regularisation.

Conceptually:

Fit the training data well, but avoid making weights unnecessarily large.

This can produce smoother, more stable models.

L1 regularisation

Another approach is:

L1 regularisation.

It can encourage some parameters to become exactly:

zero.

This can effectively remove some inputs from the model.

In some applications, this produces simpler or sparser models.

Regularisation introduces a trade-off

Too little regularisation:

model may overfit.

Too much regularisation:

model may become too simple and underfit.

So:

REGULARISATION STRENGTH

is a hyperparameter that usually needs validation.

Dropout

Neural networks often use a technique called dropout.

During training, some network units are randomly ignored on each pass.

Conceptually:

FULL NETWORK

↓

RANDOMLY REMOVE SOME CONNECTIONS TEMPORARILY

↓

TRAIN.

The model cannot rely too heavily on one precise pathway.

This can improve generalisation.

Why dropout can help

Imagine a team where every task always goes through:

one particular employee.

The team becomes fragile.

Now force different combinations of people to solve tasks.

Knowledge becomes more distributed.

Dropout has a somewhat similar intuition.

It encourages robustness across different internal pathways.

Data augmentation

We can also reduce overfitting by creating additional variations of training examples.

For images:

crop,
rotate,
change brightness.

Now the model sees many versions of the same underlying object.

This encourages it to learn:

invariant structure

rather than:

exact pixel arrangements.

Data augmentation tells the model what should not matter

Suppose we rotate a cat image slightly.

The label remains:

cat.

We are telling the model:

Small rotations should not change the classification.

Data augmentation therefore injects useful assumptions about the task.

But inappropriate augmentation can hurt

Suppose orientation actually matters.

For example:

rotated may resemble:

Blindly rotating examples could destroy label meaning.

Again, data augmentation is a design decision.

Feature selection

Another approach is to remove irrelevant or unreliable features.

Suppose a model receives:

10 useful variables

and:

10,000 random variables.

The random variables create many opportunities for accidental correlations.

Reducing irrelevant features can reduce overfitting.

Feature engineering can reduce overfitting

Good features expose meaningful structure.

Suppose instead of asking a model to infer:

day-night cycle

from a raw timestamp, we give it useful representations such as:

hour of day,
weekday.

A better representation can make the underlying relationship easier to learn with less complexity.

Simpler representations can improve generalisation

The more directly a representation captures relevant structure, the less work the model has to do.

This can reduce the temptation to exploit accidental patterns.

Model architecture and representation therefore affect overfitting together.

Cross-validation

When data is limited, cross-validation can provide a more robust estimate of generalisation.

Instead of relying on one validation split, we evaluate across several.

This helps answer:

Does the model perform well consistently, or did it simply get lucky with one split?

Data diversity is a form of regularisation

Suppose a model learns:

snow → wolf.

Now add:

dogs in snow,
wolves on grass,
wolves in forests.

The shortcut stops working.

The model is forced toward a more general representation.

Diverse data constrains the model.

This distinction matters.

Overfitting

The model learns training-specific patterns that do not generalise even to the same underlying distribution.

Distribution shift

The underlying distribution itself changes between training and deployment.

Both can produce poor test performance.

But the causes differ.

A perfectly generalised historical model can still fail in the future

Suppose a model learned the 2020 system perfectly.

Then the world changes radically in 2026.

The model may fail even though it did not overfit its original data.

This is distribution shift.

We should not call every generalisation failure overfitting.

How do we diagnose overfitting?

A classic signal is:

LOW TRAINING ERROR

and:

HIGHER VALIDATION / TEST ERROR.

The difference is sometimes called a:

generalisation gap.

A large gap suggests the model has learned aspects of the training data that are not transferring.

Generalisation gap

Suppose:

Training accuracy = 99%

Validation accuracy = 80%.

Gap:

19 percentage points.

Now:

Training accuracy = 92%

Validation accuracy = 90%.

Gap:

2 points.

The second model generalises much more consistently.

A small gap is not automatically good

Suppose:

Training accuracy = 55%

Validation accuracy = 54%.

Tiny generalisation gap.

But the model is simply poor everywhere.

That is underfitting.

We need:

good performance

and:

reasonable generalisation gap.

Bias and variance

Overfitting connects to the classic bias-variance trade-off.

Very roughly:

High bias

Model is too restrictive.

It misses important structure.

Tends toward:

underfitting.

High variance

Model changes too strongly in response to the particular training sample.

Tends toward:

overfitting.

What does high variance mean here?

Imagine collecting two slightly different training datasets from the same population.

A high-variance model learns:

very different functions

from the two datasets.

A more stable model produces:

similar predictions.

Overfit models can be highly sensitive to which examples happened to be included.

A thought experiment

Suppose we train the same model on:

Dataset A

and:

Dataset B,

both sampled from the same underlying population.

If the learned models differ wildly, the training process is unstable.

The model may be fitting sample-specific noise.

Ensembles can reduce variance

Suppose we train several models on slightly different datasets.

Instead of trusting one, we combine their predictions.

For regression:

average predictions.

For classification:

vote or average probabilities.

This is an ensemble.

Ensembles can reduce the effect of sample-specific noise.

Bagging

A famous ensemble idea is bagging.

Train multiple models on different resampled versions of the dataset.

Then combine them.

Random forests use a related idea.

Individual trees may overfit strongly.

Averaging many trees can improve generalisation.

Overfitting and interpretability

Highly complex models can sometimes fit data extremely well while making it harder to understand:

which features matter,
which relationships are stable.

Interpretability does not automatically prevent overfitting.

But examining model behaviour can help reveal suspicious reliance on shortcuts.

Feature importance can expose overfitting

Suppose a model predicts student performance.

Feature importance shows:

student ID number

as one of the strongest predictors.

That should trigger suspicion.

The feature may be capturing an accidental training-set relationship.

Interpretability can act as a diagnostic.

Shapley values can help locally

Suppose a model makes an unusual prediction.

A Shapley-style explanation shows that:

an irrelevant postcode fragment

contributed heavily.

That may reveal:

shortcut learning,
leakage,
overfitting.

Feature attribution does not prove the model is wrong.

But it can reveal what it relied on.

Overfitting and data leakage can look similar

Suppose test performance is astonishingly high.

Perhaps:

the model generalises brilliantly.

Or perhaps:

future or test information leaked into training.

Leakage can make a model appear to have solved the generalisation problem when it has not.

Always investigate surprising performance carefully.

Overfitting can happen to labels

Suppose human labels contain small inconsistencies.

A sufficiently flexible model may learn those inconsistencies.

For example:

two nearly identical cases receive different labels because annotators disagreed.

Trying to predict both perfectly may reduce generalisation.

Sometimes the correct response is not:

more model capacity.

It may be:

recognise label uncertainty.

Perfect accuracy may be impossible

If the same inputs correspond to different labels because of:

noise,
unobserved variables,
human disagreement,

then no deterministic model can classify every case correctly.

Pushing training accuracy toward 100% may force the model to exploit meaningless details.

Bayes error

In some problems, there is an irreducible level of prediction error because the available information does not perfectly determine the output.

This is sometimes related to the idea of a Bayes error rate.

The exact mathematics can wait.

The important idea is:

There may be a best achievable prediction accuracy given the information available, and it may be below 100%.

Uncertainty can be healthier than overfitting

Suppose two training examples look almost identical but have different labels.

One model becomes extremely complex to separate them perfectly.

Another says:

Class A: 55%

Class B: 45%.

The second model may be representing uncertainty more honestly.

Perfect confidence is not always desirable.

Overfitting can create overconfidence

A model that memorises training patterns may become extremely certain.

When similar-looking but genuinely new cases arrive, that confidence can be misplaced.

So overfitting can contribute to:

poor calibration.

Modern deep learning complicates the classical story

Traditional machine-learning intuition often says:

More parameters increase overfitting.

Modern deep neural networks sometimes contain far more parameters than training examples and still generalise extremely well.

This surprised researchers.

The relationship between:

parameter count,
optimisation,
generalisation

is more complicated than the simple textbook picture.

Double descent

In some modern models, test error can behave unexpectedly as model complexity increases.

Initially:

more complexity → lower test error.

Then:

more complexity → higher test error.

But beyond another point:

even more complexity → test error falls again.

This phenomenon is sometimes called double descent.

It shows that classical bias-variance intuition, while useful, is not the whole story for modern overparameterised models.

Overparameterisation

A model is overparameterised when it has enough parameters to fit the training data extremely closely, potentially even perfectly.

Modern neural networks are often heavily overparameterised.

Yet they can still generalise.

Why?

There is active research on the full explanation.

Optimisation itself provides an implicit bias

Gradient descent does not necessarily find an arbitrary solution among all possible zero-training-loss solutions.

The optimisation process may favour certain kinds of solutions.

This is sometimes called implicit regularisation or implicit bias.

The optimiser itself can influence generalisation.

Architecture also provides bias

A convolutional neural network is designed to exploit spatial structure in images.

A transformer uses attention mechanisms suited to relationships across sequences.

Architecture constrains the types of patterns the model learns easily.

This can improve generalisation even in very large models.

More parameters can make optimisation easier

Counterintuitively, having many parameters can sometimes make it easier for gradient descent to find useful low-loss solutions.

The parameter space may contain many good solutions.

So enormous model size can help optimisation.

Again:

model size ≠ overfitting automatically.

But large models can still overfit

None of this means modern models are immune.

They can still:

memorise rare examples,
exploit spurious correlations,
fail out of distribution,
fit label noise.

The practical question remains:

Does performance transfer to the situations we actually care about?

Memorisation and generalisation can coexist

A large model may:

memorise some specific training examples,
generalise well across many others.

These behaviours are not mutually exclusive.

Understanding how large models balance memorisation and generalisation is an important area of AI research.

Memorisation can create privacy risks

Suppose a model memorises:

personal information,
private text,
identifying details

from training data.

It may later reproduce them.

Overfitting therefore has implications beyond prediction quality.

It can become a:

privacy problem.

Duplicate data can increase memorisation

Suppose the same sensitive sentence appears:

thousands of times

in the training dataset.

The model has many opportunities to learn it very specifically.

Data deduplication can reduce this risk.

Overfitting and copyright

A generative model may sometimes reproduce training material extremely closely.

This raises questions about:

memorisation,
originality,
intellectual property.

Again, the boundary between:

learning general structure

and:

memorising specific examples

matters.

Data contamination

Suppose a model is evaluated on a benchmark.

But benchmark questions appeared somewhere in its training data.

The model may perform well partly because it has seen them before.

This is benchmark contamination.

It makes evaluation harder.

Contamination matters increasingly for foundation models

Large models are trained on enormous portions of available public data.

It becomes difficult to guarantee that every evaluation example is completely unseen.

So evaluating generalisation can require:

new benchmarks,
private test sets,
contamination analysis.

Test performance can still be misleading

Suppose the test data closely resembles the training distribution.

The model performs well.

But deployment occurs under:

new conditions,
new users,
new geography.

The model may still fail.

Generalisation is always relative to:

what kind of difference exists between train and deployment.

Interpolation is easier than extrapolation

Suppose training contains ages:

20–60.

Predicting for:

age 40

is interpolation.

Predicting for:

age 95

is extrapolation.

A model can generalise well within familiar regions while performing poorly outside them.

Test design should probe both when relevant.

Overfitting is relative to the future we care about

A model could look overfit according to one test and useful according to another.

Suppose it is deliberately specialised to:

Dublin households.

It may generalise poorly nationally.

But if the deployment population is only Dublin households, that may be acceptable.

There is no context-free definition of useful generalisation.

Specialisation is not always overfitting

Suppose a model is trained specifically for:

one manufacturing machine.

It learns extremely machine-specific behaviour.

Deploying it on another machine would fail.

But perhaps it was never meant to work elsewhere.

That is specialisation, not necessarily problematic overfitting.

The deployment scope matters.

The question is: what should generalise?

For every model, ask:

Across which variations do we expect the learned relationship to remain valid?

Perhaps across:

new timestamps,
new users,
new locations.

Perhaps not across:

entirely different countries,
different technologies.

Generalisation needs a defined scope.

Overfitting and the past

There is a deeper connection to the central philosophy of the course.

The training data describes:

the past.

The model exists because we want to use that past to say something about:

unseen situations or the future.

Overfitting occurs when the model becomes too attached to the exact details of that past.

The past contains both lessons and accidents

Some historical patterns represent:

persistent structure.

Others represent:

contingent accidents.

The learner does not automatically know which is which.

Machine learning is fundamentally trying to separate:

what should carry forward

from:

what should be forgotten.

Overfitting is excessive memory

One useful way to think about it is:

Overfitting is remembering too much of the specific past and learning too little of the transferable structure.

Underfitting is almost the opposite:

Remembering too little structure to describe the world properly.

Good generalisation lies between them.

This connects to Bayesian reasoning

A very flexible model can adapt strongly to limited observations.

In Bayesian terms, it may effectively place too much confidence in the specific sample.

Regularisation can act somewhat like prior structure:

Prefer simpler or more plausible explanations unless the evidence strongly justifies complexity.

This is one reason Bayesian thinking and regularisation are related.

Occam's razor

A famous principle is:

Do not multiply explanations beyond necessity.

Often paraphrased as:

Prefer the simpler explanation when two explain the evidence equally well.

This is Occam's razor.

Machine learning often benefits from a related principle:

Do not fit unnecessary complexity merely because the training data allows it.

Simpler is not always truer

Occam's razor is a useful heuristic.

But reality itself can be complicated.

A simple model may miss important structure.

The goal is not:

always choose simplest.

It is:

choose complexity justified by evidence.

Regularisation is a way of demanding evidence for complexity

Suppose adding complexity slightly reduces training error.

But it carries a regularisation cost.

The optimiser adds complexity only if the improvement is large enough to justify it.

Conceptually:

COMPLEXITY MUST EARN ITS PLACE.

Overfitting and prediction intervals

Suppose a model fits training data extraordinarily tightly.

It may produce very narrow prediction intervals.

But if it has overfit, those intervals may be falsely confident on new data.

Good uncertainty estimates should reflect generalisation uncertainty too.

Validation reveals what training cannot

Training data answers:

How well can you fit what you have seen?

Validation data asks:

Does that learning survive contact with something you have not seen?

This is why validation is central to detecting overfitting.

A practical learning curve

Imagine:

Training loss
\
 \
  \
   \
    \____

Validation loss
\
 \
  \__
     \
      \
       /
      /

Validation improves at first.

Then it worsens.

The point near the minimum validation loss may be a good stopping point.

---

### Overfitting can be subtle

Sometimes training accuracy is:

**96%**

and validation:

**94%**.

That gap may be acceptable.

Sometimes a tiny performance difference hides severe problems in one subgroup.

So generalisation should be examined beyond one aggregate metric.

---

### Group-specific overfitting

Suppose a model has abundant training data for:

**Group A**

and very little for:

**Group B**.

It may generalise well for A but overfit the limited examples from B.

Overall performance can hide this.

We may need group-specific validation.

---

### Rare classes are especially vulnerable

Suppose fraud examples are rare.

The model may memorise specific fraudulent transactions rather than learn broad fraud patterns.

New forms of fraud then go undetected.

More diverse rare-event data can help.

---

### Adversarial environments make yesterday's pattern fragile

Suppose a fraud classifier learns:

**Pattern A → fraud**.

Fraudsters discover this.

They stop using Pattern A.

Historical patterns lose predictive value.

The model may appear over-specialised, even though the environment actively changed.

Adaptive systems need continuous evaluation.

---

### Preventing overfitting is not one technique

Useful approaches include:

- more diverse training data,
- simpler models,
- regularisation,
- early stopping,
- data augmentation,
- feature selection,
- cross-validation,
- ensembles,
- careful validation design.

No single method is universally best.

---

### The appropriate solution depends on the cause

If the problem is:

**small dataset**,

more data may help.

If:

**label noise**,

cleaner labels may help.

If:

**excessive model complexity**,

regularisation may help.

If:

**distribution shift**,

retraining on current data may be necessary.

Diagnosis matters.

---

### Overfitting and foundation models

Foundation models introduce an interesting scale.

They train on huge, diverse datasets.

This can support remarkable generalisation.

But they can still:

- memorise specific sequences,
- reproduce training artefacts,
- perform unevenly in sparse regions.

Scale changes the problem.

It does not abolish it.

---

### Few-shot behaviour shows generalisation

Suppose a language model is given:

**two examples of a task**

in the prompt.

Then solves:

**a new example**.

That demonstrates a form of generalisation using learned representations and current context.

The model was not explicitly retrained for the new task.

Its existing parameters support adaptation.

---

### But benchmark memorisation can imitate capability

Suppose the "new" task appeared extensively in training.

Performance may partly reflect memorisation.

Separating:

**genuine generalisation**

from:

**training-data exposure**

can be difficult for large models.

---

### The most important question is still external

Whatever the architecture, the test remains:

> **Does the model work on genuinely relevant unseen situations?**

Not:

> How impressive is its training score?

Not:

> How many parameters does it have?

Not:

> How complex is the architecture?

Generalisation is the goal.

---

### A useful overfitting checklist

When evaluating a model, ask:

- How different are training and validation performance?
- Does validation loss begin increasing during training?
- Is the model much more complex than the evidence seems to justify?
- Is the dataset small?
- Is the data diverse?
- Are labels noisy?
- Could the model be exploiting accidental identifiers or shortcuts?
- Are related users, devices or locations present in both train and test?
- Has the test set influenced model development?
- Has the benchmark appeared in training data?
- Does performance hold across time?
- Does performance hold across space?
- Does performance hold across groups?
- Are prediction probabilities becoming overconfident?
- Would more data, regularisation or early stopping help?
- Is the problem actually overfitting, or has the world changed?

These questions help distinguish excellent training performance from genuine learning.

---

### The central idea

A machine-learning model should not merely become good at:

**what it has already seen**.

It should learn structure that remains useful for:

**what it has not seen yet**.

Overfitting happens when:

**TRAINING DATA**

↓

**MODEL LEARNS TOO MUCH SPECIFIC DETAIL**

↓

**TRAINING ERROR BECOMES VERY LOW**

but:

**NEW DATA**

↓

**PERFORMANCE DETERIORATES**.

The deeper tension is:

**MEMORY**

versus:

**GENERALISATION**.

A useful model needs enough memory to learn from the past.

But not so much attachment to the exact past that it cannot cope with something new.

> **The purpose of learning from the past is not to reproduce the past. It is to extract structure that remains useful beyond it.**

In the next lesson, we will examine the opposite failure.

Sometimes a model does not learn too much.

It learns too little.

That is **underfitting**.